In government contracts, staging and production servers are usually managed by a different team, which is very challenging when your code works well in development servers but doesn’t work properly in those other servers – you basically don’t have control nor visibility over what is happening behind scenes.
In those scenarios, it is really important to follow git best practices to ensure that the release process is not impacted. I’m talking specifically about the use of hotfix and release candidate branches, which are utilized to test and fix issues before those are merged to the master branch.
In one of our projects, we are currently following a git workflow where we can only use those branches to test our code in development servers, but are forced to merge them to master in order to test the code in the staging server to get the final sign off, because according to that team all their deployment process is automated and it uses the master branch, and requesting the deployment of a different branch will introduce a manual procedure.
In most of the cases we have been able to test in great detail and fix all bugs before we request the deployment to staging, but recently we got many performance issues in the staging server that we never experienced in the development server, and took us a long time to pinpoint and determine the cause of the problem. That impacted the release schedule, and we were not able to release other minor requests while we were solving that problem, because the master branch already contained the code that was not working, and rolling back changes manually could introduce more risks.
Testing and solving issues in hotfix and release candidate branches allow teams to release other features in parallel no matter how long other releases take to be ready for prime time.