Introduction to Branching by Quality [Branching-Merging Techniques]

There is no one best branching strategy, the branching strategy that best fits the requirements of your enterprise is the best strategy for you. The ALM rangers have comprehensive guidelines to help you decide what might work best with your enterprise. A couple of parameters you need to consider while thinking about the branching strategy are,

1. What is the length of the Iteration your team will be delivering in. Agile – Weeks, Waterfall – Months.
2. Would you have multiple versions of the applications running in production? If so, it makes good sense to branch by release.

A lot of the customers I have helped set up ALM processes have been following Agile (with this said it does not imply that this approach is not fit for enterprises that do not follow Agile), with a sprint duration of 3 weeks and a single version of the product running in production, I strongly recommend the Branch by Quality pattern be adopted in this case. In this blog post I'll attempt to explain how the branch by quality works, i will also explain some branching and merging guidelines you can follow as well as some reporting you can set up to exception handle the failure cases.

If you are enjoying the post, remember to subscribe to http://feeds.feedburner.com/TarunArora. I’ll be further exploring solutions in this space.

Branch Pattern

The branch pattern I will be discussing today is also referred to as ‘Branch by Quality’. This pattern is one of the most popular branching patterns used, and can serve as a base pattern for more complex branch strategies. In the Branch by Quality model, there are three branches (the naming convention depends on your enterprise but the role of the branch is categorized as): Production, Quality Assurance (QA), and Development.

Branch Hierarchy

Main Branch - The Main branch also known as the Development branch is where all major code development occurs. This is the branch developers will be checking their code into and out of during their daily development cycle. It is the most fluid branch of the three, and sees the most activity.

Test Branch - The QA branch is used for testing the application and achieving final application approval. The QA branch is not as fluid as the Development branch, and should not see near as much check-in/check-out activity.

Prod Branch - The Production branch contains the production code for the application. This branch should only be used to check-in/check-out fixes for hot fixes required in Production.

In Analogy imagine the Production branch is made of brick, the QA branch is made of wood, and the Development branch is made of straw. Using this model, you always want to make the majority of your changes on the softest branch, in this case, the Development branch. As you move up the branch hierarchy, things get "harder" so few changes should be made on the QA branch, and changes should never be made directly on the Production branch.

Merging Strategy

Case 1 – Sprint Merging

The Development team is working on the Main branch on Iteration 3, the Test branch currently has the code base from Iteration 2 and the Prod branch currently has the code base from Iteration 1.

Merging Process

1. Verify that there is no pending merge between Prod branch and Test branch for any Hot Fixes applied in Prod branch against P1 tickets for Iteration 1. Resolve conflicts if any and commit the changes to the Test branch.
2. Verify that there is no pending merge between Test branch and Main branch for any fixes applied in Test branch during the testing of Iteration 2. Resolve conflicts if any and commit changes to the Main branch.
3. Promote Iteration 2 from the Test branch to the Prod branch. The Prod branch can now serve as a hot fix branch for any P1 tickets raised.
4. Promote Iteration 3 from Main branch to the Test branch.
5. Main branch is now free for development of Iteration 4.

Release candidate

Ensure that the last build generated from the Test branch has been tested and signed off. This build becomes the release candidate for Production release of Iteration 2.

Best Practices

Lock the release candidate in build explorer. This ensures that the build is retained regardless if you have the build retained set up to preserve only n number of successful, partially successful and failed builds.

Build versioning – ensure that you have a meaningful build naming convention set up. I would recommend a macro similar to TeamProjectName_(BranchName, BuildTriggerType) ProductVersion.MajorReleaseVersion.IterationNumber.Revision, for example builds generated for Iteration 3 in Main branch which has a trigger type of Continuous Integration should be versioned as TeamProject_(Main, CI) 1.0.3.1

Case 2 – Hot Fix Merge

The Development team is working on the Main branch on Iteration 3, the Test branch currently has the code base from Iteration 2 and the Prod branch currently has the code base from Iteration 1. A P1 is raised in Production and a fix needs to be rolled out in Production immediately.

FAQ

In what branch should I fix the P1?

• The P1 should be fixed in the branch that is closest to the code base running in Production. In this case the P1 should be fixed in the Prod branch. By applying the fix in any other branch and rolling out the changes to production you risk releasing semi-finished or untested code from the subsequent iterations.
• Now you may argue if it is safe to work directly against the Prod branch, think again, a P1 that requires immediate attention should not be a fundamental problem in the system. In case it is a fundamental problem it should be added to the Product backlog as it may require further analysis and discussion with the customer.

Where do I test the fix?

• The developer should write a unit test to simulate the failing condition of the P1. After the fix is applied the developer should be able to see the unit test pass and also ensure that none of the existing tests are failing. In addition to that the developer should test the fix in the developer environment.
• Since Pre-prod is closest to Production. The release candidate build should then be deployed in the Pre-prod environment; the QA should be performed in the Pre-prod environment. Once the fix has been QA signed off, the fix needs to be reviewed and signed off by the end user in the form of User Acceptance Test in the Preproduction environment.
• The fix is now ready to be released in production.

What if there is another P1?

What happens in case another P1 is raised in the interim period, since the Pre-prod environment has been occupied by the testing of the current hotfix I do not have another environment to reproduce the new P1?

• This entirely depends on your infrastructure capabilities and defect rate in production. If the defect rate in production is significantly high then it is suggested that the support environment needs to be scaled out and an additional hot fix environment be used for the testing of any P1 fixes.

Merging Process

1. Reverse Merge the Prod branch to the Test branch. Resolve conflicts if any and commit the changes to the Test branch.
2. Reverse Merge from Test branch to the Main branch. Resolve conflicts if any and commit changes to the Main branch.

Release candidate

Ensure that the last build generated from the Prod branch where the fix has been applied has been tested and signed off. This build becomes the release candidate for Hot fix being rolled out to fix the P1.

Best Practices

• The merge to Test and Main branch should only be initiated when hot fix has been released in production and the user has confirmed that the fix has resolved the issue initially escalated through the P1.

Lock the release candidate in build explorer. This ensures that the build is retained regardless if you have the build retained set up to preserve only n number of successful, partially successful and failed builds.

Case 3 – Bug Fix Merge

The Development team is working on the Main branch on Iteration 3, the Test branch currently has the code base from Iteration 2 and the Prod branch currently has the code base from Iteration 1. The user testing Iteration 2 in Test branch has found an issue that seems to be valid bug.

FAQ

In what branch should I fix the bug?

The bug should be fixed in the branch that is closest to the code base the user has been testing against. In this case the Bug should be fixed in the Test branch. By applying the fix in any other branch and rolling out the changes to user testing environment you risk releasing semi-finished or untested code from the subsequent iterations.

Where do I test the fix?

• The developer should write a unit test to simulate the failing condition of the Bug. After the fix is applied the developer should be able to see the unit test pass and also ensure that none of the existing tests are failing. In addition to that the developer should test the fix in the developer environment.
• The build should then be deployed in the user testing environment; the QA should be performed in the user Testing environment. Once the fix has been QA signed off, the fix needs is now ready to be reviewed by the user who raised the Bug in the same environment.

Merging Process

Reverse Merge the Test branch to the Main branch. Resolve conflicts if any and commit the changes to the Test branch.

Release candidate

Ensure that the last build generated from the Test branch where the fix has been applied has been tested and signed off. This build is a potential release candidate for future production release in case no other changes are made with regard to Iteration 2 in the Test branch.

Best Practices

The merge from Test to Main branch should only be initiated when the bug has been signed off by the user who raised the initial issue.

Lock the release candidate in build explorer. This ensures that the build is retained regardless if you have the build retained set up to preserve only n number of successful, partially successful and failed builds.

Exception Handling

No matter how careful you are with merging there may be occasions when you accidently miss merging down a hot fix from prod branch down to Test branch and since the sprint releases are rolled out from the Test branch this will result in reintroducing P1’s that have been fixed through the hot fix release. The best way to ensure not landing in such a situation is to add some exception reports. Depending on the scale of change and the team size, this process can also be automated.

• Pending Merge Report – This report shows the merge candidates between two related branches. This report can be used by the release manager or the team in general before sprint, hot fix or bug fix releases to ensure that no pending merge change sets are accidently left out. I had written a tool “Smart Merge” using the TFS API which lets you do this and more.

• Code Churn Report – Code churn is defined as lines added, modified or deleted to a file from one version to another. It is possible to measure this at a branch level as well, this can become one of the metrics to gauge the risk involved in the release, having said that, this cannot be the only identifier to measure the level of change involved in the release, another important identifier can be the work item or bug fix severity being rolled out in the release.

I would strongly recommend reading the TFS 2010 branching and merging guide compiled by the ALM Rangers. If this results in any follow up questions, feedback or suggestions [lease feel free to add to the blog post. If you are enjoyed the post, remember to subscribe to http://feeds.feedburner.com/TarunArora. I’ll be further exploring solutions in this space.

Enjoy!!!