Load Balancing between Team Build Agents in Team Foundation Server 2008

Introduction

In Team Build 2008, while writing a build definition we need to hard-code a build agent attached to this build definition. In cases when the builds are queued automatically depending on its triggers, it gets queued to this hard-coded build agent. This raises a major problem that there are multiple free / not-busy build agents dedicated for the team project, but still there are multiple builds queued on a single build agent waiting for their turn till the other builds get completed. So we need some load balancing mechanism between the build agents, which is technically called as Build Controller in TFS 2010. But there are no direct ways to work out with this challenge in Team Build 2008. This article shows how to balance the load between build agents in Team Build 2008.

Background

We faced this challenge in the later month of the third quarter of 2009 working with Team Build 2008 and we had to come up with a solution then. There must be many more who are still working with 2008 version and facing this issue. So here you go for the solution.

Visualizing the Problem and Deciding the Expected Behavior

We can have multiple build agents for one team project in Teambuild 2008. But when builds with the same build agent (set as their default one) are queued, they form a queue in the build server explorer waiting for their turn and they get processed one by one. So the solution to this problem (keeping in mind the 2008 version) must be some application which can stop the queued builds from the busy build agents and queue them to the other build agents which are free (if any).

If you see in the above image, my team project has 3 build agents: teambuild01, teambuild02 and teambuild03. For a given instance, there are 4 builds which got queued to the same default build agent teambuild03 and waited in queue to get their status from Queued -> In Progress -> Completed. Now to give a solution, I would expect something so that, as soon as the 1^st build (Main.Continuous.BIFBuilderGenerator) in queue gets completed, it checks for the possibility of load-balancing. Having found that there are 3 builds in the same queue of teambuild03, the other two build agents teambuild01 and 02 are completely free, so each of them can be assigned a build. This application should then stop the other 2 builds from the teambuild03 and queues one-one build on teambuild01 and teambuild02. This is shown in the below image:

Implementing the Solution

Now, I will discuss how to implement the above expected behavior as a solution. This solution can be achieved in two steps:

Step 1: Writing the business logic code to identify whether load-balancing is required and if yes, which builds to stop from which build agents and queue them to which other free build agents. We will achieve this using TFS Client side API. Below is the code (explained properly with inline comments) of how to stop the builds from one agent and queue them to other free agents. Now calling the OptimizeBuilds method of class LoadBalancingTeamBuild2008 will actually do our job, but it needs two inputs which are the team foundation URL and team project name. In the next step, we will see how we can get these inputs and execute this piece of code.

using Microsoft.TeamFoundation.Build.Client;
using Microsoft.TeamFoundation.Client;
namespace RA.TeamFoundation.Build.Events
{
    public class LoadBalancingTeamBuild2008
    {
        //
        // This function finds the builds which are in queue, the agents which are free
        // and reassigns these builds to the free build agents
        //
        public void OptimizeBuilds(string strTFSServerUrl, string strTeamProjectName)
        {
            // get an instance of the tfs and build server
            TeamFoundationServer tfsServer = new TeamFoundationServer(strTFSServerUrl);
            IBuildServer buildServer = 
		tfsServer.GetService(typeof(IBuildServer)) as IBuildServer;
            IQueuedBuildSpec queueBuildSpec = 
		buildServer.CreateBuildQueueSpec(strTeamProjectName);
            queueBuildSpec.Status = QueueStatus.Queued;
            IQueuedBuildQueryResult queueBuildQueryResult = 
			buildServer.QueryQueuedBuilds(queueBuildSpec);
            // get all the builds queued for the given team project
            IQueuedBuild[] queuedBuildsArr = queueBuildQueryResult.QueuedBuilds;
            // get all the build agents for the given team project
            IBuildAgent[] buildAgentArr = 
			buildServer.QueryBuildAgents(strTeamProjectName);
            // iterate through all the build agents of this team project
            foreach (IBuildAgent buildAgent in buildAgentArr)
            {
                buildAgent.Refresh();
                // check if this build agent is free, i.e., ok to be considered
                if (IsOkToConsiderThisBuildAgent(buildAgent))
                {
                    // iterate through all the queued builds and 
                    // find out which build is perfect to be switched to other agent
                    foreach (IQueuedBuild queuedBuild in queuedBuildsArr)
                    {
                        if (IsOkToConsiderThisQueuedBuild(queuedBuild))
                        {
                            if (!AreBuildAgentsSame(queuedBuild.BuildAgent, buildAgent))
                            {
                                // confirm that the build is waiting in queue 
                                // and is ok to be switched
                                if (IsQueuedBuildWaitingForExecution(queuedBuild))
                                {
                                    // perform the job of cancelling the build from one 
                                    // build agent and queueing it in another build agent.
                                    ChangeBuildAgentAndQueueTheBuildAgain
					(queuedBuild, buildAgent, buildServer);
                                    break;
                                }
                            }
                        }
                    }
                }
            }
        }
        //
        // This function decides if it's ok to consider a given build agent 
        // for queueing new builds
        //
        private bool IsOkToConsiderThisBuildAgent(IBuildAgent buildAgent)
        {
            // returns true if build agent is enabled, i.e. reachable / available
            // and there are no builds queued on this agent, i.e. its free and NOT busy
            return ((buildAgent.Status == AgentStatus.Enabled) && 
				(buildAgent.QueueCount == 0));
        }
        //
        // This function decides if it's ok to switch the given build 
        // to any other build agent
        //
        private bool IsOkToConsiderThisQueuedBuild(IQueuedBuild queuedBuild)
        {
            // returns true if the build definition name does not end 
            // with pinned / exclude.
            // This logic may change depending upon the requirements 
            // as there are builds we never want to change their agent.
            if (queuedBuild.BuildDefinition.Name.ToLower().EndsWith(".pinned") || 
		queuedBuild.BuildAgent.Name.ToLower().EndsWith("exclude"))
            {
                return false;
            }
            return true;
        }
        //
        // This function decides if the given two build agents are actually the same
        //
        private bool AreBuildAgentsSame
		(IBuildAgent queuedBuildAgent, IBuildAgent newFreeBuildAgent)
        {
            // returns true if the build agents full path are the same, i.e. both are one.
            return queuedBuildAgent.FullPath.Equals(newFreeBuildAgent.FullPath);
        }
        //
        // This function decides if the given build is actually in queue 
        // i.e. waiting for execution
        //
        private bool IsQueuedBuildWaitingForExecution(IQueuedBuild queuedBuild)
        {
            // returns true if the given build is actually waiting in queue 
            // for its turn to come
            // such builds are ready to be switched to other agents.
            return queuedBuild.QueuePosition > 1;
        }
        //
        // This function, actually performs the action of stopping a 
        // queued build and queueing it in another build agent.
        //
        private void ChangeBuildAgentAndQueueTheBuildAgain
	(IQueuedBuild queuedBuild, IBuildAgent buildAgent, IBuildServer buildServer)
        {
            // create a new build request for this queued build
            IBuildRequest newBuildRequest = 
		queuedBuild.BuildDefinition.CreateBuildRequest();
            newBuildRequest.CommandLineArguments = queuedBuild.CommandLineArguments;
            newBuildRequest.BuildAgent = buildAgent;
            newBuildRequest.RequestedFor = queuedBuild.RequestedFor;
            // actually queue the build to the other free build agent
            IQueuedBuild newQueueBuild = buildServer.QueueBuild(newBuildRequest);
            buildAgent.Refresh();
            // stop the queued build from the existing build agent
            queuedBuild.Cancel();
            queuedBuild.Refresh(QueryOptions.All);
        }
    }
}

Step 2: How, when and who will execute the above code: This load balancing code can be called whenever a build gets completed. So, every time when the build is completed, it can check if there is any load balancing required for any other builds in queue. So we basically need a way to trigger the execution of our code at the BuildCompletionEvent of the Team Build. Here, we will discuss one of the many ways available to do this.

Write a web service with the specified syntax to receive the tfs events notification, retrieve the data required and perform the load balancing.

namespace RA.TeamFoundation.Build.Events
{
    [WebService(Namespace = "http://tempuri.org/")]
    [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
    [System.ComponentModel.ToolboxItem(false)]
    public class BuildCompletionEventService : System.Web.Services.WebService
    {
        [SoapDocumentMethod("http://schemas.microsoft.com/TeamFoundation/2005/06/
	Services/Notification/03/Notify", RequestNamespace = 
	"http://schemas.microsoft.com/TeamFoundation/2005/06/Services/Notification/03")]
        [WebMethod]
        public void Notify(String eventXml, String tfsIdentityXml)
        {
            if (!String.IsNullOrEmpty(eventXml))
            {
                // Load the XML event string and parse the required information.
                XmlDocument xmlDoc = new XmlDocument();
                xmlDoc.LoadXml(eventXml);
                // retrieve the team foundation URL
                XmlNode xmlTFSNode = xmlDoc.SelectSingleNode
			("BuildCompletionEvent/TeamFoundationServerUrl");
                string strTFSServerUrl = xmlTFSNode.InnerText;
                // retrieve the team project name
                XmlNode xmlTeamProjectNode = xmlDoc.SelectSingleNode
				("BuildCompletionEvent/TeamProject");
                string strTeamProjectName = xmlTeamProjectNode.InnerText;
                // initiate the load balancing mechanism
                LoadBalancingTeamBuild2008 loadBalancing = 
				new LoadBalancingTeamBuild2008();
                loadBalancing.OptimizeBuilds(strTFSServerUrl, strTeamProjectName);
            }
        }
    }
}

This webservice has to be hosted in the WebServices directory present physically in the team foundation server machine. E.g. D:\Microsoft Visual Studio 2008 Team Foundation Server\Web Services\ on the team foundation server machine along with the other web services. Once this is done, our webservice will also be listed among the other team foundation server services as shown below:

Now, register with the team foundation server, to invoke this web service at the BuildCompletionEvent of any build. This can be done using BisSubscribe utility as below:

D:\Microsoft Visual Studio 2008 Team Foundation Server\TF Setup>BisSubscribe.exe 
/eventType BuildCompletionEvent 
/address http://localhost:8080/RA.TeamFoundation.Build.Events/
	BuildCompletionEventService.asmx 
/deliveryType Soap /server my-server-name
BisSubscribe - Team Foundation Server BisSubscribe Tool
Copyright (c) Microsoft Corporation.  All rights reserved.
TF50001:  Created or found an existing subscription. The subscription ID is 39.

Conclusion

So, performing the above steps and using the code will make sure that team foundation server will raise a BuildCompletionEvent whenever a build completes and will execute a BuildCompletionEventService hosted by you, which performs the load balancing of builds among build agents, i.e., finds out if any build agents are free and any builds which are queued and waiting for their turn to get processed and queue these builds to the free build agents and cancel them from the busy agents.