TFS Proxy Efficiency Testing

• Download TFS Proxy Testing batch sources - 35 KB

Introduction

The structure of the development team in a company might require to set up a distributed office. Of course, not all companies can use wide Internet channels to work with a remote version control system. So, the question arises as to how we can increase the version control system speed? Microsoft Corporation proposes the Team Foundation Server Proxy application for those who use Team Foundation Server as their version control system.

The next question is the performance and benefits of using the TFS Proxy. In this article, I would like you to understand the following points:

• Which operations will be executed faster and how much faster, if we use TFS Proxy?
• Which operations do not use TFS Proxy at all?
• How traffic is shared between the local computer, TFS, and the TFS Proxy servers?
• How does the Internet channel rate impact on an operation's execution time?
• How do the various TFS Proxy settings impact on an operation's execution time?

Background

I planned this investigation when our customer demanded to use their TFS Server. The problem is our company has got a narrow Internet channel, and it is very expensive to widen it.

After we got the proposal to move all the projects on to the customer's server, we started thinking about how we can eliminate the Internet speed problem. We found some articles about Microsoft, their remote offices, and how they use their TFS Proxy. So, we thought we could use TFS Proxy as well, but our main question was whether TFS Proxy could be efficient on a slow Internet channel.

The speed of our Internet channel is 128/64 Kbps (download/upload) and, in our opinion, the local TFS Proxy should be very efficient. Also, we decided to test the TFS Proxy with wider Internet channels to test the overall system performance depending on the Internet channel rate.

Hardware and software specifications

I tested the latest version of Visual Studio and the TFS command line utilities (9.0.21022.8).

To emulate an Internet channel with various rates, I used SoftPerfect Bandwidth Manager v2.5 build 230.

Local computer configuration:

• CPU: AMD Athlon^(TM) 64 X2 Dual, Core Processor 4000+
• RAM: 2 GB of RAM
• HDD: Maxtor STM3250310AS, 250GB, SATA-II
• Windows XP SP3

TFS Proxy configuration:

• CPU: AMD Athlon^(TM) 64, Processor 3000+
• RAM: 2 GB of RAM
• HDD: Maxtor 6Y160P0, 160GB, UDMA-133
• Windows Server 2008

TFS Server configuration:

• CPU: AMD Sempron^(TM) 3500+
• RAM: 2 GB of RAM
• HDD: Western Digital WDC3200AAKS-00SBA0 320GB, SATA
• Windows Server 2003 SP2

Conditions

We tested at the following Internet channel rates (upload/download):

• 64Kbps/128Kbps
• 128Kbps/256Kbps
• 256Kbps/512Kbps
• 512Kbps/1024Kbps
• 1Mbps/2Mbps
• 2Mbps/4Mbps

This list of internet channel rates started from the rate that our company has got at present. I used the most standard rates to cover most of the cases. The speed of the inbound channel is not important, and it can be from one eight to a quarter of the outbound rate.

We tested the following TFS commands:

• Get
• Check-Out
• Undo Check-Out
• Check-In
• History
• Diff

In my company, these commands are high-usage, so I decided to test them in the first place.

We emulated the following TFS Proxy states:

• TFS Proxy enabled, 0% cached
• TFS Proxy disabled
• TFS Proxy enabled, 90% cached
• TFS Proxy enabled, 100% cached

The first alternative (Proxy enabled, 0% cached) is hardly probable, but I decided to test it to compare it with the second one.

The second one (Proxy disabled) discovers the most important information about the advantages of TFS Proxy. Also, this one I chose as the basic case to compare with all the other cases with caching.

The third one (Proxy enabled, 90% cached) represents the state of the TFS Proxy in most cases of regular use.

The fourth one (Proxy enabled, 100% cached) is the possible state of the TFS Proxy cache that can be achieved via regular cache updating on the proxy server (see here). This state should be a little bit faster than the previous, and can be realized in any software team.

For the TFS Proxy testing, we created a repository and uploaded testing data. The testing data represented a real project (see the following table for details).

Project size, MB	10.2
Cache size (whole project), MB	3.1
Total files in the project	1732
Total folders in the project	145
Total *.cs files	569
Size of *.cs files, MB	1.47

Testing methodology

To test all the TFS commands, the TFS Proxy states, and the Internet channel rates, I used the following scheme. I implemented a command file (.cmd) for each TFS command (see the example below). This file would allow setting the TFS Proxy address, executing the TFS command, and getting the time spent. Additionally, there is another .cmd file for each TFS command. It executes in a loop the first one, for several times, for all the required TFS Proxy states. Also, a single launch of the first command file allows to get the inbound/outbound traffic for each TFS Proxy state.

In my experiment, each TFS command was executed four times. The first attempt was rejected because of experimental purity, while the times of all the other attempts were added to an Excel file. So, for the 'Get' command, there were executed 16 attempts (16 = 4 attempts x 4 TFS Proxy state). Then, .cmd files for the rest of the TFS commands ('Check-Out', 'Undo Check-Out', 'Check-In', 'History', 'Diff') were executed. After that, I changed the transfer rate limit and all the tests were started again.

In a real situation, the TFS Server is remote, while the TFS Proxy Server and the TFS Clients are placed in the same network. In my testing environment, all the computers were placed in the same network (100 MBps), so I decided to use the SoftPerfect Bandwidth Manager to limit the transfer rate between both the test computer-TFS Server and the TFS Proxy Server-TFS Server to emulate a real situation.

Command file implementation

Note: It is supposed that a workplace for the testing repository was created and it was mapped to a local path (see the 'create local.cmd' file in the archive).

Below, you can see examples of the .cmd files for the 'Get' command.

@ECHO OFF
setlocal

SET Getpath="C:\Temp\tfsProxyTest1"
ECHO Trying TFS get...
ECHO.

@IF NOT [%1]==[] (@echo set proxy
@set TFSPROXY=http://tfs_proxy_server:8081)

cd %Getpath%

REM get latest source from TFS
@set t1=%time%
@echo %TFSPROXY%
tf get %Getpath% /login:server\login,password /force /recursive 
echo %t1%
echo %time%

@set TFSPROXY=

ECHO.

ECHO TFS get completed...
endlocal

Another .cmd file to execute the 'Get' command in the loop for various TFS Proxy states:

@echo OFF

echo Start: Operation GET

FOR /L %%G IN (1,1,4) DO (echo Start: without proxy
call start_get.cmd
echo End: without proxy
)

FOR /L %%G IN (1,1,4) DO (echo Start: Proxy 0% cached
call z:\delete.cmd
call start_get.cmd use_cache
echo End: Proxy 0% cached
)

FOR /L %%G IN (1,1,4) DO (echo Start: PROXY 100% cached
call start_get.cmd use_cache
echo End: PROXY 100% cached
)

FOR /L %%G IN (1,1,4) DO (echo Start: proxy 90% cached
call z:\delete_10percent.cmd
call start_get.cmd use_cache
echo End: proxy 90% cached
)

echo End: Operation GET

Additional .cmd files 'delete_10percent.cmd' and 'delete.cmd' just delete 10 percent of the cache and all the cache, respectively.

Note: All .cmd files can be found in the attached archive.

Test results

I spent several days to complete all the tests (thank goodness!). Detailed results are placed in an Excel file in the attached archive.

The 'Get' command

Note: Pastel colors on the data transfer figures correspond to a slow channel, brighter ones correspond to a fast channel.

Fig. 1 presents information about data transfers for the operation 'Get'. As follows from the figure, the slower operations are 'Proxy disabled' and 'Proxy, 0% cached', while 'Proxy, 90% cached' and 'Proxy, 100% cached' are much faster. This information conformed with the following data.

Transfer Rate / Proxy State	Proxy, 0% cached	Proxy disabled	Proxy, 90% cached	Proxy, 100% cached
64/128 Kbps	05:22,62	05:15,52	01:40,71	01:29,99
128/256 Kbps	02:37,78	02:36,54	00:56,37	00:53,38
256/512 Kbps	01:22,83	01:19,51	00:42,55	00:39,97
512/1024 Kbps	00:43,33	00:40,74	00:31,35	00:27,94
1MBps/2MBps	00:30,05	00:26,13	00:24,70	00:23,88
2MBps/4MBps	00:25,40	00:20,93	00:21,95	00:20,71

Conclusions

• The TFS Proxy helps a lot on slow Internet channel rate - the overall performance almost quadruples.
• Internet channel rate impacts on the time of operation, but it is a nonlinear dependence.
• The difference between the outbound channel rate 2 Mbps for the 'Proxy enabled' and 'Proxy disabled' modes is about 9% - do we still need TFS Proxy? ;-)
• The mode 'Proxy, 0% cached' is a little bit slower than 'Proxy disabled' because the first one should get data from the server and then send it to the client.

The 'Check-Out' command

Fig. 3 presents information about data transfers for the operation 'Check-Out'. As follows from the figure, it does not use the TFS Proxy. So, we can test only the 'Proxy disabled' state.

Transfer Rate / Proxy State	Proxy disabled
64/128 Kbps	01:27,36
128/256 Kbps	00:52,93
256/512 Kbps	00:36,56
512/1024 Kbps	00:27,72
1MBps/2MBps	00:23,38
2MBps/4MBps	00:21,25

Conclusions

• Using of TFS Proxy does not impact on the 'Check-Out' operation.
• Internet channel rate impacts on the time of operation, but it is nonlinear dependence as well.
• The difference between the outbound channel rates 1 and 2 Mbps is about 15% - how much is the difference between costs of 1 and 2 Mbps channel?

The 'Undo Check-Out' command

Fig. 5 presents information about data transfers for operation 'Undo Check-Out'. As follows from the figure, it has almost the same view as Fig. 1. It means, the states 'Proxy disabled' and 'Proxy, 0% cached' are the most slow, while 'Proxy, 90% cached' and 'Proxy, 100% cached' are much faster. See the following table and figure for details.

Transfer Rate / Proxy State	Proxy, 0% cached	Proxy disabled	Proxy, 90% cached	Proxy, 100% cached
64/128 Kbps	05:22,51	05:16,05	01:43,53	01:33,25
128/256 Kbps	02:42,26	02:38,09	01:01,47	01:01,09
256/512 Kbps	01:29,40	01:22,54	00:41,22	00:40,35
512/1024 Kbps	00:47,11	00:42,63	00:33,82	00:31,52
1MBps/2MBps	00:34,21	00:30,46	00:27,86	00:25,81
2MBps/4MBps	00:28,09	00:25,27	00:25,26	00:23,60

Conclusions

Conclusions are absolutely the same as for the 'Get' operation.

The 'Check-In' command

Fig. 7 presents information about data transfers for the operation 'Check-In'. As follows from the figure, it does not use the TFS Proxy. So, we can only test the 'Proxy disabled' state.

Table 5: Execution time for operation 'Check-In', (min:sec).

Transfer Rate / Proxy State	Proxy disabled
64/128 Kbps	01:07,15
128/256 Kbps	00:30,14
256/512 Kbps	00:17,13
512/1024 Kbps	00:10,87
1MBps/2MBps	00:07,51
2MBps/4MBps	00:06,49

Conclusions

Conclusions are absolutely the same as for the 'Check-Out' operation.

The 'History' command

Fig. 9 presents information about data transfers for the operation 'History'. As follows from the figure, it does not use the TFS Proxy. So, we can test only the 'Proxy disabled' state.

Transfer Rate / Proxy State	Proxy disabled
64/128 Kbps	01:36,80
128/256 Kbps	00:49,42
256/512 Kbps	00:25,71
512/1024 Kbps	00:14,09
1MBps/2MBps	00:08,43
2MBps/4MBps	00:05,89

Conclusions

Conclusions are absolutely the same as for the 'Check-Out' and 'Check-In' operations.

The 'Diff' command

Fig. 11 presents information about data transfers for the operation 'Diff'. As follows from the figure, it has a little bit different view from Fig. 1 and Fig. 5. The size of the transferred data by slow and fast channels is comparable. So, the difference between 'Proxy enabled' and 'Proxy disabled' should be less than in case of 'Get' and 'Undo Check-Out' operations. Let's see following table and figure for details.

Transfer Rate / Proxy State	Proxy, 0% cached	Proxy disabled	Proxy, 90% cached	Proxy, 100% cached
64/128 Kbps	01:21,03	01:15,37	00:48,40	00:46,49
128/256 Kbps	00:41,01	00:37,61	00:32,16	00:29,44
256/512 Kbps	00:30,25	00:23,25	00:23,87	00:20,80
512/1024 Kbps	00:28,58	00:19,87	00:19,11	00:16,91
1MBps/2MBps	00:24,50	00:16,65	00:16,65	00:14,50
2MBps/4MBps	00:22,68	00:15,91	00:15,48	00:13,36

Conclusions

• The TFS Proxy helps on slow internet channel rates - the overall performance almost doubles. But it is less than for 'Get' and 'Undo Check-Out' operations.
• The Internet channel rate impacts on the time of operation and it is a nonlinear dependence.
• The difference between the outbound channel rate 512 Kbps for the 'Proxy enabled' and 'Proxy disabled' modes is about 10%.
• The mode 'Proxy, 0% cached' is considerably slower in comparison with the other modes for any channel rate.

General conclusions

'Check-out', 'Check-In', and 'History' operations

• Using of TFS Proxy does not impact on these operations.

Operation 'Get'

• It seems, too much data transferred even for the 'Proxy, 100% cached' mode besides the TFS Proxy - about one third of the overall incoming data have to be got directly from the TFS Server by a slow channel.

Operation Undo Check-Out

• Size of data sent directly to the TFS Server is small - that makes sense.
• Size of received data directly from the TFS Server is huge - almost half of what is received from the TFS Proxy. It is not clear why TFS works so.
• Generally, the size of the transferred data is greater for the operation 'Get'. It is not logical, e.g., SVN version control system does not communicate with server to do 'Undo' operations.

Operation Diff

• Size of sent data directly to TFS Server is small - that makes sense.
• Size of received data directly from TFS Server is huge - almost half of what is received from the TFS Proxy.
• Operation 'Diff' has been executed with option /brief. A brief formatting prints whether the files being compared differ. In compliance with the tests, for operations 'Get' and 'Undo Check-Out', 4.7 and 5.2 Mb were received. But the 'Diff' operation received about 1 Mb, and it is too much because the size of files to compare was about 1.47 Mb. The size of the transferred data between the client and the proxy computers is about 0.5 Mb, and it is one third of the size of the compared files. It seems, the rest of transferred data is service information.
• Again, TFS is not an optimal system, e.g., Subversion does not receive any information to do the difference between the base version and the current version of a file.

So, for all operations:

• If cache is empty all the time (by any reason), you have to avoid using the TFS Proxy.
• If cache is empty, TFS Proxy gets all the data from the TFS Server and then sends it to the client.
• Always, the TFS Client sends and receives some part of information to/from the TFS Server directly (beside the TFS Proxy). The size of the information is largish, and it impacts on the overall performance of TFS. So, it would be possible to improve performance if we reduce the size of the transferred service information.
• Some operations were implemented non-optimal in comparison with some other version control systems, partially because of the architecture of TFS.

Results

TFS Proxy is great. :-)

It accelerates the speed of some high-usage operations ('Get', 'Undo Check-Out', 'Diff') and is especially useful for narrow outbound Internet channels. Unfortunately, the TFS Client sends to the TFS Server a big part of traffic besides the TFS Proxy. Some operations do not use the TFS Proxy ('Check-Out', 'Check-In', 'History') at all. It is important to keep watch on the fullness of the cached data every day to have maximum performance. The inbound Internet channel is not important for performance.

Hope, this helps.

Who can use it?

Any development team who uses a remote TFS Server.

Known issues

I do not know.

Acknowledgements

Thanks to the following people who helped me:

• Alex Maskaev
• Victor Belyavskiy

History

• Version 1.0 (2008-08-13) - Initial release.

You must Sign In to use this message board.

FAQ    Noise Tolerance Very HighHighMediumLowVery Low   Layout Expand Posts & RepliesThread ViewNo JavascriptNo JS + Preview   Per page 102550

(Refresh)

-- There are no messages in this forum --