Code Coverage for Regression Tests (Manual/Auto)

Demo zip contains  

Code Coverage For Regression Test Demo Files  

1)    Code Coverage Sample WindowsFormsApplication1 

      2) CodeCoverageXMLConverter application to convert coverage file to xml. 

      3) BatchFiles  

4) CodeCoverageProfillingFiles 

Introduction  

          Code coverage is a utility that seats in Visual studio Testing Framework. I have used Code Coverage in my previous project to identify which and how much code I have covered in my unit tests.

Table of Contents   

• Introduction
• Table of Contents
• Regression Tests Code Coverage Implementation 

     Ø  Instrument dlls or exes  to Test 

     Ø  Start Code Coverage  Monitor 

     Ø  Run Application 

     Ø  Regression Tests (Manual/Auto) 

     Ø  Stop Application 

     Ø  Stop Code Coverage Monitor 

     Ø  Review  Code Coverage File 

     Ø  Report Generation 

·  Conclusion   

·  References 

·  Development Environment 

     ·  History   

Regression Tests Code Coverage Implementation   

          First we need to create a windows application which we will use for regression tests. Create a simple windows application with a textbox and 2 buttons as Say Hi and Say Bye. In textbox we accept name of user. When user clicks on Say Hi button, show message box with text “Hi <<UserName>>”. When user clicks on Say Bye button, show message box with text “Bye <<UserName>>”. As below 

1)     Instrument DLLs /EXEs to Test: 

Open a Visual Studio 2010 or higher Command Prompt  (Start | All Programs | Microsoft Visual Studio 2010 | Visual Studio Tools | Visual Studio 2010 Command Prompt) 

C:\CodeCoverageFinal\WindowsFormsApplication1\WindowsFormsApplication1\bin\Debug  

vsinstr /coverage WindowsFormsApplication1.exe  

We can create batch file with name “Instrument.bat” to run above command
as below

call "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86

@echo off

echo Instrumention Started...

pause

vsinstr /coverage C:\CodeCoverageFinal\WindowsFormsApplication1\WindowsFormsApplication1\bin\Debug\WindowsFormsApplication1.exe

echo Instrumention Completed.

pause

 As shown in above screen application exe is instrumented and WindowsFormsApplication1.instr.pdb
file is created. 

Note: This is one time process.
If you do changes in solution then delete all pdb files then again run the
“Instrument.bat” file.   

2)     Start Code Coverage Monitor:   

To start our test run we need to spin up the code coverage monitor.  We do this by running vsperfcmd command from the Visual Studio 2010 Command Prompt.  

vsperfcmd /start:coverage /output:[AMeaningfulOutputName].coverage 

Example: 

vsperfcmd /start:coverage /output:c:\Test\RegressionTests.coverage 

/output – outputs the result to specified file.   

We can create batch file as “Start_Profiler.bat” to run above command as
below  

call "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86

@echo off

echo Starting Profiler...

pause

vsperfcmd /start:coverage /output:C:\CodeCoverageFinal\CodeCoverageProfillingFiles\RegressionTests.coverage

echo Started Profiler.

pause   

As shown in above screen profiler has started and it is recording in C:\CodeCoverageFinal\CodeCoverageProfillingFiles\RegressionTests.coverage file.  

3)     Run Application:   

You start your application by running it from specific location.  

Example : 

C:\CodeCoverageFinal\WindowsFormsApplication1\WindowsFormsApplication1\bin\Debug\
WindowsFormsApplication1.exe  

4)    Regression Tests (Manual/Auto):   

Here we run all manual/auto test cases.  

For our demo click on any or both buttons.   

5)    Stop Application: 

Shut down your app the way a user normally would.   

6)    Stop Code Coverage Monitor: 

We need to shutdown the code coverage monitor. This will cause the code coverage data to be written out to the file that we specified on the command-line in Step 2. Do this by running vsperfcmd command in the Visual Studio 2010 Command Prompt.   

vsperfcmd /shutdown

For web services/web application worker process (aspnet_wp) is created which is still running above command waits for worker process. We need to kill worker process before above command. So use iisreset to kill worker process.   

We can create batch file “Stop_Profiler.bat” for the same as below: 

call "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86
 
@echo off
 
echo Stopping Profiler...
 
pause 

iisreset 
 
vsperfcmd /shutdown 
 
echo Stopped Profiler.
 
pause 

As shown in above screen profiling has been stopped and our coverage file is populated.  

7)     Review  Code Coverage File: 

To look at the code coverage results, open the .coverage file (specified on the command-line in Step 2) in Visual Studio 2010 or higher.  If someone looking at the .coverage file has access to the source code, they will be able to see the source code coloring. This file will be very helpful to developers.  

Open the “RegressionTests.coverage” file in Visual Studio you will get details as below 

8)     Report Generation:  

Use CodeCoverageToXMLConvertor application to get xml report as below

Now you will find C:\CodeCoverageFinal\CodeCoverageProfillingFiles\RegressionTests.coverage.xml
file. This file contains many sections and gives detailed information for code coverage. I have created a xslt file "myxml.xsl" to generate report based on business requirement. You can include more sections as per your requirements. you just need to update "myxml.xsl" file. 

Edit this xml file and add below line at 2^nd line.    

<?xml-stylesheet type="text/xsl" href="myxml.xsl"?>

Open this file in internet explorer you will get a nice report for business as below  

Similarly we can profile many files at a time to generate a complete report. 

Conclusion:    

This article presented how to use code coverage during regression tests (auto/manual).    

References:  

Running Tests with Code Coverage  
  http://blogs.msdn.com/vstsqualitytools/archive/2005/06/08/426979.aspx 

MSDN: Manual Test Overview 
  http://msdn.microsoft.com/en-us/library/ms182496(VS.80).aspx 

Perform Code Coverage Analysis with .NET to Ensure Thorough Application Testing(MSDN Magazine) 
  http://msdn.microsoft.com/en-us/magazine/cc163981.aspx 

MSDN - VSInstr   http://msdn.microsoft.com/en-us/library/ms182402.aspx 

 MSDN - VSPerfCmd   http://msdn.microsoft.com/en-us/library/ms182403.aspx  

Code Covegare For Manual Tests by Steven Jean  

http://sstjean.blogspot.co.uk/2008/09/vsts-how-to-code-coverage-for-manual.html 

Running Tests with Code Coverage   

http://blogs.msdn.com/b/vstsqualitytools/archive/2005/06/08/426979.aspx 

Off Road Profiling of ASP.NET Applications   

http://blogs.msdn.com/b/graycode/archive/2005/05/10/aspnetoffroadprofilingarticle.aspx   

Development Environment:  

 VS Team System 2010 Development Edition/ Visual Studio Team System 2010 Test Edition/
Visual Studio Team System 2010 Team Suite. Or higher version of visual studio.  

History:    

9^th May 2013 Original Article 

13th May 2013 Updated some sections