Click here to Skip to main content
Click here to Skip to main content
Go to top

Code Coverage for Regression Tests (Manual/Auto)

, 16 May 2013
Rate this:
Please Sign up or sign in to vote.
This article presented how to use code coverage during regression tests (auto/manual).
This is an old version of the currently published article.

The demo zip contains:

  1. Code Coverage Sample WindowsFormsApplication1
  2. CodeCoverageXMLConverter application to convert coverage file to XML.
  3. BatchFiles 
  4. CodeCoverageProfillingFiles

Introduction

Code coverage is a utility that seats in Visual studio Testing Framework. I have used Code Coverage in my previous project to identify which and how much code I have covered in my unit tests.

In that project we used to have regression tests (Manual/auto) before deploying in production. But Business wanted to know how much code we have covered in regression tests. For that I used code coverage and generated some nice reports for business as given below. In this article I am going to tell you, how we can use code coverage to instrument DLLs or exes during regression tests (manual /auto) and generate reports based on that.

Table of Contents

  • Introduction
  • Table of Contents
  • Regression Tests Code Coverage Implementation 
    • Instrument DLLs or exes  to Test
    • Start Code Coverage  Monitor
    • Run Application
    • Regression Tests (Manual/Auto)
    • Stop Application
    • Stop Code Coverage Monitor
    • Review  Code Coverage File
    • Report Generation
  • Conclusion  
  • References
  • Development Environment
  • History

Regression Tests Code Coverage Implementation   

First we need to create a windows application which we will use for regression tests. Create a simple windows application with a textbox and 2 buttons as Say Hi and Say Bye. In textbox we accept name of user. When user clicks on Say Hi button, show message box with text “Hi <<UserName>>”. When user clicks on Say Bye button, show message box with text “Bye <<UserName>>”. As below

1. Instrument DLLs /EXEs to Test

Open a Visual Studio 2010 or higher Command Prompt  (Start | All Programs | Microsoft Visual Studio 2010 | Visual Studio Tools | Visual Studio 2010 Command Prompt)

Change directories to the directory containing windows application exe. As below:

C:\CodeCoverageFinal\WindowsFormsApplication1\WindowsFormsApplication1\bin\Debug

Use vsinstr command for each assembly you wish to instrument.

vsinstr /coverage WindowsFormsApplication1.exe  

We can create batch file with name “Instrument.bat” to run above command, as below

call "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86

@echo off

echo Instrumention Started...

pause

vsinstr /coverage C:\CodeCoverageFinal\WindowsFormsApplication1\
          WindowsFormsApplication1\bin\Debug\WindowsFormsApplication1.exe

echo Instrumention Completed.

pause

Run “Instrument.bat” file a command prompt will show up as below

As shown in above screen application exe is instrumented and WindowsFormsApplication1.instr.pdb file is created.

Note: This is one time process. If you do changes in solution then delete all PDB files then again run the
“Instrument.bat” file.
 

Start Code Coverage Monitor

To start our test run we need to spin up the code coverage monitor.  We do this by running vsperfcmd command from the Visual Studio 2010 Command Prompt.

vsperfcmd /start:coverage /output:[AMeaningfulOutputName].coverage 

Example:

vsperfcmd /start:coverage /output:c:\Test\RegressionTests.coverage 

/output – outputs the result to specified file.

We can create batch file as “Start_Profiler.bat” to run above command as below:

call "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86

@echo off

echo Starting Profiler...

pause

vsperfcmd /start:coverage /output:C:\CodeCoverageFinal\CodeCoverageProfillingFiles\RegressionTests.coverage

echo Started Profiler.

pause   

Run “Start_Profiler.bat” file a command prompt will show up as below:

As shown in above screen profiler has started and it is recording in the C:\CodeCoverageFinal\CodeCoverageProfillingFiles\RegressionTests.coverage file.

3. Run Application

You start your application by running it from specific location.

Example :

C:\CodeCoverageFinal\WindowsFormsApplication1\WindowsFormsApplication1\bin\Debug\WindowsFormsApplication1.exe

4. Regression Tests (Manual/Auto)

Here we run all manual/auto test cases.

For our demo click on any or both buttons.

5. Stop Application

Shut down your app the way a user normally would.

6. Stop Code Coverage Monitor

We need to shutdown the code coverage monitor. This will cause the code coverage data to be written out to the file that we specified on the command-line in Step 2. Do this by running vsperfcmd command in the Visual Studio 2010 Command Prompt.

vsperfcmd /shutdown

For web services/web application worker process (aspnet_wp) is created which is still running above command waits for worker process. We need to kill worker process before above command. So use iisreset to kill worker process.

We can create batch file “Stop_Profiler.bat” for the same as below:

call "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86
 
@echo off
 
echo Stopping Profiler...
 
pause 

iisreset 
 
vsperfcmd /shutdown 
 
echo Stopped Profiler.
 
pause

Run “Stop_Profiler.bat” file below prompt shows  

As shown in above screen profiling has been stopped and our coverage file is populated.

7. Review  Code Coverage File

To look at the code coverage results, open the .coverage file (specified on the command-line in Step 2) in Visual Studio 2010 or higher.  If someone looking at the .coverage file has access to the source code, they will be able to see the source code coloring. This file will be very helpful to developers.

Open the “RegressionTests.coverage” file in Visual Studio you will get details as below

8. Report Generation

Use CodeCoverageToXMLConvertor application to get xml report as below

Now you will find C:\CodeCoverageFinal\CodeCoverageProfillingFiles\RegressionTests.coverage.xml file. This file contains many sections and gives detailed information for code coverage. I have created an XSLT file "myxml.xsl" to generate report based on business requirement. You can include more sections as per your requirements. you just need to update "myxml.xsl" file.

Edit this xml file and add below line at 2nd line.

<?xml-stylesheet type="text/xsl" href="myxml.xsl"?>

Open this file in internet explorer you will get a nice report for business as below

Similarly we can profile many files at a time to generate a complete report.

Conclusion

This article presented how to use code coverage during regression tests (auto/manual).

References

Development Environment

VS Team System 2010 Development Edition/ Visual Studio Team System 2010 Test Edition/Visual Studio Team System 2010 Team Suite. Or higher version of Visual Studio.

History

  • 9 May 2013 Original article.
  • 13 May 2013 Updated some sections.

Note: Please do read, bookmark, comment, vote article.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

snehal harawande
Software Developer Datamatics Global Service Ltd.
India India
MCTS Web Applications Development with Microsoft .NET Framework 4
MCPD Windows Framework 2.0
Working in Microsoft Dotnet Technologies more that 3 yrs.

Comments and Discussions


Discussions posted for the Published version of this article. Posting a message here will take you to the publicly available article in order to continue your conversation in public.
 
QuestionCode Coverage for Manual Tests for Windows store/Mobile apps PinmemberMember 87953919-Feb-14 23:40 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web03 | 2.8.140926.1 | Last Updated 16 May 2013
Article Copyright 2013 by snehal harawande
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid