Click here to Skip to main content
14,331,901 members

Evaluation of .NET Testing

Rate this:
5.00 (15 votes)
Please Sign up or sign in to vote.
5.00 (15 votes)
19 Aug 2015CPOL
Stimulating journey on evaluation of .NET Testing over the years

Contents

Introduction

So far, I used to write the specific technology article for the development community to build the Application. Now, I'm shifting the gear into the most valuable process - TESTING.

In this competitive world, it is highly important for any software company to assure the testing throughout the life cycle. Itz for monitoring the process and methods to ensure quality and superiority.

This paper talks about the evaluation of software (automation) testing cycle in the industry, specially in .NET technology platform.

Types

In the industry, testing is categorized based on different dimension as listed below:

Level of Testing

S.No. Name of Testing Owned by
1. Unit Testing Developers; during Coding Phase
2. Integration Testing Developers with Build Team; during Application Integration Phase
3. System Testing Testers; during Application Quality Validation Phase

Intent of Testing

S.No. Name of Testing Purpose
1. Performance Testing To determine the execution speed or effectiveness of the Application and its related system resources
2. Load Testing To determine the system's behaviour under both normal and anticipated peak load conditions.
3. Regression Testing To validate the Application changes to make sure that the older programming still works with the new changes
4. Acceptance Testing To determine by the end user whether the requirements of a specification or contract are met

Patterns

Project Level Patterns

Project level patters are depicted in details at the previous section Types

Data Points Patterns

A data-driven unit test is a unit test that is run repeatedly for each row in a data source.

A common scenario for using data-driven unit tests is the use of multiple input values to test an API. Instead of writing multiple unit tests that call the API, each with a new set of inputs, or creating an array in a unit test and using looping code, you can instead write a single unit test method that exercises the API.

You can create a data-driven unit test in either of two ways:

  • Use the Properties window and set specific properties on an existing unit test.
  • Code the test as a data-driven unit test.

To configure your existing unit test, add attributes that define the data source you want it to use, the way you want that data to be accessed, and the table whose rows you want your test to use as input. It is depicted below:

[TestClass]
public class TestClass
{
    private TestContext m_testContext;
    public TestContext TestContext
    {
        get { return m_testContext; }
        set { m_testContext = value; }
    }
    [TestMethod]
    [DeploymentItem("TESTDB.MDB")]
    [DataSource("System.Data.OleDb", "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\"TESTDB.MDB\"", "Students", DataAccessMethod.Sequential)]
    public void TestMethod()
    {
        Console.WriteLine( "StudentID: {0}, Student LastName: {1}", TestContext.DataRow["StudentID"],  TestContext.DataRow["LastName"] );
    }
}

Unit Testing

Unit tests are used to exercise other source code by directly calling the methods of a class, passing appropriate parameters, and then, if you include Assert statements, they can test the values that are produced against expected values. Unit test methods reside in test classes, which are stored in source code files.

Essentials of Unit Testing

In the business context, Costing factor drives the importance of Unit Testing in any software development cycle. It is essential to identify/fix the bug early not only timing perspective but also cost factor. Cost of Testing Phase

Benefits of Unit Testing- DEV Community

As the tests need to be Fast, Robust, Readable, Focused, Deterministic and Independent, the developer community will find that it will improve Application code’s design.

  • Great Tests - Effect on production code design
  • Fast - Concise methods based on short/quick feedback
  • Robust - Design is opened to change
  • Readable - Better APIs
  • Focused - Each class and method will have a well-defined responsibility
  • Deterministic - Design decouples code from environmental effects
  • Independent - Design does not rely on shared or static state

Business Benefits of Unit Testing

  • Go To Market - with confidence on Product Stability
  • Ensures that Core Building Block are intact
  • Greater visibility on Product's technical compliance
  • High degree of Product Quality attracts the customer base

.NET Testing

Overview

Unit testing has the greatest effect on the quality of your code when it’s an integral part of your software development work flow. As soon as you write a function or other block of application code, create unit tests that verify the behaviour of the code in response to standard, boundary, and incorrect cases of input data, and that check any explicit or implicit assumptions made by the code.

With test driven development, the developers create the unit tests before they write the code, so they use the unit tests as both design documentation and functional specifications.

Top-5 Framework

 

S.No. Framework Description
1. MSTest Part of the Visual Studio Unit Testing Framework, which also has command-line tool
2. NUnit Includes GUI, command line, integrates into Visual Studio with ReSharper
3. Pex Microsoft Research project providing White box testing for .NET, using Z3 constraint solver to generate unit test input
4. XUnit Open source, community-focused unit testing tool for the .NET Framework. It supports C#, F#, VB.NET and other .NET languages; works with ReSharper, CodeRush, TestDriven.NET and Xamarin.
5. Gallilo Extensible, and neutral automation platform that provides a common object model, runtime services and tools (such as test runners) that may be leveraged by any number of test frameworks.

 

Generations

Based on Microsoft .NET Framework and Unit Testing Framework evolutions, I personally categorize into 3 generations as:

  1. Gen X - Manual
  2. Gen Y - Auto Stub/Skeleton
  3. Gen Z - Full Automation

It is represented in the diagrammatic way as:

Unit Test Generations

Use Case

To illustrate 3 generations of Unit Testing and its related process, a simple use case is taken as below:

Unit Test Use Case Reference

This example takes the details of Personal Saving Account and its related operations namely deposit and withdrawal. Respective business conditions like non-zero balance before withdrawal are meticulously added in the logic.

GenX

With my context, GenX represents the creation of Unit Test classes, methods from scratch manually. It was the initial model built in .NET frameworks. Pre-conditions are:

  • Each test class is marked with the [TestClass()] attribute.
  • Each unit test is a test method that is marked with the [TestMethod()] attribute.
  • Assert statements are used to test the values that are produced against expected values

These attributes are assigned automatically when unit tests are generated; when you code unit tests by hand, you have to add class and method attributes yourself

Implementation

.NET GenX Unit Testing phase contains three parts, namely

  1. Arrange: Set up everything needed for the running the tested code. This includes any initialization of dependencies, mocks and data needed for the test to run.
  2. Act: Invoke the code under test.
  3. Assert: Specify the pass criteria for the test, which fails it if not met.

The above steps are illustrated using a simple test method as:

[TestMethod]
public void ValidateZeroBalance()
{
    /// 1. Arrange
    PersonalAccount account = new PersonalAccount();
      
    /// 2. Act
    var result = account.GetBalance();
      
    /// 3. Assert
    Assert.AreEqual(0, result);
}

This pattern is very readable and consistent. It’s very easy to identify the different parts of the test, and to create tests for other scenarios from a root test.

Code Sample

In GenX code sample, we created PersonalSavingsAccount class based on the given use case. For convenient purpose, the main class is part of separate project, in difference with Test Class - PersonalSavingsAccountTests.

Due to the separate layer on main and test classes, the project binaries are referenced into Test class for future usage to build the test scenarios & conditions as below:

Unit Test Library Reference

 

Unit Test Class - PersonalSavingsAccountTests is written manually from scratch as per GenX model.

 

Each test condition is represented in details at each TestMethod in GenX testing frameworks.

[TestClass]
 public class PersonalSavingsAccountTests
 {
     [TestMethod()]
     public void PersonalSavingsAccountTest()
     {
         PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 900.90);
     }
     [TestMethod()]
     public void WithdrawSuccessTest()
     {
         PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 10000.11);
         account.Withdraw(500.50);
     }
     [TestMethod()]
     public void WithdrawFailZeroTest()
     {
         PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 150.50);
         account.Withdraw(0);
     }
     [TestMethod()]
     public void WithdrawFailBalanceTest()
     {
         PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 150.50);
         account.Withdraw(250.25);
     }
     [TestMethod()]
     public void DepositSuccessTest()
     {
         PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 10000.10);
         account.Deposit(200.20);
     }
     [TestMethod()]
     public void DepositFailZeroTest()
     {
         PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 100.10);
         account.Deposit(0);
     }
 }

Unit test result - Test Explorer

There are three ways to verify that a unit test has passed:

  1. Use one or more Assert statements to validate specific outcomes.
  2. Verify that no exception was thrown. It is still advisable to use one or more Assert statements.
  3. Verify that a particular exception is thrown. You can do this by using the ExpectedExceptionAttribute attribute.

In our example, the result of Unit Test Execution is listed under Test Explorer as attached:

Unit Test Result

Test Explorer explains the various test conditions and its source code, related result, stack-trace on exception, etc. It will give the complete picture of software testing for the development community.

GenY

GenY leverage the framework, which allows the platform to generate unit test stubs from your application code. It takes away the routine test creation tasks allowing a developer to focus on the highest value, writing the test itself.

One of the commonly used GenY tool is Unit Test Generator extension - Visual Studio Gallery. It helps to increase developer productivity by decreasing the set up work involved in creating new unit tests.

Implementation

Step 1: Tool Installation

Once installed, you will find the extension under Extensions and Updates in the Tools menu.

Unit Test Extensions

Step 2: Tool Usage

Right-click in your method to select Generate Unit Test.

Unit Test Extensions Menu

Step 3: Unit Test Creation

Visual Studio Unit Test Generator provides the ability to generate and configure a test project, test class, and test stub to enable you to get to write your test sooner. It provides a set of configuration options that allow you to tailor the generation to match your naming and organization schemes.

Unit Test Stub Creation

This framework is also fully configurable to support MSTest, XUnit, and NUnit so that you can choose the framework that is most suitable in your environment.

Code Sample

On running the previous process - Step 3: Unit Test Creation, Unit Test Generator framework analysis the marked class - PersonalSavingsAccount and generates the stub of the respective Test Class automatically

In our use case context of Personal Account, GenY supports the developer to create the unit test from a context menu so that the developer need not worry on setting up test project and files. Herez the auto generated test stub class for our given use case:

    [TestClass()]
    public class PersonalSavingsAccountTests
    {
        [TestMethod()]
        public void PersonalSavingsAccountTest()
        {
            Assert.Fail();
        }

        [TestMethod()]
        public void WithdrawTest()
        {
            Assert.Fail();
        }

        [TestMethod()]
        public void DepositTest()
        {
            Assert.Fail();
        }
    }
}

Along with our process, Unit Test Generator framework generates a test project and a test class—if needed—then adds the references, the name space, and the test methods.

Itz quite useful extension not only adding back functionality which had been removed but also extending the functionality by adding in support for different testing frameworks.

The limitation is that it doesn't support generating tests for internal classes and methods. It should be rather trivial for this to be fixed and an InternalsVisibileTo attribute added to AssemblyInfo.

GenZ

GenZ is the highest level of Unit Test Automation. As part of last month Visual Studio 2015 launch, IntelliTest is introduced to create Unit Test automatically that give maximum of code coverage.

Pex is a Microsoft Research project, which generates Unit Tests from hand-written Parametrized Unit Tests through Automated Exploratory Testing based on Dynamic Symbolic Execution. Pex is the inspiration for Gen-Z IntelliTest framework in Visual Studio 2015.

In traditional unit test suites, each test case represents an exemplary usage scenario, and the assertions embody the relationship between the input and output. Verifying a few such scenarios might well be enough, but experienced developers know that bugs lurk even in well-tested code, when correct but untested inputs provoke wrong responses

Implementation

IntelliTest explores your .NET code to generate test data and a suite of unit tests. For every statement in the code, a test input is generated that will execute that statement. A case analysis is performed for every conditional branch in the code.

On availability of IntelliTest framework, Visual Studio enables two menu options as highlighted below

  1. Create IntelliTest
  2. Run IntelliTest

Intelli Test Creation

“Run IntelliTest” command actually generates unit test for all methods within the specified class and proceed to execute. On running IntelliTest, API is invoked internally to guide test data generation, specify correctness properties of the code under test, and direct the exploration of the code under test.

In case you want to generate for any specific method or even start with a new one, you can use “Create IntelliTest” option.

Code Sample

As soon as the processing is done, Visual Studio will launch an another new windows called “IntelliTest Exploration Results”, that generates set of all possible unit tests based on the method parameters and which can cover the maximum of code path of the given code block. These generated test covers all possible combination of parameter values; including boundary condition.

As indicated in the below snapshot, the generated methods with it’s targets and the number of parameter (a, b) and the value that has been used for test parameter.

Intelli Test Result

If you select any of these targeted method from the IntelliTest windows, you would be able to see the Details of Generated Unit Test by the Pex. This shows how internally a generated unit test getting called and cover the certain percentage of the code.

Keeping a suite of unit tests in sync with a fast-evolving code base is a challenge. Manually adapting the code of a potentially large number of unit tests incurs significant costs. This automatic management of the suite of generated unit tests - IntelliTest, helps to resolve this challenge.

Conclusion

On deliberating with the various developers community, they felt that Unit Testing process is pain in their neck, in spite of known business & project benefits.

As the summarization of my analysis, 2 key factors that hurts the developers are:

  1. Time Consuming Process during the development cycle
  2. Unknown process/methodology to build Unit Test Classes in .NET project

Hope, this paper hints the easiness of Unit Testing in .NET paradigm and motivates them for better/wider adoption.

History

Initial Version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Ganesan Senthilvel
Architect
India India
Ganesan Senthilvel is a passionate IT leader with demonstrated 2 decades’ experience in architecture, design and implementing cutting edge solutions to address business opportunities of Enterprise applications. He earned Masters in Computer Science and Master in Business Administration. Now, he is pursing Doctorate program in Big Data. He is consistent technical contributor via COE, Blog, Whitepaper, Summit, Certification, etc. and also highly skilled at providing coaching and mentoring to internal teams and external institutes. Earned Microsoft Certifications like MCP, MCAD and financial domain certification.
He maintains his weekly blog at http://ganesansenthilvel.blogspot.in

Comments and Discussions

 
PraiseNice Article Pin
Esther_8-Oct-17 5:25
memberEsther_8-Oct-17 5:25 
GeneralRe: Nice Article Pin
Ganesan Senthilvel22-Apr-19 21:25
memberGanesan Senthilvel22-Apr-19 21:25 
PraiseInformative Article Pin
Shail Mishra12-Jul-17 18:59
memberShail Mishra12-Jul-17 18:59 
GeneralRe: Informative Article Pin
Ganesan Senthilvel16-Jul-17 17:44
memberGanesan Senthilvel16-Jul-17 17:44 
GeneralInformative Article Pin
Muthukumar Rajendran23-Aug-15 8:04
memberMuthukumar Rajendran23-Aug-15 8:04 
GeneralRe: Informative Article Pin
Ganesan Senthilvel23-Aug-15 19:19
memberGanesan Senthilvel23-Aug-15 19:19 
GeneralNice article from basic to advance concepts. Pin
hukmchand20-Aug-15 6:03
memberhukmchand20-Aug-15 6:03 
GeneralRe: Nice article from basic to advance concepts. Pin
Ganesan Senthilvel20-Aug-15 18:46
memberGanesan Senthilvel20-Aug-15 18:46 
GeneralMicrosoft are emphasising the wrong thing Pin
John Brett20-Aug-15 4:27
memberJohn Brett20-Aug-15 4:27 
GeneralRe: Microsoft are emphasising the wrong thing Pin
Ganesan Senthilvel20-Aug-15 18:54
memberGanesan Senthilvel20-Aug-15 18:54 
QuestionWhat about Mutation Testing Pin
DanielDyson20-Aug-15 1:30
memberDanielDyson20-Aug-15 1:30 
AnswerRe: What about Mutation Testing Pin
Ganesan Senthilvel20-Aug-15 18:44
memberGanesan Senthilvel20-Aug-15 18:44 
GeneralGood article for testing handson Pin
Prabhakaran Soundarapandian19-Aug-15 21:10
memberPrabhakaran Soundarapandian19-Aug-15 21:10 
GeneralRe: Good article for testing handson Pin
Ganesan Senthilvel20-Aug-15 18:41
memberGanesan Senthilvel20-Aug-15 18:41 
GeneralAwesome Article -My rating 4 out of 5 Pin
SUKUMAR.SUNDARARAJAN19-Aug-15 18:04
memberSUKUMAR.SUNDARARAJAN19-Aug-15 18:04 
GeneralRe: Awesome Article -My rating 4 out of 5 Pin
Ganesan Senthilvel20-Aug-15 18:39
memberGanesan Senthilvel20-Aug-15 18:39 
GeneralVisual Studio makes life simpler in every iteration Pin
bssrini19-Aug-15 17:43
memberbssrini19-Aug-15 17:43 
GeneralRe: Visual Studio makes life simpler in every iteration Pin
Ganesan Senthilvel20-Aug-15 18:38
memberGanesan Senthilvel20-Aug-15 18:38 
QuestionTesting Pin
Member 1192054519-Aug-15 3:56
memberMember 1192054519-Aug-15 3:56 
AnswerRe: Testing Pin
Ganesan Senthilvel19-Aug-15 4:00
memberGanesan Senthilvel19-Aug-15 4:00 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Article
Posted 19 Aug 2015

Stats

17.8K views
186 downloads
22 bookmarked