Testing Entity Framework Applications, Pt. 3: NDbUnit

This is the third of a three part series that deals with the issue of faking test data in the context of a legacy app that was built with Microsoft's Entity Framework (EF) on top of an MS SQL Server database – a scenario that can be found very often. Please read the first part for a description of the sample application, a discussion of some general aspects of unit testing in a database context, and of some more specific aspects of the EF/MSSQL combination discussed here.

Lately, I wondered how you would ‘mock’ the data layer of a legacy application, when this data layer is made up of an MS Entity Framework (EF) model in combination with a MS SQL Server database. Originally, this question came up in the context of how you could enable higher-level integration tests (automated UI tests, to be exact) for a legacy application that uses this EF/MSSQL combo as its data store mechanism – a not so uncommon scenario.

The question sparked my interest, and I decided to dive into it somewhat deeper. What I've found out is, in short, that it's not very easy and straightforward to do it – but it can be done. The two strategies that are best suited to fit the bill involve using either the (commercial) Typemock Isolator tool or the (free) NDbUnit framework. The use of Typemock was discussed in the previous post, this post now will present the NDbUnit approach...

NDbUnit is an Apache 2.0-licensed open-source project, and like so many other Nxxx tools and frameworks, it is basically a C#/.NET port of the corresponding Java version (DbUnit namely). In short, it helps you in flexibly managing the state of a database in that it lets you easily perform basic operations (like e.g. Insert, Delete, Refresh, DeleteAll) against your database and, most notably, lets you feed it with data from external XML files.

Let's have a look at how things can be done with the help of this framework.

Preparing the Test Data

Compared to Typemock, using NDbUnit implies a totally different approach to meet our testing needs. So the testing scenario described here requires an instance of an SQL Server database in operation, and it also means that the Entity Framework model that sits on top of this database is completely unaffected.

First things first: For its interactions with the database, NDbUnit relies on a .NET Dataset xsd file. See Step 1 of their Quick Start Guide for a description of how to create one. With this prerequisite in place then, the test fixture's setup code could look something like this:

[TestFixture, TestsOn(typeof(PersonRepository))] 
[Metadata("NDbUnit Quickstart URL", 
          "http://code.google.com/p/ndbunit/wiki/QuickStartGuide")] 
[Description("Uses the NDbUnit library to provide test data to a local database.")] 
public class PersonRepositoryFixture 
{ 
    #region Constants 

    private const string XmlSchema = @"..\..\TestData\School.xsd"; 

    #endregion // Constants 

    #region Fields 

    private SchoolEntities _schoolContext; 
    private PersonRepository _personRepository; 
    private INDbUnitTest _database; 

    #endregion // Fields 

    #region Setup/TearDown 

    [FixtureSetUp] 
    public void FixtureSetUp() 
    { 
        var connectionString = ConfigurationManager.ConnectionStrings["School_Test"].ConnectionString; 

        _database = new SqlDbUnitTest(connectionString); 
        _database.ReadXmlSchema(XmlSchema); 

        var entityConnectionStringBuilder = new EntityConnectionStringBuilder 
        { 
            Metadata = "res://*/School.csdl|res://*/School.ssdl|res://*/School.msl", 
            Provider = "System.Data.SqlClient", 
            ProviderConnectionString = connectionString 
        }; 

        _schoolContext = new SchoolEntities(entityConnectionStringBuilder.ConnectionString); 
        _personRepository = new PersonRepository(this._schoolContext); 
    } 

    [FixtureTearDown] 
    public void FixtureTearDown() 
    { 
        _database.PerformDbOperation(DbOperationFlag.DeleteAll); 
        _schoolContext.Dispose(); 
    } 

    ...

As you can see, there is slightly more fixture setup code involved if your tests are using NDbUnit to provide the test data: Because we're dealing with a physical database instance here, we first need to pick up the test-specific connection string from the test assemblies' App.config, then initialize an NDbUnit helper object with this connection along with the provided xsd file, and also set up the SchoolEntities and the PersonRepository instances accordingly.

The _database field (an instance of the INdUnitTest interface) will be our single access point to the underlying database: We use it to perform all the required operations against the data store. To have a flexible mechanism to easily insert data into the database, we can write a helper method like this:

private void InsertTestData(params string[] dataFileNames) 
{ 
    _database.PerformDbOperation(DbOperationFlag.DeleteAll); 

    if (dataFileNames == null) 
    { 
        return; 
    } 

    try 
    { 
        foreach (string fileName in dataFileNames) 
        { 
            if (!File.Exists(fileName)) 
            { 
                throw new FileNotFoundException(Path.GetFullPath(fileName)); 
            } 
            _database.ReadXml(fileName); 
            _database.PerformDbOperation(DbOperationFlag.InsertIdentity); 
        } 
    } 
    catch 
    { 
        _database.PerformDbOperation(DbOperationFlag.DeleteAll); 
        throw; 
    } 
}

This lets us easily insert test data from XML files, in any number and in a controlled order (which is important because we eventually must fulfill referential constraints, or we must account for some other stuff that imposes a specific ordering on data insertion). Again, as with Typemock, I won't go into API details here. Unfortunately, there isn't too much documentation for NDbUnit anyway, other than the already mentioned Quick Start Guide (and the source code itself, of course) - a not so uncommon problem with smaller Open Source Projects.

Last but not the least, we need to provide the required test data in XML form. A snippet for data from the People table might look like this, for example:

<?xml version="1.0" encoding="utf-8" ?> 
<School xmlns="http://tempuri.org/School.xsd"> 
  <Person> 
    <PersonID>1</PersonID> 
    <LastName>Abercrombie</LastName> 
    <FirstName>Kim</FirstName> 
    <HireDate>1995-03-11T00:00:00</HireDate> 
  </Person> 
  <Person> 
    <PersonID>2</PersonID> 
    <LastName>Barzdukas</LastName> 
    <FirstName>Gytis</FirstName> 
    <EnrollmentDate>2005-09-01T00:00:00</EnrollmentDate> 
  </Person> 
  <Person> 
    ...

You can also have data from various tables in one single XML file, if that's appropriate for you (but beware of the already mentioned ordering issues). It's true that your test assembly may end up with dozens of such XML files, each containing quite a big amount of text data. But because the files are of very low complexity, and with the help of a little bit of Copy/Paste and Excel magic, this appears to be well manageable.

Executing Some Basic Tests

Here are some of the possible tests that can be written with the above preparations in place:

private const string People = @"..\..\TestData\School.People.xml"; 

... 

[Test, MultipleAsserts, TestsOn("PersonRepository.GetNameList")] 
public void GetNameList_ListOrdering_ReturnsTheExpectedFullNames() 
{ 
    InsertTestData(People); 

    List<string> names = 
        _personRepository.GetNameList(NameOrdering.List); 

    Assert.Count(34, names); 
    Assert.AreEqual("Abercrombie, Kim", names.First()); 
    Assert.AreEqual("Zheng, Roger", names.Last()); 
} 

[Test, MultipleAsserts, TestsOn("PersonRepository.GetNameList")] 
[DependsOn("RemovePerson_CalledOnce_DecreasesCountByOne")] 
public void GetNameList_NormalOrdering_ReturnsTheExpectedFullNames() 
{ 
    InsertTestData(People); 

    List<string> names = 
        _personRepository.GetNameList(NameOrdering.Normal); 

    Assert.Count(34, names); 
    Assert.AreEqual("Alexandra Walker", names.First()); 
    Assert.AreEqual("Yan Li", names.Last()); 
} 

[Test, TestsOn("PersonRepository.AddPerson")] 
public void AddPerson_CalledOnce_IncreasesCountByOne() 
{ 
    InsertTestData(People); 

    int count = _personRepository.Count; 

    _personRepository.AddPerson(new Person { FirstName = "Thomas", LastName = "Weller" }); 

    Assert.AreEqual(count + 1, _personRepository.Count); 
} 

[Test, TestsOn("PersonRepository.RemovePerson")] 
public void RemovePerson_CalledOnce_DecreasesCountByOne() 
{ 
    InsertTestData(People); 

    int count = _personRepository.Count; 

    _personRepository.RemovePerson(new Person { PersonID = 33 }); 

    Assert.AreEqual(count - 1, _personRepository.Count); 
}

Not much difference here compared to the corresponding Typemock versions, except that we had to do a bit more preparational work (and also it was harder to get the required knowledge). But this picture changes quite dramatically if we look at some more demanding test cases:

OK, and What If Things are Becoming Somewhat More Complex?

Tests like the above ones represent the 'easy' scenarios. They may account for the biggest portion of real-world use cases of the application, and they are important to make sure that it is generally sound. But usually, all these nasty little bugs originate from the more complex parts of our code, or they occur when something goes wrong.

So, for a testing strategy to be of real practical use, it is especially important to see how easy or difficult it is to mimic a scenario which represents a more complex or exceptional case. The following test, for example, deals with the case that there is some sort of invalid input from the caller:

[Test, MultipleAsserts, TestsOn("PersonRepository.GetCourseMembers")] 
[Row(null, typeof(ArgumentNullException))] 
[Row("", typeof(ArgumentException))] 
[Row("NotExistingCourse", typeof(ArgumentException))] 
public void GetCourseMembers_WithGivenVariousInvalidValues_Throws
                    (string courseTitle, Type expectedInnerExceptionType) 
{ 
    var exception = Assert.Throws<RepositoryException>(() => 
                                _personRepository.GetCourseMembers(courseTitle)); 
    Assert.IsInstanceOfType(expectedInnerExceptionType, exception.InnerException); 
}

Apparently, this test doesn't need an 'Arrange' part at all (see here for the same test with the Typemock tool). It acts just like any other client code, and all the required business logic comes from the database itself. This doesn't always necessarily mean that there is less complexity, but only that the complexity happens in a different part of your test resources (in the XML files namely, where you sometimes have to spend a lot of effort for carefully preparing the required test data).

Another example, which relies on an underlying 1-n relationship, might be this:

[Test, MultipleAsserts, TestsOn("PersonRepository.GetCourseMembers")] 
public void GetCourseMembers_WhenGivenAnExistingCourse_ReturnsListOfStudents() 
{ 
    InsertTestData(People, Course, Department, StudentGrade); 

    List<Person> persons = _personRepository.GetCourseMembers("Macroeconomics"); 

    Assert.Count(4, persons); 
    Assert.ForAll( 
        persons, 
        @p => new[] { 10, 11, 12, 14 }.Contains(@p.PersonID), 
        "Person has none of the expected IDs."); 
}

If you compare this test to its corresponding Typemock version, you immediately see that the test itself is much simpler, easier to read, and thus much more intention-revealing. The complexity here lies hidden behind the call to the InsertTestData() helper method and the content of the used XML files with the test data. And also note that you might have to provide additional data which are not even directly relevant to your test, but are required only to fulfill some integrity needs of the underlying database.

Conclusion

The first thing to notice when comparing the NDbUnit approach to its Typemock counterpart obviously deals with performance: Of course, NDbUnit is much slower than Typemock. Technically, it doesn't even make sense to compare the two tools. But practically, it may well play a role and could or could not be an issue, depending on how many tests you have of this kind, how often you run them, and what role they play in your development cycle.

Also, because the dataset from the required xsd file must fully match the database schema (even in parts that otherwise wouldn't be relevant to you), it can be quite cumbersome to be in a team where different people are working with the database in parallel.

My personal experience is – as already said in the first part – that Typemock gives you a better development experience in a 'dynamic' scenario (when you're working in some kind of TDD-style, you're oftentimes executing the tests from your dev box, and your database schema changes frequently), whereas the NDbUnit approach is a good and solid solution in more 'static' development scenarios (when you need to execute the tests less frequently or only on a separate build server, and/or the underlying database schema can be kept relatively stable), for example some variations of higher-level integration or User-Acceptance tests.

But in any case, opening Entity Framework based applications for testing requires a fair amount of resources, planning, and preparation work – it's definitely not the kind of stuff that you would call 'easy to test'. Hopefully, future versions of EF will take testing concerns into account. Otherwise, I don't see too much of a future for the framework in the long run, even though it's quite popular at the moment...

The Sample Solution

A sample solution (VS 2010) with the code from this article series is available via my Bitbucket account from here (Bitbucket is a hosting site for Mercurial repositories. The repositories may also be accessed with the Git and Subversion SCMs - consult the documentation for details. In addition, it is possible to download the solution simply as a zipped archive – via the 'get source' button on the very right.). The solution contains some more tests against the PersonRepository class, which are not shown here. Also, it contains database scripts to create and fill the School sample database.

To compile and run, the solution expects the Gallio/MbUnit framework to be installed (which is free and can be downloaded from here), the NDbUnit framework (which is also free and can be downloaded from here), and the Typemock Isolator tool. Moreover, you will need an instance of the Microsoft SQL Server DBMS, and you will have to adapt the connection strings in the test projects App.config files accordingly.