How to Test Console Applications

• Download code snippet for Visual Studio 2005 - 1.27 KB

Introduction

All UI tools frequently have convenient command lines to perform automation. When you are working on a small application, you usually don't need a user interface at all.
Automated testing is very expensive, but when it goes around console applications, there is a way to avoid spending much time and effort on implementing automation. Welcome to the world of console testing.

Definitions

Solution, project and assembly are used in Visual Studio/.NET meaning.

• Tested assembly/project – project referenced by testing assembly and needs to be unit tested
• Testing assembly/project – project that performs unit testing of tested assembly

General Recommendation for Testing in Visual Studio

• Keep all your output in the same folder. If you have two executable projects that are definitely not suitable for keeping together, probably it's time to split the solution.
• If you have test data, keep it either in the resource of testing assembly or in a dedicated file copied to the output folder
• If you have a dedicated file, the evident method to copy it is to use post-build event. For example:
  
  

  xcopy "$(SolutionDir)TestData\SampleExcelTable.xls" $(SolutionDir)bin\Debug /r /y 

Using the Code

So all you need is the usual test fixture either in Visual Studio Team Edition or NUnit.
I usually name those fixtures as ProgramTests and keep them in the same testing project as other unit tests.

There are some class members used:

• TextWriter m_normalOutput; used to swap console output

Other two used to accumulate all messages retrieved during application execution are:

• StringWriter m_testingConsole;
• StringBuilder m_testingSB;

The standard test fixture methods used to replace output and snip error code are as follows:

• TestFixtureSetUp – Set working folder and replace output with mock streams. Mock streams are usual string builders and can be used to analyze output. For example, if you know that your application should output message "in progress" you can check it after execution
• TestFixtureTearDown – Restore normal output after working out whole test fixture
• SetUp – Clears string builder before running each test
• TearDown – Prints cached in string builder content to normal console. This is not a must, but it's frequently needed in development.

namespace MyConsoleApp.Tests
{
    /// <summary>
    /// Testing application BuildAdvantage.Console.exe
    /// </summary>
    [TestFixture]
    public class ProgramTests
    {

        TextWriter m_normalOutput;
        StringWriter m_testingConsole;
        StringBuilder m_testingSB;

        [TestFixtureSetUp]
        public void TestFixtureSetUp()
        {
            // Set current folder to testing folder
            string assemblyCodeBase = 
                System.Reflection.Assembly.GetExecutingAssembly().CodeBase;

            // Get directory name
            string dirName = Path.GetDirectoryName(assemblyCodeBase);

            // remove URL-prefix if it exists
            if (dirName.StartsWith("file:\\"))
                dirName = dirName.Substring(6);

            // set current folder
            Environment.CurrentDirectory = dirName;

            // Initialize string builder to replace console
            m_testingSB = new StringBuilder();
            m_testingConsole = new StringWriter(m_testingSB);

            // swap normal output console with testing console - to reuse 
            // it later
            m_normalOutput = System.Console.Out;
            System.Console.SetOut(m_testingConsole);
        }

        [TestFixtureTearDown]
        public void TestFixtureTearDown()
        {
            // set normal output stream to the console
            System.Console.SetOut(m_normalOutput);
        }

        [SetUp]
        public void SetUp()
        {
            // clear string builder
            m_testingSB.Remove(0, m_testingSB.Length);
        }

        [TearDown]
        public void TearDown()
        {
            // Verbose output in console
            m_normalOutput.Write(m_testingSB.ToString());
        }

Private method StartConsoleApplication is used to start a process in "unit tests". It waits until the console tested application that is running returns an error code of executed tested application. All output is saved in local member testingSB.

/// <summary>
/// Starts the console application.
/// </summary>
/// <param name="arguments">The arguments for console application. 
/// Specify empty string to run with no arguments</param />
/// <returns>exit code</returns>
private int StartConsoleApplication(string arguments)
{
    // Initialize process here
    Process proc = new Process();
    proc.StartInfo.FileName = "MyConsoleApp.exe";
    // add arguments as whole string
    proc.StartInfo.Arguments = arguments;

    // use it to start from testing environment
    proc.StartInfo.UseShellExecute = false;

    // redirect outputs to have it in testing console
    proc.StartInfo.RedirectStandardOutput = true;
    proc.StartInfo.RedirectStandardError = true;

    // set working directory
    proc.StartInfo.WorkingDirectory = Environment.CurrentDirectory;
    
    // start and wait for exit
    proc.Start();
    proc.WaitForExit();
    
    // get output to testing console.
    System.Console.WriteLine(proc.StandardOutput.ReadToEnd());
    System.Console.Write(proc.StandardError.ReadToEnd());

    // return exit code
    return proc.ExitCode;
}

The simplest test example is as follows:

[Test]
public void ShowCmdHelpIfNoArguments()
{
    // Check exit is normal
    Assert.AreEqual(0, StartConsoleApplication(""));

    // Check that help information shown correctly.
    Assert.IsTrue(m_testingSB.ToString().Contains(
    Program.COMMAND_LINE_HELP));
}

The first line checks that the application has worked out correctly and returned zero error code. The latter checks that console output contains command line help. It's invariant behavior for any pure console application. Empty quote specifies no arguments, otherwise you can put arguments as in usual command prompt: StartConsoleApplication("argument /switcher options").

WARNING: Arguments separated by space will be recognized as different, quote itself (") is neglected. But the current framework can be easily extended to accept multiple-word arguments.

Negative testing can be done like this: In this framework you have an opportunity to check if output messages are correct by using m_testingSB.ToString().Contains() construction.

[Test]
public void StartWithNonExistingPath()
{
    // Throw file not found. Exit code is 1
    Assert.AreEqual(1, StartConsoleApplication(
        "NonExistingMetricsTable.xls"));

    Assert.IsTrue(m_testingSB.ToString().Contains("File not found"));

    // Error code in console
    Assert.IsTrue(m_testingSB.ToString().Contains("11")); 
 
    // File name
    Assert.IsTrue(m_testingSB.ToString().Contains("NonExistingMetricsTable.xls")); 
}

Final Notes

Please pay attention to the following:

• You don't need to reference the tested project to perform testing like this: You just use command line to run the executable.
• It's acceptable both for .NET 1.1 and .NET 2.0 (Only disadvantage of .NET 1.1: you cannot reference the tested executable assembly to get its resources likewise Program.COMMAND_LINE_HELP).
• You can use the code snippet attached to this article.
• ProgramTest is quite slow and definitely will spoil your statistics of unit testing timing, but you can still keep it separately from other unit tests.
• It's good to meet the same recommendation as for unit tests: keep program tests independent from the environment, relative path, order of running and so on.
• Such program test easily can be involved in continuous integration.
• Future development of this project plans to implement timeout for test to not let whole build process to halt.

History

• 15^th February, 2007: Initial post