Click here to Skip to main content
Click here to Skip to main content

CommandLineHelper class to launch console applications and capture their output

, 23 Oct 2006
Rate this:
Please Sign up or sign in to vote.
Runs a console application and returns its output as a string. Useful for writing object-oriented wrappers around command line utilities, such as Subversion's svn utility.

Contents

Introduction

This small class provides a very simple interface for running command line applications. It traps their standard output and error streams, and returns the output as a string. It provides method overloads to either return the errors in a second string, or to throw an exception if the error stream is non-empty.

The class was written to simplify the task of writing object-oriented wrappers around command line utilities, particularly svn (Subversion's command line client utility).

There are certain complexities with trapping the error and outputs streams, which can lead to blocking behaviour and a hung application. The class circumvents these problems by asynchronously reading from the streams.

Goal

The goal of the utility is to provide a very simple interface for calling a command line process and then carrying out further processing on its outputs.

The file name and command line arguments are passed to a static method. The method launches the process, then returns the console output to the caller as a string when the process completes.

If you need something more complex, such as being able to process the outputs while they are being generated, then I would suggest looking at Mike Mayer's article "Launching a process and displaying its standard output".

Background

The utility uses the CLR's Process class to call the command line application.

ProcessStartInfo.RedirectStandardError and RedirectStandardOutput are set to trap the contents of the process' standard error and standard output streams. This allows access to Process.StandardOutput and Process.StandardError (these are both StreamReaders.)

A complication arises when reading from both streams.

The command line application will block whenever either of the streams' buffers fills up, while it waits for the Process class to make space by reading from the relevant buffer.

This causes your program to hang if it tries to read from one of the streams when the other stream's buffer has filled up. This is because the program and the command line application are each waiting for a response from the other.

The solution is to read from one or both of the streams in a separate thread. Asynchronous delegates make this a simple task.

Unit tests are provided to demonstrate the problem and its solution.

The CommandLineHelper class

Here is the declaration of the CommandLineHelper class:

using System;
using System.IO;
using System.Diagnostics;
using System.Threading;
using System.Security.Permissions;

namespace AndrewTweddle.Tools.Utilities.CommandLine
{
    [SecurityPermissionAttribute(SecurityAction.LinkDemand, 
                                 Unrestricted=true)]
    public static class CommandLineHelper
    {
        private delegate string StringDelegate();

        public static string Run(string fileName, 
               string arguments, out string errorMessage)
        {
            errorMessage = "";
            Process cmdLineProcess = new Process();
            using (cmdLineProcess)
            {
                cmdLineProcess.StartInfo.FileName = fileName;
                cmdLineProcess.StartInfo.Arguments = arguments;
                cmdLineProcess.StartInfo.UseShellExecute = false;
                cmdLineProcess.StartInfo.CreateNoWindow = true;
                cmdLineProcess.StartInfo.RedirectStandardOutput = true;
                cmdLineProcess.StartInfo.RedirectStandardError = true;

                if (cmdLineProcess.Start())
                {
                    return ReadProcessOutput(cmdLineProcess, 
                           ref errorMessage, fileName);
                }
                else
                {
                    throw new CommandLineException(String.Format(
                        "Could not start command line process: {0}", 
                        fileName));
                    /* Note: arguments aren't also shown in the 
                     * exception as they might contain privileged 
                     * information (such as passwords).
                     */
                }
            }
        }

        private static string ReadProcessOutput(Process cmdLineProcess, 
                ref string errorMessage, string fileName)
        {
            StringDelegate outputStreamAsyncReader
               = new StringDelegate(cmdLineProcess.StandardOutput.ReadToEnd);
            StringDelegate errorStreamAsyncReader
               = new StringDelegate(cmdLineProcess.StandardError.ReadToEnd);

            IAsyncResult outAR 
                = outputStreamAsyncReader.BeginInvoke(null, null);
            IAsyncResult errAR = errorStreamAsyncReader.BeginInvoke(null, null);

            if (Thread.CurrentThread.GetApartmentState() == ApartmentState.STA)
            {
                /* WaitHandle.WaitAll fails on single-threaded 
                 * apartments. Poll for completion instead:
                 */
                while (!(outAR.IsCompleted && errAR.IsCompleted))
                {
                    /* Check again every 10 milliseconds: */
                    Thread.Sleep(10);
                }
            }
            else
            {
                WaitHandle[] arWaitHandles = new WaitHandle[2];
                arWaitHandles[0] = outAR.AsyncWaitHandle;
                arWaitHandles[1] = errAR.AsyncWaitHandle;

                if (!WaitHandle.WaitAll(arWaitHandles))
                {
                    throw new CommandLineException(
                        String.Format("Command line aborted: {0}", fileName));
                    /* Note: arguments aren't also shown in the 
                     * exception as they might contain privileged 
                     * information (such as passwords).
                     */
                }
            }

            string results = outputStreamAsyncReader.EndInvoke(outAR);
            errorMessage = errorStreamAsyncReader.EndInvoke(errAR);

            /* At this point the process should surely have exited,
             * since both the error and output streams have been fully read.
             * To be paranoid, let's check anyway...
             */
            if (!cmdLineProcess.HasExited)
            {
                cmdLineProcess.WaitForExit();
            }

            return results;
        }

        public static string Run(string fileName, string arguments)
        {
            string result;
            string errorMsg = String.Empty;
            
            result = Run(fileName, arguments, out errorMsg);

            if (errorMsg.Length > 0)
                throw new CommandLineException(errorMsg);

            return result;
        }

        public static string Run(string fileName)
        {
            return Run(fileName, "");
        }
    }
}

Using the code

Opening the solution

The code was written using Visual Studio 2005. I used NUnit 2.3.0 for unit testing.

If you don't have NUnit installed, then you will receive error messages when opening the solution because the references to the NUnit assemblies cant be resolved. Ignore these and consider removing the unit testing project from the solution.

Better yet, just download NUnit. And consider downloading TestDriven.Net too. This is a very useful Visual Studio extension which integrates NUnit and other testing frameworks into the IDE.

Calling CommandLineHelper's Run methods

There are 3 overloads of the static Run method. These differ according to how you wish to handle errors being written to the error stream.

The first overload exposes the contents of the error stream as an out string parameter. You will need to check for a non-empty error string and decide how to handle this yourself.

public static string Run(string fileName, string arguments, 
              out string errorMessage)  

The other overloads will check for a non-empty error stream. If there are errors, a CommandLineException will be thrown whose Message property will contain the contents of the error stream.

public static string Run(string fileName, string arguments)  

and

public static string Run(string fileName)  

NB: With these two overloads there is no way of accessing the output stream's contents when an exception is thrown. This could be a problem if the command line process is partially successful.

For example, you might find that most items can be processed successfully, but a few items cause errors. If this could happen, then you should rather use the first overload of the Run method.

Running the unit tests

In principle, unit tests should never depend on external resources such as the file system or database.

But it's a bit hard to test code that runs command line applications without accessing the file system! So I confess - I have broken this rule...

To run the unit tests you will need to make some changes to the code.

Firstly, there is a console application which was written specifically for testing purposes. It writes hundreds of lines to the standard output and error streams alternately. You will find need to build this project. It is in the Utilities.CommandLineHelperTester project.

One of the tests uses CommandLineHelperTester.exe. The other tests all use svn, the Subversion command line client utility. If you have Subversion installed, you will only need to make changes to the paths at the top of the CommandLineHelperTester.cs file:

[TestFixture]
public class CommandLineHelperTester
{
    private const string smallVersionedTestFolder 
        = @"c:\SQR\Tools\Source\Utilities\Utilities.CommandLine";
    private const string largeVersionedTestFolder = @"c:\EstateCanePro";
    private const string unversionedFolder = @"c:\temp";
    private const string pathToTestApp
        = @"c:\SQR\Tools\Source\Utilities\Utilities" + 
          @".CommandLine.CommandLineHelperTestApp\" + 
          @"bin\Debug\CommandLineHelperTestApp.exe";
    
    
    // ...
} 

If you don't have Subversion, then you will need to find some other command line application for the unit tests to use. This will require changes to be made in various parts of the unit test file - just search for "svn".

Observing the blocking behaviour

One of the unit tests has been written to demonstrate the problem of the program hanging when stream buffers fill up.

For obvious reasons, this unit test is ignored by default:

[Test, 
Ignore("Hangs on a large working copy due 
           to blocking when the buffer fills")
]
public void RunSvnStatusCommandLineWithLargeWorkingCopy()
{
    // ...
}

Comment out the line with the Ignore attribute. Then run the unit test in the NUnit GUI (it's probably not a good idea to run the unit test in Visual Studio's Output window!)

Kill the NUnit GUI when you get tired of waiting for a response.

Creating object-oriented wrappers around command-line utilities

CommandLineHelper was written so that I could create my own library of Subversion utilities in C#. Wrapping the command line utility was a much quicker approach than learning the intricacies of the Subversion API (which would probably have required delving into C++, a language I haven't used in over 10 years, and probably the Apache Portable Runtime library as well.)

I must point out that it is also a much dirtier approach. The code is probably much more vulnerable to changes in the output format. There is a trade-off to be made, and it's important that you realise that.

I needed the utilities to automate and streamline some of my own development processes. So I was quite content with the quick and dirty approach.

Quite a few of the svn command line options now allow XML as the output format. This can make it wonderfully simple to process the results:

public class Revision
{
    public static int GetLastCommittedRevision(string workingCopyPath)
    {
        string commandLineArguments
            = String.Format(@"log --xml --revision COMMITTED ""{0}""",
              workingCopyPath);

        string xmlLogOutput 
            = CommandLineHelper.Run("svn", commandLineArguments);

        XmlDocument xdoc = new XmlDocument();
        xdoc.LoadXml(xmlLogOutput);
        XPathNavigator xnav = xdoc.CreateNavigator();
        XPathNodeIterator xiter = xnav.Select("/log/logentry/@revision");
        if (xiter.MoveNext())
        {
            return xiter.Current.ValueAsInt;
        }
        else
        {
            return 0;  /* No log entry */
        }
    }
}

In other cases, you will need to process the output yourself.

I usually use regular expressions to parse the output string and turn the results into objects.

On the other hand, these can be very cryptic and difficult to debug. You are also less likely to detect any unexpected changes to the output format. So the safer option might be to write custom code to turn the command line outputs into objects.

For what it's worth, here is an extract from the code which helps to wrap the "svn status" utility. The documentation for "svn status" can be found online at red-bean.com.

static public StatusInfo[] GetStatuses(string path, 
            SvnStatusOptions options, SvnStatusFilter filter)
{
    /* If options is null, then use a default instead */
    if (options == null)
        options = new SvnStatusOptions();

    string errorMessage;

    string svnStatusOutput = CommandLineHelper.Run("svn", 
        String.Format("status {0} {1}", 
        options.CommandLineSwitches, path),
        out errorMessage);

    if (errorMessage.Length > 0)
    {
        if (File.Exists(path))
        {
            /* "svn status" only returns statuses of working copies.
             * It generates an error message when the path is not
             * part of a working copy. It's more useful to return a 
             * status of ItemStatus.NotInAWorkingCopy instead.
             * So check for this situation and, if it's the case, 
             * create a StatusInfo record instead of throwing an 
             * exception:
             */
            if (IsANonWorkingCopyErrorMessage(errorMessage))
            {
                StatusInfo nonWCStatusInfo
                    = GetNonWorkingCopyStatusInfo();

                if (IsStatusInfoIncludedInFilter(nonWCStatusInfo,
                    filter))
                {
                    return new StatusInfo[1] { nonWCStatusInfo };
                }
                else
                {
                    return new StatusInfo[0] { };
                }
            }
            else
            {
                throw new SvnStatusException(errorMessage);
            }
        }
        else
        {
            throw new SvnStatusException(errorMessage);
        }
    }

    return ConvertSvnStatusOutputToStatusInfos(svnStatusOutput, 
        options, filter);
}

private static bool IsANonWorkingCopyErrorMessage(
    string svnStatusErrorMessage)
{
    string rxNotAWorkingCopyPattern
        = @"svn:\s'[^']*' is not a working copy";

    Regex rgx = new Regex(rxNotAWorkingCopyPattern);

    return rgx.Match(svnStatusErrorMessage).Success;
}

private static StatusInfo[] ConvertSvnStatusOutputToStatusInfos(
    string svnStatusOutput, SvnStatusOptions options, 
    SvnStatusFilter filter)
{
    string rxLinePattern;

    if (options.CheckForUpdates)
    {
        if (options.Verbose)
        {
            rxLinePattern
                = @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
                + @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
                + @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S]).."
                + @"(?<OutOfDateCode>[ *])\s*(?<WorkingRevision>\d+)\s+"
                + @"(?<LastCommittedRevision>\d+)\s+"
                + @"(?<LastCommittedAuthor>\S+)\s*"
                + @"(?<WorkingCopyPath>\S.*)$";
        }
        else
        {
            rxLinePattern
                = @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
                + @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
                + @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S]).."
                + @"(?<OutOfDateCode>[ *])\s*(?<WorkingRevision>\d+)\s*"
                + @"(?<WorkingCopyPath>\S.*)$";
        }
    }
    else
    {
        if (options.Verbose)
        {
            rxLinePattern
                = @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
                + @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
                + @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S]).."
                + @"\s*(?<WorkingRevision>\d+)\s+"
                + @"(?<LastCommittedRevision>\d+)\s+"
                + @"(?<LastCommittedAuthor>\S+)\s*"
                + @"(?<WorkingCopyPath>\S.*)$";
        }
        else
        {
            rxLinePattern
                = @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
                + @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
                + @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S])..\s*"
                + @"(?<WorkingCopyPath>\S.*)$";
        }
    }

    MatchCollection matches = Regex.Matches(svnStatusOutput,
        rxLinePattern, RegexOptions.Multiline | RegexOptions.IgnoreCase 
        | RegexOptions.IgnorePatternWhitespace);

    List<StatusInfo> statusInfoList = new List<StatusInfo>();

    foreach (Match currMatch in matches)
    {
        StatusInfo currStatusInfo 
            = ConvertMatchToStatusInfo(currMatch, options);

        bool isIncludedInFilter = true;

        if (filter != null)
        {
            isIncludedInFilter
                = IsStatusInfoIncludedInFilter(currStatusInfo, filter);
        }

        if (isIncludedInFilter)
        {
            statusInfoList.Add(currStatusInfo);
        }
    }

    return statusInfoList.ToArray();
}

private static StatusInfo ConvertMatchToStatusInfo(Match currMatch, 
    SvnStatusOptions options)
{
    StatusInfo currStatusInfo = new StatusInfo();

    /* Save the working copy path: */
    currStatusInfo.WorkingCopyPath
        = currMatch.Groups["WorkingCopyPath"].Value;

    /* Set the item's status: */
    currStatusInfo.ItemStatus = CodeToItemStatus(
        currMatch.Groups["ItemStatusCode"].Value);

    /* Set the status of the item's properties: */
    currStatusInfo.PropertyStatus = CodeToPropertyStatus(
        currMatch.Groups["PropertyStatusCode"].Value);

    /* Set the locked status: */
    currStatusInfo.IsLocked
        = (currMatch.Groups["LockedCode"].Value == "L");

    /* Set the "added with history" status: */
    currStatusInfo.AddedWithHistory
        = (currMatch.Groups["WithHistoryCode"].Value == "*");

    /* Set the switched status: */
    currStatusInfo.IsSwitched
        = (currMatch.Groups["SwitchCode"].Value == "S");

    /* Set the "up to date" status: */
    if (options.CheckForUpdates)
    {
        currStatusInfo.UpdateStatus = CodeToUpdateStatus(
            currMatch.Groups["OutOfDateCode"].Value);
    }
    else
        currStatusInfo.UpdateStatus = UpdateStatus.Unknown;

    /* Save the last committed revision and author, if available: */
    if (options.Verbose)
    {
        currStatusInfo.LastCommittedRevision
            = int.Parse(
                currMatch.Groups["LastCommittedRevision"].Value);
        currStatusInfo.LastCommittedAuthor
            = currMatch.Groups["LastCommittedAuthor"].Value;
    }

    if (options.Verbose || options.CheckForUpdates)
        currStatusInfo.WorkingRevision
            = int.Parse(currMatch.Groups["WorkingRevision"].Value);

    return currStatusInfo;
}

Other points of interest

Single-threaded apartment mode

The unit tests were running fine in NUnit's GUI interface. But I ran into trouble when I started running the tests within the Visual Studio environment.

On further investigation I discovered that within Visual Studio the tests were running in single-threaded apartment mode.

I was using WaitHandle.WaitAll to wait for both threads to finish. But according to the MSDN library: The WaitAll method is not supported on threads that have STAThreadAttribute.

I changed my code to check Thread.CurrentThread.GetApartmentState(). In single-threaded apartment mode I would poll for completion instead. This solved the problem.

Possible gotcha's

I mentioned earlier that you could run into problems if the output format changes when the command line application is upgraded.

There could also be a problem if you deploy a program based on this approach to a machine which is set up to use a different language. If the console application has been localized, then the output might differ across languages, and your wrapper will fail to process the outputs correctly.

A third problem could arise if the console application needs to prompt the user for inputs.

For example, Subversion can be set up to use the same user name and password on a particular machine. If you haven't set this up yet, then the results might be unpredictable. Since I haven't tried it out on a new machine, I don't know if it will throw an exception, or simply hang while waiting for the user to enter a user name and password.

For these reasons I suggest that you only wrap command line applications for use in a controlled environment (such as your own machine).

TO DO's

  • Add asynchronous BeginRun and EndRun methods.
  • Add support for passing data to the standard input stream.

[Please note: I'm listing these features for the sake of completeness. I have no compelling need for these features, and I don't plan to add them until I do. So if you need either of these features, don't wait for me... go ahead and add them yourself.]

History

22 October 2006

  • Initial version submitted.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Andrew Tweddle
Architect Dariel Solutions
South Africa South Africa
Andrew Tweddle started his career as an Operations Researcher, but made the switch to programming in 1997. His current programming passions are Powershell and WPF.
 
He has worked for one of the "big 4" banks in South Africa as a software team lead and an architect, at a Dynamics CRM consultancy and is currently an architect at Dariel Solutions working on software for a leading private hospital network.
 
Before that he spent 7 years at SQR Software in Pietermaritzburg, where he was responsible for the resource planning and budgeting module in CanePro, their flagship product for the sugar industry.
 
He enjoys writing utilities to streamline the software development and deployment process. He believes Powershell is a killer app for doing this.
 
Andrew is a board game geek (see www.boardgamegeek.com) with a collection of over 190 games! He also enjoys digital photography, camping and solving puzzles - especially Mathematics problems.
 
His Myers-Briggs personality profile is INTJ.
 
He lives with his wife, Claire and his daughters Lauren and Catherine in Johannesburg, South Africa.

Comments and Discussions

 
QuestionFails when input file name is C:\\Windows\System32\cmd.exe "C:\\Program Files\\abc\\abc.exe" PinmemberGanesh Satpute29-Apr-14 3:16 
GeneralWorked Great Pinmemberledtech319-May-13 4:42 
Questiontodo list Pinmemberkiquenet.com22-Jun-12 1:30 
what about Add asynchronous BeginRun and EndRun methods.?
 
thx
kiquenet.com

QuestionHow to resolve the input required at commandline PinmemberShamT20-Nov-07 7:21 
GeneralGreat stuff Pinmemberigitur1-Dec-06 1:40 
QuestionWin32Exception on Process.Start() Pinmemberleonelgalan13-Nov-06 18:48 
AnswerRe: Win32Exception on Process.Start() PinmemberAndrew Tweddle14-Nov-06 10:35 
GeneralRe: Win32Exception on Process.Start() Pinmemberleonelgalan14-Nov-06 17:39 
GeneralRe: Win32Exception on Process.Start() - Yes its a CYGWIN APP Pinmemberleonelgalan15-Nov-06 20:05 
GeneralRe: Win32Exception on Process.Start() - Yes its a CYGWIN APP PinmemberAndrew Tweddle15-Nov-06 20:10 
General.::. Good Article .::. PinmemberProgramm3r25-Oct-06 3:41 
GeneralRe: .::. Good Article .::. PinmemberAndrew Tweddle25-Oct-06 20:43 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140721.1 | Last Updated 23 Oct 2006
Article Copyright 2006 by Andrew Tweddle
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid