Contents
This small class provides a very simple interface for running command line applications. It traps their standard output and error streams, and returns the output as a string. It provides method overloads to either return the errors in a second string, or to throw an exception if the error stream is non-empty.
The class was written to simplify the task of writing object-oriented wrappers around command line utilities, particularly svn (Subversion's command line client utility).
There are certain complexities with trapping the error and outputs streams, which can lead to blocking behaviour and a hung application. The class circumvents these problems by asynchronously reading from the streams.
The goal of the utility is to provide a very simple interface for calling a command line process and then carrying out further processing on its outputs.
The file name and command line arguments are passed to a static method. The method launches the process, then returns the console output to the caller as a string when the process completes.
If you need something more complex, such as being able to process the outputs while they are being generated, then I would suggest looking at Mike Mayer's article "Launching a process and displaying its standard output".
The utility uses the CLR's Process class to call the command line application.
ProcessStartInfo.RedirectStandardError and RedirectStandardOutput are set to trap the contents of the process' standard error and standard output streams. This allows access to Process.StandardOutput and Process.StandardError (these are both StreamReaders.)
A complication arises when reading from both streams.
The command line application will block whenever either of the streams' buffers fills up, while it waits for the Process class to make space by reading from the relevant buffer.
This causes your program to hang if it tries to read from one of the streams when the other stream's buffer has filled up. This is because the program and the command line application are each waiting for a response from the other.
The solution is to read from one or both of the streams in a separate thread. Asynchronous delegates make this a simple task.
Unit tests are provided to demonstrate the problem and its solution.
Here is the declaration of the CommandLineHelper class:
using System;
using System.IO;
using System.Diagnostics;
using System.Threading;
using System.Security.Permissions;
namespace AndrewTweddle.Tools.Utilities.CommandLine
{
[SecurityPermissionAttribute(SecurityAction.LinkDemand,
Unrestricted=true)]
public static class CommandLineHelper
{
private delegate string StringDelegate();
public static string Run(string fileName,
string arguments, out string errorMessage)
{
errorMessage = "";
Process cmdLineProcess = new Process();
using (cmdLineProcess)
{
cmdLineProcess.StartInfo.FileName = fileName;
cmdLineProcess.StartInfo.Arguments = arguments;
cmdLineProcess.StartInfo.UseShellExecute = false;
cmdLineProcess.StartInfo.CreateNoWindow = true;
cmdLineProcess.StartInfo.RedirectStandardOutput = true;
cmdLineProcess.StartInfo.RedirectStandardError = true;
if (cmdLineProcess.Start())
{
return ReadProcessOutput(cmdLineProcess,
ref errorMessage, fileName);
}
else
{
throw new CommandLineException(String.Format(
"Could not start command line process: {0}",
fileName));
}
}
}
private static string ReadProcessOutput(Process cmdLineProcess,
ref string errorMessage, string fileName)
{
StringDelegate outputStreamAsyncReader
= new StringDelegate(cmdLineProcess.StandardOutput.ReadToEnd);
StringDelegate errorStreamAsyncReader
= new StringDelegate(cmdLineProcess.StandardError.ReadToEnd);
IAsyncResult outAR
= outputStreamAsyncReader.BeginInvoke(null, null);
IAsyncResult errAR = errorStreamAsyncReader.BeginInvoke(null, null);
if (Thread.CurrentThread.GetApartmentState() == ApartmentState.STA)
{
while (!(outAR.IsCompleted && errAR.IsCompleted))
{
Thread.Sleep(10);
}
}
else
{
WaitHandle[] arWaitHandles = new WaitHandle[2];
arWaitHandles[0] = outAR.AsyncWaitHandle;
arWaitHandles[1] = errAR.AsyncWaitHandle;
if (!WaitHandle.WaitAll(arWaitHandles))
{
throw new CommandLineException(
String.Format("Command line aborted: {0}", fileName));
}
}
string results = outputStreamAsyncReader.EndInvoke(outAR);
errorMessage = errorStreamAsyncReader.EndInvoke(errAR);
if (!cmdLineProcess.HasExited)
{
cmdLineProcess.WaitForExit();
}
return results;
}
public static string Run(string fileName, string arguments)
{
string result;
string errorMsg = String.Empty;
result = Run(fileName, arguments, out errorMsg);
if (errorMsg.Length > 0)
throw new CommandLineException(errorMsg);
return result;
}
public static string Run(string fileName)
{
return Run(fileName, "");
}
}
}
The code was written using Visual Studio 2005. I used NUnit 2.3.0 for unit testing.
If you don't have NUnit installed, then you will receive error messages when opening the solution because the references to the NUnit assemblies cant be resolved. Ignore these and consider removing the unit testing project from the solution.
Better yet, just download NUnit. And consider downloading TestDriven.Net too. This is a very useful Visual Studio extension which integrates NUnit and other testing frameworks into the IDE.
There are 3 overloads of the static Run method. These differ according to how you wish to handle errors being written to the error stream.
The first overload exposes the contents of the error stream as an out string parameter. You will need to check for a non-empty error string and decide how to handle this yourself.
public static string Run(string fileName, string arguments,
out string errorMessage)
The other overloads will check for a non-empty error stream. If there are errors, a CommandLineException will be thrown whose Message property will contain the contents of the error stream.
public static string Run(string fileName, string arguments)
and
public static string Run(string fileName)
NB: With these two overloads there is no way of accessing the output stream's contents when an exception is thrown. This could be a problem if the command line process is partially successful.
For example, you might find that most items can be processed successfully, but a few items cause errors. If this could happen, then you should rather use the first overload of the Run method.
In principle, unit tests should never depend on external resources such as the file system or database.
But it's a bit hard to test code that runs command line applications without accessing the file system! So I confess - I have broken this rule...
To run the unit tests you will need to make some changes to the code.
Firstly, there is a console application which was written specifically for testing purposes. It writes hundreds of lines to the standard output and error streams alternately. You will find need to build this project. It is in the Utilities.CommandLineHelperTester project.
One of the tests uses CommandLineHelperTester.exe. The other tests all use svn, the Subversion command line client utility. If you have Subversion installed, you will only need to make changes to the paths at the top of the CommandLineHelperTester.cs file:
[TestFixture]
public class CommandLineHelperTester
{
private const string smallVersionedTestFolder
= @"c:\SQR\Tools\Source\Utilities\Utilities.CommandLine";
private const string largeVersionedTestFolder = @"c:\EstateCanePro";
private const string unversionedFolder = @"c:\temp";
private const string pathToTestApp
= @"c:\SQR\Tools\Source\Utilities\Utilities" +
@".CommandLine.CommandLineHelperTestApp\" +
@"bin\Debug\CommandLineHelperTestApp.exe";
}
If you don't have Subversion, then you will need to find some other command line application for the unit tests to use. This will require changes to be made in various parts of the unit test file - just search for "svn".
One of the unit tests has been written to demonstrate the problem of the program hanging when stream buffers fill up.
For obvious reasons, this unit test is ignored by default:
[Test,
Ignore("Hangs on a large working copy due
to blocking when the buffer fills")
]
public void RunSvnStatusCommandLineWithLargeWorkingCopy()
{
}
Comment out the line with the Ignore attribute. Then run the unit test in the NUnit GUI (it's probably not a good idea to run the unit test in Visual Studio's Output window!)
Kill the NUnit GUI when you get tired of waiting for a response.
CommandLineHelper was written so that I could create my own library of Subversion utilities in C#. Wrapping the command line utility was a much quicker approach than learning the intricacies of the Subversion API (which would probably have required delving into C++, a language I haven't used in over 10 years, and probably the Apache Portable Runtime library as well.)
I must point out that it is also a much dirtier approach. The code is probably much more vulnerable to changes in the output format. There is a trade-off to be made, and it's important that you realise that.
I needed the utilities to automate and streamline some of my own development processes. So I was quite content with the quick and dirty approach.
Quite a few of the svn command line options now allow XML as the output format. This can make it wonderfully simple to process the results:
public class Revision
{
public static int GetLastCommittedRevision(string workingCopyPath)
{
string commandLineArguments
= String.Format(@"log --xml --revision COMMITTED ""{0}""",
workingCopyPath);
string xmlLogOutput
= CommandLineHelper.Run("svn", commandLineArguments);
XmlDocument xdoc = new XmlDocument();
xdoc.LoadXml(xmlLogOutput);
XPathNavigator xnav = xdoc.CreateNavigator();
XPathNodeIterator xiter = xnav.Select("/log/logentry/@revision");
if (xiter.MoveNext())
{
return xiter.Current.ValueAsInt;
}
else
{
return 0;
}
}
}
In other cases, you will need to process the output yourself.
I usually use regular expressions to parse the output string and turn the results into objects.
On the other hand, these can be very cryptic and difficult to debug. You are also less likely to detect any unexpected changes to the output format. So the safer option might be to write custom code to turn the command line outputs into objects.
For what it's worth, here is an extract from the code which helps to wrap the "svn status" utility. The documentation for "svn status" can be found online at red-bean.com.
static public StatusInfo[] GetStatuses(string path,
SvnStatusOptions options, SvnStatusFilter filter)
{
if (options == null)
options = new SvnStatusOptions();
string errorMessage;
string svnStatusOutput = CommandLineHelper.Run("svn",
String.Format("status {0} {1}",
options.CommandLineSwitches, path),
out errorMessage);
if (errorMessage.Length > 0)
{
if (File.Exists(path))
{
if (IsANonWorkingCopyErrorMessage(errorMessage))
{
StatusInfo nonWCStatusInfo
= GetNonWorkingCopyStatusInfo();
if (IsStatusInfoIncludedInFilter(nonWCStatusInfo,
filter))
{
return new StatusInfo[1] { nonWCStatusInfo };
}
else
{
return new StatusInfo[0] { };
}
}
else
{
throw new SvnStatusException(errorMessage);
}
}
else
{
throw new SvnStatusException(errorMessage);
}
}
return ConvertSvnStatusOutputToStatusInfos(svnStatusOutput,
options, filter);
}
private static bool IsANonWorkingCopyErrorMessage(
string svnStatusErrorMessage)
{
string rxNotAWorkingCopyPattern
= @"svn:\s'[^']*' is not a working copy";
Regex rgx = new Regex(rxNotAWorkingCopyPattern);
return rgx.Match(svnStatusErrorMessage).Success;
}
private static StatusInfo[] ConvertSvnStatusOutputToStatusInfos(
string svnStatusOutput, SvnStatusOptions options,
SvnStatusFilter filter)
{
string rxLinePattern;
if (options.CheckForUpdates)
{
if (options.Verbose)
{
rxLinePattern
= @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
+ @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
+ @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S]).."
+ @"(?<OutOfDateCode>[ *])\s*(?<WorkingRevision>\d+)\s+"
+ @"(?<LastCommittedRevision>\d+)\s+"
+ @"(?<LastCommittedAuthor>\S+)\s*"
+ @"(?<WorkingCopyPath>\S.*)$";
}
else
{
rxLinePattern
= @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
+ @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
+ @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S]).."
+ @"(?<OutOfDateCode>[ *])\s*(?<WorkingRevision>\d+)\s*"
+ @"(?<WorkingCopyPath>\S.*)$";
}
}
else
{
if (options.Verbose)
{
rxLinePattern
= @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
+ @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
+ @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S]).."
+ @"\s*(?<WorkingRevision>\d+)\s+"
+ @"(?<LastCommittedRevision>\d+)\s+"
+ @"(?<LastCommittedAuthor>\S+)\s*"
+ @"(?<WorkingCopyPath>\S.*)$";
}
else
{
rxLinePattern
= @"^(?<ItemStatusCode>[ ADMRCXI?!~])"
+ @"(?<PropertyStatusCode>[ MC])(?<LockedCode>[ L])"
+ @"(?<WithHistoryCode>[ +])(?<SwitchedCode>[ S])..\s*"
+ @"(?<WorkingCopyPath>\S.*)$";
}
}
MatchCollection matches = Regex.Matches(svnStatusOutput,
rxLinePattern, RegexOptions.Multiline | RegexOptions.IgnoreCase
| RegexOptions.IgnorePatternWhitespace);
List<StatusInfo> statusInfoList = new List<StatusInfo>();
foreach (Match currMatch in matches)
{
StatusInfo currStatusInfo
= ConvertMatchToStatusInfo(currMatch, options);
bool isIncludedInFilter = true;
if (filter != null)
{
isIncludedInFilter
= IsStatusInfoIncludedInFilter(currStatusInfo, filter);
}
if (isIncludedInFilter)
{
statusInfoList.Add(currStatusInfo);
}
}
return statusInfoList.ToArray();
}
private static StatusInfo ConvertMatchToStatusInfo(Match currMatch,
SvnStatusOptions options)
{
StatusInfo currStatusInfo = new StatusInfo();
currStatusInfo.WorkingCopyPath
= currMatch.Groups["WorkingCopyPath"].Value;
currStatusInfo.ItemStatus = CodeToItemStatus(
currMatch.Groups["ItemStatusCode"].Value);
currStatusInfo.PropertyStatus = CodeToPropertyStatus(
currMatch.Groups["PropertyStatusCode"].Value);
currStatusInfo.IsLocked
= (currMatch.Groups["LockedCode"].Value == "L");
currStatusInfo.AddedWithHistory
= (currMatch.Groups["WithHistoryCode"].Value == "*");
currStatusInfo.IsSwitched
= (currMatch.Groups["SwitchCode"].Value == "S");
if (options.CheckForUpdates)
{
currStatusInfo.UpdateStatus = CodeToUpdateStatus(
currMatch.Groups["OutOfDateCode"].Value);
}
else
currStatusInfo.UpdateStatus = UpdateStatus.Unknown;
if (options.Verbose)
{
currStatusInfo.LastCommittedRevision
= int.Parse(
currMatch.Groups["LastCommittedRevision"].Value);
currStatusInfo.LastCommittedAuthor
= currMatch.Groups["LastCommittedAuthor"].Value;
}
if (options.Verbose || options.CheckForUpdates)
currStatusInfo.WorkingRevision
= int.Parse(currMatch.Groups["WorkingRevision"].Value);
return currStatusInfo;
}
The unit tests were running fine in NUnit's GUI interface. But I ran into trouble when I started running the tests within the Visual Studio environment.
On further investigation I discovered that within Visual Studio the tests were running in single-threaded apartment mode.
I was using WaitHandle.WaitAll to wait for both threads to finish. But according to the MSDN library: The WaitAll method is not supported on threads that have STAThreadAttribute.
I changed my code to check Thread.CurrentThread.GetApartmentState(). In single-threaded apartment mode I would poll for completion instead. This solved the problem.
I mentioned earlier that you could run into problems if the output format changes when the command line application is upgraded.
There could also be a problem if you deploy a program based on this approach to a machine which is set up to use a different language. If the console application has been localized, then the output might differ across languages, and your wrapper will fail to process the outputs correctly.
A third problem could arise if the console application needs to prompt the user for inputs.
For example, Subversion can be set up to use the same user name and password on a particular machine. If you haven't set this up yet, then the results might be unpredictable. Since I haven't tried it out on a new machine, I don't know if it will throw an exception, or simply hang while waiting for the user to enter a user name and password.
For these reasons I suggest that you only wrap command line applications for use in a controlled environment (such as your own machine).
- Add asynchronous
BeginRun and EndRun methods.
- Add support for passing data to the standard input stream.
[Please note: I'm listing these features for the sake of completeness. I have no compelling need for these features, and I don't plan to add them until I do. So if you need either of these features, don't wait for me... go ahead and add them yourself.]
22 October 2006
- Initial version submitted.
