Run All Jobs at Once Utility

• Download project including executables - 290.69 KB

Turbo-charge your Visual Studio project builds!

How often have you wished there was a batch (command-line) command that allowed you to start several programs 'at once,' and then wait for 'all' of them to finish together as a group before continuing?

How handy would it be to be able to write a batch file that allowed you to start several VCBUILDs, having each of them run simultaneously with one another (i.e. at the same time or "in parallel" with one another), thereby fully utilizing that new dual- or quad-processor system you just bought, thereby dramatically reducing the amount of time it takes to build your product?

Well, with "RunJobs", now you can!

Background

If you have, on occasions, searched the Microsoft Visual Studio Support Forums in the past, you may have noticed that more than one person has inquired about the possibility of running multiple builds in parallel (i.e., doing a "Batch Build" but have each Solution Configuration built concurrently rather than serially one after the other), with Microsoft always replying that due to certain technical difficulties, it was not possible.

To me, this was unacceptable since, when designed (configured) properly, each Solution Project's different build configuration 'should' designate different Intermediate and Output directories, thereby allowing each of them (Debug|Win32, Release|Win32, Debug|x64, Release|x64, etc.) to be built 'in parallel' since neither was dependent on the other.

Never being the one who is satisfied with any type of "Can't be done!" response, I set about developing a simple means of accomplishing just that, thereby dramatically reducing Solution build times (especially for large complicated Solutions with many projects, each with many different build configurations).

Using RunJobs

RunJobs is a simple command-line program written in MFC that allows one to start several different programs at the same time. It is written in MFC only because that is what I personally prefer writing my Windows programs in. It is 'not' however a requirement, of course. Being such a simple program, it could easily be ported to any other language of your choice.

I shall first describe how to use it, followed briefly by how it works.

Upon starting the program either without any arguments or with the standard /? argument, the following 'Help' text is displayed, which I hope is self-explanatory in describing how to use the utility:

                     RunJobs
        Fish's Run All Jobs at Once utility
                Version 1.1.2.nnn

  Copyright (C), Software Development Laboratories
         http://www.softdevlabs.com/runjobs
                fish@softdevlabs.com

Format:

   RunJobs   <jobsfile>   [ -j nnn ]   [ -rcfile xxxxx ]


Where:

   <jobsfile>      filename of command-lines to be started
   -j nnn          maximum number of simultaneous commands
   -rcfile xxxxx   name of optional return codes file


The jobs file is a simple verbatim list of the command-lines
that you wish to issue.

Lines whose first non-blank character is '#', ';' or '*' are
considered comment lines and are ignored. All blank lines are
also ignored.

All other lines contain the exact command-line to be issued.

The -j parameter sets an upper limit to the maximum number of
simultaneously running commands.

When the jobs file contains more than this number of commands
the next job (command) is not started until one or more of the
previously issued jobs (commands) finishes first.

Specifying 0 (zero) indicates no upper limit (i.e. ALL listed
jobs will be started simultaneously regardless of how ever many
there are).

The default for -j is the value of the "RUNJOBS_DEFAULT_MAXJOBS"
environment variable if defined or the number of CPUs installed
on your system if it is not defined.

The return code from RunJobs is always the maximum returned
return code value from any of the started jobs. If the -rcfile
option is specified the return code value from each job is also
written to the specified return codes file using the following
format: "n = rc" where 'n' is the job number and 'rc' is its
corresponding return code value.


                         USAGE NOTE

Since RunJobs doesn't parse jobs file command-lines, all of your
environment variable strings on your command-line SHOULD have a
valid (non-empty) value assigned to it. Otherwise an improperly
formed command-line will be built since ExpandEnvironmentStrings
leaves the variable unexpanded whenever it cannot find it. For
example: if your specified command-line is:

    VCBUILD  %VCBUILDTYPE%  "MyProj.vcproj"  "Debug|Win32"

then VCBUILD fails (and crashes if invoked within the IDE!) with
the following error:

  "vcbuild.exe : warning VCBLD6002: invalid option Debug|Win32
                 specified. The option was ignored."

To prevent this, ensure that all environment variable strings on
your command-line are expandable.

In the sample project (which actually builds the RunJobs utility itself), the vcbuild-all.cmd batch file illustrates how RunJobs could be used to build all of your project's various configurations in parallel.

The first thing you'll want to do is copy RunJobs.exe to somewhere where Windows can find it (i.e. somewhere in your Windows search PATH).

Then simply define some additional Solution/Project configurations that call the vcbuild-all.cmd batch file (see project source) to run RunJobs with the proper jobs file which would of course be setup to issue the actual VCBUILD command for each of the different configurations you wished to be built in parallel for that particular configuration.

Refer to the supplied Visual Studio project (both the .sln solution file as well as the .vcproj project file as well as the vcbuild-all.cmd batch file and associated All-Debug.jobs, All-Release.jobs, etc, files too) for a real world example of how you might accomplish this.

RunJobs Internals

The CMyCmdLineParser class parses the command-line arguments and sets global variables to the name of the "jobs" file to be run and the desired "maximum concurrent jobs" value.

The CStdioFileEx class is used to read the specified "jobs" file and create a new entry in a CList of CPipedProcessCtl objects. The CPipedProcessCtl class is a simple custom designed class to make it trivial to spin off a secondary process via CreateProcess with its STDIN/STDOUT/STDERR files piped back to the creating process.

As each started process writes to its STDOUT/STDERR file, the separate pipe monitoring threads the CPipedProcessCtl class automatically creates redirects the output to RunJob's real STDOUT/STDERR files (which, of course, is then intercepted by Visual Studio and displayed in the Output pane).

As each process ends, the return code is retrieved (via GetExitCodeProcess) and saved, with the highest one of any of the started process' used as RunJob's own return code. In this way, one can check within one's batch files whether all the "jobs" -- as a group -- completed successfully or not.

Expected Improvement in Build-times

RunJobs, running on my dual 2.4GHz Xeon development platform (hyperthreaded to look like a quad-processor), has reduced my build time for a small, semi-complex multiple-projects / multiple-build-configurations (Debug|Win32, Release|Win32, Unicode Debug|Win32, Unicode Release|Win32, plus the same four but for the x64 platform (i.e. a total of eight build configurations per project)) from 9:48 to 5:37. An improvement of approximately 43%.

rdboss (see comments further below) writes:

This product has been a god send for us. [...] Our nightly build process has been dropped from about 20 hours down to about 10 hours!

Esteban Papp however (see comments further below) writes:

We are using it to do multiple platforms parallelized build, it is running great in a 2xQuad-Core 3Ghz, our build time was 1h and now it is 7min.

That's over '8.5x' times faster! ('almost' 10x times!) Wow!

Of course I should caution that "Your Mileage May Vary". It all depends on the complexity of your projects/solutions (how many source files each project has, how many different configurations each one has, etc), but the point is, being able to build multiple project configurations in parallel definitely does help! (which of course is why I wrote RunJobs in the first place)

In general however, based largely on the above reports, I expect you should see an approximate improvement in your build times of anywhere from 50% to over 150% or more.

Not bad for a simple command-line too, eh?   :)

Closing Remarks

Well, I guess that's it. There's really not much else to add. I hope you all enjoy using RunJobs to help speed your build times, and look forward to hearing how well it works for you.

That's it!   Enjoy faster build times with RunJobs!   :)

History

• May 2008: Version 1.0.0 released and article written

• November 2008: Version 1.0.2
  • Corrected return-code issue identified by esteban.papp
  • Converted to Visual C++ 2008 SP1
  • Statically link to MFC and CRT

• March 2009: Version 1.1.0
  • Fixed cancel propagation issue.
  • Added -rcfile option
  • Redesigned project implementation

• August 2009: Version 1.1.2
  • Fix issue where 'StartChildProcess' failure was causing the overall return-code to not get updated.
  • Fix non-unique job-number log-prefix issue identified by rdboss. (see comments)