There are several articles on CodeProject and on MSDN that deal with the redirection of Console process' input/output using pipes. Unfortunately, you might have realized that most of the time, you will receive the output of the program as a giant block of data only when the child process terminates. This is a big issue because usually, you spawn a console process that will perform a task for you, and during the execution time, you want to get a feedback.
This article provides background information on why this problem happens, and a nice solution, easy to integrate in your existing programs.
(If you're not interested in the background explanations, you might want to skip directly to Here comes the solution or to the conclusion).
So you have a nice (third-party) console program that performs a long task and prints out progression messages? Everything seems to work normally when you run that program from the command prompt... But as soon as you want to encapsulate it in a nice GUI program that will present the results to the user, things get worse, and you don't get those progression messages until the end of the subprocess.
Well, I have a good news and a bad news:
- the good news is: it's not the fault of your GUI program.
- the bad news is: most console programs behave differently when their output is redirected to pipes!
Why is that?
Let's dig in Microsoft C Runtime (CRT) Library....
printf function has an immediate effect when the program is using a real console, and seems delayed when the program is redirected to a pipe.. So, let's take a look at its source code... You will quickly find out that there is a buffering system around the stdout stream, and in order to have
_ftbuf flush that stream (= output immediately the result of your
printf), you have to have
_stbuf reach the last
return(1);. Unfortunately, when stdout is redirected to a pipe, you will discover that the
if (!_isatty(_fileno(stream))) return(0); prevents this from happening.
The Microsoft CRT considers that stdout is not a TTY when it's a pipe, and changes the buffering behaviour !
So, now there are two options:
- You can add a
fflush(stdout); after each output instruction. This works, but that requires you to have the source code of the console program and to modify/recompile it
- You want a generic solution that works with any console executable. Then follow with me a little further...
But before continuing, it's time to state these:
- This analysis is valid for programs compiled with Microsoft C Runtime Library only.
- It might not be valid with other runtime libraries, but if you see the same symptoms, that means there must be a similar buffering system.
- And finally, the vast majority of console programs out there have been developed with the Microsoft C Runtime Library.
A little deeper in Microsoft C Runtime (CRT) Library....
OK, so, what would be nice is that we cheat the CRT into thinking that stdout is still a TTY when it's a pipe.
Looking at the source code of
_isatty, we need the
FDEV flag on our file, and this only happens if a call to
Doh'! For a pipe,
FILE_TYPE_PIPE. And MSDN tells us that
FILE_TYPE_CHAR is only returned for printers and the console...
So we really need a console... Are we stuck?
My solution is to really use a console buffer that the father process will create, share, and monitor while the child process writes into it.
But there are two drawbacks that we have to solve:
- Output console buffers can only be written to, not read.
- The only way to have a console buffer is to either be a console process or to call
AllocConsole. But we don't want a console window to appear!
The first one is solved by using
ReadConsoleOutputCharacter and other console-specific functions that allow us to read information from the console buffer as if we were reading it on the screen.
The second one will be solved in a very elegant way: Instead of calling
AllocConsole from our GUI program and quickly find the window in order to hide it (like some articles suggest), we will create an intermediate stub program that our GUI program will spawn instead of spawning the target program.
This little stub program will be a real console process that will be in charge of monitoring the console buffer and flushing the data read onto its own output stream (that our GUI program will redirect to a pipe)
This solution brings two nice advantages:
- The stub program can be run hidden with the
SW_HIDE startup window option, so no console window will be visible.
- If you have already written your GUI program using redirection pipes, you can keep it! The only thing you will have to change is to insert "RTconsole.exe" at the beginning of your
A look at the RTconsole source code
The arguments to RTconsole.exe will be the original command-line, including the path to the target console program.
We build an inheritable console screen buffer and fill it with zeroes:
sa.nLength = sizeof(sa);
sa.lpSecurityDescriptor = NULL;
sa.bInheritHandle = TRUE;
HANDLE hConsole = CreateConsoleScreenBuffer(GENERIC_READ|
MAXLONG, origin, &dwDummy);
We could have used the standard console that comes with any console program, but this is cleaner and it avoids mixing the subprocess output and our own output in the same console screen buffer. The zeroes will allow us to differentiate with space character outputs. Please note also that
is possible only because we are a console application. In a GUI application, this would require calling
which would display a console popup window.
Now, we start the target process normally, sharing the same console. (RTconsole.exe itself must be started with
SW_HIDE to hide the shared console.)
si.cb = sizeof(STARTUPINFO);
si.dwFlags = STARTF_FORCEOFFFEEDBACK;
if (!CreateProcess( NULL, commandLine, NULL, NULL,
TRUE, 0, NULL, NULL, &si, &pi))
There seems to be no way to be notified on new characters arriving in the console screen buffer, that's why we now need a monitoring loop:
if (WaitForSingleObject(pi.hProcess, 0) != WAIT_TIMEOUT)
exitNow = true;
} while (!exitNow);
We exit that loop when the child process has exited, after doing an additional iteration of the loop, for the very last characters output to be taken in account. Monitoring the child process this way also should solve the problem encountered with blocking
ReadFile on 16-bit subprocesses.
In the loop, we monitor if the text cursor has moved since the last monitoring, we read the characters on screen from the last known cursor position up to the current position, we fill back with zeroes the portion we have read, and reset the text cursor to its home position.
int lineWidth = csbi.dwSize.X;
if ((csbi.dwCursorPosition.X == lastpos.X) &&
(csbi.dwCursorPosition.Y == lastpos.Y))
DWORD count = (csbi.dwCursorPosition.Y-lastpos.Y)*
LPTSTR buffer = (LPTSTR) LocalAlloc(0, count*sizeof(TCHAR));
buffer, count, lastpos, &count);
count, lastpos, &dwDummy);
Then, we analyze the characters read from the screen buffer, and convert them to lines that are written (flushed) to RTconsole's own original output stream.
These have been tested to ensure no problem is happening most of the time, but it's always good to know your weak points
There is no atomic way of reading the screen buffer and resetting it for the ongoing incoming data. So while we are reading and clearing the screen buffer, there might have been additional characters written by the child process. That's why I'm switching temporarily to
THREAD_PRIORITY_TIME_CRITICAL for a quick check to see if the text cursor has moved since then. Characters are not lost in this case because we only clear the characters we have read
Scrolling screen buffer
If the text cursor has not moved, we reset it back to its home position (0,0) to avoid letting it go down the default 25 lines or so of the screen buffer. Otherwise, the screen would start scrolling, and if it happens, then we might lose some text. Note that you could probably use
SetConsoleScreenBufferSize to enlarge the screen buffer if the target console program outputs characters too quickly.
I went a bit into details with this article, but remember, in the end:
All you have to do in your GUI application is to insert "RTconsole.exe" at the beginning of the
CreateProcess command-line and read the redirected output pipe as usual (see demo program).