Hacking the I/O Redirection Bug
Warning
I'm about to talk about modifying a Windows system in a manner that most decidedly does not meet with Microsoft's approval. I did it to a few of my systems for fun and for the educational experience; feel free to do it to yours for the same reasons. But having patched your system, don't ask Microsoft for support on that system, even for something you're sure is unrelated. And you'd be a fool to do this to an important production system.
And if you ever do this to a system other than your own, I will personally hunt you down and give you a wedgie. 'Nuff said.
Introduction
I'm an old fogey; I cut my teeth on command-line interfaces. (Actually, I'm older than that -- I cut my teeth on punch cards, but successfully made the transition to command lines.)
As anyone who has used a command line since Unix was invented knows, you can redirect standard input and output by appending a "< inputfile" or "> outputfile" to the command line.
Many people also know that on Windows, if you type to name of a file at a command prompt, and that file isn't an executable program, the system will launch the program that owns that kind of file and ask it to open the file. This is particularly handy for script files like Perl, where opening the file really means running the script. So you can run a Perl script just by typing "foo.pl".
Unfortunately, there's a bug that prevents these two very useful techniques from being used together. If you type "foo.pl >outputfile", your script bombs the first time it tries to write any output, and the output file ends up empty. This problem has been reported many times but has never been fixed, primarily because of fear of app compat (application compatibility) issues.
When the command interpreter executes a line like "foo.exe > outputfile", it opens the output file, sets its handle as the standard output of the command interpreter process, and calls CreateProcess (actually CreateProcessW, since it uses the wide character version of things). This works fine. When executing "foo.pl > outputfile", it does exactly the same thing, but CreateProcessW returns error ERROR_BAD_EXE_FORMAT. The command interpreter then calls ShellExecuteEx, which launches the appropriate program (in this case Perl.exe), passing the name of the script as a parameter. At first blush, it feels like it ought to work, so what goes wrong?
The Cause
When opening a file on Windows, you can provide a flag (lpSecurityAttributes->bInheritHandle) saying whether the resulting handle can be inherited by child processes created later. If you don't set this bit, a child process won't receive your handle. But cmd.exe correctly sets this bit when opening files for I/O redirection.
When creating a new process on Windows, you also provide a flag (fInheritHandles) saying whether the child process inherits any handles at all from the creator. cmd.exe also correctly sets this flag.
But there's no way to tell ShellExecuteEx whether to set this flag when it gets around to calling CreateProcess, and so ShellExecuteEx specifies FALSE for this parameter. You'd think that the process manager would be clever enough to let the child process inherit the standard input and output handles anyway, but that isn't the case. So the child process starts up with invalid standard input and output handles, and it's all downhill from there.
A Hypothesis
If we can just get ShellExecuteEx to pass fInheritHandles = TRUE to CreateProcessW, then all would be well. We can test this hypothesis this by running cmd.exe under a debugger, setting a breakpoint at kernel32!CreateProcessW, and running a command like "foo.pl >outputfile". When we run it, the breakpoint hits twice.
The first time is when cmd.exe is calling CreateProcess with lpApplicationName = "foo.pl", with this stack trace:
ChildEBP RetAddr Args to Child
0013fc20 4ad02d98 0017a308 001822a8 00000000 kernel32!CreateProcessW
0013fc54 4ad02df6 0017a308 00000000 00000000 cmd+0x2d98
0013fc6c 4ad05f20 0017a308 00000000 0017a308 cmd+0x2df6
0013fe9c 4ad013eb 0017a308 0017a308 00000002 cmd+0x5f20
0013fee0 4ad0f138 00000000 00000001 00000000 cmd+0x13eb
0013ff44 4ad05154 00000001 00034480 00032c98 cmd+0xf138
0013ffc0 7c817067 00000000 0007f7ec 7ffde000 cmd+0x5154
0013fff0 00000000 4ad05046 00000000 78746341 kernel32!RegisterWaitForInputIdle+0x49
The second time is when shell32.dlll, underneath ShellExecuteEx, is calling CreateProcess with lpApplicationName = "C:\Perl\bin\Perl.exe", with this stack trace:
ChildEBP RetAddr Args to Child
0013fa40 7ca03666 00000000 00000000 00187144 kernel32!CreateProcessW
0013fa94 7ca0359d 00183848 0013fab4 7ca0309c SHELL32!Ordinal159+0x347
0013faa0 7ca0309c 00000001 0013fb38 00183848 SHELL32!Ordinal159+0x27e
0013fab4 7ca02fce 00158a68 0013fb38 0013fafc SHELL32!ShellExecuteExW+0x199
0013fac8 7ca02f6a 0013fafc 00000000 0013fb38 SHELL32!ShellExecuteExW+0xcb
0013fae4 4ad13114 0013fafc 0017a308 00171b28 SHELL32!ShellExecuteExW+0x67
0013fc20 4ad02d98 0017a308 001822a8 00000000 cmd+0x13114
0013fc54 4ad02df6 0017a308 00000000 00000000 cmd+0x2d98
0013fc6c 4ad05f20 0017a308 00000000 0017a308 cmd+0x2df6
0013fe9c 4ad013eb 0017a308 0017a308 00000002 cmd+0x5f20
0013fee0 4ad0f138 00000000 00000001 00000000 cmd+0x13eb
0013ff44 4ad05154 00000001 00034480 00032c98 cmd+0xf138
0013ffc0 7c817067 00000000 0007f7ec 7ffde000 cmd+0x5154
0013fff0 00000000 4ad05046 00000000 78746341 kernel32!RegisterWaitForInputIdle+0x49
From the documentation, we know that fInheritHandles is the fifth parameter to CreateProcessW. Looking at the stack, we see that the value is indeed zero. If we set it to 1 and resume, we see that the redirection indeed works properly.
0:000> dd esp l20
0013efc0 7ca037fc 00186d34 00184a9c 00000000
0013efd0 00000000 00000000 04000400 00000000
0013efe0 00184894 00188398 001883e4 001883e4
0013eff0 00183848 00000000 00000000 00000000
0013f000 00000000 00000000 00000000 00184a9c
0013f010 001883e4 00184894 00186d34 00000000
0013f020 00000000 00000000 00000000 00000000
0013f030 00000000 00000000 00000000 00000000
0:000> ed 0013efd4 1
0:000> g
The Solution
If we go back and disassemble the code in shell32.dll leading up the the call to CreateProcessW, we see the following:
0:000> u 7ca037c7
7ca037c7 ffb5d0f5ffff push dword ptr [ebp-0xa30]
7ca037cd 53 push ebx
7ca037ce ffb5d4f5ffff push dword ptr [ebp-0xa2c]
7ca037d4 ffb5c8f5ffff push dword ptr [ebp-0xa38]
7ca037da 56 push esi
7ca037db ff7528 push dword ptr [ebp+0x28]
7ca037de ffb5c4f5ffff push dword ptr [ebp-0xa3c]
7ca037e4 ffb5c0f5ffff push dword ptr [ebp-0xa40]
7ca037ea ffb5ccf5ffff push dword ptr [ebp-0xa34]
7ca037f0 ffb5d8f5ffff push dword ptr [ebp-0xa28]
7ca037f6 ff1588149c7c call dword ptr [SHELL32!Ordinal517+0x1488 (7c9c1488)]
Since parameters are pushed onto the stack from right to left, the instruction at 7ca037db is the one that sets fInheritHandles. Since it's not pushing a constant, we can surmise that there must be some paths in the shell by which it can specify fInheritHandles = TRUE, but I haven't found them. No matter: if we can find a way in three bytes or less to push a reliably non-zero value onto the stack, we can patch the instruction at 7ca037db and be done. My favorite way of doing this is a "PUSH EBP" instruction, which is one byte long (0x55). Since EBP is being used as a stack frame pointer, it's guaranteed nonzero. So we'll just replace the three bytes at 7ca037db with 55 90 90 (PUSH EBP, followed by 2 NOPs). If we now clear the breakpoint and run, we have a command interpreter running that doesn't exhibit the bug.
0:000> e 7ca037db 55 90 90
0:000> bc *
0:000> g
We can make this permanent by creating a private copy of shell32.dll, and using a binary editor to apply this change. Having done this, we need to defeat System Folder Protection to get our changed version into the \windows\system32 directory, and reboot. (shell32.dll is on the known DLLs list, so a reboot is required). At a later date, I may post blogs on patching binaries and defeating SFP.
Epilogue
We now have a system that doesn't exhibit the bug. However, every time that Windows Update applies a patch that involves shell32.dll in any manner, it undoes our change. So we have to redo it every time.
I have a few systems on which I've been doing this for about eight years, starting with Windows 2000, and continuing with XP and 2003 server. The Windows team has never fixed this, fearing that they might break some app when it inherits more handles than expected. While I've never noticed a problem, there are at least 50,000 Windows apps that I haven't run. I also haven't upgraded any of my patched systems to Vista, so I don't know whether this would run afoul of any of its anti-malware mechanisms.