A Curmudgeon in Redmond

On Minimum-Change Programming

Think about how easy it is to make a change to a shiny new, clean, well-designed system. Now think about how hard it is to make changes to a fifteen-year old system that has been under constant maintenance. Why the difference?

Once upon a time, your fifteen-year-old monster was a shiny new system with a clean design. (At least, the people who built it probably thought it was a clean design.) So the question can be rephrased as: How did such a cute baby grow up to become such an obnoxious teenager? The answer: one change at a time.

When faced with a maintenance task, most people look for the easiest way to make the change. This is frequently dominated by internal process considerations like changing the smallest number of source files. Need to get an additional piece of data from here to there? You can either:

1. Add add an additional parameter to all of the interfaces between here and there, updating .idl files, .h files and .cpp files. Change all of the routines between here and there to propagate the additional parameter. Or,
   
2. Over here, stuff it into a global variable, and over there pull it out of the global, thus changing here and there, but nothing in between.

Most of the time, people choose the second. It's easy, and it gets the job done. Unfortunately, it's short-sighted. Every time you do this, the code base gets more complex. After several of these, you can't find the original clean design any more. Because of this, each subsequent change gets harder and harder, until you can't change anything without breaking something else.

Nevertheless, people persist in doing this. I've even seen check-in checklist forms that ask "Is this the smallest possible change?" as if that were a good thing. It's not.

So, what should we do? My maxim is:

When you're done, the code should look like you knew what you were doing when you started, whether you did or not.

In other words, when making a change, change as much as necessary to make it look like the system was designed from the start with the new requirement. If you do this, the first few changes take more time than they would otherwise. But unlike the other approach, the difficulty of changes does not increase with time.

The other problem, unfortunately, is that this is nearly impossible to do if your starting point is already a mess. This means that it becomes extremely important to resist the urge for a quick fix when the code is new, because code cleanliness is like virginity: once you've lost it, you don't get it back. To paraphrase Nancy Reagan:

Minimum-Change Programming? Just say NO.

Garbage-Collection, Finalizers and Dispose: What Every C# Programmer Should Know

Because C# is garbage-collected, I don't need to worry about cleaning anything up. Not.

If you've ever wondered why your C# application is so bloated, this post is for you.

The garbage collector in .NET ensures that you don't have to worry about freeing managed memory that you allocated. So, you can allocate strings, arrays, hash tables and the like all day long without incurring a huge memory cost. But you still need to worry about cleaning up if you're doing anything that involves resources outside .NET. So, if you opened a file, you need to close it. If you created a window, you need to destroy it. And so forth.

Now in fact, if you don't clean these things up, eventually the garbage collector will do it for you. But there's a catch. The System.Drawing.Image object, for example, has very little state inside it except for a handle to the unmanaged image. The garbage collector will think that collecting this object will only get a few bytes back, but collecting it may well result in the release of a megabyte of unmanaged memory. So you can leak lots and lots of Images, Fonts, Icons, Brushes, Regions and whatnot before the garbage collector kicks in. Meanwhile, your process has swollen to occupy many megabytes of memory.

When developing classes in .NET, you need to have two relevant goals: achieving timely cleanup in well-written programs, and preventing permanent leaking of unmanaged objects even in poorly-written code. The Dispose() method is the key to the first, and finalizers are the key to the second.

In order to understand more deeply, let's start by looking at what .NET guarantees:

The Guarantees

The .NET runtime makes you three guarantees here:

1. As long as there is any conceivable way for your object to be referenced, its memory will remain allocated. So you don't have to worry about dealing with stale references to unallocated objects. This is goodness. In determining whether your object might be referenced, the system considers the values of all global (static) variables, and the values of all local variables on the stack of every thread in the process.
2. Some arbitrary time after it becomes totally impossible to ever reference an object again, the .NET runtime will reclaim that object's memory to use for other purposes. This might happen the same microsecond that the object becomes unreachable, and might happen many minutes later. But if your process lives long enough, it will happen.
3. If your object has a finalizer, then some time after it first becomes unreachable but before its memory is deallocated, the system will call its finalizer routine.

If an object has a finalizer that hasn't been run yet, you can opt out of the third guarantee and prevent the finalizer from running by calling GC.SuppressFinalize. Similarly, you can make an object's finalizer run a second time by calling GC.ReRegisterForFinalize, but you're twisted if you do so.

IDisposable and Dispose()

There is an interface System.IDisposable, with a single method: void Dispose(). Unlike finalizers, all use of Dispose is strictly conventional. But it's a convention like "drive on the right side of the road," not a convention like "don't wear white shoes after Labor Day." So you had better understand and abide by it.

If you write an object that needs some cleanup when your client is done, implement IDisposable and code up a Dispose method. And if you create an instance of someone else's object that implements IDisposable, you should call its Dispose method when you're done.

Again, it's just a convention. But you violate it at your peril.

Your Mission

Remember your goals: timely cleanup, and no leaks.

To achieve the first goal, whenever you create an instance of a class that implements IDisposable, you must ensure that its Dispose method gets called in a timely fashion. There are two principal ways of doing this:

1. If it's only needed for the life of one method, use a using statement:
       using (MyClass myObject = new MyClass(...))
       {
           // your code here
       }
   This using statement is just syntactic sugar for "call Dispose on the way out". The Dispose method is called automatically when control hits the closing brace, or when control leaves the block in some unusual method (return, exception, goto, whatever). The using statement is exactly equivalent to coding a try ... finally statement, but less wordy.
   
2. If the disposable object is needed for the lifetime of one of your objects, then implement IDisposable yourself, and have your Dispose method call Dispose on the object you created. Note that in doing this, we've pushed the need to call Dispose up onto the shoulders of whoever created your object.

To achieve the second goal, whenever you create a class that directly wraps an unmanaged resource, create a finalizer method that releases the resource.

FxCop's Advise: The Dispose Pattern

FxCop is a wonderful tool; it can find all kinds of subtle bugs just by analyzing your code, even before you've ever run it. Here's its advise about coding classes that need cleanup behavior:

1. Inherit from IDisposable.
2. Implement IDisposable's Dispose method exactly as follows:
       public void Dispose()
       {
           Dispose (true);
           GC.SuppressFinalize (this);
       }
3. Implement a finalizer method exactly as follows:
       public ~YourClassName()
       {
           Dispose (false);
       }
4. Implement the following method, which is also named Dispose but has no relationship to IDisposable's Dispose:
       public virtual void Dispose (bool disposing)
       {
           if (disposing)
           {
               // do managed cleanup here, primarily by
               // calling other objects' Dispose method.
           }
           // do unmanaged cleanup here,
           // using P/Invokes as necessary
       }

(The code is slightly different if you're inheriting from a class that implements IDispose.)

This code strikes me as overkill. It assumes that the same class might both wrap unmanaged resources, hold references to other disposable objects, and have its own cleanup to do. It also assumes that the class might have sub- or super-classes that also have their own cleanup responsibilities.

Most of the time, however, you won't directly hold unmanaged resources, and won't have any of your own cleanup to do. So if you follow FxCop's advise, you'll write a finalizer that does nothing, and a Dispose method that keeps the finalizer from executing. I don't care very much about the execution time of this fancy no-op, but I hate wasting my brain cells writing and reading this crap. So here are my simplified rules:

Jimbo's Rules of Cleanup

1. If you need to directly wrap an unmanaged resource, then write a class that exactly wraps one resource. If you always use two different resources together, wrap each in a separate class.
2. If you're wrapping an unmanaged resource that's represented by something the same size as a pointer (usually a native pointer or handle), make your wrapper class inherit from SafeHandle (or one of its subclasses), and override its ReleaseHandle method. This covers more than 90% of all native resource wrappers.
3. If you're wrapping an unmanaged resource that's of a different size than a pointer, wrap it in a class that implements IDisposable and has the following methods:
       public void Dispose()
       {
           if (resourceIsntAlreadyCleanedUp)
           {
               CleanUpTheResource;
               FlagResourceAsCleanedUp;
           }
           GC.SuppressFinalize(this);
       }
       public void ~YourClassName()
       {
           if (resourceIsntAlreadyCLeanedUp)
           {
               CleanUpTheResource;
               FlagResourceAsCleanedUp;
           }
       }
   If this rule doesn't apply, don't code a finalizer method.
4. If your class owns other objects that implement IDisposable, then your class must implement IDisposable, and your Dispose method must call the other objects' Dispose methods.
5. If you need other cleanup behavior (flushing a buffer, for example), implement IDisposable, and have your Dispose method perform your cleanup before calling other objects' Dispose methods.
6. If you're inheriting from a class that implemented the full disposable pattern, or if you intend that others inherit from you, use the full disposable pattern.
7. The only good use for GC.SuppressFinalize is as in rule 3 above. There are no good uses for GC.ReRegisterForFInalization. And if you ever call either of these routines with a parameter other than 'this', you deserve a wedgie.

Postscript

And while I'm at it, I've often seen attempts to repair the absence of calling Dispose by sprinkling the following throughout the program:
    GC.Collect();
    GC.WaitForPendingFinalizers();
    GC.Collect();
Don't do this. Besides being horribly expensive, it's ugly. It's a lot like driving to the emergency room of your local hospital every afternoon and getting a tourniquet applied to your leg, on the off chance that you shot yourself in the foot.

Performance Rant

I've often seen code with performance problems, and watched people struggle to improve it. I've also seen people struggling to achieve optimal performance while writing code. In both cases, I've witnessed lots of folly committed in the name of performance. So here are some of my observations on the subject:

If You Haven't Measured It, You Don't Know What You're Doing

In performance work, everybody has an instinct about what will help and what won't. If you're new at the game, your instincts will be wrong 80% of the time. Once you've got 20 years of experience under your belt, you'll only be wrong 40% of the time. What this really means is that opinions aren't worth the paper they're not printed on -- you've got to measure. Measure before you make a change, measure again afterward, compare. That's the mantra. Otherwise, you can make performance-related changes all day and not get anywhere.

Model It

During the early stages of development of something new, you generally don't have a working system that you can measure. This is where modeling helps.

A model can be anything from very simple to quite complex, and frequently includes the results of small prototypes. Most of my successful models involve simple spreadsheets. For example:

During the development of the WHS Backup product, I had a design point where I said that in order to complete a certain phase of a backup, I'll need to do 10 million random reads from disk. How long will that take? So I wrote a simple prototype, discovered that you can do about 50 random reads per second on one disk, divided, and determined it'll take almost 3 days. So it was time to come up with an alternate design.

With the next design, I concluded I could accomplish the same task by reading a gigabyte sequentially, writing a gigabyte sequentially, and sorting 10 million 24-byte records, all at the same time. How long would this take? More simple prototypes determined that I could sequentially read 50 megabytes per second, but could initially write only about 5 megabytes per second. Researching that, I eventually found out how to get the write rate up to 50 megabytes too. I also discovered that there weren't any pre-canned sort routines that could sort more data than fit in memory, and offer same-day service. With more prototyping and modeling, I eventually discovered how to write one of these that worked well, too.

I was still a long way from having a working system, but at least I had rejected a design approach that wasn't going to work, and gained confidence in a different approach.

Prototype It

Often during development, you'll want to answer questions like "how fast can I reticulate splines?" This is where very simple prototypes come into play. You code up a small program that does nothing but reticulate splines, and measure it.

Other times, you'll have a working system, and be thinking about performance changes that are a lot of work. "I think it might help to make the framistats non-persistent. But to do that I'll need to write 1500 lines of code to cope with the case where the Jeffries tubes boil over." You can either:

• Spend 6 months writing and debugging the code before you find out if it helps, or
• Just change your code to use non-persistent framistats, even though the code is no longer shippable.

If you do the second, and discover that it didn't really help performance, you've just saved yourself 6 months of work. (Of course, if it did really help, you're not absolutely guaranteed that the new 1500 lines of code won't obliterate the gains, but you've at least validated that your intuition is on the right path.)

The point here is that a prototype doesn't need to be a stand-alone thing. It can be your current system, plus a few tweaks.

By modeling and prototyping, you can save yourself from going down a path that will never work well. You can find it out at design time, long before you've spent the effort to code and debug things. Models and prototypes will seldom be good enough to prove that a design will work, but even rudimentary models and prototypes can often weed out designs that won't work.

Micro-Optimization Doesn't Help

Fred is chasing a performance problem. "Oh, look here," he says. "I'm repeatedly computing the size of my window from its bounds. I should compute it once and save it." He makes the change, and goes home feeling satisfied. The next day, he wonders why life didn't get any better.

The same principle applies even if, for example, you're building the database manager where all of the time is being spent. Even then, most of the CPU time isn't being spent in your component. You still need to design such that you minimize the number of expensive calls you make to components below you, and minimize the number of expensive calls that people above you need to make in order to accomplish their jobs.

This means that for most common micro-optimizations ("Should I hoist a+b out of that loop?") the answer is "it doesn't matter for performance. Which way expresses your intent most clearly?"

Fast Enough Is Fast Enough

At least 90% of the code you'll ever write will be fast enough, even before you think about performance. If you're writing something that will be used only once, it's not worth spending a day of your time to slice a minute off the execution time. If you're writing something interactive, it doesn't help to improve things even by a factor of 10, if you're just reducing the response time from 10 milliseconds to 1 millisecond.

So, where do you spend your time? Concentrate on those programs where:

• You have real users telling you it takes too long.
• You have models or prototypes that say that performance isn't even in the right ballpark.

It just doesn't matter how much you improve the performance of your solitare game; it was probably fast enough when you started.

Cleaning Up After Your Program

You've got a big, complicated computer program and it's time for that program to exit. The user just chose the Exit menu, or he clicked on the close button of his main window, or your service has been asked to stop. What do you do? In my experience, people writing termination code fall into two major camps:

Cleanliness Is Next To Godliness

People is this camp will write lots of code to close all of their windows, ask their threads to stop, release all of their COM objects, release all of their memory, uninitialize COM and WinSock, and so forth. They'll spend a lot of time in design and debugging, ensuring that things get torn down in the right order, so that nobody depends on a module that has already been finalized.

The good news is that these people get the benefit of leak-finding tools. In theory, when their process stops, it's in exactly the same state as when it started. Any differences constitute a leak.

A Smith and Wesson Beats Four Aces

People in this camp take a different approach. They know that the OS will clean up properly when a process dies, so they just call TerminateProcess. Their termination code works first time, every time.

In a few cases, they'll introduce code during termination to send courtesy notifications to other parties, letting them know that they're dying. Presumably, these other parties would work properly without the notification, but the notification saves these parties waiting for a timeout period.

Overall, I find myself more and more drawn to the second model, mostly because code that you don't write seldom has bugs. What are your preferences?

Filler

She: You're so picky you stay awake at night wondering whether to spell anal-retentive with a dash.

He: It's not a dash; it's a hyphen.

MapPoint and GPS: Another Stupid Pill

MapPoint is a Microsoft product that's a lot like Streets & Trips, with additional features. Most interestingly for me, it includes interfaces to programmatically control it, plus an embeddable ActiveX control. Since I was putting a computer into my car, it seemed like just the thing.

In Streets & Trips 2004, and MapPoint 2004, Microsoft added GPS support. You can easily configure it such that the map tracks your current position as reported by the GPS. You can also configure it such that your view of the map is rotated such that your direction of travel is always up. Quite convenient for in-car navigation.

Unfortunately, search as I might, I could find absolutely no mention of the GPS features in the programming interface documentation. "Drat!" says I to myself. "The GPS features probably came in too late for the programming interface people to catch up." I was disappointed, but I understand how these things can happen.

So I decided I would just send Windows messages to the embedded control to convince it that I had manually pressed the UI buttons that enable GPS support. No luck; these menu options aren't in the UI for the embedded control. "No problem," I say, "I'll just start the full application via the scripting interface, and use it." So I code that up, and discover that, when started via the scripting interface, the GPS menu items exist, but are disabled.

There's a property in the scripting interface called PaneState. It returns an enum value saying which side-panel is currently being displayed, with values representing none, route planner, legend, find places and territory manager. I imagine that during the development process, a conversation like the following took place between a developer and a product manager.

Dev: What should we return for the PaneState property when the GPS pane is being displayed? Should I just invent a new value for the enum?

PM: What'll that do to compatibility with existing applications?

Dev: Well, if they don't display the GPS pane, they'll never notice the difference. If they do, they'll get an enum value they've never seen before.

PM: But won't an app break if it gets a value and doesn't know what it means?

Dev: Some app might, but most will tolerate it. Besides, it only happens if a user goes out of his way to show the GPS pane.

PM: I've got it! Let's just make it impossible to show the GPS pane if you're using the scripting interface. Then we won't have an app-compat problem.

So they go forth and implement. Out of a fear of a hypothetical user breaking a hypothetical app by turning on something new, they make the product's single most important new feature completely unusable from every app. This is not a good trade-off, and is deserving of Jimbo's Stupid Pill Of The Day award.

In any event, that was in 2004. What changed in the programming interface for MapPoint 2006? Nothing. Nada. Zilch. How about the recently-released MapPoint 2009? You guessed it: nothing. So, five long years after introduction of the GPS features, you still can't script them. (I know, life sucks.) But the worst of it is that five years later, you still can't use the GPS features at all, in any way whatsoever, if you're using scripting for any purpose.

Actually, the above paragraph isn't quite true: While you can't use the GPS features at all if you start MapPoint by creating an instance of the MapPoint.Application object, if you start MapPoint using CreateProcess and then, once it's initialized, connect to it using the GetActiveObject API, the GPS features remain enabled. You'll still need to send window messages to simulate button presses, but at least it's possible. Unfortunately, when the MapPoint team hears about this, they'll probably deem it a bug and disable it in MapPoint 2011. (Hint: If you do this, people will wait in line to give you wedgies.)

Scott Hanselman on the Difference Between Developers and Designers

I just became acquainted with Scott Hanselman's ComputerZen blog. Lots of good stuff there.

His Developer != Designer article is short and sweet, and captured my attention. Money quote:

The difference between a Designer and Developer, when it comes to design skills, is the difference between shooting a bullet and throwing it.

In case you're wondering, I'm in awe of the design skills of Kynan and Bryce, the designers on the Windows Home Server team.

Where Is Bob?

Check out the Where is Bob? blog. If you've ever worked for an antisocial, inept, incompetent, lying weasel, he's nothing compared to Bob. While presented as fact, I suspect this blog is mostly fiction. But that doesn't detract in the least from its entertainment value.

I found the link via Sergey.

Hacking the I/O Redirection Bug

Warning

I'm about to talk about modifying a Windows system in a manner that most decidedly does not meet with Microsoft's approval. I did it to a few of my systems for fun and for the educational experience; feel free to do it to yours for the same reasons. But having patched your system, don't ask Microsoft for support on that system, even for something you're sure is unrelated. And you'd be a fool to do this to an important production system.

And if you ever do this to a system other than your own, I will personally hunt you down and give you a wedgie. 'Nuff said.

Introduction

 I'm an old fogey; I cut my teeth on command-line interfaces. (Actually, I'm older than that -- I cut my teeth on punch cards, but successfully made the transition to command lines.)

As anyone who has used a command line since Unix was invented knows, you can redirect standard input and output by appending a "< inputfile" or "> outputfile" to the command line.

Many people also know that on Windows, if you type to name of a file at a command prompt, and that file isn't an executable program, the system will launch the program that owns that kind of file and ask it to open the file. This is particularly handy for script files like Perl, where opening the file really means running the script. So you can run a Perl script just by typing "foo.pl".

Unfortunately, there's a bug that prevents these two very useful techniques from being used together. If you type "foo.pl >outputfile", your script bombs the first time it tries to write any output, and the output file ends up empty. This problem has been reported many times but has never been fixed, primarily because of fear of app compat (application compatibility) issues.

When the command interpreter executes a line like "foo.exe > outputfile", it opens the output file, sets its handle as the standard output of the command interpreter process, and calls CreateProcess (actually CreateProcessW, since it uses the wide character version of things). This works fine. When executing "foo.pl > outputfile", it does exactly the same thing, but CreateProcessW returns error ERROR_BAD_EXE_FORMAT. The command interpreter then calls ShellExecuteEx, which launches the appropriate program (in this case Perl.exe), passing the name of the script as a parameter. At first blush, it feels like it ought to work, so what goes wrong?

The Cause

When opening a file on Windows, you can provide a flag (lpSecurityAttributes->bInheritHandle) saying whether the resulting handle can be inherited by child processes created later. If you don't set this bit, a child process won't receive your handle. But cmd.exe correctly sets this bit when opening files for I/O redirection.

When creating a new process on Windows, you also provide a flag (fInheritHandles) saying whether the child process inherits any handles at all from the creator. cmd.exe also correctly sets this flag.

But there's no way to tell ShellExecuteEx whether to set this flag when it gets around to calling CreateProcess, and so ShellExecuteEx specifies FALSE for this parameter. You'd think that the process manager would be clever enough to let the child process inherit the standard input and output handles anyway, but that isn't the case. So the child process starts up with invalid standard input and output handles, and it's all downhill from there.

A Hypothesis

If we can just get ShellExecuteEx to pass fInheritHandles = TRUE to CreateProcessW, then all would be well. We can test this hypothesis this by running cmd.exe under a debugger, setting a breakpoint at kernel32!CreateProcessW, and running a command like "foo.pl >outputfile". When we run it, the breakpoint hits twice.

The first time is when cmd.exe is calling CreateProcess with lpApplicationName = "foo.pl", with this stack trace:

ChildEBP RetAddr  Args to Child
0013fc20 4ad02d98 0017a308 001822a8 00000000 kernel32!CreateProcessW
0013fc54 4ad02df6 0017a308 00000000 00000000 cmd+0x2d98
0013fc6c 4ad05f20 0017a308 00000000 0017a308 cmd+0x2df6
0013fe9c 4ad013eb 0017a308 0017a308 00000002 cmd+0x5f20
0013fee0 4ad0f138 00000000 00000001 00000000 cmd+0x13eb
0013ff44 4ad05154 00000001 00034480 00032c98 cmd+0xf138
0013ffc0 7c817067 00000000 0007f7ec 7ffde000 cmd+0x5154
0013fff0 00000000 4ad05046 00000000 78746341 kernel32!RegisterWaitForInputIdle+0x49

The second time is when shell32.dlll, underneath ShellExecuteEx, is calling CreateProcess with lpApplicationName = "C:\Perl\bin\Perl.exe", with this stack trace:

ChildEBP RetAddr  Args to Child
0013fa40 7ca03666 00000000 00000000 00187144 kernel32!CreateProcessW
0013fa94 7ca0359d 00183848 0013fab4 7ca0309c SHELL32!Ordinal159+0x347
0013faa0 7ca0309c 00000001 0013fb38 00183848 SHELL32!Ordinal159+0x27e
0013fab4 7ca02fce 00158a68 0013fb38 0013fafc SHELL32!ShellExecuteExW+0x199
0013fac8 7ca02f6a 0013fafc 00000000 0013fb38 SHELL32!ShellExecuteExW+0xcb
0013fae4 4ad13114 0013fafc 0017a308 00171b28 SHELL32!ShellExecuteExW+0x67
0013fc20 4ad02d98 0017a308 001822a8 00000000 cmd+0x13114
0013fc54 4ad02df6 0017a308 00000000 00000000 cmd+0x2d98
0013fc6c 4ad05f20 0017a308 00000000 0017a308 cmd+0x2df6
0013fe9c 4ad013eb 0017a308 0017a308 00000002 cmd+0x5f20
0013fee0 4ad0f138 00000000 00000001 00000000 cmd+0x13eb
0013ff44 4ad05154 00000001 00034480 00032c98 cmd+0xf138
0013ffc0 7c817067 00000000 0007f7ec 7ffde000 cmd+0x5154
0013fff0 00000000 4ad05046 00000000 78746341 kernel32!RegisterWaitForInputIdle+0x49

From the documentation, we know that fInheritHandles is the fifth parameter to CreateProcessW. Looking at the stack, we see that the value is indeed zero. If we set it to 1 and resume, we see that the redirection indeed works properly.

0:000> dd esp l20
0013efc0  7ca037fc 00186d34 00184a9c 00000000
0013efd0  00000000 00000000 04000400 00000000
0013efe0  00184894 00188398 001883e4 001883e4
0013eff0  00183848 00000000 00000000 00000000
0013f000  00000000 00000000 00000000 00184a9c
0013f010  001883e4 00184894 00186d34 00000000
0013f020  00000000 00000000 00000000 00000000
0013f030  00000000 00000000 00000000 00000000
0:000> ed 0013efd4 1
0:000> g

The Solution

If we go back and disassemble the code in shell32.dll leading up the the call to CreateProcessW, we see the following:

0:000> u 7ca037c7
7ca037c7 ffb5d0f5ffff     push    dword ptr [ebp-0xa30]
7ca037cd 53               push    ebx
7ca037ce ffb5d4f5ffff     push    dword ptr [ebp-0xa2c]
7ca037d4 ffb5c8f5ffff     push    dword ptr [ebp-0xa38]
7ca037da 56               push    esi
7ca037db ff7528           push    dword ptr [ebp+0x28]
7ca037de ffb5c4f5ffff     push    dword ptr [ebp-0xa3c]
7ca037e4 ffb5c0f5ffff     push    dword ptr [ebp-0xa40]
7ca037ea ffb5ccf5ffff     push    dword ptr [ebp-0xa34]
7ca037f0 ffb5d8f5ffff     push    dword ptr [ebp-0xa28]
7ca037f6 ff1588149c7c   call dword ptr [SHELL32!Ordinal517+0x1488 (7c9c1488)]

Since parameters are pushed onto the stack from right to left, the instruction at 7ca037db is the one that sets fInheritHandles. Since it's not pushing a constant, we can surmise that there must be some paths in the shell by which it can specify fInheritHandles = TRUE, but I haven't found them. No matter: if we can find a way in three bytes or less to push a reliably non-zero value onto the stack, we can patch the instruction at 7ca037db and be done. My favorite way of doing this is a "PUSH EBP" instruction, which is one byte long (0x55). Since EBP is being used as a stack frame pointer, it's guaranteed nonzero. So we'll just replace the three bytes at 7ca037db with 55 90 90 (PUSH EBP, followed by 2 NOPs). If we now clear the breakpoint and run, we have a command interpreter running that doesn't exhibit the bug.

0:000> e 7ca037db 55 90 90
0:000> bc *
0:000> g

We can make this permanent by creating a private copy of shell32.dll, and using a binary editor to apply this change. Having done this, we need to defeat System Folder Protection to get our changed version into the \windows\system32 directory, and reboot. (shell32.dll is on the known DLLs list, so a reboot is required). At a later date, I may post blogs on patching binaries and defeating SFP.

Epilogue

We now have a system that doesn't exhibit the bug. However, every time that Windows Update applies a patch that involves shell32.dll in any manner, it undoes our change. So we have to redo it every time.

I have a few systems on which I've been doing this for about eight years, starting with Windows 2000, and continuing with XP and 2003 server. The Windows team has never fixed this, fearing that they might break some app when it inherits more handles than expected. While I've never noticed a problem, there are at least 50,000 Windows apps that I haven't run. I also haven't upgraded any of my patched systems to Vista, so I don't know whether this would run afoul of any of its anti-malware mechanisms.

Book Recommendation

The No Asshole Rule: Building a Civilized Workplace and Surviving One That Isn't, by Robert I. Sutton.

Executive Summary:

• Life is too short to spend it working with assholes. So don't.
• If you do, you risk becoming one yourself. So don't.

A quick, easy read. Recommended.

System.WeakReference: Solution in Search of a Problem

.NET has a WeakReference class, whose semantics appear to have been borrowed from Java. As nearly as I can tell, they're both useless.

A WeakReference is an object that holds a reference to another object. However, it holds a special type of reference, that doesn't prevent the target from being reclaimed by the garbage collector. At first blush, this seems perfect for implementing a cache -- as long as the system doesn't need the memory, we've got the object; should the system need the memory, the object goes away and we transparently re-create it next time we need it. Unfortunately, this doesn't quite work.

First, a cache is only interesting to create in the first place if two things are true:

• The objects are big enough that we can't keep them in memory indefinitely, and
• They're sufficiently expensive to create that we can't re-create them every time we need them.

We probably implement our cache using some sort of a dictionary or hash table that maps from the object identity to the cached value. In C# terms, a Dictionary<string,WeakReference>. Now, when a garbage collection occurs, the target object goes away, but the entries in the dictionary remain. Over time, we'll end up with lots and lots of entries in the dictionary, each holding a WeakReference whose target is gone. This leads to the next constraint:

• Caching using a dictionary of WeakReference's is useful only when the total population of the data subject to caching is small.

Otherwise, the dictionary will explode with useless entries.

The next problem is that not only does a WeakReference not prevent an object from being garbage-collected, it doesn't even suggest that the object shouldn't be. So the next time a garbage collection comes around, all of your otherwise-unreferenced objects are reclaimed. So your cache doesn't really hold objects until the memory is needed; it only holds objects until the next garbage-collection. This implies:

• Caching using WeakReference's isn't appropriate unless the objects are cheap enough to be thrown away (and later re-created) on every garbage collection.

I would assert that the set of problems that meet all of the above criteria is empty. After all, we just said:

• The objects are big,
• They're expensive to create,
• There aren't very many of them, and
• They're cheap to create.

Now, it's possible to get around the "not many of them" restriction with some very ugly code involving finalizers that remove entries from the hash table. (I doubt that it would be performant, but it is possible.) But I don't see how you can possibly reconcile "expensive to create" and "cheap to create".

So, has anyone out there ever successfully used Weak Reference's? If you used it for a cache, do you have measurements of how much it helped performance? If you used it for something else, what was it?

Plagarism Blog

Check out the blog Todd's World. (Go ahead, I'll wait for you to come back.)

There's lots of neat content there. Trouble is, as far as I can tell, not one word of it is original. It's all stolen from other places, including my own blog.

I don't know who Todd is; his BlogSpot profile lists him only by first name. But here's a clue for you Todd: Linking to stuff you find interesting is cool. Summarizing it, commenting on it, explaining why it's interesting, and even ripping it to shreds are all cool. Copying it word-for-word, without attribution, isn't.

It's especially galling that Google finds his copy of my words, and not mine.

Is this sort of thing common? What's appropriate to do, if anything?

SmartLabs Takes a Stupid Pill

There's a simple maxim taught in Business Management 101; Commoditize your complements. That is, whatever other people sell that makes your product more useful should be as cheap and ubiquitous as possible. Selling cars? You want there to be gas stations, tire stores, and map vendors everywhere. Selling soccer balls? You want soccer fields everywhere. Selling software? You want compatible hardware everywhere. Selling hardware? You want others to build lots of software for it.

I recently bought some INSTEON home automation equipment. (Think of it as being like X10, only it works.) If you buy their USB interface device, you can control this stuff with your computer. You can download a freeware app to do the most basic things; all well and good. But it comes with the world's crappiest user interface (see screen shot) that looks like something straight out of the 1970's. "No problem," I think to myself. "I'll just write my own."

Then the fun begins. They'll sell you a development kit for $199. Definitely pricey, but it doesn't have to save me very many hours before it's worth it. But the kicker is their sales agreement: it says right up front that they don't want me as customer unless I agree that everything about how their stuff works is confidential, I won't reverse-engineer it, if I do and learn anything, I'll tell them immediately what I learned and not tell anyone else, and so forth. All of this, even though they have publicly-available white papers describing all of the low-level details of their system. Everything is public except for what goes over the computer's USB interface.

So, what do we have here? As near as I can tell, we have:

• A company that makes its bread and butter selling remote-controllable switches, dimmers, outlets and so forth.
• I'd guess that if you asked one of their executives what their biggest problem is, he'd say that they don't have enough volume to get their manufacturing costs down.
• They publish everything you need to know to compete against them in their core business.
• Meanwhile, they also sell computer interface units. But I'd guess that these count for less than 1% of their revenue.
• They try to carefully control who does what with these interface units.

It's as if they're afraid that somebody might come along and build a better computer interface unit, jeopardizing 1% of their sales. Completely ignoring the fact that more readily available software would probably increase their core business by more than 1%.

Unfortunately, there's nothing unique about SmartLabs here; the same phenomenon repeats itself all over. I've got a Polaroid flat-screen TV, where they absolutely refuse to let me know what the infrared remote control codes are. (I reverse-engineered it, thank you.) I bought a USB FM radio receiver for my car computer, but the vendor doesn't want to let me know how to program it. (It came with yet another crappy app; so far I've partially reverse-engineered it.) I could go on for hours.

Hey SmartLabs et al, here's a clue: People who make your stuff more useful are your friends. You don't need to roll out a red carpet for them, but you should at least not get in their way.

Computer Security At Home

When I start thinking about system security at home, I find it useful to consider the lock on the bathroom door. It's an extremely weak lock, and any ten-year-old knows how to defeat it. However, in practice, it's quite effective on a wide variety of levels. Consider:

• It's strong enough to prevent accidental intrusions.
• It's a powerful symbol of expected behavior.
• It's impossible to bypass the lock invisibly and anonymously.
• When the need arises, it's easy to defeat it.

Now, consider the typical access controls on a computer system: They do a reasonably good job of preventing accidental operations. But they're quite different in the other aspects:

• They're mis-configured often enough that an access denied error isn't a symbol of antisocial behavior; instead it's an invitation to try something else.
• If you've gotten in, you might as well be invisible and anonymous. Sure there might be a security log, but nobody will ever look at it.
• The access controls are completely binary -- you're in or you're not. There's no concept of bypassing the lock if the situation warrants it.

The binary nature is especially troublesome when you have kids. They push boundaries by nature; the all-or-nothing behavior coupled with the lack on consequences for getting into something they shouldn't, means that kids tend to learn that if the computer let you do it, it must have been OK. This is completely the wrong message!

The absolute nature of the controls also renders the computer system less useful than it ought to be. While I don't want my kids poring through my tax files, I do want the option of calling a kid up and asking him to retrieve a file and deliver it to my accountant. And I really don't want to give him my password to make this happen.

Putting all this together, here's what I want:

1. I still need absolute protection against Internet-based attackers.
2. Within my home, I want a system that prevents casual or accidental access to off-limits data.
3. In unusual cases, I want anyone (inside the home) to be able to bypass the controls as needed. This should require an explicit action, that can't be undertaken accidentally or unconsciously.
4. I want to be alerted each time that someone bypasses the security, and I want the alert to include details of what was done.
5. If access to something is being repeatedly denied, I want to know that, too.

So, if a kid is systematically probing for weaknesses, I can teach him about the difference between what's right and what you can get away with. Or I can apply discipline as needed.

And what I really want is something that will automatically notify me when anything unusual is happening. A device I've never heard of appears on my network? I want to know. A computer suddenly starts sending dozens of emails per second? I want to know. Someone starts a dictionary password attack? I want to know.

But if everything is operating normally, and doing the same thing it did at this time yesterday, I want it to keep quiet.

Is this too much to ask?

My Own Security Rant

E_ACCESSDENIED? Good. We did OUR job.

One of the security groups at Microsoft used to have T-Shirts that said this; every time I saw one, my blood pressure went up. He's a story illustrating why:

Scene: A home office, fall, 2003. Our hero has just bought a new Macintosh (probably OS 10.0 or 10.1) for his wife, and is trying to configure things such that it can access shares on the existing Windows Server 2000 machine. He's wearing out a path in the carpet between the Mac and the server.

(At Mac): Access denied? I must have typed the password wrong. Let me try again. No luck.
(At Server): Maybe I set the password wrong at this end. Let me reset it.
(At Mac): Still no luck. I wonder if I have problems with the ACLs.
(At Server). The ACL looks OK. Just to be sure, let's temporarily set it to full access for everyone.
(At Mac). Still won't work.
(At Server). Oh, there's an ACL on the share as well as the directory. Let's set it to full access for everyone, too.
(At Mac). Still out of luck.

With a focused determination seen only in those afflicted with a touch of both ADD and autism, our hero flounders around, investigating every possible configuration option on both the Mac and the server. Whenever he finds some restriction that looks like it might possibly be relevant, he loosens it. Eventually, he stumbles across the Group Policy Editor on Windows, where he finds things with intriguing names like "Network Security: Minimum session security for NTLM SSP-based (including secure RPC) clients" and "Domain Member: Digitally encrypt secure channel data (when possible)". He messes around with these and other likely-looking settings, to no effect.

After a while, he has a brainstorm: Maybe some of these setting changes don't take effect until I reboot, he thinks. So, he reboots, and voilà, things work!

It's now 2:00 AM, and he's been at it four hours longer than intended. He decides to go to bed, and clean up the mess tomorrow.

The next day, he can't remember which settings he changed, and has no clue as to which of them was the relevant one. (With his luck, there probably wasn't a relevant one, but two that were each necessary.) There's no way he's going to find out which settings were relevant. "Screw it. It works; leave it alone."

There are two morals here:

First, more security is often less. Microsoft has a nice "Secure by design, Secure by default, Secure by deployment" initiative. Unfortunately, the security that the user gets is not the security that you designed, not the security that you turned on by default, and not the security that you deployed. It's the security that's left standing after he makes everything work. If there are 50 security gates in your system, and the customer's desired workload needs to make it through three of them, the non-expert administrator is likely to turn off 40 of the 50 gates before he disables the three relevant ones.

Second, it's important to realize that not every request that is denied by a security gate is in fact malicious. It sure would be nice if the system were to say "I rejected such-and-such a request for such-and-such a reason, and if you intend me to accept requests like this, you need to change the XYZ policy." Now, security experts will tell you that you shouldn't tell an attacker exactly why you rejected his request, and they're right. However, you should tell the system administrator, in case it wasn't really an attack, but someone trying to get legitimate work done.

When I'm supreme dictator of the universe, anyone who writes code that returns E_ACCESSDENIED will be required to create an event log entry saying what software component did it, why, and what setting to change in what piece of UI if this wasn't the desired outcome.

Postscript: I think it was the "Domain Controller: Digitally encrypt secure channel data (always)" policy, and I think it's because early versions of Mac OS X shipped with a version of SAMBA with a bug. But I'm not sure on either account.