Makin' Pie

Well, must be the season for wrapping up development cycles. As evidenced by other entries (by Alex and Keith) I’m not the only one looking forward to the winding-down of another successful product.

Only our development cycle ended in July, so we’re a few months late. Sue me.

The story in a nutshell: my company has a legacy app, or rather several legacy apps, that together power a comprehensive suite of tools for a very specific industry. One of these legacy apps forms the backbone of the rest of the application—in essence, it builds the data warehouse upon which the rest of the apps rely. Now, as it turns out, this app is also where the lion’s share of the business logic is kept, as building this particular data warehouse isn’t a trivial task. (Trust me on this one.) Well, this app works, but it has problems. It’s slow, for starters. Like painful slow. Like old-lady-pushing-a-heavy-cart-down-the-middle-of-a-crowded-aisle-in-the-supermarket slow. It’s also convoluted as all hell, brittle, and people are actually afraid to work on it. It’s also what yours truly has spent the last year of his life on.

Rebuilding it, that is, from scratch.

Huh. That just takes the cake, doesn’t it? After all, rebuilding from scratch is, oh, number 1 on the list of things not to do according to the venerable Mr. Spolksy. But, we figured, the reasons Mr. Spolksy spoke of don’t really apply to us:
1) All those “hairs” on the code? They’re not bugfixes, they’re bugs.
2) Architecture? What architecture?
3) Inefficient? Oh, boy.

Actually #2 and #3 were closely related. Yes, the code was factored wrong, but in a very pervasive way. Essentially the thing was built to process record-by-record when it needed to be built according to a batch process. It was supposed to be built for raw, burn-yer-eyeballs speed, and here it is pulling down result sets like there’s no tomorrow, mangling the data, and passing it back across the wire when all that could have been done in the database. You know, in sets. I hear relational databases are good at that sort of thing.

Nope, nope, we said, it’s all gotta go. Right now we’re processing twenty entities a second; by June we’ll be pushing five hundred!

I suppose you’re waiting for this, so I’ll get it over with. Past deadline and over budget, and that creaky old legacy app is still outperforming us. It’s like we went down to the Ferrari dealership, bought a 308, and got smoked on the way home by a Ford Festiva.

Why did this happen?

For several reasons, at least one of which is far too sordid to elaborate on here. But the one I’m going to talk about (eventually) is that we weren’t just making software, we were makin’ pie.

Ever see the Chunkin Punkin festival? It’s where people, for some reason, compete to see whose device (catapults, trebuchets, hydraulic boom-cannons, whatever) can throw a pumpkin the farthest. Well, every year it seems, someone decides Damnit, I’m-a gonna out chunk all them other sonsabitches by a mile, and (this seems especially to be the case with the boom-cannons) over-engineers some monstrosity that accelerates the pumpkin so fast that it disintegrates almost instantly, sending a spectacular if decidedly non-aerodynamic mess a mere fifteen feet or so. Contestants call this “Makin’ Pie”.

Well, that’s what we did. In our quest to produce the most robust, elegant, well-factored code the world has ever seen, we wrote software that does more work than the code we meant to replace. By a damn sight. The new version does something like twenty times the amount of work the old one does.

Now of course we can fix it. We’ve got a better, more stable, more flexible architecture, more amenable to changes. But it’ll take time, bags of time. And we’re already past deadline.

Take-home lesson: next time, we’ll focus on what the software is supposed to do more than on how we want it to be.

I’ll let you know how that goes.

Great post. I've been

Great post. I've been victim to this a number of times. What are good methods for combating this kind of over-engineering? It seems to me that the agile camp may have some good ideas here. Get it in front of the user/customer as quickly as possible, iterate, rinse, repeat. That way you don't spend too much time working on something that isn't going to matter. Of course I have yet to implement that method and see how it works.