Lesson Learned

Skip the first solution(s)

Some people are really fast. They receive a problem set, spend a short time pondering, quickly come up with a solution and then sit down at their computer to begin coding. Sometime later, give or take some trial and error, the code is complete, maybe even tested and then submitted to source control and QA for black box testing. Managers and other bystanders happily point out: This is one fast developer!

Here's the problem: Fast development does not necessarily mean good development. Joel Spolsy hinted at that in his Hitting the High Notes: The quality of the work and the amount of time spent are simply uncorrelated.

Now, I would guess though that the really good people (no matter how fast they are) understand that the first solution is not necessarily the best one, even if it does provide the correct result. This means that even after having arrived at a solution, one does not stop, but rather look at other approaches, which just might turn out to be much, much better.

For sake of a simple example let's think about Fibonacci numbers. That's right, you maybe heard about them last in some algorithms and data structures class and probably never had to deal with them since.

In short Fibonacci numbers have these values: F0 = 0, F1 = 1, and Fi = F(i-2) + F(i-1) for every i > 1. The series of Fibonacci numbers starts like this then: 0, 1, 1, 2, 3, 5, 8, 13, ...

Hm, so a quick glance reveals that for every Fibonacci number for values greater than 1 we will need to know the values of the two Fibonacci number before that. That's clearly recursive!

The implementation might follow something like this then:

function fibRec(long input): long
if input < 2
return input
else
return fibRec(input - 2) + fibRec(input - 1)
end function

Now again, this is the first and quick solution. The problem is that this is also an extremely inefficient solution. Yes, it returns the correct result, but for increasingly big input values, the running time quickly becomes absolutely prohibitive. The running time grows in fact exponentially: Depending on your computer's performance and your compiler implementation, you probably will start losing patience waiting for the result of, say fibRec(60).

This can be vastly improved by using an iterative approach:


function fibIter(long input): long
long prev = -1
long result = 1
long sum = 0
for each number i from 0 to input
sum = result + prev
prev = result
result = sum
end for
return result
end function

The running time of this algorithm is linear - much faster than the first approach. This is a very solid approach, even for pretty big input values.

It takes a little bit more code than the recursive solution (i.e. writing the recursive routine might be faster), but is really similarly straightforward. Spending some more time and thought can bring up an even better solution for this problem, one with a logarithmic running time, outperforming the straightforward, iterative approach. Feel free to google this, I am sure this can be found pretty easily.

The point of all this: I think we can really think of it as a general principle. Part of being good at what you do is to not stop at the first solution that might present itself, but always look for alternative ways. This may often be easier said than done, considering schedule pressures, etc., but it definitely has been my experience that the first solution is most often really not the best one. Even if it is though, there's no way of knowing, unless we can compare it with others.

Unicows.dll Removes Unicode

UnicoWS.dll does not Unicode enable your program in Win9x. It does not add Unicode to Win9x (it works by converting Unicode to ANSI, i.e. removing Unicode). However, what unicows.dll does is an extremely useful thing: it allows your new Unicode (wide char) program to "work" on old Win9X/ME machines that don't support the Unicode APIs.

Because of the fog of trepidation surrounding Unicode/charset issues, I think the Microsoft announcement of MSLU made the right choice in not overemphasizing what MSLU, the "Microsoft Layer for Unicode on Windows" (unicows lib and dll), does not do. They let the discerning reader figure it out somewhere down in the article where they say it is "not a panacea" and vaguely spell it out under the second point made there. Hey, you can't start out by shouting what your product does not do just because people might think so.

I have never used unicows, but every time I hear about it people seem to suggest that it gives you Unicode on Win9x, so I take the bait and research into it only to be reminded again of what it actually is. What set me off to write this article was a response to a post on JoS as follows:

Does it make sense to *not* compile for Unicode these days? The old excuse was "Unicode doesn't work on Win9x", but now it does as long as you link in the MSLU library. So why not compile for unicode right off, and just forget that ANSI/MBCS exists? Chris Tavares

If you slightly reworded it 'The old excuse was "Unicode builds don't run on Win9x"' then I would agree, and to be fair that may be what the poster meant. But wherever I hear about unicows it often seems to come across as adding Unicode to Win9x.

On old Win95, 98 and ME boxes, your unicows linked (wide char _UNICODE) program will generally operate in the ANSI locale code page with anything involving Windows messaging and APIs. That means all the filename and directory APIs, text on Windows controls etc. Characters not supported in the system ANSI code page will be lost when they are passed into or retrieved from APIs.

But this is not as bad as it sounds because users generally don't have multilingual expectations on those old machines (not to mention fonts were limited). So MSLU answers the main concern of having a single executable that works on all Win32 platforms.

I can't write an article about unicows and not mention one of my 3 favorite bloggers, Michael Kaplan, who as far as I can tell is the guy behind MSLU (though he gives credit to others) and is a great presence on newsgroups, always humble and helpful (and his site has a section on MSLU). He was also mentioned by Joel Spolsky recently as the guy who helped Joel figure this Unicode thing out (some time ago).

The reason I am sensitive to this unicows issue is that I have worked with Unicode on Win9x. There are a couple of Wide Char APIs that have been available since Windows 95, namely GetTextExtentPoint32W and TextOutW, and I built an edit control that works internally in UTF-8 using these APIs, but that is another story.

Are bad Requirements our fault?

Over the past week or so, I've been reading Marcus Buckingham's First, Break All the Rules. I caught him at a conference last year and finally got around to picking up both of this books, so I've been looking forward to this.

Anyway, I'm approximately half way through it and while I was reading this morning, I was struck by something. Let me quote:

Conventional wisdom asserts that good is the opposite of bad, that if you want to understand excellence, you should investigate failure and then invert it.

He goes on to support this by pointing out that we tend to study things if we want to prevent them: student truancy, drug use, etc, etc. And I found this interesting when applied to some of my recent projects. Everyone tends to focus on "what went wrong and how do we fix it" instead of "what went right and how do we do more of it?"

As is common in the software world, we complain quite a bit about not having a specification. We talk about it to anyone who will listen... and I was actually doing it this morning. Maybe we need to change our perspective a bit. As John pointed out in his recent post, most of our customers and even we take things for granted in how they should work, how we perceive the world, and what we expect from our people and our tools. When we are working with something as intangible and complex as software, we need to focus on making things a bit more tangible and descriptive for our users. We have to make sure that they can understand some of the key things... not *how* we're doing something, but what the tradeoffs are.

I believe that we need to educate our customers on not only what is possible, but what is impossible given their constraints. Not only could this help clarify requirements, but it might completely eliminate silly requirements like "Build an auction system on par with Ebay".

Strange case of two system locale ANSI charsets

Are you technically familiar with how system locale ANSI charsets work on Windows? I thought I knew enough to get by... until recently. You may know that there are two primary ANSI locale settings, one of which requires a reboot, but do you know when it comes to the distinction between these two settings that the MSDN docs get it wrong, that the Delphi 7 core ANSI functions get it wrong, or that you cannot set the system code page in C/C++ using setlocale?

In Windows XP Regional and Language Options, you can set "Standards and formats" on the first tab, and "Language for non-Unicode programs" on the third tab, the latter requires a reboot (it is similar on previous Windows OSes). The weird thing is that both of these mess with the ANSI locale code page.

Windows APIs are built around two types of text strings, ANSI and UNICODE. The UNICODE charset (Wide Char) is pretty straight-forward because it is not affected by what the locale settings and language are. The ANSI charset always supports the 128 ASCII values, but can have different ways of utilizing the high bit of the byte to support additional characters. In single-byte charsets, the upper 128 values are assigned to the additional characters like European accented characters, Cyrillic or Greek characters. In East Asian double-byte charsets, special lead byte values in the upper 128 are followed by a second byte to complete the character. "Double-byte" is actually a misleading term because characters in double-byte strings can use either one or two bytes. An ANSI charset is implemented as a "code page" which specifies the encoding system for that charset.

The ANSI charset used by the computer changes according to the locale the computer is configured to, but what is the computer's locale? Well, no one is terribly clear on that! The Windows API GetLocaleInfo allows you to get information about either the "default system locale" or the "default user locale." The MSDN article then goes on to refer to the "current system default locale," and the "default ANSI code page for the LCID," as opposed to the "system default–ANSI code page." I have yet to discover how the User/System differentiation works although presumably user logons retain certain aspects of the Regional and Language Options. Anyway, I would say it is anything but clear.

According to MSDN for Microsoft C++, a C program can use setlocale( LC_ALL, "" ) to set itself to use the "system-default ANSI code page obtained from the operating system" rather than plain ASCII, and then all multi-byte string functions will also operate with that code page. However, it turns out that this code page is actually the one from "Standards and formats" in the computer's Regional and Language Options. I call this the "setlocale" charset.

Meanwhile, all ANSI Windows APIs and messages operate according to the ANSI code page from the "Language for non-Unicode programs" setting. This setting governs the real ANSI system locale code page which you can find out with the GetACP Win32 function. This is the default code page used in MultiByteToWideChar when you specify CP_ACP. I call this the "GetACP" charset.

When these two code pages are different such as U.S. English setlocale charset and Japanese GetACP charset, many programs used internationally exhibit bugs you won't see otherwise. For example, Delphi core source code SysUtils.pas uses a SysLocale based on the setlocale charset in many of its Ansi string functions like AnsiPos and CharLength, while implicit ANSI/WideString conversions and other Ansi functions like AnsiUpperCase happen according to the GetACP charset.

Even Winzip prior to release 8.1 did not parse pathnames correctly with different setlocale and GetACP charsets. WinZip couldn't zip Japanese filenames that contained an ASCII backslash as the second byte of the Shift-JIS character because the string functions were treating the double-byte ANSI strings as single-byte ANSI (Windows-1252) strings.

There is a reason that Windows has these two system charsets. The locale info for a locale's "Standards and formats" is provided via ANSI API in a particular charset (is this what MSDN vaguely referred to as the "default ANSI code page for the LCID"). The OS cannot provide Japanese standards and formats such as weekday names in a Western European ANSI charset. So, a programmer is supposed to interpret that locale info's text strings according to the locale info's charset, even if the machine locale charset is different. But I have not found this documented properly, and I don't think many people know about this. Delphi got it wrong and the Microsoft C++ documentation is not clear on it. I think at this point, Microsoft developers are inclined to want to forget about these issues and focus on Unicode.

Experiences and clarifications are welcome!

We didn't need documentation back then!

Ever been in that situation? A product has been in development, from the ground up for three to four years. There have been several sales and the product is overall regarded as rather successful. In fact, it grows much bigger than initially expected. An infrastructure forms around it, staffing demands grow, etc. etc. until at some point the company is big enough that an actual HR department is justified. Then, the core developer decides to move on and is off to greener pastures. Ouch.

It gets worse: He or she did in fact never bother to create any elaborate documentation of the architecture or any important design decisions. It's all in his head.

Well.

I overheard a similar description the other day. Interestingly, the speaker concluded his story with the following statement: "We did not need to create documentation for anything back then." Nodding, chuckling ... then the conversation moved on to other subjects.

I could not forget about this though. On the one hand, I do empathize with the spirit: You're having a high tech startup, so you work hard, really hard to get out that product. You have only one or two people hacking away on the code. You're exploring new territory and have to hurry, because the competition could enter your market any moment. You really a) have no time for documenting, b) see no reason, because the application is far from complex at that point, and finally c) only need a minimum of documentation simply because there is not a large team involved and thus pretty few things to be communicated. No time for breaks, because you and your team are busy inventing the future and selling the half-ready program to cash-wielding prospects who would prefer to have the final result tomorrow. It's a pretty exciting environment.

Not creating effective documentation in that early stage of a product is a horrible mistake.

Here are just some reasons for this:

  • As more and more requirements are revealed software tends to gain and not lose complexity.
  • The software (or rather, the code base) will end up being in use far longer than the original developers could possibly anticipate.
  • Teams will grow, requiring a great information overhead. It can be very painful process, trying to discover the existing knowledge about architecture, design or features.
  • People will move on. They may move within the company or sign on somewhere else. Either way, the transition can be quite slow and inefficient. If the former main developer is suddenly busy running the company he won't have much time explaining database table structures.
  • The early knowledge may well be core knowledge, if the early development focused on the most important features of the application.

There is of course no easy answer, because some of those earlier objections may well appear quite valid. For some time anyway. Then, you better have a plan on how to transition smoothly to a good documentation process. Otherwise you might just find yourself in an unlucky scenario five years down the road.

Choose the Right Tool for the job

Over the years, I've worked with a number of systems which aggregate data in some way. These have included simple RSS aggregators, the importation of multiple XML formats, and even interacting with a multiple of bug tracking systems. Regardless of the domain, there is a simple concept here: There are a particular set of actions we wish to perform on the data, but quite often the data sources have a variety of data structures.

For some reason, most people immediately jump to the "simple" solution of building a inheritance heirarchy, each with a distinct class/structure beneath it to a specific data structure. This can work fine for quite a few situations and ends up being quite elegant in patterns such as the Data Access Object and others where you can completely encapsulate the data structure and simply forget about it. Unfortunately, this doesn't work in all scenarios.

For example, I'm involved in a large scale Java system where we are retriving XML data structures from a series of different sources (think aggregation). When we started there were twelve different sources in two different formats and the solution looked simple, we would simply import each of the structures and be done with it.

Since I've been bitten by this before, I proposed a different solution. Instead of doing the import in a single step, we would use a Two-Step Pattern to first convert each data structure into our custom structure and simply import that. We've been able to implement this by building a single importer class and creating a new XML-to-XML XSL transform for each new data source.

Just in time too, within 2 months, we were up to 7 different formats from 237 different sources...

Pave the cowpath

At my alma mater, Ball State University, there is a dirt walkway affectionately known as the cowpath. It is a makeshift shortcut from one of the dorms (er, residence halls) to classrooms on the other end of campus. It emerged out of need and student's tendency to take the shortest line between two points. The cowpath has existed for decades, and unless something has changed in the last five years, it remains a dirt trodden path.

What does any of this have to do with producing and managing software? Put simply, when we observe people wearing a path that we didn't anticipate, like when they use our software in interesting or peculiar ways, we have a few options:

  1. Slap the user's wrist and tell him not to mickey with the software
  2. Analyze if the software's user interface created some confusion
  3. Observe the user closely to understand his motivation

If you do #2 and #3 you will most likely choose to pave the cowpath instead of discouraging the user from doing what he wants to do. While I wouldn't advise retrofitting a spreadsheet into a flight simulator, developers and project managers should keep an open mind about creative ways that people repurpose software. A practical example is the Friendster software product. From presentation notes given by Danah Boyd to the Supernova Conference:

What was successful about Friendster had nothing to do with its original purpose or design [dating]. Instead, users saw it as a flexible artifact that they could repurpose to reflect their social practices. As i learned how people embedded Friendster into their daily lives, i was fascinated by how it manifested itself as so many different tools to so many different people. Some saw it as an information gathering tool, allowing them to learn about friends and strangers. Others saw it as a performance tool and a venting site. It was also used as a gaming device, a distribution channel for the drug dealer, an anti-depressant for the voyeur, a popularity contest. Many also saw it as a cultural artifact necessary for all water cooler discussions.

At first Friendster resisted the use cases they didn't ordain, but gradually they saw value in what their user base wanted from the service. They were fortunate to have a flexible system, which helps. They revised their software in minor ways (pave the cowpath), but mostly they just needed to get out of the way.

The next time your users surprise you with the way they (mis) use your software, consider if the possibility that the user knows something you don't. As a consequence you may tap into new markets for your software, and at the very least, by adapting your software to the needs of the user you will make their experience more pleasant.