Interview: The Wall Street Programmer

This is the fourth in a series of interviews we're making available to the CodeSnipers community. It's quite a bit different from our previous ones which focused on Open Source developers, Project Mangers, and general community contributors. This gentleman is a bit more fired up and has something to say. Without further delay... we have the Wall Street Programmer.

When your blog launched in November '05, you titled it "Thoughts, insights and justified profanity from a Wall Street Programmer". What are a few aspects that make software development on Wall Street different from other places and/or industries?

Sofware development on Wall Street seldom follows the fads and keyword technologies of proper software shops. Only now is the season of the dying breed of C++ developer being replaced by a new wave of Java centered managerial decisions. This is enirely too late as compared to other software developer domains. As I've stated in my post about traders, much of the software here is seen as a necessary evil...a means to an end. Just write the thing in whatever way is fastest, make sure it's relatively stable and runs fast, and shove it out the door...

How long have you worked on Wall Street and what types of firms have you worked for?

Now now...tsk, tsk. I don't like to have the blog have any 'absolutes' in it. Following this decision, I refrain from releasing my definite age, duration of career, duration of career on Wall Street, or name...although there has been some turbulance with the later...as many readers have probably witnessed. This has nothing to do with conflict of employment, but rather gives me flexibility of not only content, but also of administrative things like blog termination without a second thought...the first attempt at which produced too many cheers of 'encore' to be ignored. Let's just say I joined Wall Street before any derivative of George Bush sat on the throne, and many of the people I once hired have long since been in positions of hiring others. I have worked in companies as small as 10 people, and in as-big-as-they-come international investment banks - both in equities, and in fixed income.

What the fundamental benefit/drawback of working in a smaller organization versus a large organization?

One of the reasons I started the blog in the first place was to release on post: the one giving telling newbie Wall Street programmers to start their careers in small firms. The post was very cathartic, and proved to be one of the more popular, as expected.

"[1] Favor the smaller company to the international investment bank.

This list is not in any particular order... aside from this item. I cannot stress enough my favor of getting entry-level programmers starting in smaller Wall Street firms.

When you arrive here, you will feel the requirement to throw most of what you learned in academia out of the window. No one will tell you this, but you will start noticing how things are done, as opposed to how you were taught they should be done.

While international investment banks such as Goldman Sachs, Morgan Stanley, Lehman Brothers, and Merrill Lynch will provide you with security, you will not be able to absorb anywhere near the information you would if you were to work for a smaller company. The single most important thing I attribute to my longstanding success (ahem) on Wall Street is the fact that I started my career working for a small firm (less than 15 programmers…about 20 traders).

Here’s a typical entry-level candidate with a good understanding of Computer Science, a medium grasp of C++, and absolutely no foundation in financial products:

After 2 years of working at Goldman, Morgan, Lehman, or Merrill, the candidate reaches a pretty good level of understanding of financial products He also gains a substantial amount of interoffice social skills (a result of office politics), a thorough respect for a slower than snail release process, and an anticipation of the requirement to have 2 weeks worth of meetings in order to approve an environment variable change in a configuration file. His debugging skills probably improve, but his programming skills are now completely in the toilet.

After 2 years of working at a small company, the candidate reaches a painfully thorough understanding of a small area of the financial world (whatever the company was focusing its business model on). He gains the same level of interoffice social skills (politics cannot be avoided…anywhere). He also gains excellent debugging skills by analyzing raw Solaris core dumps on a daily basis, and gains environment foundation skills by constantly tweaking Rendezvous channels, Sybase settings, Solaris process heap sizes, XML configuration files, and logging frameworks – in a live production environment (because DEV sucks everywhere…always). He gains an understanding of ‘trader-speak’ – something which will serve him greatly no matter where he goes, learns to build and deploy code to production within minutes of noticing a problem, and picks up about 4 new programming languages including C (because most API layers are in C), some shell language (because you just have to), Perl (because shell sucks), and a REALLY good understanding of SQL (using sqsh, or isql from command line).

I’ve known folks who have been working at investment banks for more than 15 years, joining straight out of school, and simply don’t compare with a mid-level developer coming out of a small company with 3 to 4 years of experience. I (and most senior developers) have a long track record of hiring mid-level developers coming from small companies. Their practical skills are orders of magnitude higher than those of candidates crawling out of investment banks."

The (very long winded) answer to your question sits on top of the advice list.

On your blog, you mention that you've been on both sides of the recruiting process. What specific attributes make a candidate stand out during your hiring?

I've written about this as well, but that was geared toward experienced developers. While I do have four sets of questions which I ask candidates depending on their experience level, the correct answer is not the thing which makes a candidate stand out. My most memorable interviews (99% of which resulted in an extended offer to the candidate) were the kind of interviews that 'degenerate' into a conversation or a debate between just two IT guys. The only way this is achievable is if the candidate can take a topic on which a given question is based, say threads, and run with it... from threads to the crappy thread implementations on early Linux systems, to past experiences of processes dumping core because of some poor locking mechanism implementations, to debates over the benefits of multicore processors. I want to see the candidate think...not about the O(n) factor of some algorithm, or how many phone booths there are in New York, but about interesting and insighful things related to the question I bring up. If the candidate can reach this state of conversation with me, he is in a much better position than a candidate who can blast out just the correct answers. By the way, my question set geared towards hard-core developers doesn't even have correct answers. Everything there is geared towards starting a debate.

Compensation. You've mentioned quite a few times the huge compensation potential that exists on Wall Street in terms of both salary and bonuses. What sort of packages have you seen for junior/mid/senior developers and what sort of expectations were involved in each role?

I've stated in one of my earlier posts that an developer coming out of school can grab somewhere around $50-60k in base salary. I say nothing about bonuses in the early years, because there are so many parameters influencing these during the beginning of a career, that everything I quote will be an outlier in some firm. Also, the numbers I mention are concerning the typical role of a enterprise-software-developer. By this, I mean a candidate who will spend his time writing something related to trading software - probably client-server code - in some combination of C/C++/Java/ or Perl. I am not in a position to quote salaries of DBA's, QA folks, or System Administrators. After 4 or 5 years of work on Wall Street, a solid candidate can expert something around $100k in base salary plus some bonus. One must realize that Wall Street is in New York City, and $100k means nothing in NY. This is the place where on easily dumps $40 a day on mediocre lunch, and rent hovers around the $2500 a month range for a one bedroom closet-sized dwelling. I haven't met many people making over $300k as a developer. At this point, they either change roles towards a more business-centered area, or they stay at a certain 'ceiling' base salary, and get raises in bonuses alone.

In many of your posts, you describe a high level of stress prevading just about every aspect of your life. How long do developers normally stick around Wall Street?

I see first signs of people burning out after 2 years. Many do not make it through their first jobs, and go for advanced college degrees, just so they can revert back to the warm womb of academic life. I, myself, have been tempted to return for my PhD many times. There is definitiely some kind of a ceiling, and I am probably reaching it. There are very few faces older than 45 who are still coding...and anyone over 40 is usually consulting rather than sitting around one place for too long.

In early December, you announced that your blog was going offline. There was quite a bit of conjecture that you had been outed, why did you shut it down? The announcement caused quite an uproar, why did you end up coming back?

As I've stated in my 'return post', I started the blog on the verge of making a decision as to whether to leave Wall Street or not. I thought I had finally reached that decision and closed down the blog on the day I decided to anounce my departure to Big Brother. I couldn't really write about being a Wall Street Programmer if I no longer was one. As it turned out, I got recognition of being burned out (which was more important than one realizes), and a month long paid vacation to regain my cool. We'll see how it turns out...I may yet leave if I don't feel right about coming back after this 'vacation'...but it'll probably be a battle - companies don't like to give up experienced people.

Finally, if you could give exactly one piece of advice to developers not on Wall Street, what would it be?

There was a post which someone else wrote literally days before I intented to write on the subject. This was 'Fear of the code' written by Andrey Butov. The entire 'who is he really?' fiasco began after I posted the link to this article. I did so because the writer was dead-on as far as what it feels like to joins that first job here...even about wanting to cry after the first day is over. Many developers will realize this, but will refrain from sharing it for obvious reasons. Aside from that, the only advice is to not put too much stress on 'proper academic foundation', since the best developers I've hired came from state or city colleges, try to start your career with a smaller company - even if it means a shorter duration than it would be otherwise, and involve yourself in the IT sphere - publish articles, read articles, participate in discussions...stay fresh. Many programmers here are stuck in their little "I'm a Java coding monkey" role without realizing it. The candidate who can have a conversation with me about a recent Dr. Dobbs article, or can explain Web 2.0 (please don't...I've had enough...), will always be the type of person I would want to work with.

- Cheers
- Wall Street Programmer

About Wall Street Programmer: He is an older-than-you-would-like-to-be software developer on Wall Street, who after years of seeing college kids and experienced developers strive to make a career for themselves on Wall Street, decided to start an anonymous blog through which he can not only share his widsom in the hopes of getting waves of more knowledgeable interview candidates, but also to spew occasional misanthropic profanity about the daily grind of being a software developer on Wall Street.