Wednesday, March 31, 2010

Don't be the slowest gazelle

Every morning in Africa, a gazelle wakes up. It knows it must outrun the fastest lion or it will be killed. Every morning in Africa, a lion wakes up. It knows it must run faster than the slowest gazelle, or it will starve. It doesn't matter whether you're a lion or a gazelle -- when the sun comes up, you'd better be running.
-- proverb attributed to Roger Bannister (first man to break the 4 minute mile)
At a startup, there are inevitably moments when you question whether or not you're moving too fast. From defects, regressions, and poor performance to missed requirements, and reimplemented requirements. Now you have more customers, you're investing more in marketing, you can't afford the mistakes you used to be able to tolerate. Your startup team wants to begin establishing slower, more measured process for greater predictability and fewer mistakes. You might think about hiring a more "experienced" program manager or project manager, maybe you think about moving to a waterfall process. Although it seems counter-intuitive, you must not slow down... you need to speed up. In his post, Speed up or slow down? lean startup evangelist Eric Ries refers to this as the speed-up-or-slow-down-moment. Eric advocates for speeding up -- "The day-to-day process that startups build should also attempt to maximize speed of learning." Not only do I strongly agree with the need to speed up, I believe you need to speed up regardless of your company's current size, age or market cap. You need to focus on staying lean and agile -- or suffer the fate of the slowest gazelle...

I'd like to use Eric's broadened definition of a startup, with minor modification:
"A startup is a human institution designed to create a new product or service under conditions of extreme uncertainty."
-- Eric Ries
I've struck out new. While this may be a useful modifier for the traditional view of a startup, I strongly believe it is not a precondition for applying many of the lean startup principles. All businesses, regardless of product or service (old or new), are operating under conditions of extreme uncertainty. Over half the companies in the Dow Jones Industrial Average, still a key benchmark for performance of the US Equities Market, were added in just the past 20 years. Even if you think you've moved beyond startup classification, there are lions stalking you -- competitors, regulators, patent trolls, your customers' whims. You need to be faster than all of them.

In 2007, at my previous company Vonage, we were found to be infringing on Verizon's patents. Verizon sought and won an injunction from our use of those patents -- which included technology at the core of Vonage's call processing infrastructure. I will abstain for now from editorializing on the sanity of these rulings but, I wrote about my general sentiments on a similar patent issue here. Vonage managed to appeal and receive a stay for the injunction. This stay set the timeline for our designing patent workarounds. A patent workaround is basically a new way of getting something to work that can not be read on the patent. A judge then needs to approve the workaround. If you think involving business owners in designing software is hard, try working with lawyers... Then, drop a judge in as your QA team. Only if the judge doesn't approve your workarounds, it is basically lights out. Goodbye 2mil customers; farewell $50mil monthly revenues; goodnight future potential for pivots.

Prior to beginning the patent workarounds, it took months to deploy substantial changes to our call processing system. The planned deployment cycle was 6-8 weeks; 3-4 weeks for QA plus another 3-4 weeks to deploy. Inevitably, between week 6 and week 8, a critical defect would be found (memory leak, call loop, etc), the code would be rolled back and the cycle would begin again.

Gaining agreement to move faster with code deploys with QA and Operations was initially met with antagonism - "well, if your coders didn't put any bugs in the code, it would get deployed faster." Once our survival as a company was inextricably tied to our ability to modify and deploy our core processing system, that agreement became substantially less complicated. We had already begun to develop code quality testing and automated integration tests. The manager of the team ran to Best Buy for a switch to build a network that the development team could run automated deployments on. We worked with the QA team to define a < 1 work day set of tests to be ran for the most frequently used features. Then, we (substantially) automated the deployment and production integration testing of the call processing system.

In around 2 months, we went from taking months to deploy the system, to at the height of the workaround madness, deploying the entire system of ~100 machines in less than 24 hours -- for a system supporting 2 million customers making calls next door, across continents, to the other side of the world and most importantly to 911. All systems have bugs and defects, but, when you can fix those bugs and redeploy on a daily basis the impact of any one bug is substantially reduced. Most importantly, we learned what we needed to do to release quickly and often, with lower stress and higher quality thanks to fewer regressions, better automated testing and faster identification of bugs caused by rare or edge cases. By hitting those edge cases early on in the process, we knew we had a stable base to build from if there was a ruling that required any new change.

Being able to "run faster" helped ensure Vonage's future in a very uncertain time. No matter how big your company is, there is no way of knowing where the "lion" is going to come from. Rest assured they're out there; will you be ready to outrun them when they pounce? For a technology company, choosing to speed up improves the likelihood you'll be able to outrun your competitors, react to regulators, work around predatory patent litigation and most importantly react to your customers' needs. Don't be the slowest gazelle.

* As a footnote, Vonage did end up settling with Verizon before the workarounds came to a court decision. Even though we were confident we could quickly modify our code to whatever specs were needed to qualify for a workaround, in an appeals situation you're not guaranteed the judge will accept your workaround. A single day without using our call processing technology would have essentially put Vonage out of business.

Monday, March 1, 2010

On becoming a practicing software engineer

If you're a recent or soon-to-be college grad (or maybe you realized your undergrad degree in Art History ain't gonna pay the bills) and you are passionate about computers and computer programming, here are my tips for becoming a successful practicing software engineer.  Many of these things probably aren't the things they taught you in your college programming classes, but all of these are important.
  1. Practice! Practice! Practice! You learn to code by, um, reading and writing code! If you don't have much experience and want to get started, find an open source project you care about and contribute a patch or two.  The "View Source" feature of Web Browsers and the open source movement are 2 of your greatest assets when learning to code.  Use them! As someone who started his career copying BASIC programs from Compute! and Byte magazines, I can't tell you how great it was to discover the magical "View Source" menu item in Netscape.  Open Source projects will teach you about packaging, style guidelines, automated testing, bug tracking, and version control -- while also giving you much needed practice.

    Don't forget, your code doesn't *work* until someone else uses it.  If you can't work at a startup and need to get code into the hands of users quickly, open source projects are a great way to go.  I still recommend going with the startup, though.  At Knewton, everyone from summer interns to new full-time engineers ship code (that customers actually use) in their first couple weeks of starting.  I'd like to get this down to the first day.

    In your early practicing, make sure to develop really strong habits.  I learned the most of my habits from Steve McConnell's Code Complete and Kent Beck's eXtreme programming.  Write code others can support if they need to, but try to make it so they don't need to support it :)

    I'd also recommend checking out PragDave's Code Kata site to work on solving problem:

  2. Work at a startup! You will learn more in your first month at a startup than you will in your first year in any other company.  The first company I worked for was a 3-person shop in Syracuse, NY.  I learned everything from how to become a practicing software engineer, to how to be a customer support person, estimate and bid consulting jobs, write requirements, QA, write user manuals, configure SQL Servers, configure IIS Servers, configure linux firewalls ... If your first job entails you being handed requirements that you then write code for and hand "over the wall" to QA - run!

    Don't just take my word for it.  Here's what one of our former interns who recently graduated and landed a sweet job in CO had to say:
    "I had NO experience as a coder.  You guys gave me a LOT.  In fact, more skillz than you can really understand. Things that transferred over beyond Ruby, RUnit, Rails, MySQL unix commands (I'm loving that I actually understand how to use the CLI in my Ubuntu set up btw...), etc - more the ability to take a bunch of instructions I barely understood and google my way/solve my way to a solution."
    Chris Dixon also has a couple nice posts on this topic:
    Joining a startup is less risky than you think.

    More specifically, work for my startup:

    If there's nothing interesting to you there, check out these couple of other sites:

  3. Don't be afraid to make mistakes! or Perfection is for popes and Chinese emperors.
    In 1999, I got my first big job at an online brokerage (Datek) in NYC.  This was back when everyone, probably including you, traded online and doubled their money daily - basic fundamental laws of economics changed... until they didn't.  The day of my first big release, I took down the ability to login to the production site at market open.  This was, as my CTO at the time reminded me, "reeeeaaaaaaalllllly baaaaaddddd."  However, we talked through the problems and took the appropriate steps to ensure I couldn't make the same mistake again (it involved improper DB connection handling, lack of performance testing, deployment timing and rollback procedures).  If he fired me, maybe I'd feel differently about making mistakes.  Fortunately, for me, he understood these types of mistakes will happen -- as long as you're willing to grow from the mistakes and not repeat the same mistake twice.

    Don't forget that coding is a creative endeavor; typically, there is no one correct solution.  Be prepared to try several.

  4. Reality...zen and the art of "boring" tasks.
    You studied sexy problems in school, you know how to solve nine-queens efficiently, find shortest path in a graph, compute Chebyshev distance in a metric space...the large part of your day as a software developer will not be spent working on such problems. More than likely you will spend a day on a far wider range of tasks that are comparatively less interesting in an engineering/problem-solving sense. These lower-level tasks are generally more simple and yet each decision that is made in their execution can be evaluated and perhaps improved upon. Design details as small as method signatures, naming conventions, loop constructs or recursion, tail-recursion or not (maybe even that's too sexy!), are both extremely important and regularly overlooked. A large program is built on many lines of code. Each line contains required prior decisions to produce. An appreciation for these small details will contribute a lot not only to the quality of the program as a whole, but to the education of the coder.

    If you're not already in a coding-related field, look for ways to make your current job more efficient through automation.  This can be as simple as creating access databases, word mail merges and batch files to automate tasks that used to take you hours or days to complete.  This is actually how I got my start coding professionally, by building MS Access database apps that made week-long tasks takes hours (mass mailings to customers at an HVAC rep and students at the SU Masters of Public Administration program).

  5. Finish!
    You don't get any points for effort.  You need to finish what you start.  Half-written, incomplete code atrophies very, very quickly.  If you find yourself starting more than you finish, you need to revisit the scope of your problems.  Code against smaller problems, but finish the code!
While not everyone is destined to be a great coder, if you're interested in learning how, I strongly recommend the list above.  I'm not certain this is an exhaustive list; I'd love to hear any other suggestions.  Good luck, and have fun!