The Lean Startup - Eric Ries [99]
Why did we add a new version of a gem in production without testing?
Answer: We didn’t think we needed a test in these cases.
Proportional investment: Bennett and Jim—Write a unit or functional test in the API and CMS that will catch this in the future.
Why do we add additional gems that we don’t intend to use right away?
Answer: In preparation for a code push we wanted to get all new gems ready in the production environment. Even though our code deployments are fully automated, gems are not.
Proportional investment: Bennett—Automate gem management and installation into Continuous Integration and Continuous Deployment process.
Bonus—Why are we doing things in production on Friday nights?
Answer: Because no one says we can’t and it was a convenient time for the developer to prepare for a deployment we’d be doing on Monday.
Proportional investment: Tony—Make an announcement to the team. There will be no production changes on Friday, Saturday, or Sunday unless an exception has been made and approved by David (VP Engineering). We will reevaluate this policy when we have a fully automated continuous deployment process in place.
As a result of this Five Whys session and the proportional investments we made, our deployments are easier, quicker, and never again will our process allow a developer to place gems into production systems with unintended consequences. Indeed, we have not had another issue like this. We strengthened our “cluster immune system” as you would say.
Without the Five Whys, we would have never discovered all of the information we did here. My guess is that we would have told that one developer to not do stupid things on Friday nights and moved on. This is what I emphasized earlier, where a good Five Whys session has two outputs, learning and doing. The proportional investments that came out of this session are obviously valuable, but the learnings are much more subtle, but amazing for growing as developers and as a team.
ADAPTING TO SMALLER BATCHES
Before leaving the topic of building an adaptive organization, I want to introduce one more story. This one concerns a product that you’ve probably used if you’ve ever run your own business. It’s called QuickBooks, and it is one of Intuit’s flagship products.
QuickBooks has been the leading product in its category for many years. As a result, it has a large and dedicated customer base, and Intuit expects it to contribute significantly to its bottom line. Like most personal computer (PC) software of the last two decades, QuickBooks has been launched on an annual cycle, in one giant batch. This was how it worked three years ago, when Greg Wright, the director of product marketing for QuickBooks, joined the team. As you can imagine, there were lots of existing processes in place to ensure a consistent product and an on-time release. The typical release approach was to spend significant up-front time to identify the customers’ need:
Typically the first three to four months of each annual cycle was spent strategizing and planning, without building new features. Once a plan and milestones were established, the team would spend the next six to nine months building. This would culminate in a big launch, and then the team would get its first feedback on whether it had successfully delivered on customers’ needs at the end of the process.
So here was the time line: start process in September, first beta release is in June, second beta is in July. The beta is essentially testing to make sure it doesn’t crash people’s computers or cause them to lose their data—by that time in the process, only major bugs can be fixed. The design of the product itself is locked.
This is the standard “waterfall” development methodology that product development teams have used for years. It is a linear, large-batch system that relies for success on proper forecasting and planning. In other words, it is completely maladapted for today’s rapidly changing business environment.
Year One: Achieving Failure
Greg witnessed this breakdown in 2009, his first year on the QuickBooks