The Lean Startup Movement and the Quant Uprising

I'm building my first new product in over a decade. I started my career back in 1993 as a black box tester on Media 100. I joined the team just as the first engineering prototypes were coming in from manufacturing, well prior to shipping v1.0. After that I helped create Adobe ImageStyler 1.0 in 1998, and then Adobe LiveMotion 1.0 in 2000.

On the suggestion of a coworker I picked up "The Lean Startup" by Eric Reis which lead to a whole lot of book reading (see my other post here).

As my team and I began applying the tools such as Problem and Solution Interviews, mock web sites with A/B testing, and in product analytics, I had a growing sense of unease. A long time ago, in a place far, far, away, I studied chemistry, a fair amount of psychology, and statistics. I've also been deeply personally involved in the evidence based medicine movement and often find myself reading meta-analyses at the Cochrane Collaboration.

The tools from the LSM were generating a fair amount of structured data, some qualitative, and some quantitative. As my coworkers attempted to communicate their findings, they started committing data abuse. They would run a survey on an incredibly small population and attempt to assign meaning to differences that were beyond any reasonable threshold of noise. Or make a graph comparing data without appropriately normalizing it, rendering the results meaningless. Or score aspects of qualitative interview to allow quantitative comparison without controlling for the lack of rigor in how we conducted the interviews.

In "The Lean Startup", Eric talks about Vanity Metrics. But he only looks at some small ways to bias your data so you can lie to yourself. As I continued to deepen my research through reading more books and talking to our researchers, I realized that there are many, many ways to lie to yourself via these methods. 

I had to sit back and collect myself.

Vexed that these new tools I was so inspired by when I read "The Lean Startup" can be so easily abused.

I then realized that data driven decisions (aka The Quants) is a whole continuum of tradeoffs. On the quick and dirty side we have some of the tools I used 12 years ago: Informal customer chats, launch and pray, intuition. On the deep quant side you have extensive surveys, large scale A/B testing, structured and appropriately mediated customer interviews (using tools from the social sciences to avoid bias). For medical work you get all they way to double blind control studies as the gold standard.

But moving across this spectrum greatly changes the amount of time and cost associated with making a decision. In the business world, it can often be cheaper to fail (or make a wrong decision earlier) then to delay action until you have the quants figure it out. 

That was the key part for me: Failure can be cheaper then learning through experiments.

Your job as a leader is to understand the risks and choose the level of rigor and science that is appropriate to the situation.

The Oakland A's helped revolutionize baseball by adding in some quant tools to the player recruitment process. This gave them a huge first mover advantage. It didn't take very long for the other teams to adopt the same tools, therefore rendering that lead negligible. But now just to be in the game you'd better be using those tools or you'll be at a distinct disadvantage. (http://blogs.hbr.org/davenport/2011/09/six_things_your_company_has_in.html) 

The Lean Startup Movement (LSM) tools introduce a little bit more quant (and therefore rigor, discipline, and skill) with the hope of reducing risk. But they are a tradeoff. They take a bunch of work. Work that could be spent elsewhere. You need to decide if you are in a game like baseball where all your peers are using these tools. If they are, you'd better dust off your copy of "The Lean Startup" or just go home.

But you also need to be realistic about what you'll get from the tools. They are incredibly rough approximations of tools that have been used in the sciences for years. They are full of bias and inaccuracy, just (hopefully) slightly less then the techniques you were using before (and of course slightly better then your competitors). On the flip side, you may be making a decision that is so high risk that you want significantly more rigor then the LSM tools.

I strongly believe that managers who use Quants appropriately will kick ass in the long run. But it's that 'appropriately' part that's hard. As a community we are collecting, quantifying, and tabulating more data then ever. 

Do you have the skills to choose the appropriate level of rigor for the decision at hand?

I didn't used to. I'm getting better though. My next post will go into details as to how I buffed up my inner quant so I can select the appropriate tool for the task at hand.

Till next time,

Chris

Recovery from CFS/Post Exercise Syndrome

Back in my original post about my electric bike conversion I mentioned that I had CFS, aka Chronic Fatigue Syndrome. CFS is a poorly defined health condition that in my opinion actually covers a number of very different health conditions. I'm glad I never really accepted it as a diagnosis from the rheumatologist that mentioned it to me, but instead kept scouring my life and health for anything that could impact my energy level.

I'm happy to say I no longer suffer from it.

While it was a slow slide, it hit me hard in the fall of 2008. I just had my first big release as a new engineering manager; planned a wedding and got married; and I started flight training that summer. By the time the wedding rolled around I was beat in a way I had never been tired before. No amount of coffee or sleep would help.

It took me 4 years to figure out all the contributing factors and rehabilitate myself. During that time I met with 7 different doctors trying to get a handle on all the different causes that would contribute to being run down.

I can be a tenacious bastard when I have a goal in sight.

At one point I was working with an endocrinologist and still couldn't handle aerobic exercise. 10 minutes with my heart rate above 105 bpm and I would need to sleep a couple of hours later. I had to ask him "Do you have any more ideas, is there anything else _you_ can do for me now?" after a long pause he admitted no.

Which was a good thing because then I went searching for more answers. In the end I had to address the following things to get my energy back:

• Allergies
• Sleep Apnea
• Stress Management
• Sleep Quality
• Vitamin D levels (I was near the level that causes rickets)
• Stress response to exercise

The last one was the most interesting and least documented of the set. After I had addressed the previous five items, my baseline energy level was great, but any aerobic exercise knocked me flat. Dr. Emily Cooper at Seattle Performance Medicine was the person who helped my through that. It took 3 months of very precise interval based aerobic exercise at the gym to work through the fatigue. At the beginning I had to plan my days to allow me to take a nap after the exercise, or work out in the evening so I could just go to sleep.

After several months I noticed I no longer needed to nap. I was dumbfounded. I kept trying to tickle the dragon, but nope, I was solid for the first time in years.

So now I ride the electric bike for fun and pleasure.

Lean Startup Book Roundup

Earlier this year I got involved with using Lean Startup techniques to help with a new business inside of Adobe. As is my normal style, I read a large number of books to help get my head around the techniques and build up a base of knowledge I could use in the future.  

As an entrepreneur/intrapreneur you have to be able to lie to yourself a little. Otherwise you would just stay at home and not pursue the new idea that you have. But lying to yourself until you ship your product to the marketplace can be expensive and emotionally devastating.

For those of you new to the Lean Startup Methodologies, they are tools that help you stop lying to yourself and check in with reality at all phases of developing your business. My own path with these tools has been somewhat backward. I originally thought we were further along with our business when I picked them up. But as I applied a tool that I thought was appropriate for the phase of development (for instance a retention graph for our private beta), reality would come through and we need to go a step earlier in the chain to find the problem. We finally ended up all they way back at the beginning.

Here are the books I read and what I took away from them.

"The Lean Startup" by Eric Reis : http://theleanstartup.com/

This is a very inspirational book. It goes pretty fast and doesn't get bogged down. One of the most influential portions for me was the discussion of Vanity Metrics. These are ones that are generally easy to measure and create, but almost always lie and don't provide real actionable data.

I consider this book a great seed of ideas, but too thin on actual implementation and case studies. Read it, get inspired, get familiar with the terms and concepts, then move on to a different book for implementation.

"Running Lean" by Ash Maurya : http://runninglean.co/

I consider this the next book in the series. A key tenet to stop lying to yourself is to go out and interview potential customers in as neutral a way as possible (it it very easy to influence people so they tell you what you want to hear).

Ash does a great job step by step explaining how to use a Lean Canvas (a derivation of the Business Model Canvas) to help clarify your ideas, find target customers, and then go out an do qualitative interviews. He has a lot of examples of how he used these tools to clarify his thinking and provides good guidance when you use them yourself. 

"Lean Analytics" by Alistair Croll  (Author) , Benjamin Yoskovitz : http://leananalyticsbook.com/  

This book is good when you have an idea of where you are going. While they talk some about the early qualitative interviews you need to get out and do (in their Empathy stage), I found "Running Lean" much more detailed and useful for doing that. But "Lean Analytics" does point out many common problems in interviewing techniques that aren't really addressed in "Running Lean" and therefore is a nice complement.

"Lean Analytics" has my favorite graphic in it where they list the major lean methodologies that are kicking around and compare and contrast them. They then go ahead and invent their own. That just underscores the fact that LSM is just like Agile Engineering. It's much more about philosophy and culture then any specific dogma. Dogma is good when getting started as it can help prevent you from heading into the weeds, but as you get familiar with the concepts you can be flexible.

This book was my first introduction to many different revenue generation models (I read it before "Business Model Generation") and is great for that. Not only does it cover the models, but also what metrics should be tracked, and has many case studies as to how these were applied in the real world.

The book is also awesome for coming up with "The One Metric That Matters". It can be easy to lose focus with all the things going on, and this is a reminder to regularly pick out what you should be targeting, figuring out how to measure it, and then head for it.

One thing I was disappointed about is that they don't really address anything about the statistical and/or business significance that you can attach to numbers. When starting out at the beginning of a product, you normally have very small sample sizes and it is very easy to be mislead by your data. When working with small sample sizes you need to be vigilant when to consider your quantitative data suspect and just throw it out lest it influence your thinking (see "Thinking, Fast and Slow" chapter 14). Stay tuned for a blog post on this problem area.

"The Startup Owner's Manual" by Steve Blank and Bob Dorf : http://www.stevenblank.com/startup_index_qty.html

I really wanted to like this one. It's intentionally not done in a narrative style and I found it hard to digest. While I occasionally would reference it, I found myself using the other books more often. The book does call this out at the beginning. I think the book would work much better in a classroom setting then how I used it. I found the explanations of _how_ just not fitting how my brain works. The why parts are great.

The book does have a great discussion of the Customer Development methodology (which helped spawn Eric Ries' thinking in "The Lean Startup) and is good for that reason alone. The other parts are useful, and it has some good case studies. I just opened it to refresh my mind while writing this and I saw a few nuggets that I liked.

On a dorky note, I had a really hard time with the graphic design of the book. I didn't like the main serif font chosen, and I found the mixture of serif, and sans serif fonts jarring. I also didn't like the use of whitespace or even the general typographic layout (the leading felt off, particularly in bullet lists). But I've worked at Adobe for 15 years so I'm probably more sensitive to these things.

"Business Model Generation" by Alexander Osterwalder and Yves Pigneur : http://www.businessmodelgeneration.com/

Speaking of design, the graphic design of this book is luscious. Do NOT buy this on Kindle. The physical book has a wonderful layout along with useful and interesting imagery. As a business and tech person who has a foot in the visual arts, I found that the graphic design really helped pull me into the book and keep my attention. 

If you have a hard time sitting down and plowing through page after page of text, just go and experience this book. You'll need the tools in the other books at some point, but this will get you started in thinking like an entrepreneur.

This book provides a common language (and visual language) for communicating and reasoning about all the key parts that go into a business, not just a product. It can be used for any kind of business, so certain parts didn't make sense for us, but it was a good start. In the end we ended up using the Lean Canvas from Ash Maurya since it was adapted to fit our space better than the generic canvas in this book.

This book really helped my team come to agreement on terminology and allowed us to communicate more clearly with each other. A canvas is much quicker and more effective then a powerpoint with a long presentation.

As you can see, no one book does it all. That's a good thing since this is a big thing to wrap your head around, and frankly there is no one true way.

I just picked up a copy of "Disciplined Entrepreneurship" (http://disciplinedentrepreneurship.com/) that has come out of MIT Sloan Business school. I'm thrilled to see the East Coast represent! I grew up in Newton, MA and started working in tech back there before relocating to San Francisco and then Seattle. I've only read the forward, but he starts by recommending all those books above and pointing out that there is no single book to cover all the steps.

Happy Reading!