Tuesday, August 27, 2013

The Lean Startup Movement and the Quant Uprising

I'm building my first new product in over a decade. I started my career back in 1993 as a black box tester on Media 100. I joined the team just as the first engineering prototypes were coming in from manufacturing, well prior to shipping v1.0. After that I helped create Adobe ImageStyler 1.0 in 1998, and then Adobe LiveMotion 1.0 in 2000.

On the suggestion of a coworker I picked up "The Lean Startup" by Eric Reis which lead to a whole lot of book reading (see my other post here).

As my team and I began applying the tools such as Problem and Solution Interviews, mock web sites with A/B testing, and in product analytics, I had a growing sense of unease. A long time ago, in a place far, far, away, I studied chemistry, a fair amount of psychology, and statistics. I've also been deeply personally involved in the evidence based medicine movement and often find myself reading meta-analyses at the Cochrane Collaboration.

The tools from the LSM were generating a fair amount of structured data, some qualitative, and some quantitative. As my coworkers attempted to communicate their findings, they started committing data abuse. They would run a survey on an incredibly small population and attempt to assign meaning to differences that were beyond any reasonable threshold of noise. Or make a graph comparing data without appropriately normalizing it, rendering the results meaningless. Or score aspects of qualitative interview to allow quantitative comparison without controlling for the lack of rigor in how we conducted the interviews.

In "The Lean Startup", Eric talks about Vanity Metrics. But he only looks at some small ways to bias your data so you can lie to yourself. As I continued to deepen my research through reading more books and talking to our researchers, I realized that there are many, many ways to lie to yourself via these methods. 

I had to sit back and collect myself.

Vexed that these new tools I was so inspired by when I read "The Lean Startup" can be so easily abused.

I then realized that data driven decisions (aka The Quants) is a whole continuum of tradeoffs. On the quick and dirty side we have some of the tools I used 12 years ago: Informal customer chats, launch and pray, intuition. On the deep quant side you have extensive surveys, large scale A/B testing, structured and appropriately mediated customer interviews (using tools from the social sciences to avoid bias). For medical work you get all they way to double blind control studies as the gold standard.

But moving across this spectrum greatly changes the amount of time and cost associated with making a decision. In the business world, it can often be cheaper to fail (or make a wrong decision earlier) then to delay action until you have the quants figure it out. 

That was the key part for me: Failure can be cheaper then learning through experiments.

Your job as a leader is to understand the risks and choose the level of rigor and science that is appropriate to the situation.

The Oakland A's helped revolutionize baseball by adding in some quant tools to the player recruitment process. This gave them a huge first mover advantage. It didn't take very long for the other teams to adopt the same tools, therefore rendering that lead negligible. But now just to be in the game you'd better be using those tools or you'll be at a distinct disadvantage. (http://blogs.hbr.org/davenport/2011/09/six_things_your_company_has_in.html

The Lean Startup Movement (LSM) tools introduce a little bit more quant (and therefore rigor, discipline, and skill) with the hope of reducing risk. But they are a tradeoff. They take a bunch of work. Work that could be spent elsewhere. You need to decide if you are in a game like baseball where all your peers are using these tools. If they are, you'd better dust off your copy of "The Lean Startup" or just go home.

But you also need to be realistic about what you'll get from the tools. They are incredibly rough approximations of tools that have been used in the sciences for years. They are full of bias and inaccuracy, just (hopefully) slightly less then the techniques you were using before (and of course slightly better then your competitors). On the flip side, you may be making a decision that is so high risk that you want significantly more rigor then the LSM tools.

I strongly believe that managers who use Quants appropriately will kick ass in the long run. But it's that 'appropriately' part that's hard. As a community we are collecting, quantifying, and tabulating more data then ever. 

Do you have the skills to choose the appropriate level of rigor for the decision at hand?

I didn't used to. I'm getting better though. My next post will go into details as to how I buffed up my inner quant so I can select the appropriate tool for the task at hand.

Till next time,


  1. Chris, interesting to hear your perspectives on the continuum between the extremes of "quant" and "launch & pray"--I've been wondering about this dichotomy lately. A mixed strategy with the best of both seems most appealing. Techniques like A/B testing do allow you to quantify the potential value of design deltas, but they can also encourage local hill-climbing instead of innovation. Reminds me of Richard Hamming's motto, "The Purpose of Computing is Insight, not Numbers." Looking forward to your "Buffed Quant" blog! --MLB

  2. It will be useful for you to find out more about scholarship essay writing. It helped me a lot when I was in college.

  3. Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This siteseems to get a good amount of visitors

  4. You can learn a lot when you communicate with professionals. For that reason, it will be extremely helpful for you to visit this website https://www.essay-company.com/essay-writer if you need to make your writing skills better

  5. How to write a good essay? I know a lot from https://samedayessay.com/buy_essay
    For instance, Avoid explicitly stating the point of your essay. It’s far less effective when you spell it out for someone. Delete every single “That’s when I realized,” “I learned,” and “The most important lesson was...” It's unnecessary, unconvincing, and takes the reader out of the moment. Instead, let them read between the lines and interpret the meaning of your story on their own. You shouldn’t have to say anything like, “And that’s how I learned to stand up for myself,” because the admission's officer should already know. Oftentimes when you watch a movie, an actor’s expression, sigh, or closing of a door speaks louder than words.

  6. We can help you whenever and where ever you want, We are giving services all over world and especially in Australia in following catagories: Assignment help – University of Sydney, Help with assignments in Melbourne – Deakin University, online assignment help – Queensland University of Technology, Perth assignment writing help – Edith Cowan University, Canberra assignment help – University of Canberra, Assignment help services – Flinders University and many more.

  7. I was skeptical about the essay writing services of dissertation help. But, when they delivered my essay 5 days before the deadline, they won my heart. Upon request, the writers sent me the outline for my college essay that they’ve prepared before writing my paper. Other companies do not provide students with an essay outline. I was unsure if they could work on my paper from scratch. To my surprise, dissertation help provided the best outline and delivered a plagiarism report for free!

  8. Get the assignment help from the professional online writing services that are ready to help you at any moment.

  9. How Lemeridian funding service  grant me a loan!!!

    Hello everyone, I'm Lea Paige Matteo from Zurich Switzerland and want to use this medium to express gratitude to lemeridian funding service for fulfilling his promise by granting me a loan, I was stuck in a financial situation and needed to refinance and pay my bills as well as start up a Business. I tried seeking for loans from various loan firms both private and corporate organisations but never succeeded and most banks declined my credit request. But as God would have it, I was introduced by a friend named Lisa Rice to Le_meridian funding service and undergone the due process of obtaining a loan from the company, to my greatest surprise within 48hrs just like my friend Lisa, I was also granted a loan of $216,000.00 So my advise to everyone who desires a loan, "if you must contact any firm with reference to securing a loan online with low interest rate of 1.9% and better repayment plans/schedule, please contact Le_meridian funding service. Besides, he doesn't know that am doing this but due to the joy in me, I'm so happy and wish to let people know more about this great company whom truly give out loans, it is my prayer that GOD should bless them more as they put smiles on peoples faces. You can contact them via email on {lfdsloans@lemeridianfds.com Or lfdsloans@outlook.com} or Text through Whatsapp +1-989 394 3740.


  10. شركة نقل عفش من الرياض الى الامارات شركات نقل عفش من الرياض الى الامارات
    شركة نقل عفش من الرياض الى الاردن شركات نقل عفش من الرياض الى الاردن

    شركة نقل عفش من الرياض الى المدينة المنورة شركة نقل عفش من الرياض الى المدينة المنورة
    شركة نقل عفش من الدمام الى مكة شركة نقل عفش من الدمام الى مكة