Tuesday, August 27, 2013

The Lean Startup Movement and the Quant Uprising

I'm building my first new product in over a decade. I started my career back in 1993 as a black box tester on Media 100. I joined the team just as the first engineering prototypes were coming in from manufacturing, well prior to shipping v1.0. After that I helped create Adobe ImageStyler 1.0 in 1998, and then Adobe LiveMotion 1.0 in 2000.

On the suggestion of a coworker I picked up "The Lean Startup" by Eric Reis which lead to a whole lot of book reading (see my other post here).

As my team and I began applying the tools such as Problem and Solution Interviews, mock web sites with A/B testing, and in product analytics, I had a growing sense of unease. A long time ago, in a place far, far, away, I studied chemistry, a fair amount of psychology, and statistics. I've also been deeply personally involved in the evidence based medicine movement and often find myself reading meta-analyses at the Cochrane Collaboration.

The tools from the LSM were generating a fair amount of structured data, some qualitative, and some quantitative. As my coworkers attempted to communicate their findings, they started committing data abuse. They would run a survey on an incredibly small population and attempt to assign meaning to differences that were beyond any reasonable threshold of noise. Or make a graph comparing data without appropriately normalizing it, rendering the results meaningless. Or score aspects of qualitative interview to allow quantitative comparison without controlling for the lack of rigor in how we conducted the interviews.

In "The Lean Startup", Eric talks about Vanity Metrics. But he only looks at some small ways to bias your data so you can lie to yourself. As I continued to deepen my research through reading more books and talking to our researchers, I realized that there are many, many ways to lie to yourself via these methods. 

I had to sit back and collect myself.

Vexed that these new tools I was so inspired by when I read "The Lean Startup" can be so easily abused.

I then realized that data driven decisions (aka The Quants) is a whole continuum of tradeoffs. On the quick and dirty side we have some of the tools I used 12 years ago: Informal customer chats, launch and pray, intuition. On the deep quant side you have extensive surveys, large scale A/B testing, structured and appropriately mediated customer interviews (using tools from the social sciences to avoid bias). For medical work you get all they way to double blind control studies as the gold standard.

But moving across this spectrum greatly changes the amount of time and cost associated with making a decision. In the business world, it can often be cheaper to fail (or make a wrong decision earlier) then to delay action until you have the quants figure it out. 

That was the key part for me: Failure can be cheaper then learning through experiments.

Your job as a leader is to understand the risks and choose the level of rigor and science that is appropriate to the situation.

The Oakland A's helped revolutionize baseball by adding in some quant tools to the player recruitment process. This gave them a huge first mover advantage. It didn't take very long for the other teams to adopt the same tools, therefore rendering that lead negligible. But now just to be in the game you'd better be using those tools or you'll be at a distinct disadvantage. (http://blogs.hbr.org/davenport/2011/09/six_things_your_company_has_in.html

The Lean Startup Movement (LSM) tools introduce a little bit more quant (and therefore rigor, discipline, and skill) with the hope of reducing risk. But they are a tradeoff. They take a bunch of work. Work that could be spent elsewhere. You need to decide if you are in a game like baseball where all your peers are using these tools. If they are, you'd better dust off your copy of "The Lean Startup" or just go home.

But you also need to be realistic about what you'll get from the tools. They are incredibly rough approximations of tools that have been used in the sciences for years. They are full of bias and inaccuracy, just (hopefully) slightly less then the techniques you were using before (and of course slightly better then your competitors). On the flip side, you may be making a decision that is so high risk that you want significantly more rigor then the LSM tools.

I strongly believe that managers who use Quants appropriately will kick ass in the long run. But it's that 'appropriately' part that's hard. As a community we are collecting, quantifying, and tabulating more data then ever. 

Do you have the skills to choose the appropriate level of rigor for the decision at hand?

I didn't used to. I'm getting better though. My next post will go into details as to how I buffed up my inner quant so I can select the appropriate tool for the task at hand.

Till next time,
Chris




26 comments:

  1. Chris, interesting to hear your perspectives on the continuum between the extremes of "quant" and "launch & pray"--I've been wondering about this dichotomy lately. A mixed strategy with the best of both seems most appealing. Techniques like A/B testing do allow you to quantify the potential value of design deltas, but they can also encourage local hill-climbing instead of innovation. Reminds me of Richard Hamming's motto, "The Purpose of Computing is Insight, not Numbers." Looking forward to your "Buffed Quant" blog! --MLB

    ReplyDelete
  2. It will be useful for you to find out more about scholarship essay writing. It helped me a lot when I was in college.

    ReplyDelete
  3. Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This siteseems to get a good amount of visitors

    ReplyDelete
  4. You can learn a lot when you communicate with professionals. For that reason, it will be extremely helpful for you to visit this website https://www.essay-company.com/essay-writer if you need to make your writing skills better

    ReplyDelete
  5. How to write a good essay? I know a lot from https://samedayessay.com/buy_essay
    For instance, Avoid explicitly stating the point of your essay. It’s far less effective when you spell it out for someone. Delete every single “That’s when I realized,” “I learned,” and “The most important lesson was...” It's unnecessary, unconvincing, and takes the reader out of the moment. Instead, let them read between the lines and interpret the meaning of your story on their own. You shouldn’t have to say anything like, “And that’s how I learned to stand up for myself,” because the admission's officer should already know. Oftentimes when you watch a movie, an actor’s expression, sigh, or closing of a door speaks louder than words.

    ReplyDelete
  6. I was skeptical about the essay writing services of dissertation help. But, when they delivered my essay 5 days before the deadline, they won my heart. Upon request, the writers sent me the outline for my college essay that they’ve prepared before writing my paper. Other companies do not provide students with an essay outline. I was unsure if they could work on my paper from scratch. To my surprise, dissertation help provided the best outline and delivered a plagiarism report for free!

    ReplyDelete
  7. Get the assignment help from the professional online writing services that are ready to help you at any moment.

    ReplyDelete
  8. How Lemeridian funding service  grant me a loan!!!

    Hello everyone, I'm Lea Paige Matteo from Zurich Switzerland and want to use this medium to express gratitude to lemeridian funding service for fulfilling his promise by granting me a loan, I was stuck in a financial situation and needed to refinance and pay my bills as well as start up a Business. I tried seeking for loans from various loan firms both private and corporate organisations but never succeeded and most banks declined my credit request. But as God would have it, I was introduced by a friend named Lisa Rice to Le_meridian funding service and undergone the due process of obtaining a loan from the company, to my greatest surprise within 48hrs just like my friend Lisa, I was also granted a loan of $216,000.00 So my advise to everyone who desires a loan, "if you must contact any firm with reference to securing a loan online with low interest rate of 1.9% and better repayment plans/schedule, please contact Le_meridian funding service. Besides, he doesn't know that am doing this but due to the joy in me, I'm so happy and wish to let people know more about this great company whom truly give out loans, it is my prayer that GOD should bless them more as they put smiles on peoples faces. You can contact them via email on {lfdsloans@lemeridianfds.com Or lfdsloans@outlook.com} or Text through Whatsapp +1-989 394 3740.

    ReplyDelete
  9. Do you often need to quickly write an essay? In this case, the article and essay will help you. https://paramountessays.com/dissertation-editing

    ReplyDelete
  10. Need to write the best opinion essay? By following throught the list of opinion essay topics you can find something you want to write about.

    ReplyDelete
  11. Hi to everybody, here everyone is sharing such knowledge, so it’s fastidious to see this site, and I used to visit this blog daily. ExcelR Data Analyst Course

    ReplyDelete
  12. But you also need to be realistic about what you'll get from the tools. They are incredibly rough approximations of tools that have been used in the sciences for years. They are full of bias and inaccuracy, just (hopefully) slightly less then the techniques you were using before (and of course slightly better then your competitors). On the flip side, you may be making a decision that is so high risk that you want significantly more rigor then the LSM tools. heavy bed sheets , queen comforter sets , queen bed sheet sets , nishat linen bridal bed sheets , bedding sets uk , cotton razai , apricot sofa covers , velvet bed sheets , razai cover online , towel wholesale market in karachi

    ReplyDelete
  13. Wonderful article! This blog makes me to learn new thinks. Thanks for your content. Best digital marketing course I am suggesting here. This field having more job opportunity in future.

    ReplyDelete
  14. Thanks for this awesome information. If you need an attractive website design for your business, visit OGEN Infosystem and also get SEO Services for your business promotion.free article submission site

    ReplyDelete
  15. Thanks a lot for sharing kind of information. Your article provides such great information with good knowledge. Digital Marketing Training in Pune

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. Very interesting article.
    I once also started writing from small blogs and now I work https://manyessays.com/order-dissertation-online and I am very satisfied.
    I love to write essays, articles, essays and coursework.
    I am constantly learning new information and so I develop.

    ReplyDelete
  18. 4 years ago. Beautiful, sensitive, kind and eyes as deep as the sea. But, he was going through a divorce and I could see he needed to heal https://remarriagemag.com. So we remained friends and kept contact. My hopes were after he healed and became stable we could move forward.

    ReplyDelete
  19. Thanks for The Amazing Blog.
    We’re one of The Best Food License Consultant in Hyderabad.

    ReplyDelete
  20. I really appreciate this wonderful post that you have provided for us. I assure this would be beneficial for most of the people.
    data analytics training in hyderabad

    ReplyDelete
  21. We also provide interview questions and answers according to industry standards along with resume writing. We also give you the opportunity to practice mock interviews and exams. You can get a job learning Online Marketing if you have no prior experience
    Working on real-life case studies Add industry-recognized training to your resume and LinkedIn profile Work with top multinational companies.

    digital marketing course in Bangalore

    ReplyDelete
  22. Digital marketing training online and certification in Google Analytics Individual Qualification Google Ads Display Certification Google Ads Measurement Certification Shopping Ads Certification Google Ads.
    digital marketing training online

    ReplyDelete