Saturday, October 5, 2013

The Lean Startup Movement from a Decision Science perspective

First off, my apologies to actual Decision Scientists. I have no formal training and just recently learned that the area I'm fascinated by actually has a name.

There are a lot of anecdotes out there about how wonderful all the different Lean Startup methodologies are. If you go and read the Amazon book reviews, you'll see lots of comments about how it changed someones life. 

What you won't see is any data showing they actually help.

I'm a skeptic at heart. Studies on businesses are notoriously difficult to do, and while I could see value in the tools and techniques, I really wanted a deeper scientific basis for all the hype.

I finally managed to find it within the Decision Science literature. Daniel Kahneman has done a great survey of key ideas and concepts in "Thinking, Fast and Slow". It's a long book, but for the purposes of this discussion we really care about part 3: Overconfidence. "The Signal and the Noise" by Nate Silver also talks a fair amount about our inability to do forecasts well. Lastly, "Naked Statistics" provides another view.

The Lean Startup Methodologies (LSM) are designed to help you answer two questions:
1) Should I even bother building a full product?
2) If I do build the product, should I continue with incremental changes, or pitch the whole thing and start over?

As an entrepreneur with a product or business idea you are going be subject to three different cognitive illusions. These are things that are common to all of humanity, and there is little you can do to train them away. The three illusions I see as being the most problematic are Optimism Bias, Domain Expertise Confidence; Confidence in Prediction in an unstable environment.

Optimism Bias
Humans in general tend to think they perform better then average. In particular, entrepreneurs tend to be even more optimistic then the general population about their ability to beat the odds. For instance the base rate of failure for a new business is 65%. But the vast bulk of entrepreneurs consider their chance of failure at 40%. This delusion can help us make it through the day, but it can also cause you to stick with something for far too long. (Kahneman, Daniel (2011-10-25). Thinking, Fast and Slow (p. 256). Farrar, Straus and Giroux. Kindle Edition. )     

Domain Expert Overconfidence
Everyone is over-confident about their ability to predict the future. Doesn't matter how much of an expert you are in your field, you are overconfident. Time and time again we see research that shows a crappy mathematical model that is informed by expert opinion is almost always better than either one alone. (http://www.rff.org/Events/Documents/Lin_Bier.pdf, all of "The Signal and the Noise").

The second part here is apropos to a problem I'm wrangling with at work. I'm working on a product that is all about market creation. By definition, my ability to research and learn about my market is terribly limited since the market doesn't exist. Learning the Lean Startup tools improves your metacognitive ability to see your own weakness in expertise and therefore adapt to it. (http://gagne.homedns.org/~tgagne/contrib/unskilled.html)

Confidence of Prediction
When you ask yourself "Do I need this feature in my product?", the question you are really asking is "Will adding this feature to my product add enough value to my business in the future to justify the (opportunity) cost now?". That is attempting to forecast the future, something humans can be good or bad at depending on how quickly they get feedback about their decisions. Firefighters and nurses, who get almost instantaneous feedback on the quality of their forecasting are able to build a great amount of skill in this area. Those of us running in longer time scales fall prey to building confidence in our predictions, but not actually improving our accuracy. Think of how much that sucks for a moment. You can be in a field for years, making forecasts, assuming you are getting better, but in reality, you aren't. (Kahneman, Daniel (2011-10-25). Thinking, Fast and Slow (p. 240). Farrar, Straus and Giroux. Kindle Edition.)

Essentially it all boils down to the fact that you are going to feel really confident about your idea and plans, but that confidence is not based on actual hard data. It is instead an illusion being fed on how we perceive and think about the world. You can't use your gut 'Confidence' check to know if you really have a product or not at the get go, we're just not wired that way.

But by acknowledging our limitations, we can figure out ways to work around them.

So how do you deal with this trifecta of Overconfidence in your idea?

To quote Steve Blank: "GET OUT OF THE BUILDING" and go talk to customers.

That's it.

All the different LSM out there have different suggestions on how to go about defining a market segment, finding customers, and how to talk to them. But it boils down to talking directly to customers to counteract your overconfidence. They all reccomend a progressive approach in how you do your research. You start with wide open investigational interviews, and as you learn more (and validate or invalidate your overconfident beliefs) you start using more structured interviews to gather more accurate data (but less wide ranging) including things like surveys and prototypes. You can even run A/B value prop testing with a mock web site and google adwords.

That said, you need to be cautious about over-generalizing your results. We humans love to see patterns and over value things we see and measure, even if they have low confidence.
 (Patrick Leach (2006-09-15). Why Can't You Just Give Me The Number? (Kindle Locations 1911). Probabilistic. Kindle Edition. )

Launching - Answering the "Persevere or Pivot" question

These pre-launch experiments have a whole range of cost, accuracy, and specificity. On the low end you have informal unstructured interviews. This is great for proving things our early on, and also allowing you to find a better business idea then the one you thought of originally. On the high end of cost and complexity, you have large scale polling that you do right (i.e. random distribution of customers, proper question wording to avoid bias, etc). These aren't going to find you that better business idea, but can provide a very accurate and specific answer to the question you pose. 

At some point though (and this will be very specific to you and your circumstances) you'll need to sit back and decide to stop pre-launch experments. When do you stop? When failing in market will be better then continuing to run experiments. For example, if you have a large existing business (lets say $100M in annual revenue) that you are thinking of disrupting with a new revenue model, you probably want to do the more formal methods since spending $50,000 on a polling firm and taking the time is small relative to the risk. But a 6 person startup with no actual revenue yet? Depends on the size of your market. If you only have 100 customers, you may want to do more upfront work because running experiments on customers can cause them to get grumpy and leave. But if you have a larger target market, it's fine to lose some customers while you work things out. If you are looking for a more decision science based approach, see Chapters 11 and 12 in Patrick Leach (2006-09-15). Why Can't You Just Give Me The Number?. The different market sizes account for the different approaches in the LSM books.

Once you get a product in market, you are still subject to the same overconfidence illusions around forecasting. This is where the second part of the LSM stuff kicks in: Analytics and Rapid iteration.

I'm totally thrilled to watch market after market get disrupted by rapid prototyping. On the hardware side we had FPGA's come along in the 90's that allowed really interesting products to be built without the capital outlay needed for an ASIC. On the SaaS side, AWS/DevOps/Harware as software movement has added nimbless to that field. Outside of computing, the revolution around rapid prototyping, 3D printing, and cheap CNC tools (like CNC plywood routers) has drastically changed things. Even the repatriation of hard goods manufacturing is occuring because it allows businesses to iterate faster (http://www.nytimes.com/2011/10/13/business/smallbusiness/bringing-manufacturing-back-to-the-united-states.html?pagewanted=all ). 

How can overconfidence get you after launch? Go read the opening chapters of "The Startup Owners Manual" to learn about how Webvan's overconfidence caused them to ignore the metrics they were getting and fail big.

The steps at this point are:
1) Ship iteration of business (this includes ad copy, market segment, marketing webite and materials, actual product)
2) Observe behavior using quantitative metrics
3) Use that to drive qualitative discussions with customers
4) Make a hypothesis and modify product/web site/ad copy
5) Repeat

It's easy to get analytics wrong. Eric Ries labeled these 'Vanity Metrics'. These are metrics that are pretty much guaranteed to give you the answer you want (generally up and to the right). But much like qualitative interviews, there is a broad spectrum of accuracy and complexity around implementation. For that first launch you don't need much. Just a retention chart that is keyed off the activity that drives your engine of growth is enough to shake your confidence. You are looking for analytics that help you detect the huge problems in your overconfident assumptions. You aren't at the point where you care about 3% improvement in a number or running a linear regression on your data.

Don't know what metrics to track? Grab a copy of Lean Analytics (http://leananalyticsbook.com/). They breakdown a large number of different business models and what you should be looking at to decide if you should throw in the towel or not.

How quick should your iterations be? As quick as possible without pissing off your customers or partners. For instance, if you are growing rapidly you should iterate quickly (daily even?). As a portion of your customer base, those people irritated by all the change will always be a shrinking proportion of your total base since you are getting new customers at a very fast rate. I personally (overconfidentaly and untested of course) think you need to be willing to lose your early customers and therefore shouldn't worry about them.

One last moment of reflection. These are all really cool tools. But if you've already decided on a course of action, the value of any new information may be zero since it will not change your mind. In that case, just make your decision and go on. I like these tools (in particular stochastic modeling), but in all honesty, if you crack open Bayesian theory and run the numbers, they only help increase your odds of a good outcome by a small amount. This is due to the huge amount of raw luck and chance that exists in the world. A lot of this is outside of our scope of control (I feel for everyone who launched a new business right before the great recession).

So have fun and enjoy yourself!

Books:
"The Lean Startup"
http://theleanstartup.com/ 

"Lean Analytics"
http://leananalyticsbook.com/ 

"Thinking, Fast and Slow"
http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555 

"Naked Statistics" 
http://www.amazon.com/Naked-Statistics-Stripping-Dread-Data/dp/0393071952 

"The Signal and the Noise"
http://www.amazon.com/The-Signal-Noise-Many-Predictions/dp/159420411X 

"Why Can't You Just Give Me The Number? …Guide to using Probabilistic Thinking to Manage Risk and to Make Better Decisions"
http://www.amazon.com/Guide-Probabilistic-Thinking-Decisions-ebook/dp/B0029F2STA 

"The Startup Owners Manual"
http://www.amazon.com/The-Startup-Owners-Manual-Step-By-Step/dp/0984999302 

23 comments:

  1. Nice information you can make this more informative by searching or getting the knowledge about Kaizen Training

    ReplyDelete
  2. This windows shortcut keys are very useful this helps user to avoid the use of mouse and touch pad. windows shortcut keys list Which helps user to improve his/ her speed in his/ her work.

    ReplyDelete
  3. Thank. An interesting point of view.
    Here you can get even more information http://essaylab.co.uk/ will help you write articles, 24 hours a day, which saves you time.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. How Lemeridian funding service  grant me a loan!!!

    Hello everyone, I'm Lea Paige Matteo from Zurich Switzerland and want to use this medium to express gratitude to lemeridian funding service for fulfilling his promise by granting me a loan, I was stuck in a financial situation and needed to refinance and pay my bills as well as start up a Business. I tried seeking for loans from various loan firms both private and corporate organisations but never succeeded and most banks declined my credit request. But as God would have it, I was introduced by a friend named Lisa Rice to Le_meridian funding service and undergone the due process of obtaining a loan from the company, to my greatest surprise within 48hrs just like my friend Lisa, I was also granted a loan of $216,000.00 So my advise to everyone who desires a loan, "if you must contact any firm with reference to securing a loan online with low interest rate of 1.9% and better repayment plans/schedule, please contact Le_meridian funding service. Besides, he doesn't know that am doing this but due to the joy in me, I'm so happy and wish to let people know more about this great company whom truly give out loans, it is my prayer that GOD should bless them more as they put smiles on peoples faces. You can contact them via email on {lfdsloans@lemeridianfds.com Or lfdsloans@outlook.com} or Text through Whatsapp +1-989 394 3740.

    ReplyDelete
  6. Is your cooker Gone bad and you want our service at affordable cost then make a call visit through our website and we will be there in a few minutes to serve you.

    Cooking Range Repair in Dubai

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Very interesting article.
    I once also started writing from small blogs and now I work https://manyessays.com/order-dissertation-online and I am very satisfied.
    I love to write essays, articles, essays and coursework.
    I am constantly learning new information and so I develop.

    ReplyDelete
  9. TVrepair.ae can be repaired and maintained effectively for TV in repair Dubai service and you do not have to buy a new one. We are the TV repair company that offers a cost-effective way. We have experts who can find out why your appliance has failed to perform a function. So don't worry about it. Just connect with us.

    ReplyDelete
  10. Thanks for The Amazing Blog.
    We’re one of The Best Food Business Consultant in Hyderabad.

    ReplyDelete
  11. Eagle Technical services are a highly professional company all over Al Nahda. We can provide you the best service at a lower cost for a House wall painting service, Wall painting service, Interior wall painter, and Wall Painting Al Nahda. Do you want good service? Call me today and get great services.

    ReplyDelete
  12. If you are searching on searching engine how to get free freelance visa in UAE Dubai? Then no worry freelancer visa Dubai is here give you these services. So, just contact us through our website.

    ReplyDelete
  13. If you find it difficult to solve examples, you math games for 4th grade can ask your relatives for help, who will be happy to play fun math with you. Good luck!

    ReplyDelete
  14. Adobe
    As the first Data Science professional in my organization of around 200 people,cctv services Millennium Plaza Hotel Tower, Office: 1504, Sheikh Zayed Road, Dubai, United Arab Emirates.
    Contact number - 045864033
    For help Free Pickup & Delivery

    ReplyDelete
  15. Thanks for sharing the informative content with us. Have you kitchen cabinets repair, cupboards making, Door installation and repair. You can visit for any carpentry Services in Dubai

    ReplyDelete
  16. You shared with us an amazing collection of creative works. Without a doubt, you'll attempt some of your ideas and contact us. curtain rod installation near me

    ReplyDelete
  17. 5 Useful Printer Troubleshooting Tips You Tell Me It's pretty much as you tell me and it's pretty easy to troubleshoot printer problems. I want to thank you for sharing this blog. Call us for More information toll-free number 045864033.

    ReplyDelete
  18. A range of interior design services is offered by the company, including planning, Carpenters Sharjah, and others. The types of buildings designed to inspire a sense of safety and security are the most prevalent architectural styles in cities. For further information, call us at +97145864033.

    ReplyDelete
  19. MacBook Repair Dubai offers MacBook repair and service in Dubai. MacBook repair services include MacBook battery replacement, MacBook hard drive replacement, MacBook RAM replacement, MacBook Air replacement, MacBook pro replacement, and MacBook Air screen replacement. contact us at +97145864033.

    ReplyDelete
  20. Great post. Thanks for sharing such a nice blog. Its helpful for everyone Keep blogging like this. If you are looking for Buy Bulk Gmail Accounts. Our website provides 100% authentic accounts.

    ReplyDelete
  21. Nice Post
    Do you need best specialist Best Physiotherapists Near Me and Best Chiropractors in Jaipur are both healthcare professionals who specialize in the assessment and treatment of musculoskeletal conditions.

    ReplyDelete
  22. Nice Article

    Do you need Physiotherapy in Dwarka? Consult with some of the top physical therapists at a facility in your area run by Prayaas Physiotherapy Clinic in Dwarka.

    ReplyDelete