Better Late Than Never — Learning From Our Data

image

In the course of our first year at Upstart, we made plenty of mistakes – that’s the nature of startup life.  Some mistakes are an inevitable part of trying to do something completely new, but others are avoidable with a little forethought.  I recently realized we had made a mistake of the avoidable type, but I wanted to share what I’ve learned so that other young companies can learn from our experiences.

At Upstart, we are really believe that the key to success for a startup is learning quickly from your experiences and adapting to those learnings.  This concept is best described in Eric Reis’ book The Lean Startup – a definite must-read for entrepreneurs.  Putting this belief into practice means seeing everything we do as an experiment with a hypothesis and a way to test the results.  Of course, key to testing results is data.

If you are really trying to learn from your experiences – data is the lifeblood of your efforts.  In our early days we focused on building dashboards out of our production data.  After a few requests to our Product Manager for data pulls – he built me a quick and dirty dashboard of data.  Of course I built a spreadsheet that could take a copy of that table, apply a whole bunch of formulas, and generate some very useful graphs.  I took a snapshot once a week so we could see historicals and compare progress.  And every few days I sent a new email asking: “can you add X to the dashboard?”

Don’t get me wrong – that data was very valuable and we learned a lot from it.  But there were several flaws in our strategy.  Asking a new question of the data meant asking engineering to run new queries and update the dashboard.  This was usually lower priority than product features and so they were always on the bottom of the request pile.  And as much data as we were capturing in the product – it was inadequate to answer some of the more difficult questions we wanted to ask.

To try and resolve this problem, we recently decided to implement Mixpanel.  We recently finished our implementation and started collecting data and my only response was – “we should have done this a year ago!”  At the time paying for an external solution to this felt excessive and unnecessary – we are after a startup trying to be lean.  But it’s becoming very clear how much easier it will be for us to analyze and understand our data now.  And I can ask as many questions of the data as I want without ever having to talk to an engineer!

In the spirit of learning from mistakes as well as experiments – here are the three things I learned from this experience.

1. Start with a plan.  Our early data strategy was essentially to add some views into our production database – but we never really thought about what data we would need to capture to analyze our key metrics.  Without a plan for capturing the data you want to learn from (versus what’s needed to operate the product), it’s easy to miss critical data.

2. Buy, don’t build.  I honestly wish we had started using Mixpanel from our first launch.  It would have been much easier to implement with so many fewer things happening on the site and we would have gotten into the habit of making sure we were tracking all relevant events and data earlier.  It may feel expensive – but if data is critical to what you’re doing it’s worth investing in the right tools to capture and analyze it – without spending valuable engineering time to build them.

3. Capture everything. As we’ve gone through the process of defining what data to track – we keep expanding the list as we find new questions we’d love to ask.  There’s little downside to tracking more data point – and you’ll be surprised how quickly “we’d never need that” can turn into a critical question from the CEO.  The answer to the question “should we capture X” is pretty much always yes.