Data and Insight. When Bad Data is Worse Than No Data. A Straight Talk With Lean Startup Expert Laura Klein

If you have less than 20 users, it may be a sign that you need to continue market research to ensure your product and intended market fit together. Market research is easy, you think to yourself. So you grab your clipboard and stack of product surveys, and hit the town’s coffee shops to gather some good data.NEWSFLASH: Gathering useful data is not as simple as talking to customers to figure out what they want.Laura Klein, author of UX for Lean Startups, taught me how to properly conduct user research when I met her at the Lean Startup Conference in San Francisco. Klein helps product teams learn from their users and turn those insights into products people love and use.With the Bay Bridge and Alcatraz Island in clear view, Laura gave me a breakdown of quantitative versus qualitative data and how it relates to Lean Startup.She believes that in the process of gathering data, it is crucial for startups to know the simple truth: no data can sometimes be better than bad data.She explained it like this….Bad data, which could be interviewing the wrong market segment, not asking the right questions, or trying to get quantitative results without enough users, will make you believe you are on the right track when you’re not (which is bad).No data, which means you are still working under your assumptions, keeps you open minded and will prevent you from going off in one specific direction while thinking that you know everything (not as bad). As long as you’re aware of your assumptions and are working to correct them, this is less dangerous than letting bad data trick you into thinking you know what you’re doing.In order to understand how the collection of data impacts entrepreneurs who are trying to develop their product or service, I asked Laura to get into more detail on the different types of data that are usually collected, and how the process impacts business decisions.User research data can be divided into two general categories: quantitative and qualitative. Let’s take a deeper look at what each entails and what can be done to gain customer insight.Quantitative Data is information about what is happening; it’s information that can be measured and written down with numbers. Entrepreneurs can gather this type of data when they have a decent number of users to track with analytics. Most startups with a brand-new product, however, are nowhere near enough users to necessitate data-based analytics.For startups with thousands of users, thousands who can therefore provide statistically significant amount of quantitative data, tactics like A/B Testing and data analytics for such large data sets are great.With large data sets, come a lot of information. The data provides information such as when they clicked this, where they clicked after that, and when they click after that. Sure, you might have all this data, but you might miss why people are doing what they are doing.Quantitative data can help us see what they are doing, but we also need ask why they would do that.In other words, you can say “this number is going up,” or “this number is going down.” Next, you can predict that it’s going up or down because of x, y, and z reasons. You can even change that element and see what happens to the data. Yes, that number might change, but you still don’t actually know why it went up or down in the first place, nor after the change.Qualitative Data is typically descriptive data, and as such is harder to analyze than quantitative data. Qualitative research is useful for studies at the individual level, as well as finding out, in depth, the ways in which people think or feel (e.g. ethnography or customer development interviews).When you have less than enough users for quantitative research to be statistically significant (which typically requires thousands of users) and you can’t muster enough data to find insights, you primarily have to rely on qualitative research.This approach would focus more on gaining customer opinions and feelings about your product or service. Of course, you should still use qualitative research once you’re also collecting quantitative data, but before you have quantitative, you have to rely entirely on qualitative.When you perform qualitative research, approach it more scientifically than many people do. Scientific methods don’t only pertain to calculations and statistical significance; they’re also important in the documentation of the process. Qualitative research is too often thought as something done casually, like “Let’s go out and chat to some people.”Instead, write down ahead of time what your hypotheses are, and stick it up on the wall. Jot down your beliefs, then follow it up with detailed observations of what you saw, heard, and experienced from those you interacted with. Although qualitative research doesn’t deal with numbers, treat it with the same scientific mindset as you would quantitative research.So, it’s true that you can only really get useful quantitative data when you have a decent number of users. However, this quantitative data alone is not enough to understand the WHY. In other words, you can only use quantitative data when you have enough users, but even then, there is still value in gathering qualitative data. Therefore, ideally, when you have enough users, you should use both qualitative and quantitative.Do you have any good stories of trying to collect user data? Share in the comments!