Small Data beats BIG Data when Innovating

The infatuation with big data and machine learning has clouded critical knowledge that can only be discovered through “Small Data”.

While doing work for a manufacturing IoT start up, that captured and used a narrow set of machine generated data, I became curious about the concept of little or small data. I thought my search would provide wisdom on capturing small quantities of narrowly defined data, what I discovered was much more interesting.

Upon researching this subject I found some early work on the subject (among others see: Mark Bonchek’s HBR piece “Little Data Makes Big Data More Powerful” ) But the material was thin and it has primarily focused on personal feedback loop tools like using your FitBit to track and improve activities and calorie count. There was nothing obvious on its applicability to business until I stumbled across a very intriguing book by Martin Lindstrom titled “Small Data”, a book that was nothing I expected when I first opened it up.

Let me jump ahead (hopefully this will make sense soon by the end of the blog) to a book I just finished by Clayton Christensen (of Innovator’s Dilemma fame), Competing Against Luck. This is a great book on innovation I highly recommend, and worth additional blogs, but there is one concept that was heavily promoted in the book that echoed in my mind, because I knew I had read about it before…In Small Data.

In Competing Against Luck, Christensen and his co-authors caution against depending on big data for innovation development. Big data identifies and provides analysis on similarities based on statistical occurrences in your target environment. Thanks to big data Tesla can know that individuals who live in wealthy communities, shop at REI, donate to the Sierra Club are very likely to be receptive prospects for their car. But this information gives Tesla little insight into what that individual wants or dislikes in their car, or – since the books was to addressing how to innovate, Christensen makes the case that big data does not help discover these needs, or the “Job the product is being hired to do”. Big data also mis-direct product organization to create a single product that targets a mythical average customer, presumably the largest potential opportunity for their product – when in reality there is no one single person who is the average customer. [The book retells the story of the US AirCore’s effort to improve Cockpits by designing it for the average pilot which resulted in it fitting no one]. Only by having detailed investigations to the problems people are trying to solve and understanding the intimate and emotional reasons behind those needs is an organization able to control and effective deliver innovations.

Although Christensen never used the term “Small Data” it was clear he and Martin Lindstrom were speaking of the same point. Reading Competing Against Luck brought me back to this insightful little book by Lindstrom. Throughout “Small Data” Lindstrom, an internationally recognized branding expert, described how consumers desires are hidden in the details of small data that can only be captured when delving intimately into the personal world of the customer and uncovering the subtle even subconscious signals that were hidden there. Information that is captured by observing what magnets are on Russian family’s refrigerators, or how Americans behave in an elevator hold the clues of what will matter to a community of customers. He illustrates this effectively in his story of how Lego was initially misguided by big data analysis, nearly to their ruin, and how an 11 year old’s worn pair of sneakers lead to the reversal of strategy that has made Lego the largest toy manufacture in the world. Lindstrom also points out the dangers we face in relying on big data and missing the important details gained from an intimate investigation of people and their lives.

Neither Christensen or Lindstrom are suggesting big data analysis should be eliminated, but they are making a strong case that it is not the tool to drive innovation. Great innovative product resulted from solving a need that customers have, needs that are usually connected to deep desires and emotions. Discovering these emotional connections to a solution is the road to innovation. The techniques to discover this is not easy (and worth another blog and it is also raised in another great book “The Four Steps to the Epiphany” ) But it is not through Big Data.

One additional thought I had on this subject was inspired by Lindstrom’s warning. Can the influence and attraction of big data have a collective dampening effect on Innovation throughout society? If those in a position of investing in new ideas, or approving new projects are committed to following the guidance of big data we face the danger of great innovation being dismissed. I hope all heed the wisdom and advice Christensen and Lindstrom put forth in Competing Against Luck and Small Data.

Leave a Reply

Your email address will not be published. Required fields are marked *