Ian Ash, one of the co-founders of Dig, provides his point of view on a trend that will impact research in 2019: the marriage of market research and data science.

The end of market research?

Data science is the latest trend that is being hailed as the end of traditional market research. However, like the hype around social media listening before it, data science has inherent limitations. These limitations will keep data science from fully replacing traditional market research anytime soon.

All approaches have limitations

Data science methods allow you to analyze variables available in your data. However, when you don’t have available observations to draw from, you immediately hit limitations on what you can achieve. Simple examples include when you don’t have observations about a potential innovation or a price-point that has never been executed in-market. Because data science uses real-world data as the key input, it is challenging to make projections for things that don’t exist. Similarly, data science does not help us to understand underlying motivations.

The limitations of market research are well known and frequently cited, including: achieving sufficiently representative samples, accuracy of findings and the assumption that people will actually do what they say they will do. This last limitation has garnered increased attention as the popularity of behavioural economics has grown.

A fusion approach is better

Better insights can be achieved by merging traditional market research and data science methods into holistic approaches. A good example of this is leveraging sales/customer data + research data to refine predictions. These methods can include (but are not limited to):

• Sales and volume prediction models that use behavioural data + market research data through trade-off models (Discrete Choice, MaxDiff and Conjoint) create far more reliable predictions for product innovation than either method on its own.
• Segment prediction models that incorporate behavioural data from data science and attitudes and perceptions from traditional market research to predict segment membership create far more actionable segments than either method on their own.
• Price elasticity models that merge historical in-market volume movements + observed movements from market research data allow for a model that is grounded in reality and experimental to include prices never executed in market.
• Product optimizations that use both revealed preference modeling techniques from data science and choice-based conjoint methods from market research allow us to recommend changes that will fully optimize a product’s feature mix.

Why these approaches work so well

The examples outlined all work because they have the following in-common:

1. Data science on big data sets allows for reliable identification of key variables, relationships and trends.
2. Market research ‘fills in the blanks’ from data that doesn’t yet exist.
3. The market research methods employed lend themselves to techniques than can leverage behavioural data effectively. Behavioural based methods (like trade-offs) often work best, but other methods (like segmentation) can also work well when designed in a way that accounts for appended behavioural data.

Individual level data is better

In our experience, the most robust methods allow for individual-level observations where customers can be both interviewed and observed. Appending individual-level data from a customer database to a well-designed market research study allows us to refine predictions and compare actual results in post-analysis refinement of the model.

Some agencies (including Dig Insights) have already embraced this future-state by building data science and market research cross-functional teams and identifying the methods that best combine both. The trend of fusing DS + MR will accelerate as clients find more and more that data science alone can’t replace market research and that market research alone lacks the robustness of data science approaches.

Market conditions are driving this trend

Further accelerating this trend will include the growth of Mobile-First Market Research designs and dashboarding technologies/designs that can better visualize the relationships between market research and behavioral datasets.

The agencies that will own the future are those that realize it isn’t ‘us OR them’, it’s actually ‘us AND them’. Data science and market research are complimentary approaches to finding better insights that take the best from both and overcome the limitations of each.