In the innovation testing industry, there is debate around how to show stimuli – is it better to test your ideas as written text or to develop visuals?

The argument for testing visuals:

• Visuals are more engaging and improve the respondent experience.
• If the visuals are launch ready, they are a closer representation of what consumers will see in market.
• Visuals can help to convey a truly new idea for which respondents may not have a mental reference.

The argument against testing visuals:

• Developing visuals adds significant time and cost, often requiring resources from cross-functional teams or outside agencies.
• The process of developing launch-ready visuals forces teams to redirect their energies from ideation to execution.
• Draft visuals may misrepresent the idea, driving stronger or weaker results vs. what we will see with the market-ready visuals.

This paper reports the results of a parallel test where the same ideas were tested as text only vs. tested with packaging visuals.

Methodology

The research was fielded with 600 people from the UK who consumed a chocolate bar in the past 4 weeks. The exercise was framed as testing chocolate bar flavors. We split the sample into two demographically balanced groups where the only difference was the stimuli.

Calculating Idea Score

To understand the performance of the flavors across the two studies, we used our Idea Score which is a composite metric based on Interest and Commitment scores:

Interest Score – Participants were exposed to 14 chocolate bar flavors in the Upsiide platform one at a time in randomized order. Interest is the percentage of people who swiped right or clicked on each product, indicating liking.

Commitment Score – When people like two products, they are shown head-to-head. Commitment is the percentage of times each product won this tradeoff.

The Idea Score has been calibrated using a Hierarchical Bayesian (HB) Linear Model to optimally predict in-market sales.

Click here to experience the chocolate bar study as a respondent.

Click here to experience the results dashboards for the chocolate bar study.

The results

Consistent overall Results (Fig. 1)

The Idea Scores are highly correlated (r=0.80) across the two groups; the top 4 chocolate bar flavors are consistent across both groups: Cookie Dough, Nutella, Toffee Crunch, and Orange. The lowest performing chocolate bar ideas are also consistent across the two groups.

The role of visuals in communicating unfamiliar concepts (Fig. 1)

The one key difference that emerged was a mid-tier flavor, Smores, which performed significantly better when presented with packaging visuals vs. text only. Our hypothesis is that awareness of smores is quite low in the UK. In this case, the image helped consumers to understand the flavor.

When smores is removed from the data set, the overall correlation improves to r=0.90.

The risk of visuals miscommunicating familiar concepts (Fig. 2)

We also see that the fruit flavors consistently perform better when tested as text only vs. packaging visuals (though the ranking remains consistent). Our theory here is that the tested graphics did not effectively communicate the flavor appeal of chocolate / fruit combinations. For example, the Fruity flavor has the largest gap in idea scores (+24 higher in the text-only test) between the two groups. While we can’t know why the packaging visuals underperformed the text-only representation, a viable hypothesis is that the packaging visuals placed too much focus on the fruit and not enough focus on the chocolate.

Conclusions

Testing text-only ideas vs. ideas supported by visuals yields similar results. This is great news for teams who want to make their innovation process more agile. It is usually possible to forgo visuals when testing innovation. In addition, draft visuals may undersell ideas, resulting in poor performance and uncertainty about how to improve the ideas.

This may be especially true if you are testing draft visuals vs. in-market products that have fully developed visual languages. In this case, it’s best to represent your potential innovations and in-market products with text-only descriptions. The exception to this is when testing truly new ideas with low familiarity. Here visuals may help consumers to understand what you are offering.

Our general recommendation is to test text descriptions if you don’t have market-ready visuals. A separate study can be used to test package designs for the strongest ideas.