Creating a world-class bourbon is not magic. All it takes is disciplined cycles of creation, research, learning, and recreation. This approach was refined and validated by our sister company, the Eureka! Ranch, over the past 40 years working with tens of thousands of companies, from multinationals to craft startups. It’s also how we win awards with our craft distillery, Brain Brew Custom WHISKeY in Newtown, Ohio. No matter what size distillery you run, you can use good research practices to improve your products’ taste and sales.
Proof that Research-Driven Development Works
Quantitative research has directly helped us win competitions. Eighteen months after Brain Brew opened, we won two of six Double Gold medals at the North American Bourbon Championship. We’ve since won many Double Golds in San Francisco, and scores of 92, 95, and 97 from the Ultimate Spirits Challenge.
Research also helps us win in the marketplace. We used this system to create Noble Oak Double Oak Bourbon, the fastest-growing super premium bourbon in the U.S., according to our estimates, and Dexter Three Wood, which was named one of the 12 best bourbons by Ultimate Spirits.
It Starts with Meaningful Uniqueness
After conducting more than 40 years of research, we have found that the metric most predictive of success for new products is meaningful uniqueness. It correlates at the 95% confidence level with awareness, trial, repeat rate, and total sales. In spirits, this means that to win in today’s crowded marketplace you need your products to both taste amazing (meaningful) and offer customers something they can’t get anywhere else (uniqueness).
We quantify meaningful uniqueness by asking customers on a 0 to 10 scale how likely they are to buy the product and how new and different they perceive it to be. A weighted score over 5 works well for large companies with lots of marketing money. However, our studies find that for craft distilleries, a score over 7 triples the odds of igniting word-of-mouth awareness and thus multiplying the impact of more limited marketing budgets.
Product Testing Options
There are two product testing options: single product and paired comparison.
Single product testing has consumers rate each product in the absolute on various parameters, including purchasing intent, whether a product is new and different, and sensory traits. These results can be compared against databases (Eureka! Ranch has the largest whiskey and new product databases in the world) or benchmarks. The results from single product testing can be used to create risk-adjusted five-year sales forecasts valuable when pitching to investors or banks for funding by using market research modeling techniques like Fourt-Woodlock trial and repeat models, Bass diffusion models, and Monte Carlo simulations.
Paired comparison blind testing involves consumers tasting two unidentified products blind and comparing them head to head. It’s most commonly executed by pairing a new, in-development product with an existing marketed product. Paired comparison test results are an invaluable tool when developing a new spirit, providing a “reality check” on the status, strengths, and weaknesses of your proposed product. Learnings can be used to improve your product so it can be tested again. This process is repeated until you meet your success standard.
Designing a Paired Comparison Test
The testing process begins with deciding which product to test against. In early testing, we keep it simple by testing two of our product samples against each other. This process is repeated until we get a product we believe in. At this point we move to testing our product versus a well-known brand that costs more than we anticipate selling ours for. Our goal is to beat them 60/40 in blind tasting. By testing the products without brand identification, we learn how good our product is in the absolute without the influence of marketing or reputation.
You can pick whatever brand you wish. You can test versus a local competitor whom you respect or a national brand that is well-respected. Because our distillery is focused on higher-end spirits, when testing a bourbon we commonly use Blanton’s. When testing a single malt we often use The Macallan 12. With rye, we often use Sazerac Rye. I realize that testing against high-end products makes testing more expensive, but it’s well worth it when you show the data to distributors or buyers.
How to Do a Paired Comparison Test: The Best Option
The most effective way to execute a paired comparison test is to use one of the commercial rapid research systems. These systems make digital sample data collection fast and easy, as consumers simply touch buttons on a tablet. Questions often include: whiskey drinking habits and practices, purchase intent, whether a product seems new and different, and a collection of sensory statements that diagnose strengths and weaknesses of your product. Examples include: richness, easy drinking, complexity, tastes more expensive, tastes like it’s been aged longer, aftertaste, bite and burn, and so forth.
Most importantly, digital systems automate analysis and statistical testing, making results much more credible with potential investors, distributors, and retailers. They can also instantly visualize your results in pie charts.
How to Do a Paired Comparison Test: The Fast and Free Option
I recognize that quantitative product testing is foreign to most craft distilleries. To that end, our Brain Brew Crew worked with Greg Lemon, who leads the Eureka! Ranch Research Division, to create a fast and easy way for craft distilleries to do paired comparison testing. The method can be executed in your tasting room over a weekend, or at a local bar or restaurant (depending on your state alcohol laws) with virtually no equipment.
The process involves handing consumers two small samples. We usually pour about 1/2 ounce of each sample into a disposable or glass shot glass. We hand the first sample to the customer and say, “This is the left hand sample.” We then do the same with the other sample, to the right hand. We usually offer each sample neat, with a small glass of ice for consumers to add to the whiskey if they wish.
After they’ve tasted both, we ask them which product they would be most likely to buy: left hand, right hand, or a tie. We tally their answer as a hash mark on a clipboard we carry while doing the test. We go on to ask which is more new and different: left hand, right hand, or tie. Then, we ask a collection of sensory questions: which is smoother, which tastes more expensive, which has the most complexity, etc., marking all responses on the clipboard. Lastly, we ask them for any advice they would give us on the two products. We write down what they say as precisely as we can on our clipboard.
When we’ve gotten to 30 people, we stop and tabulate the results. Each choice of right hand or left hand earns that sample one point. Ties are scored 1/2 point each. We then take the total for right hand and left hand and divide each by the total sample number of 30. Statistics indicate that with a test sample of 30, you need to win 68% to 32% to be significant at the 95% confidence level. Said another way, you need roughly 20 out of 30 people to pick your product for a significant win.
To assess meaningful uniqueness, we look for a win on both purchase intent and new and different. We then dig into the sensory traits to understand our strengths and weaknesses versus the competitor. Early in the development process we start with small base sizes of 30 to make testing fast and easy. As we get further along in the development process, we move to larger base sizes of 100 to 200 to validate our results.
Learning Cycles Are the Pathway to World Class
Following each test, we review the data and make the following decisions: 1) Should we go ahead with this product? 2) Should we change it, and if so, how should we change it? And 3) Should we archive this product?
In Ohio, we can distribute directly to bars. This makes testing very easy. When we are deep into developing a new product or brand, we’ll do two or three paired comparison tests three or four nights a week. The data has an amazing ability to accelerate our team’s willingness to change a product. Nothing stops mindless debate more than learning that your product just lost 70/30!
When we get to a “wow” product, we then move to paired comparison testing of packaging and brand positioning. Our goal is to increase consumer appeal when we add our branding to our product. Importantly, when doing branding and packaging testing, we show consumers only the front and back of the package. We do this because, when you are on the shelf, that is all 95% of consumers will know about your brand.
When Investing My Money I Want to Know the Truth
Creating a spirit brand and taking it to market requires the investment of time, energy, and money. At Brain Brew Distillery, testing is not optional. It took nine months to validate our first products. Today, our systems for development and testing are more refined. It now takes us about two months from raw idea to final product and packaging.
Testing is an iterative process; the best way to learn how to do it is to do it a lot, so get out there and start testing!
