OK, settle down. I’m not saying that Multivariate testing is “flawed.” It’s a perfectly useful way to test. However, I recommend against looking for the right “recipe” before you know what your audience is hungry for!

By “recipe,” I’m referring to the popular approach to multivariate testing where a page is first broken into its discrete components (e.g. navigation, call to action, image, headline, etc.). Then, testers create different versions of all the components identified. Finally, all the variations are thrown into a multivariate experiment that runs until the best combination of variations of page components is accidentally found.

To illustrate, I’ll tell you a true story that a client told me. This client subscribed to a testing software, and also paid the software company’s “professional services” team to build and execute some tests. The vendor (who shall remain nameless) ran a few tests, and got the client some conversion rate lift.

The client’s complaint, however, was that a few winning tests took 6 months to run (!) and in the end didn’t deliver any business insights about why the winning variations performed better.

One of their tests I audited was a multivariate test run on the client’s product page template. The “recipe” approach was taken, and the page was broken into the following components:

1. page background
2. incentives
3. trust badges
4. miscellaneous messaging
5. main navigation
6. page layout

Each of these 6 components had 1 Variation to challenge the Control. That means there were 12 total Variations including the Control in the experiment.

47 days later, the test was ended due to being inconclusive. OUCH! The winning variation was at a confidence level of only 68%, so the experiment was nowhere near being finished. Worse yet, the winning variation, had it been statistically confident (e.g. 95% or higher), would have only given the client a 6% conversion rate lift. And because there was no stated hypothesis, there would have been no customer insight, except that the audience preferred a random combination of page elements. How exactly do you leverage that insight to make more money? 🙁

Frustrated but inspired, I ran a follow-up multivariate test on the same product page template that reached statistical confidence in 9 days and reported a 40% conversion rate lift! How did I get such a nice lift in a much shorter time frame, while still using a multivariate approach? By NOT using the “recipe” approach.

The vendor had cut the page up into ingredients, and tried substituting different ingredients (e.g. a white page background instead of a gray page background) hoping to find a winning recipe. That is a flawed approach because it’s so random. It isn’t driven by any analysis or hypothesis.

For my test, I picked two page components of the vendor’s 6 that I believed would have the most impact on the performance of the page for that client’s traffic, on that type of page, at that point in the sales funnel. I chose ‘page layout’ and ‘trust badges,’ and ran 3 challenger combinations against the Control. My point is that I analyzed and theorized my way to a simpler test that got results, got them fast, AND taught the client and I something about the site’s target audience. For example, one useful insight we got was that the audience responded to security badges in the middle of the sales funnel, which isn’t always the case.

I’m going to leave you with a nice quote that sums up why I believe the recipe multivariate approach is flawed, and why doing a bit of up front thinking can make your multivariate tests much more effective.

Optimization starts with in-head factors, not on-page factors.

– Linda Bustos

Hypothesize about what’s going on in the heads of your prospects BEFORE you start tinkering with minor on-page factors. Or, to finish with the culinary metaphor, take an educated guess at what your prospects are hungry for before you craft the perfect recipe. 🙂