Multivariate Testing for the Resource Constrained

I posted nearly a year ago about how multivariate testing is a flawed method when you use a “recipe” approach. And I haven’t written much about multivariate testing (MVT) on my blog, except to get tactical and talk about how I’d use MVT to experiment with testimonials.

In the spirit of fairness, there are plenty of ways to leverage MVT for business benefit, and plenty of testing scenarios that call for multivariate experiments. One such use case occurred to me when working with a client that was what I call “resource-constrained.” Continue reading

Why The “Recipe” Multivariate Test Approach is Flawed

chef with recipe with red x overlayOK, settle down. I’m not saying that Multivariate testing is “flawed.” It’s a perfectly useful way to test. However, I recommend against looking for the right “recipe” before you know what your audience is hungry for!

By “recipe,” I’m referring to the popular approach to multivariate testing where a page is first broken into its discrete components (e.g. navigation, call to action, image, headline, etc.). Then, testers create different versions of all the components identified. Finally, all the variations are thrown into a multivariate experiment that runs until the best combination of variations of page components is accidentally found.

To illustrate, I’ll tell you a true story that a client told me. Continue reading

How To Optimize Customer Testimonials Part 2: Designing A Test

In my Part 1 post about optimizing customer testimonials, I laid out my CAPP framework for thinking about the various aspects of a testimonial and the variables you might improve on and/or test.

In this post, I want to illustrate how I would apply this framework using a multivariate test to see which optimized combination of factors would increase conversions the most. Continue reading