OK, settle down. I’m not saying that Multivariate testing is “flawed.” It’s a perfectly useful way to test. However, I recommend against looking for the right “recipe” before you know what your audience is hungry for!

By “recipe,” I’m referring to the popular approach to multivariate testing where a page is first broken into its discrete components (e.g. navigation, call to action, image, headline, etc.). Then, testers create different versions of all the components identified. Finally, all the variations are thrown into a multivariate experiment that runs until the best combination of variations of page components is accidentally found.

To illustrate, I’ll tell you a true story that a client told me. Continue reading →