Multivariate Testing for the Resource Constrained

I posted nearly a year ago about how multivariate testing is a flawed method when you use a “recipe” approach. And I haven’t written much about multivariate testing (MVT) on my blog, except to get tactical and talk about how I’d use MVT to experiment with testimonials.

In the spirit of fairness, there are plenty of ways to leverage MVT for business benefit, and plenty of testing scenarios that call for multivariate experiments. One such use case occurred to me when working with a client that was what I call “resource-constrained.”

This client was really smart about online marketing and testing, but their design/development resources were practically non-existent. They existed, but they were never available to build test variations.

This seems to happen when the Marketing organization doesn’t have dedicated front-end development resources. When front-end development resources belong to the IT or Product organizations, they are seldom available to help with Optimization. Too bad, but I digress…

Because this client wasn’t able to build page variations for A/B/n split testing, we fell back on multivariate tests because we could make easier page changes, and still be testing in an iterative fashion.

Based on working through these tests with the client, I’ve come up with my top 5 elements to use in MVT testing if you don’t have front-end development resources available to you.

1) Headlines

This should go without saying, but headlines are fantastic elements to test in general. In MVT testing, it’s simple to write multiple headlines and try them with different page combinations. In testing tools like Optimizely or Visual Website Optimizer, or any testing tool that takes a WYSIWYG approach, you should be able to just click and type!

When thinking about headlines in a MVT test, let your creativity (or your copywriter) loose. A really simple approach you can take is writing versions of your main headline that are logic/evidence based (Our widget can withstand temperatures of 345° Farhenheit), and testing against versions that are emotion based (Our widget’s quality is there for one reason – to protect your family.)

2) Form/element alignment

This one really surprised me the first few times I tested it because it had more impact on conversion rate than I expected. Without redesigning your page layout (which requires front-end development resources that you don’t have), you can often simply move page elements from left to right to center.

For example, if your landing page has a lead capture form, try versions with the form aligned to the left vs. versions with the form aligned to the right vs. centered. Depending on the mix of browser types (tablet, mobile, etc.), you may find that form-left is a winner because smaller screens don’t even display your form!

Another test I’ve run is on eCommerce product pages where the product image is on one side, and the product “action box” (price, call to action, quantity, etc.) is on the other side. I’ve gotten pretty big lift by testing whether the product image should be left-aligned or right-aligned above the fold.

Again, this should be achievable in many modern testing tools without taxing your scarce web resources.

3) Call to action

Call to action (buttons, especially) are easy to throw into multivariate tests. If you have a designer, they can create different button designs that you can upload into your testing tool. If your CTA buttons are rendered via CSS, you can probably tweak the button label and color without outside help.

call to action buttonI believe CTA label (e.g. does it say “Act Now,” or “ACT FAST,” or “Buy Now »”?) gives the best ROI for testing your calls to action in a multivariate test because you can create variations with very little effort, yet they often have a surprising impact on your target audience’s willingness to take the action your desire.

4) Value Proposition Statement

Value Proposition Statement is another great page element to add into multivariate tests. It’s also something you can test without those scant front-end development resources. A value proposition statement is the concise statement about why prospects should do business with you instead of your competition.

uvp on homepage screenshotBecause it’s just text, or sometimes a fancier graphical treatment of text, value proposition statements are relatively easy to test. And because they answer a crucial question your prospects have, they often add a lot of boost to your conversion rate!

MarketingExperiments has an impressive body of research on this topic, and I’ve had some success with testing value proposition, too.

5) Credibility Indicators

Credibility indicators are graphical elements that add credibility (via proximity and relevance) to your page elements. Examples are security badges or simple lock icons, industry award logos, a marquis client list, a phone number prominently displayed, and more.

grayscale client logosThese elements often increase conversion rate because the added credibility they bring “tips the scales” in your prospect’s mind, and helps her feel confident taking the desired action.

Because they are often simple text or graphics, they are great for adding into multivariate tests, and seldom require development resources to implement. You can test:

  • combinations of your page that do/don’t have a certain credibility indicator
  • different sizes of indicators (to see how small is too small to have impact, and how big is so big that it distracts)
  • “grayscale” images vs. full color images
  • the order of graphics from left to right
  • the location or proximity to your point of action

Try These…But Not All At Once

Remember: Just because you can test all 5 elements pretty easily, doesn’t mean you should test them all at once. For every new element you put in a test with combinations, the number of variables jumps, and you can end up having a test that takes too long to reach conclusion.

Test a handful of these elements, with a handful of variations for each. If you’ve got a lot of traffic and decent conversion rate, try 3x3x3=27 combinations, for example. Just think about what assets you have already built, which ones you can modify on your page, and which ones take the least resources to test. And don’t forget to prune as you test!

That way, you’ll be able to design a test that gets impressive results, even without the help of internal (or external) resources.

What elements do you like to test in your multivariate experiments?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>