While case studies about conversion rate lift and increases in revenue from testing get a majority of the “press” these days, there is something far less sexy and far more important to be thinking about: The order in which you execute your test ideas.

I know, that’s way more boring than, Learn how company X increased their sales by 4,000% with one simple change!!!!! I don’t blame anyone for wanting to share exciting test results, but what if you burn an hour on an webinar and find out you’ve already made the change they’re talking about? Or, you’ve already tested that change and it didn’t make a difference on your marketing?

That’s why we’re going to talk about something “boring” that is guaranteed to help you get results over a longer time frame. Prioritization – the way you apply resources to Optimization work to get the best ROI.

What the Road Map Document Is

Commonly referred to as an Optimization “road map,” your prioritization documentation is going to be a crucial focal point as your testing practice matures beyond those first few experiments. The most common format is a matrix where tests are rows, and columns are metadata about that row’s test.

The road map document usually tracks an experiment from its inception (idea formed), through development, and until final results (hypothesis proven/dis-proven). It can be an archive of past results and insights that can inform the testing program and remind folks of what’s already been tested.

When tests are just ideas, the road map helps prioritize which of them to work on first. When tests are in development, the road map can help track the status of Optimization efforts, which is crucial when there are multiple tests in development at the same time. When tests are finished, the road map is a quick reference on what pages have been tested, what the experiment findings were, and the business impact of the test results.

Ways to Prioritize Tests

Once you’ve captured a group of test ideas in your road map, it can become overwhelming due to the number of choices available. And while it’s tempting to simply “work from top to bottom,” you will get a nagging feeling that you’re not impacting your business as you should be due to lack of clarity about priority.

Also, an un-prioritized road map will invite arguments over which to test first because stakeholders with the loudest voices, or the highest paid person’s opinion (HiPPO) will win out instead of logic.

To avoid subjective decision-making, I started prioritizing road maps with simple rating systems like “high, medium, low” business impact. This evolved into a 2-part scoring system that attempted to balance business impact with ease of execution.

Now, my client road maps have evolved to use (still pretty simple) weighted scoring formulas involving multiple attributes like resource use, technical complexity, traffic volume, “political” value, potential for broad insights, risk, and more.

Example 1

Let’s look at a basic example [click to enlarge]:

In this example, 6 test ideas are captured and assigned scaled scores on attributes of:

1. Estimated Test Duration – a short test duration will be scored high; a long test duration will be scored low. This scoring should be based on prior estimation calculations that take into account pageviews/day, historical conversion rate, estimated conversion rate lift, and number of experiments
2. Ease of execution – An easy test scores high; a complex test scores low. #yougetthepicture
3. Business Impact – This has to do with how an improvement in a KPI could impact top-line or bottom-line of the company. A shopping cart test would score high and a homepage test might score lower because the “line of sight” to revenue isn’t as clear on the average homepage.
4. Cost of traffic – This has to do with the predominant traffic source. If you traffic is “owned,” meaning people coming to your homepage as Direct because you’re a household name, then the cost of traffic scores lower. If you’re a start-up in a competitive space buying PPC ads, your cost of traffic score will be high.
5. Requires Legal Approval – This is fictional but assumes some sort of operational constraint. The scale is a simpler binomial 0 or 1. Needing approval is a hassle, so a score of 1 is subtracted from the total score.

The formula, then, is :

Estimated Test Duration + Ease of execution + Business Impact + Cost of traffic – Requires Legal Approval = Priority Score

By using the above formula, you’ll find (in a very objective way) that your campaign landing pages are the best pages to test even though you have to go through Legal hoops. The reason is mainly due to traffic volume, simplicity of tests, and the fact that you’re paying for that traffic.

Example 2

Let’s now tweak the example road map [click to enlarge] to account for a different business environment and priorities. Let’s assume that going through Legal approval is a HUGE hassle that really slows your testing program down. Your whole Marketing organization really wants to avoid this slowdown, so you can weight your formula to account for this

Estimated Test Duration + Ease of execution + Business Impact + Cost of traffic – 10*Requires Legal Approval = Priority Score

By applying this new, weighted formula, and of course sorting by Priority Score “descending,” you now see that your campaign landing pages are lower priority because they slow your Optimization program down to a crawl. The business needs to see progress and results, not email threads about indemnity, so you’ll de-prioritize those campaign landing pages.

Example 3

In this last example [click to enlarge], we’ll assume that Legal approval is slight hassle, paid traffic is under-performing and killing your Marketing budget, and you’re behind in your quarterly quota of experiments run, necessitating that you speed up your testing cadence. Isn’t Digital Marketing fun? 😉

4*Estimated Test Duration + 2*Ease of execution + Business Impact + 3*Cost of traffic – 4*Requires Legal Approval = Priority Score

This final weighting really dials the prioritization into specific needs. Fast-running tests will score high, simpler tests will float to the top, paid search remains a priority, and the slowdown of getting Legal approval is also accounted for.

Now, you see that 2 of your 3 campaign landing pages are still a priority due to the weighting of paid search traffic, and the slower-running shopping cart test may have to wait until next quarter. This is OK because it’s exactly what the business context is demanding of you as an Optimization Manager.

Of course, the weighting and attributes in this post are just examples, and you’ll need to listen to what your Marketing Organization is asking of the testing program in order to have the right prioritization and weighting in place.

It’s a bit of extra work, but ultimately worth it when you can quell internal debates about which tests are most important at any given time. Making prioritization objective gets all stakeholders on the same page about what tests you’ll be working on soonest, and sets more realistic expectations.

Set up a spreadsheet of your own with specific attributes, and play around with a weighted formula. I’d love to hear how it goes!