As the testing and Conversion Optimization of websites becomes more widely accepted and practiced, the technologies that enable testing (A/B and Multivariate testing software) are getting better and cheaper. There is good competition in the testing tool market, so it’s a great time to be a company looking to start a formal testing and Optimization program.

With all the choices for testing technology, a common mistake is for companies to simply buy/license a testing tool and assume that they can “handle” the needs of testing and be successful.

Many companies that go down this road fail after a short time due to the difficulties of setting up valid tests, getting measurable results, and coming up with new testing ideas. Even companies that are somewhat successful at taking on testing without any outside help tend to “plateau” after 6-12 months and aren’t able to get sustainably positive test results.

While I can understand a CMO’s desire to reap the touted rewards of Optimization “on the cheap,” this post will explain why you need staffed Optimization expertise to be successful in a formal testing program.

Whether the expertise is “in house” or outside consulting, a marketing organization that wants to be successful needs to invest in people skills as much as they need to invest in the right testing platform. Avinash Kaushik has argued in his books on Web Analytics that a company should spend 10% of its budget on tools, and 90% on people. I believe a similar approach makes sense in Optimization!

To aid my argument, I will discuss 8 specific values that an Optimization Expert will bring to your testing efforts to enable success. [Disclaimer: I am one such “Optimization Expert,” working as a consultant for a testing software vendor. However, this is my personal blog, and this is NOT a plug or pitch!].

1. Understanding of Business – An Optimization expert will gain a solid level of understanding of your business over time. The depth of knowledge of your business will vary between an “in house” expert and a consultant, but ultimately both will need to know your business in order to optimize it. This is of obvious value. The more people on your team that understand your business, the more innovative ideas you’ll receive on how to improve the bottom line.
2. Education – A good Optimization expert will educate your marketing organization, and perhaps your whole company, on the ins and outs of testing. He or she will help you understand what can be tested, what can’t be tested, how long tests take, why some tests are better than others, risks, and more.
3. Testing Strategy – An Optimization expert will lead the crafting of your company’s testing strategy. This is crucial if you want to leverage testing as a long-term tactic for achieving business goals. This expertise will often take the tangible form of an Optimization “road map,” which is something I plan to post about soon. The strategy road map will help guide the organization through many tests to make sure everyone is on-strategy, knows the priorities, and that testing resources are well utilized.
4. Test Design – Your Optimization expert of choice will bring their expertise in test design to ensure that experiments are structured and instrumented to run quickly, map to business goals, and provide clean, trust-able results. There’s a lot more to this; probably an entire post. The proof of their expertise will be evident in their creation of Test Plan documents; arguably the most important of all the documents used in Optimization.
5. Test Asset Guidance – Your expert will provide tangible value when they help guide the creation of necessary test assets. Most tests require some new content, whether it’s layouts, images, videos, copy, or user experiences. Designers, copywriters, and other practitioners will be the ones creating this new test content, but the Optimization expert should drive their creativity in a direction that supports the hypothesis, will get bold results, etc.
6. Test Platform Expertise – We made it all the way to #6 without mentioning technology (!), which again speaks to the fact that expertise (not just a tool license) is needed to get the most of your testing technology investment. An Optimization expert may not be technical enough to understand the testing platform at the “code level,” but they need to have what I call an ‘Architectural Understanding’ of how the platform functions. Also, some level of understanding of the math behind the platform is pretty important.
7. Test Analysis & Interpretation – The results of even the most successful test are pretty useless without the expertise to interpret the results and make recommendations based on the collected data. Without an Optimization expert on your team, you won’t be very good at answering the crucial “So, what do we do next?” question. If you’re evaluating an expert, ask to see examples of their test reports and analyses.
8. Test Evangelization – Because the practice of Optimization is so relatively new, you may also need expertise to help “sell” the concept throughout your organization. This is especially true at large enterprises, where testing needs to be piloted, then evangelized throughout various business units to drive adoption and ROI. Having an expert on staff or on retainer will help you with promoting the benefits of testing to many different types of stakeholders.

As I was writing this list, I realized it would be very difficult to go out into the talent pool and find an expert who could really nail every one of these areas of expertise. Be realistic as you look at talent; no one can be good at everything, and everyone’s expertise is on a spectrum.

Still, the 8 areas I’ve laid out would make a nice checklist for interviews, and because you’re realistic you’ll know that you’ll have to make a tradeoff or two to find the right expert for your specific company, culture, and needs.

Feedback welcome if there are areas of expertise I’ve missed, or other comments!