I recently read a fascinating paper written by some folks at Microsoft called “Online controlled Experiments at Large Scale.” Skip to the end if you want a link to the paper.
The paper’s topic was how Microsoft has scaled its testing and optimization program on the Bing search engine. It was written with somewhat of an engineering bent, so it wasn’t 100% relevant to my world. The kinds of optimization tests I design and conduct are neither automated nor conducted at such massive scale.
However, the authors laid out 3 “Testing Tenets” that they felt were instrumental to the development (and scaling) of their testing program. These simple rules were fantastic to read and 100% relevant to the world of Marketing Optimization and testing!
I enjoyed them so much that I’m going to summarize them for you, with commentary on why you might want to steal them ;-).
Optimization is difficult. I’m often amazed that good, sound testing ever occurs on websites due to the fast-evolving complexities of digital marketing, web development, and doing business across the digital landscape.
One of the reasons that Optimization is difficult is because it’s extremely cross-discipline by nature. It’s part marketing, part user experience, part design, part analytics, part web development, and part statistics.
Anything that cross-discipline requires a great deal of coordination to make sure all the specialists involved in conducting a test are doing their parts efficiently.
Another challenge of cross-disciplinary digital work is that accountability is sometimes missing, or not strong enough. This is a major reason why a lot of corporate testing programs don’t grow from “skunkworks” into something more accepted and robust—testing efforts become inefficient due to a lack of shared understanding about who is responsible for what and when. And maybe more importantly, who takes command when things aren’t running smoothly? Continue reading