I recently read a fascinating paper written by some folks at Microsoft called “Online controlled Experiments at Large Scale.” Skip to the end if you want a link to the paper.
The paper’s topic was how Microsoft has scaled its testing and optimization program on the Bing search engine. It was written with somewhat of an engineering bent, so it wasn’t 100% relevant to my world. The kinds of optimization tests I design and conduct are neither automated nor conducted at such massive scale.
However, the authors laid out 3 “Testing Tenets” that they felt were instrumental to the development (and scaling) of their testing program. These simple rules were fantastic to read and 100% relevant to the world of Marketing Optimization and testing!
I enjoyed them so much that I’m going to summarize them for you, with commentary on why you might want to steal them ;-).
One of the reasons that Optimization is difficult is because it’s extremely cross-discipline by nature. It’s part marketing, part user experience, part design, part analytics, part web development, and part statistics.