With all of the quality blogs devoted to Conversion Optimization tips and best practices, I’ve not come across any posts related to the different kinds of documents required to be successful in a conversion optimization and testing effort.

Being a bit of a “document geek,” I will give you my thoughts on important documents and how to use them to set yourself up for success.

Why You Should Care About Documents

If you’re not a self-professed Document Geek like me, you may be wondering why this post is worth reading. Documents are boring to some, so how can documentation make your Conversion Optimization efforts more fruitful?

Because well designed and maintained documents contribute to accountability, repeatability, efficiency, and scalability.

If you want your testing efforts to be high quality, and especially if your optimization crew is a bit “junior,” documentation adds accountability to the process you’re using and those who are paid to follow the process.

As with any business process, you want it to be repeatable and therefore predictable. This helps with budgeting, resource allocation, project planning, and nearly everything related to a business. Use of high-quality documents and templates, along with a crafted business process, will ensure that your optimization and testing process looks relatively consistent over time as you move from test to test.

While people often grumble about documents because they’re “just more paperwork to fill out,” the fact is that documentation, when done correctly, should save you effort. People often overlook the fact that using solid documentation avoids errors, vagary, and re-work that are often the most painful effort that goes into projects. Documentation that captures the necessary data and functions as a check list saves you time and effort in the long run, thereby increasing efficiency.

While most initial optimization efforts, like running a simple A/B test or two, can survive without proper documentation, the goal is do more than survive; it’s to flourish! As soon as you try to ramp up your testing program to more complex experiments, involving more resources, capturing more data that impacts more coworkers, you’re now dealing with “scale,” and it will quickly become painful without the right documents. For scalability, your process and your documentation need to be top-notch!

The 6 Types of Optimization Documents

Now that all the document “haters” have stopped reading, we can review the six categories of optimization documents 😉

Note: This post deals only with documents that would be created as part of an optimization process. There are often dozens of documents referenced by an optimization process that were created for other marketing purposes. Some examples might be customer survey data documents, marketing personas, and marketing strategy documents that provide data about messaging, value proposition, channel mix, etc.

1. Discovery Documents

The first type of Conversion Optimization document you might call a “Discovery” document. Discovery is a project management term for the initial phase of a project where you must discover or uncover information necessary to flesh out the problem that needs to be solved.

This type of document should summarize the research you’ve done at the front end of your Conversion Optimization, or test, project. It should focus on the known business goals, KPIs, and constraints related to the site/page you’re going to test. Also, it should get everyone on the same page about the target audience you’re trying to convert.

As mentioned earlier, you may pull data from dozens of marketing documents to arrive at this summary, but condense it all in one document that is specific to the optimization and testing you want to do.

2. Analysis Documents

No test should ever be undertaken without first doing a good deal of data analysis related to the page/experience you want to optimize. Generally, this analysis is done on web analytics data, but there are other data sources like AdWords or call tracking that could certainly come into play.

The point of the analysis is to find insights in the data that drive testing ideas: what pages to test, how to test them, in what order, with what sample size, etc. To show your team that you’re not testing based on opinion, your analysis document should capture and summarize what analysis was done, what problems are indicated by the data, and what the “downside” is of these problems. For example, a high bounce rate on a landing page means $X/month of wasted marketing investment.

3. Recommendation Documents

While the Analysis and Recommendations related to an optimization project are sometimes combined into a single document, I’m breaking them out in this post because their aim is truly different.

If the Analysis document is the “why” something should be changed on a site, the Optimization Recommendation document is the “how” it should be changed. Optimization Recommendations are often grouped together in a single document in order to capture the larger picture of multiple recommended changes across the site.

If multiple recommendations are grouped together, this type of document then becomes a “road map” or “queue” of prioritized recommendations starting with the most valued test, and decreasing in value as the list goes on.

Finally, this document could also take on some aspects of project management to capture what resources will be needed, risks, constraints, milestones, etc.

4. Design Documents

Once your team has agreed to make an optimization-related change, or conduct an experiment, Optimization Design documents are the way you capture requirements, specifications, instructions, and content in order to build and conduct the test.

The Optimization Design document should contain “scientific method” things like problem statement and hypothesis. It should also capture the framework of the proposed experiment, like sample size, estimated duration, test goals, and test framework (e.g. full factorial multivariate).

This document will likely also contain design artifacts like wireframes, mockups, or new copy. The amount of detail varies from company to company, but I’ve seen some very detailed, formal design documents that contain approved design, imagery, and copy, leaving only markup to be written.

Finally, there may be elements specific to your testing tool and how it needs to run the experiment. An example might be a set of javascript tags, code placement examples, and specific pages that your IT department needs to modify.

5. Results Documents

Once the change has been made, or the experiment completed, you’ll of course want to do the accountable (and often fun) thing and report results back to stakeholders. Enter the results document, which summarizes what was tested, why, how, and with what results. There might also be financial impact calculations estimating the “upside” of an improved KPI like funnel conversion rate or revenue per visitor.

As with any good experiment, you should also report back on whether the hypothesis was proven or dis-proven. Either way, there are learnings to be gleaned from the data (e.g. a focus on “loss aversion” is more persuasive than a focus on features) and conveyed to your stakeholders so they can make more educated decisions in their jobs going forward.

Finally, the results document is the appropriate place to state any ideas you have for follow-up tests, new hypotheses, or when the results should be validated in the future.

6. Archival Documents

The sixth category of “archival” is often overlooked by beginners and experts alike. Once you finish a test and get the win or the learning, it’s easy to assume that you’ll remember the details in the future, but the fact is that it’s extremely difficult to retain that information once your optimization program is up and running.

The other thing to keep in mind is that it’s not just your personal memory of testing that matters; it’s the collective “organizational memory” that needs to be documented. After all, you may leave your post, and someone else will come in and want to know what’s been done to-date.

Archival documents should summarize the basics of each experiment run: time frame, hypothesis, examples of new content tested, type of experiment, results, and interpretive learnings.

Over time, your team’s testing “knowledge base” will become an extremely powerful marketing asset (and competitive advantage) that various team members can draw on as they plan new campaigns, new experiments, or new site experiences.

What Did I Miss?

This post was an interesting mental exercise to pull together a list from my various experiences working for agencies as well as my own consulting practice. I tried to keep it general in order to provide an overview, but did I miss any major categories? Are there any specific documents that you find indispensable in your optimization work? Share!