An earlier post reviewed how to create a hypothesis. If you’re new to testing, though, you might wonder why you need a hypothesis at all.
When thinking about site tests, there are two approaches you could take: Guess and Check; or Hypothesis Driven.
Guess and Check
Guess and check is literally that – choose something, make a change and test it to see if the change helps. This type of testing is based on a best guess approach to what will improve the site.
Sometimes you’ll be lucky and find a change that helps. More often, though, you won’t see a big improvement.
The reason that this approach is less likely to work is your audience. The success of changes to a site is entirely dependent upon the audience’s reaction. The guess and check approach doesn’t consider the audience in any depth. So the changes are a best guess rather than audience driven.
Hypothesis Driven
The second approach is to think about the details of the test and form a hypothesis. When you create a hypothesis, you consider your audience’s point of view.
A hypothesis can be thought of as a testable short story that describes user behavior when a change is introduced.
A hypothesis might be structured as, “Changing the hero image copy to focus on the design benefit rather than technical benefits will increase revenue by 10%”.
A hypothesis states what’s changing (the copy) and the predicted audience behavior (10% increase in revenue).
While it doesn’t appear in the hypothesis statement, the most important part of constructing a hypothesis is why you think it will work.
You can answer why a hypothesis will work when you are knowledgeable about your audience. You can take what you know about your audience, eg their goals and motivations, and use this information to design changes you anticipate the audience will respond well to.
Testing with a hypothesis is also useful for increasing your knowledge about your audience.
For instance, if you test new copy and it works, it enriches your existing ideas about your audience. If it doesn’t work, it also adds to your knowledge of your audience. You had an idea as to why it should work but that idea was wrong. So now you need to reconsider what you know about your audience. You can use this information to make your future tests stronger.
Without a hypothesis, a winning or failing test lacks context. It’s hard to interpret because the test wasn’t created with the audience and site in mind. Instead of enriching already known information, the test results are isolated data points.
Hypotheses are also useful because you can use them to prioritize site changes. If you have many changes you want to make to a site, look at what hypotheses are most worthwhile and run those tests first. You get the most impact for your testing efforts.
Finally creating hypotheses is a good way to get support from the people you work with. A series of hypotheses that ties into your site strategy will give people confidence that you know what you’re testing, why you’re testing it and what outcome you anticipate.