Testing the Boost website – semantic differentiation

By Nathan Donaldson

Tags: Agile , Design

We are constantly evolving the Boost website, the enhancements we make are largely based on customer feedback and the results we get from user tests. The iterative approach we are taking has offered up many benefits, but the biggest one is the ability to change the direction we are going in quickly and painlessly.

We recently modified the homepage just two weeks after releasing a new design. This was as a result of IntuitionHQ tests we conducted, along with face to face testing. For the sprint that finished last week we decided to do a semantic differentiation test on the homepage, and it has thrown up some more food for thought.

Firstly, what is semantic differentiation? It’s a technique for measuring the connotative meaning of concepts, which when I first came across the concept meant very little to me. So in terms that make sense to me, it is a way to measure the impression that our homepage makes on visitors to the site. You create a set of opposing terms and ask people to rate the Boost homepage against those terms on a scale.

When it came to creating a set of terms, they had to be things that mattered to us, and reflected how we wanted to be viewed by current and potential customers. We came up with five sets of opposing terms, with none of them necessarily being negative:

• Established/New
• Conventional/Cutting edge
• Cost-effective/Luxurious
• Niche/All-inclusive
• Creative/Technical

We asked twelve people (who we harassed whilst they hid from the torrential downpours Wellington was assaulted by last week) to look at our homepage and mark on a scale where they thought Boost fit. We also got them to do this for three competitor’s homepages, here are the results:

And so you can make head or tail of what this actually means:

• Dotted line: How we would like to be perceived
• Red line: How we are actually perceived according to the people who studied our homepage
• Black line: How competitor number 1 is perceived according to the people who studied their homepage
• Green line: How competitor number 2 is perceived  according to the people who studied their homepage
• Blue line: How competitor number 3 is perceived  according to the people who studied their homepage

So based on this feedback the way we are perceived is close to the way we want to be perceived in some areas, but we also have some work to do in others (for example, we have been around for 13 years, so would like to fare better on the established side of things). This has given us plenty of ideas and a solid direction to move in for the time being, or at least until the next round of tests.