You Think the Red Button is Better, But How Do You Know?

Answer:  Multivariate Testing (or MVT)

So what is it?  According to Wikipedia:

…multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous split tests or A/B tests performed on one page at the same time. Split tests and A/B tests are usually performed to determine the better of two content variations, multivariate testing can theoretically test the effectiveness of limitless combinations…In a nutshell, multivariate testing can be seen as allowing website visitors to vote with their clicks for which content they prefer and will stand the most chance of them proceeding to a defined goal.

Two Beaconfire staffers, Shiloh and Jo, recently won our Leadership & Innovation Award for their work on MVT, and I had the opportunity to sit down with them and find out more about it.

“The goal of MVT is to find the best combination of elements on a page to meet the goal of site,” explained Shiloh.  “One thing that differentiates MVT from traditional A/B testing is that you can measure the effects of the interaction of multiple elements.”

Jo followed up, “But the overall goal of the testing process is determined by the client.  Beaconfire’s clients tend to focus on things like increasing donations or building their lists, but any measureable activity can be an appropriate goal.  Any page where an organization is trying to generate more activity, like a campaign landing page or micro-site or a blog, can provide a good goal for MVT.”

“The nice thing about an MVT testing platform is that, unlike traditional A/B email testing, you don’t have to start by segmenting your list or tying your test to an email campaign,” noted Shiloh.  “You just need access to the testing tool.”

Jo explained, “Here at Beaconfire, we use a platform called Optimost, which includes a web-based control console that allows you to define what you’re testing and organize the test and the relevant elements.  Optimost both hosts the testing code and provides code to embed in your site to activate the test or track its results.”

Shiloh pointed out, “Optimost is great, because it’s specialized for MVT and doesn’t interfere with your web traffic or existing analytics package.  Your site visitor clicks on same URL, and the element swapping happens through Javascript.  The tool analyzes only the elements you’re testing and provides a specialized report, and your traffic data remains consistent because the rest of user’s visit is unaffected.  But you don’t have to subscribe to something like Optimost to do MVT.  Google offers a free tool, Google Optimizer, that allows you to do some simple testing.”

“MVT is really helpful because design is an artistic process, so it can lead to some guess work about what will really be effective,” Jo remarked.  “MVT provides conclusive data about the design elements and language that will actually motivate your audience to do what you want them to do.”

“It’s all about data-driven decision making,” concurred Shiloh.  “Everyone has theories about what works best, design-wise, for their constituents.  MVT takes all the guess work out, and it simplifies and accelerates decisions by providing objective resolution to disputes between usability, design, color palate, and stakeholders.”

“MVT also allows you to test accepted Web ‘best practices’ to see if they’re really valid for your audience,” added Jo, “which can be very useful in providing justification for what you’re doing, particularly if what you suggest runs counter to conventional Web wisdom.”

So do Beaconfire’s MVT experts have any advice for organizations considering MVT?

“Watch out for seasonal effects,” Shiloh remarked.  “The way your users behave in July may not be the same as the way your users behave in December.  So if you’re trying to optimize for year-end giving, you need to make sure you’re testing at year-end, too.”

“There’s also a familiarity effect,” added Jo.  “Sometimes we’ll discover that a new element may be more objectively intuitive and may perform better with new users, but frequent site visitors are so accustomed to the old way that it continues to perform better for them, even if it’s objectively ‘worse.’  So you need to know what your goals are and who you’re trying to serve.”

“Actually, that’s the most important thing:  organizations should be clear up front about what they’re trying to measure,” Shiloh noted.  “This is particularly important if you’re doing MVT on the organization’s home page.  Home pages often serve a variety of competing needs:  getting donations, increasing engagement, promoting events, list building, providing news, information about the organization, and advocacy opportunities, etc.  We recommend that organizations begin by focusing on one primary goal and testing around it.”

As Jo stated, “If you aren’t testing the right thing, you won’t get the right information.”

“Organizations also have to be willing to engage the process, keep an open mind, and take risks,” continued Jo, “but the fact that we’re ‘testing’ makes it easier to do that.  We’re not committing to change the site; we’re just testing out an idea.  Testing mitigates risk, provides freedom and can help facilitate the internal approval process, even if senior management is skeptical.”

“And in the end, all results teach us something, even if what you learn is that a particular hotly-debated element makes no difference at all, because now you know,” concluded Shiloh.   “You always learn something valuable, if sometimes unexpected, from the MVT process.”