A First Look at Google Website Optimizer

The really big analytics news for small businesses is that Google is now making available in beta a split-test and multivariate testing tool, the Google Website Optimizer (http://services.google.com/websiteoptimizer/) at no charge that is "pretty good," in the words of one leading industry observer who doesn't pass out compliments easily.

I took a good look at the tool at the Emetrics Summit and was quite impressed. It allows you to set up split-tests, that is, measuring any number of variations of one element on a webpage, such as headline or a button. It also allows multivariate testing, that is, testing several variations of several different elements at the same time, such as 3 or 4 variations on an order button, plus several variations of a headline, plus several variations of a "hero shot" (a testing industry term for the major graphic) -- all at the same time.

Set-Up Process

I was quite impressed with the careful set-up process that the tool offers. Unlike the first generation of many software programs, Google is careful to provide a step-by-step process to guide users through each phase of creating and naming areas of the page to study, then generating and placing JavaScript, then adding the variations of each element, and finally testing the set up for accuracy and beginning the test. It is a superbly-designed process for what is a complex procedure!

Dashboard

As the test begins to accumulate data, this data is shown in the dashboard. There's a graphic area that shows how much better or worse a particular variation is performing above the original control version, the baseline without any alterations. Red means poorer performance; green indicates better performance. To the right of the graphic portion are statistics which show the confidence level of the results, number of results showing this particular variation, etc.

Flaws in Google Website Optimizer

It is a wonderful tool, especially for a 1.0 version. However, I see two flaws.

1. The graphic portion of the dashboard doesn't indicate confidence level. Too many untrained users will see green as good and red as bad and take the good -- even though the statistical confidence level is too low (under 85% to 90%, for example) to draw statistically valid conclusions. I strongly recommend that the portion to the right of the median point of the graphic show as yellow until a certain minimum confidence level is reached and only then turn green. We're working with unsophisticated marketers. Let's give them all the help we can.
2. The multivariate tool doesn't include the Taguchi method of experiments, thus it takes much, much longer to get statically accurate results than it should. This is a huge flaw, in my opinion, that must be fixed to give this tool the kind of potential that it seems at first blush to offer.

Importance of the Taguchi Method for Multivariate Testing

Let me explain why the Taguchi method is so important. Simple A/B split testing of 2 or 3 or 4 variations of a single element require 2 or 3 or four versions to be run simultaneously, with the traffic split between them. When enough traffic comes through, the results will have enough statistical validity to form conclusions. So far, so good.

But if you have 5 or 10 important elements of a landing page to test, it will take weeks or months to sequentially test first element one, then element two, then element three, and so on. Enter multivariate testing, that is, testing of several elements of a webpage at the same time. Instead of 2 to 4 versions being tested simultaneously, you have to conduct many, many tests simultaneously to cover all the possible variations. For example:

• Four elements
• Each with four variations

requires

4⁴ tests = 4 x 4 x 4 x 4 = 64 tests

This takes a long time to get enough data to be statistically significant. What Taguchi did for Japanese industry seeking to optimize manufacturing processes, was to develop a mathematical array that defined the least possible number of tests necessary to reach a conclusion considering the number of variables and variations being tested. This cut down the number of necessary tests hugely and made possible the total quality management revolution.

But Google Web Analyzer doesn't use the Taguchi method of experiments. Rather it requires you to collect data for many, many unnecessary experiments, with the result that it takes much more data and much more time to reach a statistically meaningful result. Cynics observe that Google makes money every time they drive a click-through to your landing page, so it's against their interests to reduce the number of tests necessary to get a meaningful result. While I'm not that suspect of Google's motivations, the tool does need to incorporate the Taguchi method to make it widely useable for multivariate testing.

Conclusion

Overall, it looks like a great tool that small businesses can use to get started in testing at no cost. Of course, the real cost is in the time and expertise to use the tool to learn what is going on with your site. For that, you may want to hire a consultant that can do some testing work for you. All in all, Google Website Optimizer is real step forward for small businesses! Strongly recommended.