Dan Wood: The Eponymous Weblog (Archives)

In my previous posts, I mentioned Google's Website Optimizer. I got quite a few reactions to that, so I thought I'd follow up a bit.

First of all, I should say that I had heard of it for a while, but it wasn't until I read Always Be Testing did I realize how important and useful of a tool it is. If you are thinking about this, I recommend the book highly.

The goal that I am testing is this: do people download the program after visiting the main Sandvox page? The reason this is the goal is because it's the most obvious action for somebody to take. Sure, I could track sales, but that's a bit trickier. One problem is that about half of our users buy Sandvox from the store that is built into the application. It's just a webview, so it's possible that the cookies from the optimizer would carry through, but if the visitor was using Firefox — which many are — it's not as easy.

Plus, as you might expect, more people will download a free trial than will end up purchasing something. And this kind of testing requires big numbers to get any statistical significance, so a test with a higher base conversion rate is going to get some conclusive results a lot more quickly.

(So if the page you are measuring is not getting at least several hundred visitors and conversions daily, testing is going to take you a long time.)

A few months ago, I realized that all I really cared to measure for conversions was people visiting our website with a Mac. There's no point in measuring random visitors from other platforms. (About 16 percent of visitors to the main Sandvox page are on other platforms.) What I did was to use PHP and only include Google's JavaScript that you put at the bottom of the tracked and goal pages. Something like this:

<?php
$userAgent = $_SERVER['HTTP_USER_AGENT'];
if ((''==$userAgent) || (FALSE !== strpos($userAgent,'Macintosh')))
{
?>
	<!-- insert GWSO stuff here -->
<?php
}
else
{
	echo "\n\n<!-- No GWSO -->\n\n";
}
?>

I've tried to take notes over the months as I've worked with the tool to adjust our main Sandvox web page. It might be interesting for some folks to read about what we did. Here are handful of our recent experiments:

• Moved our Apple Design Award badge from the top of the sidebar to below the download/buy buttons.
• Various experiments in button colors and fonts. We used to have flat, dull, gray buttons. Brightly colored, glossy buttons did much better.
• Experimented with the font of our body text. I was expecting sans-serif to win, but Serif did better. Later, we did another experiment and the Optima font we used did slightly better.
• We did several experiments to tweak the headline and sub-headline for Sandvox.
• Our website used to be more greenish than blue. Blue did better. (We actually have an entirely new design in the works for release soon.)
• We tested whether to run a "box shot" as if the software were something you could pick up in a computer store (you can't) vs. the icon, and the icon won.
• How about a nice little perspective screeenshot instead of the icon? Nope, the icon still rules.
• We did some variations on our list of sandvox features: one or two columns, large or small icons, etc. Smaller icons and one column was better.
• We had a lot of "copy" that I tried just taking out, and it was better to have it gone. Sayonara!
• Would a bigger Apple Design Award badge convince more people to download? Nope.
• We originally had our "slide show" of websites built with Sandvox up near the top. I tried a smaller version above the download button in the sidebar, and also just move it way out of the way, and the version where the images were way out of the way did best. I guess people don't want images to get in the way of reading the juicy details!

Anyhow, the experiments continue. I'll probably have to start a bunch over again when we get our new design in place, since the new metrics and colors will probably impact some of the choices we made.

See you at WWDC!