Vol. 1, No. 30
The Importance of Being Tested

If you have a large site, or even a small one, with some scripting functionality, it's a very good idea to expose it to a small sample audience early on. That way, you can get feedback before inviting the whole world in and finding out 10,000 users later that there's a nasty bug in your code. Whether you're running a complicated bulletin-board system with membership, authentication, rankings, auctions, real-time chat, and a Flash-based interface, or merely a site with a few pages and a dash of JavaScript navigation, it's impossible to overstate the importance of testing your site. (Well, not impossible, but it'd require a couple more newsletters, and maybe a touring sketch comedy troupe to hammer the point home.)

So, test your site. If you don't, you may fail to detect a vital flaw in your e-commerce site until it has: shipped all your inventory, free of charge, to your competitor; spammed your customer base and then deleted your records; and resulted in several class-action lawsuits. But even if there's no possibility of a major flaw, and your livelihood isn't riding on the success of your site, it's still highly worthwhile to show your site to a few people before you launch, just to make sure that they are happy with the interface, navigation, and the background color. It never hurts to make sure you haven't misspelled anything either.

The basic idea of testing is to simulate as closely as possible the real use your site will be subjected to. Gather a bunch of people, put them in front of computers, and set them loose on your site. First, have them use the site normally and see if it makes sense to them, and if they can intuit an easy-to-follow progression through the site. If they get confused and can't figure out how to register, how to return to the home page, or how to log out, you probably need to tweak things a bit.

Next, encourage your testing team to try out as many different functions as they can, and to do unusual and unpredictable things. What happens when they try to print out pages? What happens when they log out and then hit the back button? What happens when they mistype the URL? The more eyes you have looking at the site and the finer-meshed your net, the less likely it is that problems will slip through. (Use common sense, though: five testers will probably catch almost as many bugs as an army of 500, and it'll take a lot less time to read through their feedback.) Try to have your testers work on as many different platforms as possible. How does it look using Opera 3.3 with JavaScript turned off and caching cranked up to the max? How does it look in IE 5, through a firewall, on a black-and-white screen? What happens if the user has a very slow connection and no mouse? What happens if they hit Stop midway through the download? Encourage your testers to try new and bizarre things — give them a list if they're not creative types.

If you can, it's very helpful to look over the testers' shoulders — literally — as they use your site. It gives you an opportunity to observe sticky points that the testers themselves may not be aware of. If it's a big and crucial rollout, you may even want to do a little time-and-motion study: videotape the testers as they cruise around the site, and review. Where do they seem to get confused? What could you make more efficient?

You can also prepare a little survey for the testers to fill out when they are finished. But be warned: surveys tend to be less useful than you might think. Testers often feel compelled to please, and they may gloss over aspects of the site that were really problematic, or spend pages expounding the details of aesthetic judgments that you don't care really about.

When choosing testers, try to get a nice cross-section of your target demographic. If it's a site for computer novices, don't just ask your hacker friends to test it out. If it's for kids, get real kids to use it. If it's all in French, don't expect Hungarian testers to understand. If it's — well, you get the picture. Chances are, you'll get more objective feedback from strangers than if you just get your mom and sisters to test it. Your mom and your sisters are nice, and they're smart, and they may in fact comprise your ideal target audience, but they love you very, very much, and that means they will tell you your site's wonderful, even if it is a dire landscape of broken images, dead links, browser-unsafe colors, and hideous 10 MB background GIFs.

HINTS, POINTERS, AND TIPS 'O THE TRADE:

Assuming you even have the space to do so, it's a good idea to have all your testers testing in the different rooms, at the times. No kibitzing. Every user for him or herself.

Try to get a new batch of testers for each new version of your project, so they're not biased by what they've already seen.

If you're really clever, you can incorporate checkpoints into the CGI code on your site that automate the testing process a little by detecting and logging patterns and trends of usage. Not necessary, but definitely clever.

Homemade ice cream can be a lot of work, but oh boy, is it ever worth it.

RESOURCES:

Webmonkey's Mike Kuniavsky's guide to user testing

Handcrafted Archive