Usability Testing Basics:

How to effectively choose your test subjects

Overview: Any user testing is only as effective as the users you choose. This document give you some basics for avoiding trouble.

Choosing your test subjects carefully and in an unbiased manner is the key to getting meaningful results out of your user testing. If you choose users that aren't representative of your actual end users, you are literally wasting your time. If you choose end users that are biased, or who don't interact well, you're sunk as well.

In any testing scenario, including this course, you have to always balance idealistic wishes with practical reality. Sure, you'd love to have a pool of thousands of eager, willing and highly-motivated test users to choose from, with detailed demographic and background information for each. Dream on. In real life, both time and resources are limited, and compromises have to be made. At the same time, it's clear that if your test users are not representative of actual end users, you are very likely wasting the limited resources that you have. It's a tricky problem, but paying a little attention here can go a long way. Here are some guidelines to consider as you plan how to recruit and motivate your test users:

• Make sure that your target pool of potential testers is representative of your end users. The best way to arrange for this, of course, is if they actually are the (potential) end users for your product...people that you believe or (even better) have actually said that they would buy/use a product like yours if it existed.
• Just because you have SOME end users doesn't mean you're good. If your end user population is diverse (in age, in background, in experience, whatever) then you'll want to make sure your cover the entire spectrum of possible end users in your testing group. Maybe you app works well for the 18 year old CS geeks that you want are planning to sell your app to, but the even more important (wealthy!) group of 30-55 year old professionals that are also in your target doesn't get it. If you test only with CS geeks, you're heading for a disaster.
• The whole point of the constructive interaction technique that we are using for interaction testing is that we gain insight into user thinking and interpretation of our GUI via the problem-solving conversation between them as they work together through the interface. Obviously this works best if they talk...a lot! Gender-based power relationships or shyness can get in the way here, so my personal approach is to purposefully and conscientiously engineer testing pairs. At first glance, you might say that not doing just random pairing introduces bias, but this is not so: we are not trying to scientifically prove that our GUI works for all individuals, but rather, that it can work for all users inside our target group. What matters is that all potential users types are tested...and not how they are paired. Here are some specific tips:
  • Women work best. I'm sorry gentlemen, but it's true: in general women testers tend to be more collaborative and talk more as they work together to solve things. Of course, you can't just test women (unless, of course, your app is targeted ONLY at women), but take advantage of this and pair women together, particularly if you are dealing with a pool of total strangers, i.e., nobody knows anybody. As strangers, men tend to be more reserved and silent with each other, while women seem to engage and open up more quickly.
  • Use existing friendships. Whether women or men, people that already know each other are more open and will tend to collaborate and talk more as they work. With this in mind, try to recruit befriended pairs from the start: on your "application form" or when you ask people to participate, just straight up ask if they have one or two friends that they'd like to work with. This is also a great way to grow your testing group fast: because people like to work with folks they know, they'll recruit their friends for you!
  • Watch out for power inequities. Think about it: if you're sitting in a testing lab partnered with a full professor, a congressman, or other "person of status", then you're going to act differently. Maybe you'll be more reserved...or maybe more out there. But you'll be different. This "social power" effect works on much more subtle levels as well. Avoid pairing people where you think a social power differential might exist: sizeable age differences, big differences (perceived or actual) in expertise or experience, differences in social status (homeless vs. professional), etc. To avoid this, of course, you need good background info on your participants; that's what the pre-test questionnaire (below) is for.
  • Give them both control during the test. Or have them switch off. Since you've worked so hard to try to pair people to avoid power differentials, try to avoid re-introducing them in the testing environment by giving one tester more control than the other. This can be tricky: they person how controls the keyboard controls what answers get entered; the person that controls the mouse have a clear advantage when it comes to exploring the GUI. Some things I've done: connect two mice! Most O/S's allow this...both mice fight over control of the cursor. And usual the testers quickly settle on who's going to do most of the driving...but having that second mouse allows the other to jump in opportunistically. Another way is to ask them to switch off with controlling during the test...although I prefer to avoid this, seeing as it breaks the flow.
• Watch out for monetary motivations. One obvious way to build up a huge pool of possible test subjects, of course, is to put out an ad that says "sign up and be in the running to get paid $50/hour for testing software applications". This is, in fact, what many larger organizations do. And it's fine...but dangerous. If you have a pool of people who are JUST in it for the money, you're asking for trouble. These folks will often say whatever it takes to get chosen for a testing round...but then may not be the most motivated. Plus, you'll get only people who need the money in your pool...hardly representative of the well-off users you're targeting with lots of software! At the same time, providing some little sign of appreciation is really nice: some cookies as a snack when they arrive, a thank-you card, whatever. Goes a long way...