Usability Testing Basics:

Creating a high-quality Lab Manual

Overview: Creating a clear, concise lab manual that directs users through a series of specific domain-level tasks is absolutely vital for successful usability testing. This document tells you how!

A "lab manual" is the term we'll use for the document that you give your test users to guide their exploration of the interface that you're testing; other sources might call it a "test guide", "user worksheet", etc. The basic idea is simple: You are going to bring your users into the testing lab and tell them nothing much about the software they'll be working with outside of it's overall purpose, e.g., "We have a cool new drawing app for your to try out today". Then you'll sit them down, wire them up as needed...and then what? Well, that's when you give them the lab manual. It is just a brief intro paragraph that sets the stage, followed by a set of domain level tasks that you want them to accomplish. They work through all the tasks using your application, and when they hit the last page that says "Wow, you're done! Awesome and thanks!", then they are done and you can shut things down. Some key pointers:

• FIRST: Explicitly discuss and write down your specific goals for this usability test. Decide what exactly it is you want to test here. If it's just a general more or less isolated test of an application, you'll probably just focus are the key functionalities of the app. It's important to articulate these very clearly; ask yourself some questions to move this along: What is the main, central thing(s) that users want to accomplish quickly and easily with this app? What functionalities are those that absolutely positively must work, or else users will dump the app? On the other hand, maybe this test is part of a larger testing regime associated with a full development effort. In this case, your targeted functionalities might be very specific, e.g., those associated with a newly develop/modified module, or functionalities that had significant problems during a previous round of testing, or specific functionalities that you are told to test by the development team. Whatever the case, very clearly outline the functionalities you want to test before you start writing up the lab manual!! You'll need to have them clearly in mind to design a good lab manual...and you'll need to explicitly review these testing goals later, when you report out your results.
• Ok, now that you know where you are heading, we can write the manual:
• Start with a separate cover page that just has the title on it. This is key: first, they can't read ahead before you start the test; second, you can clearly see on your video record when they turn that page and start. A nice clean starting point for your analysis.
• Start the main section with an intro paragraph that sets up the context. Something like "Imagine you are traveling abroad in Europe. One of the most tricky challenges you'll face when traveling is figuring out how to manage local transportation, e.g., how best to get from your hotel to a recommended restaurant elsewhere in the city. The TravelGuru app is here to help. In the following pages, you'll be asked to use the app to find out key information during your imaginary trip. In particular, let's start with the assumption that you're staying in the Hotel Jardinier on the outskirts of Marseilles; this is what we've set as your current location. Have fun!". See what I mean? Sets up the context, gives a clue what the app is and what they'll be using it for.
• Next you'll have a series of exercises. Use the testing goals you articulated at the start to drive these "test tasks"...obviously the tasks should exercise exactly those functionalities that you are curious about. ) This means that you have to think about, discuss, and settle on what these key functionalities are!! Then design little tasks that test these functionalities. You yourself (as an expert) should be able to complete them all in 1/3 of the time you expect users to figure them out in. Make it all flow within the imaginary scenario you intro'd at the start; so between questions you have some connective stuff like "now that you've found a good restaurant and eaten, you decide that you want to go see a live theatre performance. Use the app to..." See what I mean? Keep the story going...
• Vital!! Your tasks should be described in terms of the end-user domain; specifically, they should never include interface level instructions for accomplishing the task. Example of good user-level prose: "You have heard there is an outstanding seafood restaurant called Mustafa's down by the harbor in Marseille. Go ahead and use the app to find out how best to get there from your hotel". See? No hints about how to do it, just tells them what domain-level information they need to find out. Example of bad prose: You want to get to a harbor restaurant called Mustafa's. Use the "find location" option under the File menu to enter this information, then click on column headings in the resulting list to sort results in whatever order you like". Here, the prose totally blows it, giving away key hints on how to use the interface...stuff that should be "intuitive" and is exactly what you're trying to test. Again, focus on describing WHAT to do in terms of end-user goals, rathen that HOW to do it using your interface.
• When possible, focus the task around some very specific goal, and ask the user to actually write down something documenting their findings in the lab manual (where you've provided blanks, etc. for this). So "when you've found Mustafa's write down (a) the fastest way to get there, including estimated cost and time; and (b) the cheapest way to get there, including estimated cost and time". Or maybe, for a picture editor: "When you have achieved the effect shown in the sample image at the right, write down the filter(s) that you used, and the parameters you configured for each."
• Put exercises on separate pages. Again, this makes it very clear on your recording when they are done with one and transitioning to the next one! You literally see them turn the page.

Ok, following these guidelines should give you a solid lab manual. Of course, you'll learn tons of new lessons along the way. Remember: you goal is to give clear description of WHAT you want them to do, without ever giving hints on HOW you want them to do it using the interface.

To give you a sample, here is a lab book that I dug up from some user testing I did on the Cardiovascular Simulator teaching tool that I've mentioned at times during lecture.

You will often want to submit your lab manual for review to teammates or a supervisor. To allow them to give you effective feedback, you need to educate them (a) on the application and (b) on your goals and strategy for this particular testing round. The outline of what you need to present follows:

1. Cover sheet for your packet. Team name, members, date, project/submission title, class name.
2. Intro the App. Anybody reviewing your lab manual needs to have a good understanding of the software you will be reviewing. Give a quick concise statement of what the app is supposed to do overall. Talk about the market, who the target audience is, what the status of the app is (in design, already released), etc and who the company is that is producing it. Then bullet out and briefly explain the broad key features of the app. This is basically what you'd find under "key features" in the app store or other advertising for the app...what cool things it can do.
3. Intro the testing plan. Ok, we understand the app halfway well; now explain what exactly you're up to with this test that you're making the lab manual for. Give a general overview statement of the test and its purpose, e.g., "We are testing a deployed app that is well-established in the market; the goal is to review usability of critical features, analyze issues and make recommendations for resolution". Next, bullet out the "targeted functions" that you'll be testing. Start by explaining how/why you selected the set of targeted functions, then bullet them out and explain each in one-sentence. Finally, close the section with how you expect testing to go down: how long users are intended to take, how many users/pairs, what kinds of users (characteristics) you'll be using, and maybe some mention of particular "problem areas" of the UI that you are worried about (and hence will be testing most carefully).
4. Next attach your lab manual. This is the packet you'll actually give the users when they come in for testing... and which you are submitting for evaluation.