Due Dates: At Final
Points: Final Usability Analysis/Report: 100pts; Presentation: 50pts = 150 total pts All points modified by peer review factor
Rules: Same teams as for lat project.

Overview: Based on your initial user studies, you have now designed and built a fully-functional piece of software. Now it is time to see how thoughtfully and carefully your design and implementation was executed! As we've discussed in class, one would normally start out with a few expert reviews to iron out the worst bugs, and then go on to do a series of usability tests (e.g. constructive interaction, pairs of end users) to check out your application with a fine-toothed comb. Since our timeline is slipping a little bit in the last weeks, and because we've already had a decent taste of the usability testing, we will trim this down to decompress things a little bit: we will just do several expert reviews, followed by a refinement cycle to arrive at the final result.

Expert Reviews

In lecture, we outlined the concept of an expert review in general, and then discussed several specific variations on this theme. Recall that I had strongly recommended that teams do a rudimentary expert review or two earlier in the process --- at the pre-alpha release phase, or for specific modules/components --- and some of you may have done so. Let's do several formal expert reviews at this stage to refine your prototype somewhat. Here are the relevant details:

• Select the type of expert review that you feel would be most appropriate; most likely this will be a "Golden Rules" review. Then you'll need to find at least three suitable experts. You can definitely get more reviews if you like, but exactly three of them must be chosen from *within* the CS477 class (teams of four need at least four). This ensures that everyone in the class does one review for some other team.
• The "experts" could all be fellow 477 students --- but of course from a different team. You could also additionally find outside usability experts; this doesn't necessarily mean a real professional usability consultant (though that would be great!), it could just be someone who has extensive experience developing software for end-users. Your buddy who has hacked together a few web pages won't do. Find someone who you can reasonably argue (and you'll have to do this in your final report/presentation) has excellent qualifications to comment critically and usefully on your design.
• Have your experts do an expert review based on one of the frameworks we discussed (see the variations we discussed in class, e.g., Golden Rules analysis, etc.). What you'll give the experts is the review framework you'd like them to use, e.g., a nice little page summarizing the Golden Rules and the process of doing an expert review. You will also give him/her a background document that contains (2-3pages):
  1. an intro to your application (use the intros from your written reports: has overview of the domain/market,
  2. a second section detailing the specific aims or goals of your Thirstbuster product so that the expert knows what exactly your software of trying to achieve (main functionalities),
  3. a third section where you discuss carefully who the end-users of your product will be (summarize demographics info from your user studies) so that the expert knows as precisely as possible what perspective to take in reviewing the product.
  4. Then you would typically have a full user flow outlined for the expert. Think of this as you "lab manual" for the review. Sure, the expert is going to explore the app fully on his/her own motivation, but you need to give some guidance on how a "typical user flow" works, or what the key activities in the app are. This is just a series of short items, that detail the key activities. So for instance you might have: "Creating an account --- users should be able to create an account by entering their name, email, desired password, and a credit card". And then later "Users should be able to create, populate and place a new drink order. Details include <list out detailed functionality for this action, e.g., can add drinks, delete drinks, select drinks from menu, cancel order, etc". In essence, these activity descriptions are ordered in such a way as to represent a full walk-through of all aspects of your app. They are the "route description" for your expert to use as a hand rail in working through your product.
  5. Finally, a fourth short section details any particular areas of concern that your development team might have, i.e., specific aspects of the UI that have caused lots of discussion in your design phase, seem particularly iffy, or that you are otherwise worried about. Just gives the reviewer a special heads-up to look at those areas with particular care.
  Make sure this "instruction packet" is part of the appendix to your final deliverable!
• For experts internal to the CS477 class, I'll expect a formal, detailed written report. Learning to produce a good, credible expert review is a valuable skill. This isn't extremely difficult or arduous, it has:
  1. An intro section: The expert starts the report by formally introducing the situation and his/her credentials. Something like: "This document represents an expert review of the <insert name> application, which is a <web-app/mobile> app aimed at....etc". Then talks about credentials: "My expertise in performing this review is based on <detail background and experience...courses, degrees, experience>". To perform this review, I was given a <the review framework, the background doc, and describe how access was given to the app>
  2. Summary of review task and goals: Here the expert re-states the information given about the application, summarizes as he/she understands it. So a summary of the domain, end-users, and the main functionalities of the app.
  3. Outcomes Overview: Here the expert overviews what was actually done. "As requested, I performed an expert review of the application based on the <whatever> framework. Working over the course of (xx hours/days), I began by downloading the application to my <describe platform>, following the instructions given by the team. I then...etc etc". A paragraph.
  4. Detailed results: Here is the meat of the review! There is considerable flexibility on how this part is put together, but one excellent approach for a Golden Rules review is just to literally have one subsection for each Golden Rule (I like to put the name of the rule, followed by the rule itself in italics at the start, to remind all readers what the rule is). In that subsection you just a short intro sentence or two (did you find lots of trouble there or not, etc.), followed by a bunch of bullets for issues found. Some experts like to start each bullet with a priority/severity rating of some sort. In each such bullet, you'd describe the potential problem you noted. Plus you might give your "expert opinion" on how the problem could be solved/avoided. So I'd expect eight sections, each with comments and suggestions.
  5. A conclusion. Just a paragraph that first summarizes your findings, e.g., looking good overall with just minor problems; or maybe: major problems, to the point of not being able to effectively review it; or something in between. It then prioritizes the problems found, pointing the team at what the expert thinks are the most serious of the issues found, the ones that (in expert's opinion) must absolutely be solved first. Finally, the conclusions states recommendations for moving forward. Things like "Ready for intensive user testing" or "Ready for user testing after high priority items above fixed" or maybe "Needs major problems fixed, then one or more additional experts reviews to ensure it's refined to point where end-user testing can commence".
  6. For the purposes of this class, a perfectly reasonable review write-up should be 2-3 pages. It should be professionally formatted: you will be graded on the quality of your review by me...and the requesting team.
• For experts EXTERNAL to the CS477 class: Because this is done on good will and not for juicy payment, it is unreasonable to ask an external expert to write a full detailed report, as one would do in real life. We will settle for something less formal: Simply ask your expert to do the review (on their own time, without your interference), and then organize his/her notes in a 1-2 page sketch based on the review framework (e.g. Golden Rules) that you asked him/her to judge the design on, and/or specific problems noted. As an example, if you do a "Golden Rules Review", I would write up a report template for my expert, with brief instructions on top, and then listing out all eight of the golden rules (perhaps with a little explanation to jog memory, if expert is not currently in this class); the expert can then just edit this document and bullet in his/her commentary with respect to the various rules.
• When the review comes in, go over these notes with your expert verbally, adding your own annotations to them (perferably in a different color) until you understand what he or she is getting at. In real life, this is typically a phone call after the dev team has had a chance to read and digest the review.

Note that you may choose to "stagger" the expert reviews over time: You could get one or two experts going, see if you can get quick results from them...use those results to fix some major problems...and then do your last review on the revised application. This can be a nice strategy, as you try to see some improvement over time. On the other hand, you might choose to do all three at once, and then concentrate on the points of consensus between the experts (the more you have, the better, if you're doing this stategy).

You will then analyze the results of the various expert reviews as a whole. You will use the expert review feedback to improve your prototype in as many ways as possible. And finally, you will describe the expert review process, summarize the results of all the expert reviews and your response (app improvements) to them in your final report and presentation.

Detailed Deliverables Specs:

1. Specs: Final Report Writeup and Final Presentation