HCDE 517: Usability Studies (Winter 2022)

Course Description

HCDE 517 will introduce you to usability testing and to usability research as a user-centered design strategy. The course takes a process approach; you will learn how to define your audiences and issues, create investigative procedures that answer your questions, administer the procedures, analyze the results, and report your findings effectively.

Though we use primarily examples of computational systems in this course, many of the methods, strategies, and approaches apply to products in various media (software, hardware, or services). You are encouraged to bring a variety of examples into the class.

It is important that you keep up with the assignments and that you seek help — from the teaching team, your teammates, or your classmates — if you are struggling.

Learning Objectives

At the end of HCDE 517, you should be able to:

• Understand and explain to others what usability testing and usability research are and what they can contribute to a design effort
• Analyze the usability issues that a product has and prioritize those that merit investigation through a usability test; analyze the various audiences for the product and prioritize those that are most critical at the current moment
• Design a usability test that answers the questions you have for the audiences of interest
• Administer the test, analyze the results, and report the findings effectively
• Explain how usability issues or analysis might inform the design of new products or services

Resources

Slides and notes

Slides/notes handouts from the lectures: Generally made available via the Canvas site (under files), after our class meeting.

Papers

Unless otherwise announced, papers will be posted on the Canvas site and/or linked to from this syllabus.

Textbooks

There are two required textbooks:

• Measuring the User Experience, Tom Tullis & Bill Albert. I will be using the second edition, though if you find the first edition, that should also be okay. Available online: EBook Central, O'Reilly Academic, UW Libraries Link
• Handbook of Usability Testing, Jeffrey Rubin & Dana Chisnell. You'll want the 2^nd edition (the first is pretty dated). Available online: EBook Central (limited number of users), O'Reilly Academic, UW Libraries Link.

Both should be available in the UW Bookstore.

Course discussion site

One of the best ways to get help during the course will be via the course Slack workspace we have created. You are encouraged to respond to (comment on) other students' messages. Don't worry about giving wrong information: the instructors, or your peers, will post corrections if necessary.

If you have something of a private nature that you don't want to share with the other students, feel free to send a message to one of us individually.

Class policies

Collaboration, academic honesty, and integrity

This quarter, you will be asked to contribute to the success of your colleagues' projects by providing them with feedback on their works-in-progress. Your participation in workshop activities will be graded for the quality, usefulness, and insightfulness of your comments. You will also be collaborating with colleagues on assignments.

If you receive assistance on an individual assignment, please indicate the nature and the amount of assistance you received. If you have more experience with some of the material and are willing to help other students, please feel free to do so. Just remember that your goal is to help teach the material to the classmate receiving the help.

The essence of academic life revolves around respect not only for the ideas of others, but also their rights to those ideas and their circulation. It is essential to take the utmost care that the ideas and expressions of ideas of other people always be appropriately attributed, and, where necessary, cited. For writing assignments and anywhere in print, when ideas or materials of others are used, they must be cited. Any direct quotes must be cited. In any situation, if you have a question, please ask. Such attention to ideas and acknowledgment of their sources is central not only to academic life, but life in general.

Course conduct is governed by the University of Washington Student Conduct Code and the College of Engineering Academic Misconduct Policy. You should also acquaint yourself with the HCDE Plagiarism Policy. Failure to properly cite or attribute an idea will result in a zero on that assignment and a report to the College of Engineering; severe or repetitive instances may result in more severe consequences.

To reiterate, the collaboration policy is as follows. Collaboration in the class is encouraged — you can get help from anyone as long as it is clearly acknowledged. Use of work — yours or others', from this course or similar courses — from previous quarters is not allowed. The authorship of any assignments must be in your own style and done by you, even if you get help. Any significant help must be acknowledged in writing; any use of others' words or ideas must be cited and attributed appropriately.

Classroom Rules

I like a relaxed classroom where everyone feels comfortable. You are welcome to bring drinks or snacks to class (though, due to COVID restrictions, will need to eat during breaks). You can bring your laptop to class. I would rather have you come to class and listen with one ear than not come to class at all. As a courtesy to others, be sure to put your phone on silence/vibrate. Coming late to and/or leaving early from class is okay, as long as you don't disturb your classmates. Ask questions at any time, and if you have some expertise in a particular topic, please to raise you hand and share it with the class. If you aren't comfortable doing so, you should also feel free to post it to the Slack workspace!

Our primary purpose in the classroom is to interact and learn from each other. While non-participation (sleeping; checking email) is permitted, we urge you to consider its impact on your fellow students. Class is a lot more fun when everyone is engaged; a few visibly disengaged people can suck the energy out of a room. We reserve the right to ask you a question to try to reengage you if you seem to have checked out, but will try not to do this in an embarrassing way.

Contacting the instructors

Our prefered method of contact (for quick questions, to let us know you will miss class, to schedule a meeting, etc.) is Slack or email. I (Sean) don't check my voicemail or Canvas messages regularly. When emailing:

• Please put "[HCDE 517]" in the subject line. This helps us distinguish your messages from the many other, but typically less urgent, emails we get.
• We try to reply to all email as quickly as possible, but please allow up to 48 hours.
• Please email both instructors unless it is a private concern. If you don't, you may wait considerably longer than 48 hours for a reply.
• We (and you) get a lot of email. Please use proper punctuation and capitalization. Please make any questions clear (or we may not answer, thinking it is just an "FYI" message).

I want to emphasize that your feedback is welcome. I may adjust the course based on what you say, I may save your comments for next year, or I may disagree, but in any case, your thoughts are appreciated.

Student rights

Please read the HCDE statement on student rights.

Acccess & Accommodations

Your experience in this class is important to us. If you have already established accommodations with Disability Resources for Students (DRS), please communicate your approved accommodations to us at your earliest convenience so we can discuss your needs in this course.

If you have not yet established services through DRS, but have a temporary health condition or permanent disability that requires accommodations (conditions include but not limited to; mental health, attention-related, learning, vision, hearing, physical or health impacts), please contact DRS at 206.543.8924, uwdrs@uw.edu, or disability.uw.edu. DRS offers resources and coordinates reasonable accommodations for students with disabilities and/or temporary health conditions.

Reasonable accommodations are established through an interactive process between you, your instructor(s), and DRS. It is the policy and practice of the University of Washington to create inclusive and accessible learning environments consistent with federal and state law.

Religious Accommodations

Washington state law requires that UW develop a policy for accommodation of student absences or significant hardship due to reasons of faith or conscience, or for organized religious activities.

The UW’s policy, including more information about how to request an accommodation, is available. Accommodations must be requested within the first two weeks of this course using the request form.

Student Conduct

The University of Washington Student Conduct Code (WAC 478-121) defines prohibited academic and behavioral conduct and describes how the University holds students accountable as they pursue their academic goals. Allegations of misconduct by students may be referred to the appropriate campus office for investigation and resolution.

Harassment will not be tolerated, and consequences will include immediate removal from the Slack workspace and other course tools. I also emphasize the importance of respecting your classmates' (and teaching team's) boundaries around how and when they wish to be contacted.

I am not an official required to report under UW's Title IX plan. This means I am not legally required to report misconduct or crime; if you share a problem with me, I will work with you to determine next steps. An exception is if anyone involved is under 18. You also can reach out to UW confidential advocates or SafeCampus.

Work in the Course — Getting a Grade

Each assignment is designed to test your achievement against one or more of the learning objectives. Different assignments emphasize different learning objectives. Please note that some grading will be subjective in nature.

There are three categories of assignments:

• Reading discussion / presentation
• Class participation
• Project

In addition, you will complete a final self and team evaluation report, reflecting on your learning and participation across the quarter.

Late assignments

It your responsibility to make sure that each assignment has been turned in on time. Except by prior arrangement, we do not grade late assignments, though we will return them with feedback. For personal emergencies, we will work out an alternative plan with you and your team if you contact us before the due date.

Quality of Communication Assignments

All written assignments for this class must be of high quality: thoroughly proofread, well organized, and stylistically appropriate for the context. If in doubt, err on the more formal, polished, professional side. Writing quality will be a graded component of every written assignment, presentation clarity and engagement will affect your grades on all presentation assignments. In the workplace, if your results or ideas are of high quality but your communication is not, no one will be able to benefit from the work you do.

Grading

The graded work in the course will be weighted roughly as follows to determine a final percentage grade.

Reading presentation & discussion	10%
P1. Preliminary proposal	Credit/No credit
P2. Interaction map	10%
P3. Usability study plan	15%
P4. Usability study kit	15%
P5. Study results (divided among presentation & report)	35%
Self + Team-evaluation Report	10%
Class participation	± 5%*

* possibly more for particularly constructive participation. Attendance is not graded, but it is a lot easier to participate if you are here.

Also see the HCDE grading policy.

Course Outline

In this course, we will cover usability studies, including their role in the design process, what questions they can answer, and some of the different evaluations and tests you might peform. We will discuss how to plan and conduct usability tests, and how to report results from usability studies. To practice these skills, we will use in-class workshops and an accompanying, quarter-long project.

Success in the Course

From my prior experience teaching HCDE 517, the most common places students run into trouble are: (1) not starting soon enough, (2) letting team issues fester, and (3) not negotiating a clear and reasonable scope with clients at the start.

The most common pitfall is not starting on any portion of the project soon enough. You'll be working with participants and (possibly) real world clients, and with technology that may have issues. Recruiting almost always takes longer than you think, people may not show up, or there may be a chance snow storm. The website you are studying might go down for maintenance. The app you are evaluting might auto-update to a new version, ruining the script for your evaluation. If you start early, you'll have time to manage these setbacks. If you don't, you won't.

Additionally, as relative newcomers to this material, you tend to need some time with your data (especially any qualitative data) to draw out a strong analysis and good discussion. Doing a study at last minute and then rushing your writeup and analysis tends to have poor outcomes.

Second, you'll be working in teams and it is important to attend to team dynamics. Early in the quarter, you should have a discussion in your team in which you review and negotiate communication expectations and your goals for the course and project. As the quarter progresses, check in, see how things are going, and adjust as necessary.

Good luck and welcome aboard!

Course Schedule and Important Dates (view on separate page)

Week	Dates	Topic	Readings	Assignments due
1	4 January	Course overview Introductions What is usability? History of usability? Learning & Memory	Required Katie Hafner. “Wanted by the Police: A Good Interface,” NY Times. W Craig Tomlin. “UX Killed Usability.”
2	11 January	Planning & considerations for usability testing Why do we need usability testing? Usability testing within the UCD process Identifying usability concerns Usability inspection methods Choose teams + Projects Workshop: Brainstorming usability concerns	From the books (required) Tullis & Albert, “Introduction” Rubin & Chisnell, Chapters 1 -3 Required Jeff Sauro, “What's the Difference between A Heuristic Evaluation and a Cognitive Walkthrough?” Stephanie Rosenbaum, “Usability Evaluations Versus Usability Testing: When and Why?” Jakob Nielsen, “Usability Heuristics” (ch5), Usability Engineering. Optional John D Gould & Clayton Lewis, “Designing for Usability: Key Principles and What Designers Think.” Research-Based Web Design & Usability Guidelines, from Usability.gov. Extend topic: Heuristic evals for specific settings Menno de Jong and Thea van der Geest, “Characterizing Web Heuristics” Jennifer Mankoff, Anind K. Dey, Gary Hsieh, Julie Kientz, Scott Lederer, & Morgan Ames. 2003. Heuristic evaluation of ambient displays. CHI 2003. Enrico Bertini, Silvia Gabrielli, & Stephen Kimani. 2006. Appropriating and assessing heuristics for mobile computing. AVI '06. Raina Langevin, Ross Lordon, Thi Avrahami, Benjamin Cowan, Tad Hirsch, and Gary Hsieh. Heuristic Evaluation of Conversational Agents. CHI 2021. Extend topic: Cognitive Walkthroughs in other Settings Aaron R. Lyon, Jessica Coifman, Heather Cook, Erin McRee, Freda F. Liu, Kristy Ludwig, Shannon Dorsey, Kelly Koerner, Sean A. Munson & Elizabeth McCauley The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability. Implementation Science Communications 2 (2021).	Read project descriptions Sign up for reading presentation
3	18 January	Planning for a usability study, cont. More methods for identifying usability issues Selecting representative users	From the books (required) Rubin & Chisnell, “Develop the Test Plan” (ch 5) & “Set Up a Testing Environment” (ch 6) Tullis & Albert, “Background” (ch 2) Required Code of Conduct of the User Experience Professionals Association (2013) Martin Schmettow. Sample size in usability studies. Communications of the ACM. 2012. Steve Krug, “Usability Testing on 10 cents a day” (ch 9) from Don't Make Me Think: A Common Sense Approach to Web Usability. New Riders 2000. Optional Jakob Nielsen. (2007). “Fast, Cheap, and Good: Yes, You Can Have it All.” More on sample size Jeff Sauro. (2010). “Why You Only Need To Test With Five Users (Explained).” Laura Faulkner. (2003). “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.” Jakob Nielsen. (2000). “Why You Only Need to Test with 5 Users.” Joe Dumas, “How Many Participants in Usability Test is Enough?” Extend topic: Accessibility Web Accessibility in Mind Petrie & Bevan. “The evaluation of accessibility, usability and user experience” in The Universal Access Handbook. Reframing Accessibility for the Web Design for Autism	P1. Preliminary proposal
4	25 January	Forming an initial test plan What are the responsibilities of a tester? What are general considerations to drive a test plan? What is essential and what is optional when preparing a meaningful study?	From the books (required) Rubin & Chisnell, “Skills for Test Moderators” (ch 4) & “Find and Select Participants” (ch 7) Tullis & Albert, “Planning a Usability Study” (ch 3) Required Medlock MC, Wixon D, McGee M, & Welsh D. “The Rapid Iterative Test and Evaluation Method (RITE): Better Products in Less Time” Walji et al. “Are Three Methods Better Than One? A Comparative Assessment of Usability Evaluation Methods in an EHR?”, IJMI 2014 Optional Susan Dray & David Siegel. “Penny-Wise, Pound-Wise: Making Smart Trade-offs in Planning Usability Studies.” Jeff Sauro. “What goes into a good usability test plan?” Extend topic: Usability, children, and aging RL Franz, C Munteanu, BB Neves, R Baecker. Time to Retire Old Methodologies? Reflecting on Conducting Usability Evaluations with Older Adults. MobileHCI '15. J Nielsen. Seniors As Web Users. "Usability Testing with Kids and Teens", Usability.gov. Extend topic: Usability and automation 99% Invisible: Children of the Magenta (Automation Paradox, pt. 1) 99% Invisible: Johnnycab (Automation Paradox, pt. 2)	P2. Interaction map Draft: in class Final: Sunday, 30 Jan
5	1 February	Preparing a test with measurable results What test materials do we need for a usability study? What will be measured? How do we create successful data collection instruments? Workshop: Study plans	From the books (required) Rubin & Chisnell, “Prepare Test Materials” (ch 8). Tullis & Albert, “Performance Metrics” (ch 4), “Issues-based Metrics” (ch 5), “Self-reported Metrics” (ch 6). Required Joe Dumas, “Usability Testing Methods: Subjective Measures,” in The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, 3rd ed. MS Andreasen, HV Nielsen, SO Schroder, J Stage. 2007. “What happened to remote usability testing?: an empirical study of three methods.” CHI 2007. Optional Jeff Sauro & Joe Dumas. “Comparison of Three One-Question, Post-Task Usability Questionnaires,” CHI 2009. Hyewon Suh, Nina Shahriaree, Eric B. Hekler, Julie A. Kientz. “Developing and Validating the User Burden Scale: A Tool for Assessing User Burden in Computing Systems” CHI 2016. Extend topic: The Hamburger Icon	P3. Study plan Draft in class Final by Sunday, 6 Feb
6	8 February	Conducting a usability study What is “thinking aloud”? What is key to successful facilitation? How do we collect data from a usability study? Workshop: Study kits	From the books (required) Rubin & Chisnell, “Conduct the Test Sessions” (ch 9). Tullis & Albert, “Behavioral and Physiological Metrics” (ch 7). Required Michael Hawley. “User Interview Techniques, Guidelines for Obtaining Better Results,” User Experience 8(3). Judy Ramey. “Methods for successful "Thinking Out Loud" procedures.” Michael Margolis. “How to hack your body language for better interviews,” Design Staff. Optional M. Ted Boren & Judy Ramey. (2000). “Thinking Aloud: Reconciling Theory and Practice,” IEEE Transactions on Professional Communication 43(3). Mie Norgaard & Kasper Hornbaek, “What Do Usability Evaluators Do in Practice? An Explorative Study of Think-Aloud Testing,” DIS 2006. Extend Topic: Invisible OSs, Recognition, and Recall R Budiu. Memory Recognition and Recall in User Interfaces Nielsen Norman Group. J Pavlus. "No UI" Design's Next Move: Fake UI. FastCompany Dan Norman and Bruce Tognazzini. How Apple Is Giving Design A Bad Name. FastCompany Extend topic: Additional Techniques Still & Morris. (2010). “The Blank-Page Technique: Reinvigorating Paper Prototyping in Usability Testing,” IEEE Transactions on Professional Communication 53(2).	P4. Study kit Draft: In class Final: Sunday, 12 Feb
7	15 February	Data analysis & reporting Validity and Reliability Analysis and Reporting Workshop: Data analysis	From the books (required) Rubin & Chisnell, “Analyze Data and Observations” (ch 11). Tullis & Albert, “Combined and Comparative Metrics” (ch 8). Required Chauncey Wilson. “Analyzing and Reporting Usability Data,” STC Usability SIG Newsletter. Michael Hughes. “Rigor in Usability Testing,” Technical Communication, Fourth Quarter (1999): 488-94. Optional Jeff Sauro & James Lewis. “Correlations Among Prototypical Usability Metrics: Evidence for the Construct of Usability,” CHI 2009 Laurie Kanter, Deborah Sova, & Lori Anschuetz. “Organizing Qualitative Data from Lab and Field: Challenges and Methods,” UPA 2005. Extend topic: Voice UI Extend topic: Large data in usability Walter C. Mankowski, Peter Bogunovich, Ali Shokoufandeh, & Dario D. Salvucci. “Finding Canonical Behaviors in User Protocols,” CHI 2009. A/B Testing Mary C. Burton, Joseph B. Walther. The Value of Web Log Data in Use-Based Design and Testing. Journal of Computer Mediated Communication 2006.
8	22 February	Presenting your findings Discuss details for the final assignments How can data and findings be meaningfully shaped for stakeholders? Reporting and communicating results Usability experiences in sociotechnical systems, can things be “too” usable? (& other dilemmas)	From the books (required) Rubin & Chisnell, “Report Findings and Recommendations” (ch 12). Required Kerry Rodden, Hilary Hutchinson, & Xin Fu. “Measuring the User Experience on a Large Scale, ” CHI 2010. Sheryl Ehrlich, Karen Cross, Wilson Chan, Darcey Imm. “More for Less: A Novel Hybrid Method to Maximize the Impact of Research,” Ext. Abstracts CHI 2003. Dave Yeats & Locke Carter. “The Role of the Highlights Video in Usability Testing: Rhetorical and Generic Expectations,” Technical Communication, May 2005: pp. 156-162(7). Amy Schade. 2013. “Making Usability Findings Actionable: 5 Tips for Writing Better Reports.” Optional Jakob Nielsen. 2005. “Formal Usability Reports vs. Quick Findings.” Chauncy E. Wilson & Kara P. Coyne. 2001. Tracking Usability Issues: To Bug or Not to Bug?, ACM interactions. Extend topic: Technical choices that affect usability Toby Biddle. Site Speed and Usability
9	1 March	Guest lecture: (Tentative - TBD) Reporting Examples Usability in the field, and moving forward What are the benefits & challenges of conducting usability activities in the field? What are other tools and approaches?	Required Erin Friess, 2010. “The Sword of Data: Does Human-Centered Design Fulfill its Rhetorical Responsibility?” Design Issues 26(3). Greenberg & Buxton, “Usability Evaluation Considered Harmful (Some of the Time),” CHI 2008. Extend topic: Biometrics Jeremy Kahn "Expedia Reads Your Mind (and Face) to Beat Rivals" (Video)
10	8 March	Course wrap up & Presentations	Optional Janice Redish. 1995. “Are we really entering a post-usability era?” SIGDOC Asterisk J. Comput. Doc. 19(1): 18-24. James Kelway. “Usability is dead.” N Tractinsky, AS Katz, D Ikar. 2000. “What is beautiful is usable,” Interacting with Computers 13(2): 127-145.	P5a. Presentation P5b. Report Self & Team Evaluations

Acknowledgements

In this version of HCDE517, I extensively draw from and/on materials developed by Rebecca Destello, Andy Davidson, and Judy Ramey.