HCDE517: Usability Analysis - Schedule - W23

• Carey Goldberg. “Design Flaws In Electronic Health Records Can Harm Patients, Study Finds,” WBUR.
• W Craig Tomlin. “UX Killed Usability.”

Overview from the books (required)

• Tullis & Albert, “Introduction”
• Rubin & Chisnell, Chapters 1 -3

Usability evaluations - Required

• Jeff Sauro, “What's the Difference between A Heuristic Evaluation and a Cognitive Walkthrough?”
• Stephanie Rosenbaum, “Usability Evaluations Versus Usability Testing: When and Why?”
• Jakob Nielsen, “Usability Heuristics” (ch5), Usability Engineering.

Guidelines for usability - Optional

• John D Gould & Clayton Lewis, “Designing for Usability: Key Principles and What Designers Think.”
• Research-Based Web Design & Usability Guidelines, from Usability.gov.

Extend topic: Heuristic evals for specific settings

• Menno de Jong & Thea van der Geest, “Characterizing Web Heuristics”
• Jennifer Mankoff, Anind K. Dey, Gary Hsieh, Julie Kientz, Scott Lederer, & Morgan Ames. 2003. Heuristic evaluation of ambient displays. CHI 2003.
• Enrico Bertini, Silvia Gabrielli, & Stephen Kimani. 2006. Appropriating and assessing heuristics for mobile computing. AVI '06.
• Raina Langevin, Ross Lordon, Thi Avrahami, Benjamin Cowan, Tad Hirsch, & Gary Hsieh. Heuristic Evaluation of Conversational Agents. CHI 2021.

Extend topic: Cognitive Walkthroughs in other Settings

Aaron R. Lyon, Jessica Coifman, Heather Cook, Erin McRee, Freda F. Liu, Kristy Ludwig, Shannon Dorsey, Kelly Koerner, Sean A. Munson & Elizabeth McCauley. The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability. Implementation Science Communications 2 (2021).

Planning for Usability Testing

• Required
  • Rubin & Chisnell, “Develop the Test Plan” (ch 5), “Set Up a Testing Environment” (ch 6), “Find and Select Participants” (ch 7)
  • Tullis & Albert, “Background” (ch 2)
• Optional
  • Steve Krug, “Usability Testing on 10 cents a day” (ch 9) from Don't Make Me Think: A Common Sense Approach to Web Usability. New Riders 2000.
  • Jakob Nielsen. (2007). “Fast, Cheap, and Good: Yes, You Can Have it All.”

Ethics

• Required: Code of Conduct of the User Experience Professionals Association (2013)
• Optional
  • Ethics of UX Research. 2014.
  • Ethics of Studying Digital Contexts. 2020.

Number of participants

• Required: Martin Schmettow. Sample size in usability studies. Communications of the ACM. 2012.
• Optional
  • Jeff Sauro. (2010). “Why You Only Need To Test With Five Users (Explained).”
  • Laura Faulkner. (2003). “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.”
  • Jakob Nielsen. (2000). “Why You Only Need to Test with 5 Users.”
  • Joe Dumas, “How Many Participants in a Usability Test is Enough?”

Extend topic: Accessibility

• What are the differences between universal design, accessibility, and inclusive design? SayYeah.
• Web Accessibility in Mind
• Petrie & Bevan. “The evaluation of accessibility, usability, and user experience” in The Universal Access Handbook.
• Reframing Accessibility for the Web
• Introduction to Web Accessibility
• Design for Autism

Usability test plans & roles

• Required
  • Rubin & Chisnell, “Skills for Test Moderators” (ch 4)
  • Tullis & Albert, “Planning a Usability Study” (ch 3)
• Optional
  • Susan Dray & David Siegel. “Penny-Wise, Pound-Wise: Making Smart Trade-offs in Planning Usability Studies.”
  • Jeff Sauro. “What goes into a good usability test plan?

Other methods, considerations about how many?

• Medlock MC, Wixon D, McGee M, & Welsh D. “The Rapid Iterative Test and Evaluation Method (RITE): Better Products in Less Time”
• Walji et al. “Are Three Methods Better Than One? A Comparative Assessment of Usability Evaluation Methods in an EHR?”, IJMI 2014.

Extend topic: Usability, children, and aging

• RL Franz, C Munteanu, BB Neves, R Baecker. Time to Retire Old Methodologies? Reflecting on Conducting Usability Evaluations with Older Adults. MobileHCI '15.
• J Nielsen. Seniors As Web Users.
• "Usability Testing with Kids and Teens", Usability.gov.

Extend topic: Usability and automation

• 99% Invisible: Children of the Magenta (Automation Paradox, pt. 1)
• 99% Invisible: Johnnycab (Automation Paradox, pt. 2)

Preparing test materials (required)

• Rubin & Chisnell, “Prepare Test Materials” (ch 8).

Metrics and measures

• Required
  • Tullis & Albert, “Performance Metrics” (ch 4), “Issues-based Metrics” (ch 5), “Self-reported Metrics” (ch 6), “Behavioral and Physiological Metrics” (ch 7)
  • Joe Dumas, “Usability Testing Methods: Subjective Measures,” in The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, 3^rd ed.
• Optional
  • Jeff Sauro & Joe Dumas. “Comparison of Three One-Question, Post-Task Usability Questionnaires,” CHI 2009.
  • Hyewon Suh, Nina Shahriaree, Eric B. Hekler, Julie A. Kientz. “Developing and Validating the User Burden Scale: A Tool for Assessing User Burden in Computing Systems” CHI 2016.

Extend topic: Remote testing

• Björn Völkers. A beginner’s guide to remote usability testing
• MS Andreasen, HV Nielsen, SO Schroder, J Stage. 2007. “What happened to remote usability testing?: an empirical study of three methods.” CHI 2007.
• Remote Moderated Usability Testing. Nielsen Norman Group.
• Moderated and Unmoderated Think Aloud Comparison. (2015)

Extend topic: The Hamburger Icon

Conducting test sessions

• Rubin & Chisnell, “Conduct the Test Sessions” (ch 9).
• Michael Hawley. “User Interview Techniques, Guidelines for Obtaining Better Results,” User Experience 8(3).
• Michael Margolis. “How to hack your body language for better interviews,” Design Staff.

Think Aloud Testing

• Required: Judy Ramey. “Methods for successful "Thinking Out Loud" procedures.”
• Optional
  • M. Ted Boren & Judy Ramey. (2000). “Thinking Aloud: Reconciling Theory and Practice,” IEEE Transactions on Professional Communication 43(3).
  • Mie Norgaard & Kasper Hornbaek, “What Do Usability Evaluators Do in Practice? An Explorative Study of Think-Aloud Testing,” DIS 2006.

Extend Topic: Invisible OSs, Recognition, and Recall

• R Budiu. Memory Recognition and Recall in User Interfaces Nielsen Norman Group.
• J Pavlus. "No UI" Design's Next Move: Fake UI. FastCompany
• Dan Norman and Bruce Tognazzini. How Apple Is Giving Design A Bad Name. FastCompany

Extend topic: Deception in UI/UX

• Press me! The buttons that lie to you BBC Future
• Adar & Teevan. Benevolent Deception in Human Computer Interaction. (2013)

Extend topic: Blank Page Technique and Paper Prototyping

• Still & Morris. (2010). “The Blank-Page Technique: Reinvigorating Paper Prototyping in Usability Testing,” IEEE Transactions on Professional Communication 53(2).
• Paper Prototyping is a Waste of Time
• The Magic of Paper Prototyping

Analyzing Data

• Required
  • Rubin & Chisnell, “Analyze Data and Observations” (ch 11).
  • Tullis & Albert, “Combined and Comparative Metrics” (ch 8).
  • Chauncey Wilson. “Analyzing and Reporting Usability Data,” STC Usability SIG Newsletter.
  • Michael Hughes. “Rigor in Usability Testing,” Technical Communication, Fourth Quarter (1999): 488-94.
• Optional
  • Jeff Sauro & James Lewis. “Correlations Among Prototypical Usability Metrics: Evidence for the Construct of Usability,” CHI 2009
  • Laurie Kanter, Deborah Sova, & Lori Anschuetz. “Organizing Qualitative Data from Lab and Field: Challenges and Methods,” UPA 2005.

Extend topic: Voice User Interfaces & Usability

• Voice Interaction UX: Brave New World...Same Old Story. Nielsen Norman Group.
• Is Voice UI the future of user experience design? UX Magazine

Extend topic: Large data in usability

• Walter C. Mankowski, Peter Bogunovich, Ali Shokoufandeh, & Dario D. Salvucci. “Finding Canonical Behaviors in User Protocols,” CHI 2009.
• A/B Testing
• Mary C. Burton, Joseph B. Walther. The Value of Web Log Data in Use-Based Design and Testing. Journal of Computer Mediated Communication 2006.

Reporting Results

• Required
  • Rubin & Chisnell, “Report Findings and Recommendations” (ch 12).
  • Dave Yeats & Locke Carter. “The Role of the Highlights Video in Usability Testing: Rhetorical and Generic Expectations,” Technical Communication, May 2005: pp. 156-162(7).
  • Amy Schade. 2013. “Making Usability Findings Actionable: 5 Tips for Writing Better Reports.”
• Optional
  • Jakob Nielsen. 2005. “Formal Usability Reports vs. Quick Findings.”
  • Chauncy E. Wilson & Kara P. Coyne. 2001. Tracking Usability Issues: To Bug or Not to Bug?, ACM interactions.

Extend topic: Additional Approaches

• Kerry Rodden, Hilary Hutchinson, & Xin Fu. “Measuring the User Experience on a Large Scale, ” CHI 2010.
• Sheryl Ehrlich, Karen Cross, Wilson Chan, Darcey Imm. “More for Less: A Novel Hybrid Method to Maximize the Impact of Research,” Ext. Abstracts CHI 2003.

Extend topic: Technical choices that affect usability

• Toby Biddle. Site Speed and Usability

Critiques

• Erin Friess, 2010. “The Sword of Data: Does Human-Centered Design Fulfill its Rhetorical Responsibility?” Design Issues 26(3).
• Greenberg & Buxton, “Usability Evaluation Considered Harmful (Some of the Time),” CHI 2008.

Extend topic: Biometrics

• Jeremy Kahn. "Expedia Reads Your Mind (and Face) to Beat Rivals."

Various takes on the state of usability over time (optional)

• Janice Redish. 1995. “Are we really entering a post-usability era?” SIGDOC Asterisk J. Comput. Doc. 19(1): 18-24.
• James Kelway. “Usability is dead.”
• N Tractinsky, AS Katz, D Ikar. 2000. “What is beautiful is usable,” Interacting with Computers 13(2): 127-145.