Project Description
You will design an experiment to evaluate and compare the effectiveness of two tools or processes that can be used to accomplish the same task, report how their usabilities and user experiences differ and suggest improvements. You will collect and analyze your own data, and create a visual presentation, individually or in pairs, to report your findings.
The presentation must contain at least one statistical model and at least 2 substantial visualizations. You will use the data and visualizations to support a hypothesis or argue for a solution to a problem. During the final weeks of this class, you will present your findings to the class. The presentation will be graded by both the instructor and your peers on its clarity, design aesthetics, and creativity.
Phase 0: Forming Groups (5 points)
Form groups of 2-3 people for this project. You and your group members will work closely both inside and outside of class on this project. If you would like your group members assigned to you, please reach out to the instructors.
What to submit: Names of the members in your group.
Phase I: Idea Proposal (5 points)
For this project, you will be designing an experiment to compare the usability or user experience of two products or processes that people can use to accomplish the same goals. To start, brainstorm with your group members which products you want to investigate and identify the goal your users want to accomplish.
Example (1) People often need to schedule meetings among multiple people. Busy schedules and different time zones of participants can make scheduling a difficult task. Tools like when2meet and Doodle help groups schedule meetings. We plan to compare the ease-of-use and user satisfaction of these two tools for this project.
Example (2) There are many social platforms for people to message their friends. Many people use multiple messaging platforms for different purposes. Tools like Discord, Facebook Messenger, and Slack are popular tools we use to communicate with friends and family. For this project, we propose to compare the frequency of use for these three messaging platforms amongst people in different age groups. We also investigate the type of activities or conversation topics people engage in when using these platforms.
What to submit: A paragraph describing what you plan to do for the project that includes the names of the products or processes you want to evaluate, and the user goal or task based on which you will evaluate the tools.
Note that you can still change your mind on what you want to study after submitting your proposal. The purpose of this exercise is to encourage you to brainstorm with your group early on in the process.
Phase IIA: Experiment Design (10 points)
It’s time to design your experiment. You will later recruit participants to participate in your experiment, so design the experiment in a way that makes it a pleasant experience for your participants. Also, to make the experiment more viable, design it so you can reliable recruit and collect data from approximately 20 people. For example, try not to design an experiment that has 10 conditions, because to obtain meaningful results, you would probably need a LOT of participants.
What kind of experiment do you plan to conduct? Will it be a survey or a usability test? How many conditions will you have? Do you plan to conduct a within-subject or between-subject experiment? For example, if you are comparing the usability between when2meet and Doodle, will you ask all your participants use both tools and compare their satisfaction (within-subject) or will you ask half of your participants to use when2meet, and the other half to use Doodle (between-subject)?
What are your independent and dependent variables? Please use at least 2 measures of usability as your dependent variable. One of them should be an objective measure such as the time the participant took to complete the task. One of them should be a subjective measure such as user satisfaction or frustration level. You can consult the NASA TLX for some examples of subjective measures (https://humansystems.arc.nasa.gov/groups/tlx/).
What to submit: (1) A summary of your experimental design, including the method (e.g., survey, interview, card sorting), the independent/dependent variables (e.g., time to complete a task, frustration level), the experimental conditions (e.g., how many conditions? Is it within- or between-subjects?), and the number of participants you plan to recruit.
(2) A description of your experimental procedure, including the questions you will ask participants and the order in which you will ask them. If you are asking participants to complete a task and observing them as they do it, describe how you plan to set the session up, and what metrics you will capture when you observe the session. If you are using Qualtrics or other survey platforms (e.g., Google Forms) for your experiment, submit a PDF (or other text documents) of your experiment.
Phase IIB: Revise Experiment Design (10 points)
Your instructors will provide you with feedback on your experiment design and procedure, just as how you might receive feedback on your work from your fellow UX researchers and developers in the real-world. Revise your experiment based on the feedback you received. This revised experiment should be launch-ready.
What to submit: A summary of the revisions you made to your experiment to address the corresponding feedback. Note any additional changes you’ve made to your experiment.
Example:
Feedback: Please omit the negative phrasing in Question #3 “What features did you not think is important when you are using the product?”
Revision: We have reframed the question to “Which of the following features do you find important?” to increase its clarity.
Phase IIIA: Data Collection Progress Report (5 points)
You will have time to have your classmates participate in your surveys and collect data from them in Week 10 (April 11). For this project, aim to collect data from around 20 people. We also expect you to recruit some additional participants from your social network (e.g., your friends outside class, family members, etc) to obtain a more representative sample.
What to submit: A progress report stating how many participants you’ve collected data from so far and how many more you plan to recruit. Describe how you will finish data collection by April 18th.
Phase IIIB: Submit Collected Data (5 points)
We will be learning how to visualize and statistically analyze the data you collected.
What to submit: Please submit your collected data in a CSV file. If you collected data via Qualtrics, you can download the CSV data files directly from Qualtrics.
Phase IV: Submit Presentation Draft (20 points)
Your presentation should include the following components:
- Your Research Question
- Description of Target Users and Goals
- Description of Experimental Design
- Description of Participant (Who did you recruit? What are some of their characteristics?)
- Description of Experimental Procedure (What did the participants do in your study? What questions did they answer? What tasks did they have to accomplish?)
- Two Substantial Data Visualizations
- Statistical Analysis of Results: Specify the type of test you conducted, and discuss the significance of findings in plain language.
- Description of Key Takeaways of Your Study
- Suggestions for Improving the Tool or Process
What to submit: A draft of your presentation including the 9 items described above. Powerpoint or PDF or any other preferred format is acceptable. The content of the 9 items can be in draft form (e.g., your visualization can be a sketch of the actual visualization, your statistical analysis can be a description of what you plan on doing, and the key takeaways can be notes or bullet form).
We will simply check that the 9 items are present in your draft. You will get full points for this submission if they all exist in one form or the other. We will not evaluate your submission based on quality, aesthetics, or accuracy. The purpose of this is to help you manage your time and progress toward the final project.
In Class Presentations (30 points)
You will present your project in class. Each group will be able to give a 5-minute presentation to the class on their project. (Expected to be timed. You will only have 5 minutes).
Component 1: Group Member Effort Evaluation (10/30 points)
On the day of the presentation, your group members will fill out a group-member-effort evaluation form to share their perspective of your contribution to the project with regards to your participation in developing ideas, your willingness to discuss with others, your ability to cooperate with other members in the group, your enthusiasm in the project, your effort accomplishing your share of the project, and your familiarity with the relevant course materials to the project.
Component 2: Instructor Evaluation (10/30 points)
On the day of the presentation, the instructor will also evaluate your presentation based on its clarity, design aesthetics, creativity, proper references to relevant in-class materials, and thoughtfulness.
Component 3: Peer Evaluation (10/30 points)
On the day of the presentation, your peers will also evaluate your presentation based on the same criteria as the instructor. Your peers will also be prompted to list one thing they liked about your presentation and one suggestion for improvement.
Note that you will submit your evaluation of your classmates’ presentations as Assignment 8, peer critique of presentations. Please write your evaluations thoughtfully.
Phase V: Submit Revised Presentation and Summary of Changes (10 points)
Based on the feedback you received from your peers and the instructors, make final revisions to your presentation and indicate points of improvement.
What to submit: Submit the final presentation along with a list of changes. The instructor will evaluate your presentation based on the accuracy and appropriateness of your content, as well as considering how you have addressed the suggestions for improvement.