Usability Evaluation
Discover usability issues with a social-networking app prior to product launch
📌 PROJECT SCOPE
Client: POP, a local startup
Timeframe: 12 weeks
My Role: UX Researcher
Team: Mia Eltiste, Metinee Ding, Ayu Larasati, Janani Ravikumar
Methods: Usability Testing, Prototyping, Heuristic Evaluation, Competitive Analysis, Subject-Matter-Expert Interview
Tools: Figma, Miro, Airtable, Zoom
Project Overview
🚀 CLIENT KICKOFF
The POP app facilitates friendships based on shared interests for University students. POP’s long-term goal was to improve the mental health of university students by providing meaningful social connections.
🔎 OBJECTIVES
Understand the pain points of students and their social network at college
Assess the usability and user experience of the POP app
Prioritize recommendations so POP can address users’ deepest pain points
✏️ NOTES
At the time of our assessment, POP had a working prototype in Figma where we conducted our heuristic evaluation as a team. Shortly after, POP released their app on TestFlight and has since launched at the University of Texas at Austin.
Methodology
Heuristic Evaluation
Competitive Analysis
User Interviews
Affinity Diagram
Data Analysis
📋 Heuristic Evaluation
The project started off with the startup’s Founder coming to speak to our class. They had already designed out all their screens and had a working prototype in Figma. Our class started with a Heuristic Evaluation so we could hand off some things to work on while the rest of the project progressed.
💡 UX tip: A heuristic evaluation is a great way to grab low-hanging fruit and identify predictable usability issues. You can use Nielsen’s 10 Heuristics for Interface Design to assess user interface (UI) interactions, but don’t let that limit you. These are considered “rules of thumb” because they can be seen again and again in UI designs. If a particular finding doesn’t fit into a category that doesn’t mean it won’t be an issue — it will just require more research on your part to back it up.
We evaluated two different “sections” of POP’s app:
The onboarding process that every user will go through one time to create their account
The matchmaking and chatting process that users will go through repeatedly as they interact with the app
Our team went through the prototype together and took notes using Airtable so we could tag notes by the possible 10 heuristics violated and mark the level of severity.
One other team member and I documented all the findings into one slide deck. We included the relevant screen, what heuristic(s) it violated, the severity of the issue, and our team’s recommended changes.
Heuristic evaluation findings
The most heuristic violations were:
Visibility of system status (29 issues)
Consistency and standards (18 issues)
User control and freedom (18 issues).
1. Keep users informed
POP should provide feedback to clear up issues and help users know they’re on the right track.
POP should have very clear indicators around their most important tasks (getting a match and messaging a match) otherwise users may stay confused and abandon the app.
We recommend POP add features like a progress bar and confirmation feedback to keep users aware of what’s going on.
Why? Humans like to be in the know. Predictable actions and providing confirmation can help build trust between the user and an app.
Heuristic referenced: Visibility of system status
2. Give users control
POP should provide users the option to quickly fix mistakes or make changes later.
POP should allow users to change their mind (un-matching with someone) at any time of a user’s interaction.
We also recommend POP allow users to edit as much of their profile as possible, including avatars and codenames.
Why? Humans make mistakes or change their minds. When an app doesn’t allow changes, users can feel trapped and dissatisfied.
Heuristic referenced: User control and freedom
3. Be consistent
Especially as users are learning a new app, it’s important to be consistent so users know what to expect and feel safe to explore.
POP should follow industry conventions with match-making as many of its target users are familiar with apps like Tinder and Bumble.
We recommend POP use a similar information hierarchy on profiles and only allow messaging after matching.
Why? Humans prefer not to work. When an app doesn’t follow what users expect, it can be hard to learn or a barrier to use.
Heuristic referenced: Consistency and standards
🏅 Competitive Analysis
Our team looked at typical matchmaking apps like Bumble. At the time of the study, they were the only dating app offering platonic connections. We compared POP, Bumble, Meetup, Friender, Nextdoor, and Meet my dog.
Competitive analysis findings
The competitive analysis showed POP had a unique proposition but a very similar idea was previously tried and saw limited success. Research on the desirability of the app and assessing product-market fit should be major priorities.
1. Exclusive membership
POP stands apart in its membership being exclusive to university students
Both direct and indirect competitors had an open community approach
Nextdoor has seen success by keeping things exclusive based on neighborhood and POP could see similar benefits
2. Anonymous profiles
Anonymity was the biggest differentiator across all competitors but may bring novel issues
Anonymity increases the cognitive load users will need for remembering information about matches
Anonymity has effects on people’s behaviors and should look into especially around how individuals will be treated online and safety for offline meeting
3. Friender tried it already
Friender would be a direct competitor, but it seems the company is no longer active
Friender is the most similar to POP, encouraging platonic relationships, without anonymity
Ultimately, Friender was not successful which may be indicative of POP’s low product-market fit
With social network businesses, the company’s value is proportional to the network of users and if there are already companies out there with the networks built, users may not be motivated to abandon a product with far more users
👥 User Interviews
💡 UX tip: User interviews are one of the most powerful research methods companies can do. You don’t need a ton of participants or resources to gain insights, as long as you know how to ask the right questions. A UX research expert (like myself) knows how to avoid priming respondents, dig deeper into “why,” and avoid their own confirmation bias.
Methodology
Screener - We selected a diverse group of individuals that varied in their age (freshmen-seniors), their residency (in & out - of state, international students), and transfer students
Remote Moderation - This project was started in January 2020 and so our team had to move our interviews to be conducted online using Zoom and Figma prototype
The Process
We chose six tasks for users to do during the usability test
Since we were a team of four, we all covered four sessions, two moderating and two sessions of note-taking
We created an interview script, a deck for users to reference tasks and likert scale choices, and a codebook for the notetaker
🗂️ Affinity Diagram
💡 UX tip: Affinity diagrams help you sort through all the data you’ve collected. I like to think of affinity diagramming as finding the descriptive statistics (like the average) of qualitative data. The process helps you pull out themes from research without hard-coding qualitative responses that can take more time (and isn’t as necessary of practice in industry as it is in academia).
The Process
We grouped things by which task/screen it was associated with. I made sure each sticky referenced the task and participant id before importing them to miro
Within each task, we grouped things by their impact: positive, negative, and neutral.
After discovering themes across multiple interviews, we worked together to synthesize the data and agree on insights. From these insights, we created our first suggestions that would later become more polished recommendations for the POP team.
💡 UX tip: Make a codebook in Excel first. Splice together comments and the participant/task id using the concatenate formula. Color-code in Excel, and then import to Miro.
📊 Data Analysis
The Findings
POP was a fairly usable app since it was pretty similar to matchmaking apps. Since POP’s target user was young university students, most were pretty advanced tech users.
Users struggled most with new features that were different from other matchmaking app conventions.
Usability and user interface issues were minor inconveniences for users. Users struggled with the purpose of the app.
💥 Recommendations
Overall project findings
POP was a fairly usable app
Some novel issues may come up with features like anonymous profiles
1. Conduct further research on the desirability of POP
Direct further research on desirability over usability - a perfectly usable app does no good if users don’t want to use it
Focus on pain points of users to assess product-market fit
Explore further revenue stream - the liability universities would need to take on may keep them from partnering officially with POP
2. Design for the effects of anonymity
Allow users to create meaning and uniqueness in their profiles to encourage an emotional connection to the app
Lower the effort needed to make and keep connections with other users
Reduce the users’ cognitive load associated with anonymous profiles to keep them engaged
3. Follow industry standards
Provide confirmation screens for user actions to reduce confusion
Avoid information overload by reducing content displayed to users
Refer to conventional patterns of feature placement to enhance learnability
4. Include a comprehensive onboarding experience
Comprehensive onboarding should clarify issues users had with anonymity and privacy
Highlight differentiators to increase user engagement and desirability of POP
💭 Reflection
Industry Takeaways
POP was a fairly usable app so our recommendations to improve the user experience felt nonessential. The problems we discovered during our user interviews were rooted in problems with the desirability of the app and not understanding users’ real pain points.
This project demonstrated the limitations of evaluative research when generative research was not done to assess the initial product-market fit.
Personal Takeaways
Coming from a startup background, I recognized some problems with POP’s app that are traditionally outside of a UX researcher’s scope. This required me to balance helpful insights that weren’t strictly UX-related.
I gained experience delivering what the client asked for while still including appropriate findings outside of the requested project deliverables.