THIS CLIENT PROJECT WAS COMPLETED WITH A GROUP OF FELLOW HUMAN FACTORS GRADUATE STUDENTS AT BENTLEY UNIVERSITY
Vempathy is an online application for user testing that uses artificial intelligence to expedite usability research. The application can quickly create reports, automated highlight video reels of the participant experience, and track emotional responses in journey maps.
As a fairly new product in the industry, Vempathy wanted to conduct usability research to find out if their product resonates potential customers.
The scope of our study focused on whether their marketing website is effectively communicating their unique features, whether their pricing and package structure make sense to users, and the usability of the actual product, which involves setting up an account and working through the setup of a project trial.
Type of Research: Desktop platform Usability Testing :: Duration: 3 month
I was a member of the Bentley University research consulting team that designed and completed a research study to investigate whether Vempathy’s product and pricing would appeal to UX researchers.
I contributed to usability testing and created our presentation style ( i.e.consulting team logo). For research, I helped define research goal, design test plan and conducted usability testing (i.e.wrote heuristic evaluations, including competitive and design reviews. Conducting moderated and unmoderated remote testing.).
For this study, we conducted moderated, remote usability test sessions with 14 UX researchers using UserZoom software. The participants typically conduct 3-4 research studies a month, and none of them had any experience with Vempathy. Qualifying participants selected for this study included:
UX design and research practitioners (1),
UX researchers (10),
UX Research managers (3)
Industry experience ranging from senior level (5–10 years) and management level experience
1. HEURISTIC EVALUATION/EXPERT REVIEW
As a first step, our team conducted a heuristic evaluation, which is an inspection method for finding the usability problems in a user interface design (Nielsen and Molich, 1990; Nielsen 1994). Our team evaluated the marketing, pricing, and onboarding pages.
2. USABILITY TESTING
Next, we conducted usability testing to compare our heuristic/expert review findings with actual user observations. We conducted 14 remote usability sessions, each one a 60-minute think-aloud session. Each user was asked to explore Vempathy, set up an account, and create a project, all while sharing their feedback.
To conclude the usability testing, we used the System Usability Scale (SUS), a post-survey questionnaire administered at the end of a usability test session. The survey is a reflection of how study participants perceive the usability of the product (website, app, etc,) as a whole.
We used the following scale (inspired by Jeff Sauro’s severity scale) to prioritize the issues we identified:
Critical: Leads to task failure and extreme frustration.
Major: Occasional task failure for some users; causes the delay in completing tasks, irritation, and confusion.
Minor: Able to complete tasks; causes slight irritation and frustration; cosmetic.
Win: Successfully completed task, positive finding.
Observation: Observed behavior, interesting, could be something to research later.
There were a large number of findings, some very minor and easily fixed, and others that were more concerning. If you are interested in seeing the full report, you can download a pdf here. But for the sake of brevity, I’ll outline 3 selected global issues below.
First, there were a few issues related to the design of affordances on the website. An affordance in UX is a visual characteristic of an element on a website that lets a user know that this element can perform some function. For example, this underlined text suggests that it’s a link. So it can be pretty frustrating if you clicked on it and nothing happened. It doesn’t prevent you from continuing through this page, but you might have a sense that something is not working as planned — we rated this a minor issue.
Vempathy’s product had a few instances of this — for example, in the image below, the blue text above the input field looks like a link, but it was actually just static text:
On the other hand, links like “Start 30-day Free Trial” have no visual indicator to show that this text would behave differently from the rest of the static text in the price box:
And in other cases, the words “click to copy” were used to indicate that users could copy a link, but a button would have been a more suitable affordance:
Although we classified these examples as minor issues, there were enough of them that caused confusion to the test participants. On a positive note, they are all easily fixed.
At various points in their process, participants expressed concern about the overall value they would be getting from Vempathy. This finding was classified as an observation.
On the homepage, they wanted more information on the AI analysis, and on the pricing page, participants were confused about what exactly each package offered. We found that the researchers we tested with define projects in different ways — some of them use a blend of testing to make up one project, and others had a specific number of users they would want to test, and weren’t sure if there were restrictions on that as part of one project:
“I want to know the cost and if it’s actually worth it”
Vempathy’s package descriptions were not easily understood by our test users. Adding more detailed information would help potential customers understand exactly what would be included in each package, and prevent the need to contact Vempathy for sales support in order to make a decision.
AI and Emotional Analysis
The biggest issue we found was that many participants were somewhat skeptical about the effectiveness of AI.
The potential customers of Vempathy are UX researchers, who are an analytical group — they want to see how AI is used within the product, and understand exactly how it contributes to the automated reporting.
“I don’t expect artificial intelligence to be that good, but I’m
intrigued by emotional analysis.”
We also found that participants really wanted to understand the emotional analysis that Vempathy offers. The website offers a few graphics that try to communicate the value, but participants were looking for more information. They really want to see it in action:
"I want to see what the emotional analysis looks like. That’s
the sole reason I would use this tool, and I would like to see
how effective and insightful it is. To do this looks like I need
to run an entire test. I wish there was a way to “preview”
what an analysis looks like, before I run an entire test, or
even sign up for the trial.”
Some of the study participants were able to preview the reporting, and while they were very interested in the concept, they didn’t fully understand what they were seeing:
“I’d like to get more information about what these metrics
mean. I’m sure they’re interesting, I just don’t know what
“At the end of the day, you’ll need to translate this
information to the higher-ups.”
Vempathy is ahead of the market with their product, and therefore needs to build a bridge between people’s understanding of the current methods of research and what Vempathy can add to it. It was interesting to me that this was the biggest takeaway from our study, and yet it was not a usability issue. Nonetheless, it was an important concern to raise with the stakeholders of Vempathy since it could be a major barrier to the adoption of the product.