Usability Test Objectives

Our client, Terrence Nevins, is the founder and CEO of Royal Cause. The overall goal for Royal Cause is for users to engage with the mobile application for three reasons:

1. To make small adjustments in their behavior to support causes they care about
2. To gain social recognition for making intentional choices to support their causes
3. To be part of an instant network of others that share their causes

Currently, Royal Cause mobile application is scheduled to go-live in Q1 of 2016. Before the go-live date, our client would like to kick start a feedback loop for measuring and improving usability. Terrence’s goal is to have the confidence to make changes that will have significant and real impact on improving user experience, based on third-party usability test results. 

The goal of this study was to evaluate the usability of the Royal Cause mobile application in order to provide action-based recommendations to drive adoption and application usage. As such, the focus of our testing was on onboarding and the ability of users to find businesses that support their cause(s) in the local community. This relies on the “Explore” feature, with the objectives that users will:

• successfully find the information they need to ‘execute’ an activity (e.g., find a location nearby, view business hours, etc.)
• have an intuitive search experience that produces actionable results
• see the relationship between businesses and their cause(s)

Interaction Map of the Mobile App

Creating interaction map enabled us to have an overall review of this app, while getting familiar with what features and information it had, especially knowing how users can go through all the interfaces and interact with the app.

Usability Test Plan

We planned 2 pilot studies and 8 formal usability tests. The results and feedback collected in pilot studies would be used to revise our test plan.

Through pilot studies, we found several issues in the test plan:

1. There is no clear stopping point for task 1, which may confuse participants and make the task difficulty to control.

2. Rating a task difficulty on a scale of 1-10 may put more pressure on participants because of the broad range.

3. The instruction about what the app does should be more concrete and consistent among participants so that they will have a basic knowledge about the app before tasks. 

According to the feedback, we revised the usability study plan and script, and printed it before tests.

Test Setting up

There are 4 people in the usability test room:

• The participant
• Primary Facilitator
• Secondary Facilitator
• Note Taker

The equipment include:

• one cameras to record the participant's cellphone screen
• one iPad to record the cellphone screen and connect to the TV, so that the screen will be projected on the TV
• one laptop for the secondary facilitator to take notes

Data Analysis

The goal of data analysis includes:

• Finding out information architecture issues of the app.
• Finding out interface design issues that may challenge or confuse users.
• Summarizing features that could be added to or deleted from the app.

The data we collected in the usability tests included notes, videos and radios. We analyzed them through affinity diagram. 

We also transcripted the data, coded it according to a pre-specified rule, and stored it in an Excel-based database. Inside the database, we classified data according to different sessions and different subtasks, and whether it was participants' comment or their action. We also noted the impact of each observation as positive or negative, low, moderate or high.

Recommendations

Based on the result of data analysis and affinity diagram, we summed up several recommendations for the product.

1. Onboarding process is too long

The on boarding process is actually not difficult, but too long and tedious for users. It has a lot of instructions and forces users to select based on their preferences even before they start using this app. This long process might make users feel impatient and quit the app during the onboarding process. At the end of signing up, there is no clear information showing that users have already completed the onboarding process.

Our recommendation is to allow users to tour the app without providing personal and preference information, and save some instructions when users actually start using corresponding features of the app. It's also necessary to provide a confirmation at the end of signing up, indicating users that they have completed onboarding.

2. Icons are quite confusing

The icons in the app are quite confusing: 1) some icons don't give clear intuition what will happen if users click them; 2) some icons fail to show affordance if they are clickable or responsive. For example, on the left screen, there is a purple person icon and a green person icon. Several participants had no idea what they mean and what's their differences. On the right screen, there is Impact Stats, below are 6 icons. A lot of participants tried to click those icons but found it's actually not responsive. However, these icons look similar to those clickable icons (just like the person icons on the left), which is very confusing.

Our recommendation is to re-design icons, and give more affordance to responsive icons. If necessary, combine icons with words.

3. Refine the search feature

Currently the app allows users to search for business and Nonprofit, but they have to search separately. Some participants indicated that current search function was not flexible enough.

Our recommendation is to combine business and nonprofit searching, while enabling users to search according to different causes.

4. Provide rich information on business and nonprofit

• The information about business and nonprofit provided by Royal Cause is quite limited, especially considering the amount of photos. Participants said they would like to see what the business or nonprofit is like by viewing its photos. They were also curious and wanted to know more details about how businesses are supporting their causes.

Our recommendation is to add photos to describe businesses and nonprofits, while providing detailed factual information or news about how the business supports nonprofits.

5. Other recommendations

Other usability issues we noticed and the corresponding recommendations include:

• Long loading time: it takes a long loading time to get search results or jump from one page to another, which may cause users to feel impatient and quit using this app. Therefore, it's helpful to improve the performance of the app and increase the response speed.
• Implement data policy or privacy policy, since currently we can't find such policy in the app.

View more details in our final usability test report.

Reflections

Through this usability testing project, we have got lots of takeaways:

• We finished 8 tests in 2 days, which was difficult, but it was a good way to keep the whole process consistent among different participants. Besides, we were also able to apply what we learned from last test into next few tests quickly.
• Before the test, we should pre-specify the note taking format, otherwise it will take a lot of time and efforts analyzing unstructured notes.
• Set expectations on change management and impact during test process. Our client updated several features of thee app during our test process, causing part of our test results less meaningful for them. However, continuous change on one app is quite common, especially for Royal Cause which is still in its development phase. What we need to do is communicating with the client and make sure the app won't update while we are actually conducting the test, so that we can ensure the app's look and features are the same among different participants.

Thanks for reading!

Qin

Team collaboration

Team members: 

Kristen Carter, Laura Eise, Qin Jiang, Drew Nevins

My Role: Moderator, Note taker

• developed the test plan and test kit with team members
• participated in 2 pilot tests and 8 formal usability tests, and served as note taker
• developed the final test report and presentation slides with team members

MORE PROJECTS