PROJECT INTRO & OUTCOME

I designed a SaaS Scorecard application while interning at VMware. The SaaS Scorecard was aimed to solve the problem that VPs at VMware didn't have a tool to get a quick glance of SaaS health. Since VMware is aggressively heading to SaaS business, it has become a high-priority need. 

I conducted this project independently, under the mentorship of two designers. I have completed 4 design iterations, and came up with a high-fidelity prototype. VPs loved the product and decided to ship it immediately.

Time  Jun 2016 - Sept 2016 (12 weeks)                      Role  User Researcher & Interaction Designer

The Problem

"I'm not able to quickly know the overall health of SaaS products."

Paul, the target user of this project, is a VP from VMware. The company is aggressively heading to SaaS industry. However, as a VP, Paul is not able to quickly know how the SaaS products are running. He has met two major problems:

• PROBLEM 1:  There isn’t a single tool that provides an integrated view of how SaaS products are performing for VPs like Paul.          
• PROBLEM 2: Currently the reports are manually created, so there is time lag on reporting.

Paul wants an iPad Application that could help him to quickly know the overall health of SaaS products.

The process

To address the design challenge, there were two main questions to answer:

• How to measure overall SaaS health?  --> Competitor Analysis
• What information/data will be useful for VPs to view on the Scorecard?  --> User Interview

DOMAIN RESEARCH FINDINGS

Competitor Analysis

This list contains current SaaS analysis/monitoring tools. However, they are not able to solve the target user--Paul's problem. 

1. None of the products provides a comprehensive view of how SaaS products perform. 
2. They target specifically on finance managers or ops managers, and provide detailed metrics or real-time monitoring data. 

User Research Insights

User Scenarios: When will Paul use the product?

User Scenario I: Daily Check

• Mobility and visibility
• Track possible changes

“We made a decision during last weekly meeting, can I see that change happening? Is someone taking care of it?”

User Scenario II: Weekly Health Meeting

• Convey clear information 
• Make important decisions 
• Review the influence 

“I'm seeing something I like, how can we increase that? I'm seeing some issues, what's the plan to address it?”

User Scenario III: Quarterly Board Meeting

• Message up
• Comprehensive view

“How is the business running?”

Verifying assumptions: Does it have to be a scorecard ipad application?

Scorecard is different from traditional dashboard in that Scorecard doesn't show the actual numbers for each metrics. Instead, it shows unified scores ranging from 1-100. It's similar to the credit score metaphor, which is calculated based on various factors and ends up in one score.

According to user interview, VPs need to take care of multiple SaaS products at the same time. They care about the overall health of each SaaS, as well as how the health status changes over time. They want high-level data without spending time going through detailed metrics.

A Scorecard application maintains one overall health score for each SaaS product, which makes it easier and faster for VPs to gain an overall picture of SaaS health. Comparing to a dashboard with all SaaS metrics, a scorecard will be better. Besides, it will also be easier to compare the changes of the overall performance since it's just one score, rather than various metrics. 

Use cases: How will Paul use the SaaS scorecard?

I summarized a use case table to describe how will target users use the product. Priority ratings were used for engineers to determine what features to develop first.

Design ideation

The Challenge: HOW TO CALCULATE THE GENERAL HEALTH SCORE?

Through user research, I verified that the "scores" matched target users' requirements. Based on competitor analysis, the SaaS metrics can be categorized into Customer, Finance, Marketing and Operation. There are various metrics under each category. I selected 4-6 key metrics under each category to come up with the Customer Score, Finance Score, Marketing Score and Operation Score. The General Health Score can be calculated based on the 4 sub scores. 

The weight of each metrics can be decided based on the company strategy. Thus, the importance of certain metrics will be reflected in the scores.

information architecture

design iterations

I went through 4 design iterations, and each iteration took about one week. 

design decision 1: the scorecard on homepage

The scorecard on Homepage shows general SaaS health information and helps users quickly know what's going on. The scorecard should be clean and insightful.

• Users felt they couldn't get a lot of useful information from the color bar, and it was too eye-catchy. Thus, I removed the color bar.
• Users hoped to be able to quickly check the change of general score on Homepage. Thus, I added the line chart showing general score trend on each card.
• After adding the general score trend, users still felt it was hard to interpret or gain insightful information from the trend line. The problem is that a single line without any measurement or comparison is hard to build contexts. Thus, I added the general score goal as a dash line in the trend line chart. After that, users could better understand how well the SaaS products were running by comparing with goals.

Design decision 2: the global navigation

Global navigation was for users to quickly switch among different SaaS products without jumping back to homepage. In the previous version, the global navigation was fixed on the left side of each page. In the final version, I put the global navigation into the hamburger menu.

• During the user test, all users were quite familiar with hamburger menu and its interaction patterns. They didn't have problems finding the navigation.
• It’s not likely that users will frequently switch from one SaaS to another, since different SaaS products are not comparable according to user interview. Therefore, it won't be a problem if we fold the global navigation into the hamburger menu instead of constantly showing the navigation on the page.
• The hamburger menu can be folded, so it saves a lot of space. There will be more space for charts and diagrams.

DESIGN DECISION 3: the metrics overview

The Metrics Overview was a new master card that I added on the single product Overview page during the final design iteration.

• During the usability test, the most frequent question users asked was how to calculate the general health score. It was difficult for them to trust the score without knowing the low-level metrics. Therefore, I thought of this way to show all the metrics under each category, so that users can view them at one place. It also provides the color-coded information of whether each metrics has become better or worse. Users can pay attention to those metrics that are in trouble. The feedback I got from user test was pretty positive. Users thought this master card was pretty insightful for them.

FINAL Hi-fidelity Prototyope

View Interactive Prototype on InVision

Results

After seeing the SaaS Scorecard prototype, VPs at VMware wanted it immediately. A group of engineers have been assigned to this project so that it can be shipped before the end of this year.

Gallery: Ideation Process Sketches

Whiteboard Ideation

User Flow

Low-fidelity Prototype

Poster

We had a poster presentation among all the VMware interns in this summer. I designed this poster for the presentation and it got 80 votes.

Project Reflection

This is my first Interaction Design Internship, and I have kept learning from my mentors and peer designers. The most important things I have learned include presentation skills, agile design skills and dashboard design experience.

Presentation skill is the core skill every designer should have. During my internship, I have presented to target users, peer designers, and stakeholders who are not designers. I learned to tailor content based on my audience. While preparing slides and speech, I always tell myself to think whether my audiences will be interested in this topic, what my audiences want to take away from my presentation, whether my audiences are able to understand it, and whether the logic behind the whole story is correct.

I also practiced agile design skills. I have run 4 design iterations in 1 month. I spent the first 3 days in a week designing the prototype and spent the rest days in that week running heuristic evaluation or usability test. This method turns out to be very effective. I was able to quickly evolve my design based on users feedback and constantly find out usability issues.

I have got some experience designing dashboard through this project as well. The key point I realize is to be careful while making visualization decisions. Pay attention to the priority of information to be shown on the dashboard. Do a lot of user research to know how users interpret it. Be careful of the usage of color and legend.

It was definitely a fun and meaningful summer for me.

Thank you for reading.

Qin

photo with our ceo

MORE PROJECTS