UX Research | Usability Testing | HTC Creative Labs 'Viveport'

Team Members: Omari Stringer, Zimu Guo, Adrian Che, Vera Chen, Anmol Rajal Anubhai

My Role: Planned and prepared the usability study guide, recruited participants, conducted 'think aloud' sessions, conducted semi structured interviews, designed user feedback survey form, coded findings, prepared affinity diagrams, distilled insights and presented findings

Mentors: Prof. Sean Munson and Ms. Angela Sharer (Sr. UX Researcher @ HTC Creative Labs)

Location: HTC Creative Labs Office, Seattle

Purpose and Background 

The Viveport store creates an interactive and immersive environment for customers to preview new games and manage their VR game libraries. 

The goal of the study was to evaluate different scenarios in which users may browse the Viveport store and to identify any usability issues that they face when interacting with the store using VIVE VR equipment. 

We met Ms. Angela Sharer and discussed the key pain points that their team (Design & Engineering) face when interacting with the Viveport. After discussing the pain points with her as well as learning about the specific features that they are seeking feedback on, we decided to focus on the following key scenarios:

Scenarios

• Browsing the Viveport store

• Finding specific games that participants want

• Making a purchase

• Bookmarking a game (save for later)

Research Questions

We next decided to focus on a fixed set of research questions. We made sure that our research questions had a fixed scope while still allowing us to learn about and explore the different pain points that our set of scenarios might reveal. Below listed are our research questions:

• What usability issues do users encounter in the Viveport store?
  
• Can users successfully navigate VR scene previews to purchase a game?
  
• What are common errors? What is the error rate?
  
• Where do people struggle in the purchasing process?
  
• What do users think of the VR game previews?
  
• Can users successfully use interface components such as ‘filters’ to view different categories of games?
  
• How do users feel about shopping in Viveport VR store as compared to purchasing games on the Viveport website?

Designing the Study Guide

We had to next design an effective set of methods that would help us understand our participant's experience as well as the problems that they face while interacting with the VR Space.

Challenges

We were facing a number of challenges when designing a usability study guide for this VR space. The main challenge was us not being able to see our participant's experience in the VR space in real time. Thus, we realized that methods like ethnographic study were not going to help us. After studying a variety of methods, we decided to use task analysis, think - aloud sessions, pre study interviews, post study interviews and feedback survey to understand our participant's experience in great depth.

Think Aloud Sessions

Verbalization has three levels. Levels 1 and 2 focus on the participant's short term memory and only require them to verbalize their current activities while Level 3 makes use of their long term memory. Level 3 verbalization asks the participant to connect their responses with past experiences. We decided to ask our participants to engage in Level 1 and 2 verbalization.

There are two methods of conducting think aloud sessions: 1) Classical 2) Relaxed. The Classical method requires the researcher to be very brief in his responses and prodding. Since this is a relatively new form of technology, our participants might need more support/help, we thought that a relaxed method would be more helpful.

Open and Close Ended Interviews

We also decided to conduct a pre-study interview and a post-study interview with more open-ended questions. The pre-study interview would introduce the researchers, the purpose of the study, and some basic background questions on participants’ pre-existing knowledge. The key goal was to familiarize the participants with the study.

The post-study interview would focus on asking more in-depth questions. The goal was to study aspects of the participants’ experiences with Viveport such as their levels of comfort, ease of use, and overall impression. We also wanted to answer questions that our participants might have for us.

Surveys

We decided to use a quick post-study survey to acquire some quantitative data. This data would help us accurately measure parameters such as comfort, entertainment value and ease of navigation.

A combination of methods would allow us to answer our main research questions comprehensively and triangulate any specific usability issues with the store as well as gather participant views about the Viveport store. 

A full list of our data collection plan can be found here: https://goo.gl/GziaUu

Pre - Test Interview Questions

These are the questions that we asked our participants before beginning the study. The goal was to put them at ease.

• What is your favorite genre of game? (action, educational, puzzle, music)?
• When is the last time you played a VR game?
• Do you own a VR system?

Post Task Questionnaire

Below listed are our post - study survey questions:

• Rate your level of comfort using Viveport: 1 - Extreme Discomfort, 2 - Discomfort, 3 - Some discomfort at particular points, 4 - Comfort, 5 - Extreme Comfort
• Did you find Viveport easy to use and navigate? Yes/No. Please state your reason
• Would you recommend Viveport to your friends?
• Did you understand the terminology used in Viveport? For example: What did the different genres in filters or sort types mean? Yes/No. Please state your reason
• How was your overall experience: [Rate on a scale of 5]
• Was Viveport efficient to use? Could you accomplish tasks such as buying games quickly and easily in viveport? Yes/No with reason.

Post-Test Interview

Below listed are our post study interview questions that we used to learn more about their experience:

• What did you encounter as your biggest difficulty?
• Was it easy to learn or start using how to navigate the system?
• What worked well
• What didn’t work well
• Were there any moments of discomfort? If so, when?
• What was your overall experience with this system?
• Would the VR previews motivate you to use the Viveport store?
• Would you use the Viveport store to purchase games?

3.7 | Task Analysis

We designed four key tasks that would engage our participants with the different features of the Viveport and in turn also help us study their experience using each. Each task was designed so that it would have a specific end goal while still allowing the participant to explore the virtual space.

Sort games using different filters

• Task: Now you wish to see more games. We would like you to use the different search options to sort and play the “view” or “preview” for the games. a) Please sort games by price b) Please sort games by recommendations.
  
• Goal: To test the effectiveness of using filters and  the “sort” feature.
  

Bookmark

• Task: You seem to really enjoy a game and would like to bookmark it for future reference. How would you do so?
  
• Goal: To test if the participants are able to add games to ‘bookmark’ and find their saved games.
  

Find the “Cat Sorter”

• Task: Use the filters to search for the game “Cat Sorter”.
  
• Goal: To test the effectiveness of the “filters” feature
  

Preview

• Task: Enter the game preview and play the game.
  
• Goal: 1) To check whether the participants are enjoying the previews, and whether they realize the difference between the preview of a game and the game; 2) To see if the previews can motivate the participants to buy the game.
  

Purchase

• Task: You now wish to buy this game using your preset credit card. How would you do so?
  
• Goal: To test if the purchase flow is intuitive enough for participants to make the purchase successfully.
  

Environment, Equipment, & Logistics

The study was conducted at the HTC Lab in Pioneer Square, Seattle. The room was equipped with Lighthouses (infrared cameras for VIVE system), the VIVE head mounted display, controllers, computer, associated equipment, and plenty of space for participants to freely move around. We also used equipment such as microphone, camera, and screen capture software to record our study sessions. 

Facilitation Approach | Empathizing with our participants

Our facilitation approach was to try and put our participants at ease and ensure that they are in a relaxed environment. Using a relaxed Thinking Aloud protocol, we intended to help the participants in cases when they might feel that they have reached roadblocks. We realized that since the Viveport is a new form of technology, it might cause some levels of discomfort to certain participants or the VR experience might prove to be intimidating for some. Hence, we decided to start with an open ended interview in order to put our participants at ease. The open-ended interview also aimed at exploring all the possible explanations for certain user behavior patterns.

Usability Test Results 

Next we present to you some of our findings. We have divided our findings into the following four categories based on the information architecture of the store:

• VR Previews
  
• Filter/Sort functionality
  
• Navigation & Viveport Experience
  
• Bookmarks

We have also rated our findings on a severity scale of 4. Our scale is as follows: 

• Level 1: Cosmetic
  
• Level 2: Moderate
  
• Level 3: Major
  
• Level 4: Critical
  

In the next section, I take you through some of our key findings. 

VR Previews | Interactions

Participants were unable to understand the type of interactions that were possible in a VR preview

Severity Rating: 3

No. of participants who faced the issue: 80%

User Quotes:
“Was that the preview that I was looking at?”
“I didn’t know I could walk around in this preview!” 
“How am I supposed to interact except of course looking around? Or is that the case with this game in particular?”
“Still not sure exactly what to do!” 
“No, it’s not working. I thought it was” Couldn’t figure out what objects can be interacted with in the previews”. 

Navigation & VR Experience | Joy

User Quotes:

• ‘Other VR experiences aren’t as interactive as this!’
• ‘There are a surprising number of games here!’
• ‘I think the store was fine as you just had to look a little left or right to find the options.’

Post-Task Survey & Interview Results

• 60% of participants did not understand the icons used in the Viveport Store
• 40% of the participants would recommend Viveport to a friend
• Generally rated a positive and comfortable experience
• 60% of participants said the store was efficient to use (purchase flow)
• People like the previews, do not understand how to interact with them

Recommendations

Below listed are some of the recommendations that we made in order to address the issues that the participants were facing.

Recommendations | VR Previews

Standardize or create guidelines for all VR Previews

• Create fixed standards for all elements of the previews to ensure ease of comprehension
  

Create a VR Preview icon on each game thumbnail

• Users should be able to learn if a game has VR Preview or not by just looking at the game thumbnail

Recommendations | Filter & Sort

Enhance the Filter & Sort menu

• Add tooltips or more noticeable iconography for discoverability
  
• Add more granular options in filter settings
  
• Combine the filter and sort menus 

Recommendations | VR Environment

Reduce Motion Speed

• Reduce/Remove the “bounce” animation when selecting titles
  

Enhance Visibility of Wallet Balance

• Show the wallet balance on-screen in the 'profile information' section
  
• Help the user establish his budget while browsing
  

User Personalization

• Allow users to customize store background / ambient environment