UX Research | Usability Testing | HTC Creative Labs 'Viveport'

Team Members: Omari Stringer, Zimu Guo, Adrian Che, Vera Chen, Anmol Rajal Anubhai

My Role: Planned and prepared the usability study guide, recruited participants, conducted 'think aloud' sessions, conducted semi structured interviews, designed user feedback survey form, coded findings, prepared affinity diagrams, distilled insights and presented findings

Mentors: Prof. Sean Munson and Ms. Angela Sharer (Sr. UX Researcher @ HTC Creative Labs)

Location: HTC Creative Labs Office, Seattle

Purpose and Background 

The Viveport store creates an interactive and immersive environment for customers to preview new games and manage their VR game libraries. 

The goal of the study was to evaluate different scenarios in which users may browse the Viveport store and to identify any usability issues that they face when interacting with the store using VIVE VR equipment

We met Ms. Angela Sharer and discussed the key pain points that their team (Design & Engineering) face when interacting with the Viveport. After discussing the pain points with her as well as learning about the specific features that they are seeking feedback on, we decided to focus on the following key scenarios:

Scenarios

  • Browsing the Viveport store
  • Finding specific games that participants want
  • Making a purchase
  • Bookmarking a game (save for later)

Research Questions

We next decided to focus on a fixed set of research questions. We made sure that our research questions had a fixed scope while still allowing us to learn about and explore the different pain points that our set of scenarios might reveal. Below listed are our research questions:

  • What usability issues do users encounter in the Viveport store?
  • Can users successfully navigate VR scene previews to purchase a game?
  • What are common errors? What is the error rate?
  • Where do people struggle in the purchasing process?
  • What do users think of the VR game previews?
  • Can users successfully use interface components such as ‘filters’ to view different categories of games?
  • How do users feel about shopping in Viveport VR store as compared to purchasing games on the Viveport website?

Information Architecture Map

Before designing our tasks and deciding our research methods, it was crucial for us to gain a complete and thorough understanding of the Viveport. We studied Viveport and designed an information architecture map of the same. 

(That's my team mate Omari on the right, engaging in a think - aloud session and sharing his experience with us!)

The information architecture map that we designed is shown below: 

Recruitment

Next, we discussed our recruitment plan with our stakeholders to ensure that our plan was respectful of the resources that they had set aside for this project. I like to make sure that my stakeholders and team members are on the same page at every step. It is important for us as a team to understand as well as respect resource constraints. 

Our recruitment guide had three key considerations (as mentioned on the right).

Designing the Study Guide

We had to next design an effective set of methods that would help us understand our participant's experience as well as the problems that they face while interacting with the VR Space.

Challenges

We were facing a number of challenges when designing a usability study guide for this VR space. The main challenge was us not being able to see our participant's experience in the VR space in real time. Thus, we realized that methods like ethnographic study were not going to help us. After studying a variety of methods, we decided to use task analysis, think - aloud sessions, pre study interviews, post study interviews and feedback survey to understand our participant's experience in great depth.

Think Aloud Sessions

Verbalization has three levels. Levels 1 and 2 focus on the participant's short term memory and only require them to verbalize their current activities while Level 3 makes use of their long term memory. Level 3 verbalization asks the participant to connect their responses with past experiences. We decided to ask our participants to engage in Level 1 and 2 verbalization.

There are two methods of conducting think aloud sessions: 1) Classical 2) Relaxed. The Classical method requires the researcher to be very brief in his responses and prodding. Since this is a relatively new form of technology, our participants might need more support/help, we thought that a relaxed method would be more helpful.

Open and Close Ended Interviews

We also decided to conduct a pre-study interview and a post-study interview with more open-ended questions. The pre-study interview would introduce the researchers, the purpose of the study, and some basic background questions on participants’ pre-existing knowledge. The key goal was to familiarize the participants with the study.

The post-study interview would focus on asking more in-depth questions. The goal was to study aspects of the participants’ experiences with Viveport such as their levels of comfort, ease of use, and overall impression. We also wanted to answer questions that our participants might have for us.

Surveys

We decided to use a quick post-study survey to acquire some quantitative data. This data would help us accurately measure parameters such as comfort, entertainment value and ease of navigation.

A combination of methods would allow us to answer our main research questions comprehensively and triangulate any specific usability issues with the store as well as gather participant views about the Viveport store. 

A full list of our data collection plan can be found here: https://goo.gl/GziaUu

Pre - Test Interview Questions

These are the questions that we asked our participants before beginning the study. The goal was to put them at ease.

  • What is your favorite genre of game? (action, educational, puzzle, music)?
  • When is the last time you played a VR game?
  • Do you own a VR system?

Post Task Questionnaire

Below listed are our post - study survey questions:

  • Rate your level of comfort using Viveport: 1 - Extreme Discomfort, 2 - Discomfort, 3 - Some discomfort at particular points, 4 - Comfort, 5 - Extreme Comfort
  • Did you find Viveport easy to use and navigate? Yes/No. Please state your reason
  • Would you recommend Viveport to your friends?
  • Did you understand the terminology used in Viveport? For example: What did the different genres in filters or sort types mean? Yes/No. Please state your reason
  • How was your overall experience: [Rate on a scale of 5]
  • Was Viveport efficient to use? Could you accomplish tasks such as buying games quickly and easily in viveport? Yes/No with reason.

Post-Test Interview

Below listed are our post study interview questions that we used to learn more about their experience:

  • What did you encounter as your biggest difficulty?
  • Was it easy to learn or start using how to navigate the system?
  • What worked well
  • What didn’t work well
  • Were there any moments of discomfort? If so, when?
  • What was your overall experience with this system?
  • Would the VR previews motivate you to use the Viveport store?
  • Would you use the Viveport store to purchase games?

3.7 | Task Analysis

We designed four key tasks that would engage our participants with the different features of the Viveport and in turn also help us study their experience using each. Each task was designed so that it would have a specific end goal while still allowing the participant to explore the virtual space.

Sort games using different filters

  • Task: Now you wish to see more games. We would like you to use the different search options to sort and play the “view” or “preview” for the games. a) Please sort games by price b) Please sort games by recommendations.
  • Goal: To test the effectiveness of using filters and  the “sort” feature.

Bookmark

  • Task: You seem to really enjoy a game and would like to bookmark it for future reference. How would you do so?
  • Goal: To test if the participants are able to add games to ‘bookmark’ and find their saved games.

Find the “Cat Sorter”

  • Task: Use the filters to search for the game “Cat Sorter”.
  • Goal: To test the effectiveness of the “filters” feature

Preview

  • Task: Enter the game preview and play the game.
  • Goal: 1) To check whether the participants are enjoying the previews, and whether they realize the difference between the preview of a game and the game; 2) To see if the previews can motivate the participants to buy the game.

Purchase

  • Task: You now wish to buy this game using your preset credit card. How would you do so?
  • Goal: To test if the purchase flow is intuitive enough for participants to make the purchase successfully.


Environment, Equipment, & Logistics

The study was conducted at the HTC Lab in Pioneer Square, Seattle. The room was equipped with Lighthouses (infrared cameras for VIVE system), the VIVE head mounted display, controllers, computer, associated equipment, and plenty of space for participants to freely move around. We also used equipment such as microphone, camera, and screen capture software to record our study sessions. 

Facilitation Approach | Empathizing with our participants

Our facilitation approach was to try and put our participants at ease and ensure that they are in a relaxed environment. Using a relaxed Thinking Aloud protocol, we intended to help the participants in cases when they might feel that they have reached roadblocks. We realized that since the Viveport is a new form of technology, it might cause some levels of discomfort to certain participants or the VR experience might prove to be intimidating for some. Hence, we decided to start with an open ended interview in order to put our participants at ease. The open-ended interview also aimed at exploring all the possible explanations for certain user behavior patterns.

Affinity Diagramming and Data Analysis

We used our video recordings to go over all our study sessions in depth. We took detailed notes of participant quotes, feedback and problems that they had conveyed while going through the tasks. We noted them down on separate 'post it notes'. We then designed a coding guide in order to analyze our findings and note any trends across our notes. We then made an affinity diagram using the same guide. Our codes covered topics such as themes, emotions, and key pain points conveyed. 

Example:

For example, when exploring the preview of a game, one of the participant said “Wow, this is awesome.” It was a positive opinion (P) while using the Preview feature (3). The participant was happy, thus the emotion detected was marked as a lowercase 'a'. So we marked this quote as P3a. 

Usability Test Results 

Next we present to you some of our findings. We have divided our findings into the following four categories based on the information architecture of the store:

  • VR Previews
  • Filter/Sort functionality
  • Navigation & Viveport Experience
  • Bookmarks

We have also rated our findings on a severity scale of 4. Our scale is as follows: 

  • Level 1: Cosmetic
  • Level 2: Moderate
  • Level 3: Major
  • Level 4: Critical

In the next section, I take you through some of our key findings. 

VR Previews | Mental and Conceptual Model Mismatch

The 'VR previews' is a unique feature that is provided by the Viveport. Using this feature, participants can play a game first and then decide if they would like to buy it or not. 

One problem that was noted was that the 'Gulf of Evaluation' was too wide. I say this because 60% participants expressed how they couldn't fully comprehend all the possible interactions that were possible in a preview. They thought that they could only experience the environment of a game using the preview feature. Thus, their mental model of a preview did not match the preview's actual conceptual model. No 'help' section was provided as well.

Severity Rating: 2

No. of participants who faced the issue: 60%

User Quotes:
“I have very little context for what a VR Preview means. Is it the environment or the things that I interact with?”
“Am I looking up reviews?


VR Previews | Discoverability

Participants were unable to locate the preview button.

Severity Rating: 2

No. of participants who faced the issue: 20%

User Quotes:
“For the game that I am viewing, I can’t see the preview button somehow!”


VR Previews | Joy

Participants enjoyed being able to test a game before deciding if they would like to purchase it or not. They said that this was particularly useful for a new space such as VR.

No. of participants: 60%

User Quotes:
“Basically I wanna try before I buy, right.”
“Wow, this is awesome.”
“Now we’re in the Cat Sorter universe. How delightful!”
“Oh, so now I find the progress bar”
“The VR preview was really cool”
“I mean I won’t buy a 10 dollar game without experiencing it!”
“Definitely a preview over just a video of the game. Since VR is so new I think it is important!”

VR Previews | Interactions

Participants were unable to understand the type of interactions that were possible in a VR preview

Severity Rating: 3

No. of participants who faced the issue: 80%

User Quotes:
“Was that the preview that I was looking at?”
“I didn’t know I could walk around in this preview!” 
“How am I supposed to interact except of course looking around? Or is that the case with this game in particular?”
“Still not sure exactly what to do!” 
“No, it’s not working. I thought it was” Couldn’t figure out what objects can be interacted with in the previews”. 

Filters | Comprehension

Participants were unable to understand the filter labels and icons (Visual Design Language is difficult to decipher)

Severity Rating: 3

No. of participants who faced the issue: 40%

User Quotes:

“The filters were easy to find but the icons were ambiguous”

Filters | Information Architecture

Participants were unable to fully understand the information architecture because of issues such as redundancy. There was a 'Vocabulary problem' because of which they couldn't understand some of the options that the filter provided.

Severity Rating: 3

No. of participants who faced the issue: 60%

User Quotes:
“Interesting is if I point it here, it will show ‘Recommended’, if I point it here (another similar icon), there is nothing here.”
“This appears to be the only menu” (referring to filter)”
“Is that efficient? Is that as much as you can filter?”
“The ones that don’t have a price listed next to them, are those free?”
“Now I find a menu, that’s not intuitive” 

Filters | Discoverability

Participants were unable to spot the filters because of them being located on the very left of their periphery of vision.

Severity Rating: 3

No. of participants who faced the issue: 80%

User Quotes:
“That takes a little while to figure out”

Navigation & VR Environment | Legibility

Participants found it difficult to read the text in the VR environment.

Severity Rating: 2

No. of participants who faced the issue: 40%

User Quotes:
“I find it hard to read the text”
“There are some signifier or affordance that are not clear.”

Navigation & VR Experience | Layout

Severity Rating: 2

No. of participants who faced the issue: 40%

User Quotes:
“I’m not really crazy about that design” (referring to stack of videos on main content window)

Navigation & VR Experience | Language - Vocabulary Problem

Vocabulary Problem is when the phrases/words that the designer uses to refer to certain actions are very different from the words that the user tends to use to refer to the same actions.  

Severity Rating: 2

No. of participants who faced the issue: 60%

User Quotes:
“I don’t have a high level of trust because I’m seeing some duplication”
“This says download, I thought I can directly download for free. Now you say join now when you hover there, then I have to pay or select subscription.” 

Navigation & VR Experience | Animation

Severity Rating: 2

No. of participants who faced the issue: 80%

User Quotes:
“There is a lot to see here all of a sudden”
“When I select, it like bumps me through all the options instead of replacing and selecting it. That’s a little bit disorienting. It’s not so smooth”
“That jumping is really distracting” 

Navigation & VR Experience | Discoverability

Severity Rating: 2

No. of participants who faced the issue: 60%

User Quotes:
“I think having things left and right, I didn’t notice either side of my peripheral. I would say if they are in my (sight) periphery, I would know quicker”
“I am not sure how to find the genres!”


Navigation & VR Experience | Assistance

Severity Rating: 2

No. of participants who faced the issue: 80%

User Quotes:
“The four squares icon for the genres didn’t have anything say below it (tooltip) When you hover it should have some text.” 

Navigation & VR Experience | Experience Expectation Mismatch

Severity Rating: 3

No. of participants who faced the issue: 80%

User Quotes:
‘It’s a typical window interface just that it’s in 3D which didn’t seem very natural to me (store VR environment)’
‘It’s just steam in 3D literally and that doesn’t make sense to me! If you are going to make it in 3D come up with a new shopping experience.’


Navigation & VR Experience | Joy

User Quotes:

  • ‘Other VR experiences aren’t as interactive as this!’
  • ‘There are a surprising number of games here!’
  • ‘I think the store was fine as you just had to look a little left or right to find the options.’

Bookmarks | Discoverability

Severity Rating: 3

No. of participants who faced the issue: 20%

User Quotes:
“So I assume that the game has just been added already or….?” (After clicking the yellow bookmark button)”

Post-Task Survey & Interview Results

  • 60% of participants did not understand the icons used in the Viveport Store
  • 40% of the participants would recommend Viveport to a friend
  • Generally rated a positive and comfortable experience
  • 60% of participants said the store was efficient to use (purchase flow)
  • People like the previews, do not understand how to interact with them

Recommendations

Below listed are some of the recommendations that we made in order to address the issues that the participants were facing.

Recommendations | VR Previews

Standardize or create guidelines for all VR Previews

  • Create fixed standards for all elements of the previews to ensure ease of comprehension

Create a VR Preview icon on each game thumbnail

  • Users should be able to learn if a game has VR Preview or not by just looking at the game thumbnail

Recommendations | Filter & Sort

Enhance the Filter & Sort menu

  • Add tooltips or more noticeable iconography for discoverability
  • Add more granular options in filter settings
  • Combine the filter and sort menus 

Recommendations | VR Environment

Reduce Motion Speed

  • Reduce/Remove the “bounce” animation when selecting titles

Enhance Visibility of Wallet Balance

  • Show the wallet balance on-screen in the 'profile information' section
  • Help the user establish his budget while browsing

User Personalization

  • Allow users to customize store background / ambient environment
Copyright © All rights reserved.
Using Format