Mixed Reality Design Labs & MRTK v1

May 2017 – Present, Ecosystem Experience Design | Analog | Microsoft

Background

When I first joined the HoloLens design team, I was excited about the new canvas and started learning how to build apps for HoloLens. As I go through the journey of creating Mixed Reality apps, I realized that there are no common control sets for the basic interactions and UI that are usually expected from the platform level or through SDKs. I found that a lot of developers and designers are experiencing similar pain in creating common mixed reality interaction patterns and user interfaces from scratch.

For example, a lot of developers were having trouble making a simple button work with proper visual feedback. Everyone wanted to make a beautiful button that looks like the one in the HoloLens OS shell. However, the only button UI that people could use was a 2D button provided by Unity. For the developers and designers coming from the 2D world, new elements such as collider, rigid body, shader, material, and z-depth were big challenges.

Because developers had to create their own version of the common interactions and interfaces (e.g. Gaze Cursor, Button, Bounding Box), it created inconsistent experiences in Mixed Reality apps as well. I realized that it is crucial to have common building blocks for foundational interactions and UI for both MR developers and consumers.

Role

I started researching crucial common UI controls that are mostly used in existing apps and requested by developers from the community. I organized them into visual graphics and presentations to help our team socialize the current situations and design challenges across the organization.

I worked with colleague technical designers to create the initial sets of foundational building blocks such as Interactable Object, Button, and Bounding Box. In this process, I helped the team to achieve visual and interaction quality that matches the Hololens OS shell through design direction, design integration, prototyping, and validation. I created example scenes to help our team experience and test out the features.

Helped the team to define and prioritize the list of the building blocks for the developers
Worked with technical designers to build extensible and modular building blocks that people can easily customize and infuse own brand identity
Created example scenes that demonstrate how to use the building blocks
Polished the visual and interaction design of the controls
Reviewed Pull Requests on GitHub for UX related items
Created documents that explain technical details and design guidelines
Created videos with Mixed Reality Capture to promote MRTK’s features
Engaged with the developer community through multiple channels including GitHub, Slack, and Mixed Reality Academy
Drove re-branding effort from HoloToolkit to MRTK

Output

Common UI controls

When we build the UI controls, we made sure to make them modular and extensible so that developers can easily customize and bring their own brand identity into their app experiences.

MRDL’s Holographic Button made with modular components

Publication on GitHub

We created the design team’s open-source repository ‘Mixed Reality Design Labs‘ on GitHub. This allowed us to publish experimental components and iterate fast.

Sample apps

To demonstrate the use of common UI controls and interaction building blocks, we created sample apps with a complete end-to-end experience.

Periodic Table

This app demonstrates the use of Interactable Objects and Object Collections. It shows a good example of Microsoft’s Fluent design elements such as spatial typographic layout using depth, motion, and materials.

Lunar Module

Example of using two-handed gesture and manipulation. The app shows how to provide UX flow for spatial mapping room scanning.

Outcome

Responses from the community

MR community appreciated easy-to-use building blocks for frequently used crucial interactions and UI. With published open-source UI controls and sample apps, developers and creators started sharing their own creations and blog posts.

Periodic Table app with Bing Speech API’s translation service

Migration to MRTK

With positive responses and requests from the community, the team decided to migrate the building blocks to MRTK v1 (HoloToolkit). We polished and ported UX building blocks from Mixed Reality Design Labs to MRTK v1 (HoloToolkit). During this process, we collaborated with the MRTK engineering team and community members.

Pull request examples:

Medium articles on MRTK’s building blocks

To introduce new building blocks ported to MRTK, I wrote articles on Medium. To make it easier to understand, I created visual assets including videos, screenshots, and diagrams. This Medium post became one of the most popular articles on mixed reality, in the AR/VR designer/developer community. Eventually, I presented the content at Microsoft Build conference, AWE, and Unity Unite Berlin 2018.

MRTK: Open-Source Building Blocks for Windows Mixed Reality