Mixed Reality Design Labs & MRTK v1
May 2017 – Present, Ecosystem Experience Design | Analog | Microsoft
When I first joined the
For example, a lot of developers were having trouble making a simple button work with proper visual feedback. Everyone wanted to make a beautiful button that looks like the one in the
Because developers had to create their own version of the common interactions and interfaces (e.g. Gaze Cursor, Button, Bounding Box), it created inconsistent experiences in Mixed Reality apps as well. I realized that it is crucial to have common building blocks for foundational interactions and UI for both MR developers and consumers.
I started researching crucial common UI controls that are mostly used in existing apps and requested by developers from the community. I organized them into visual graphics and presentations to help our team socialize the current situations and design challenges across the organization.
I worked with colleague technical designers to create the initial sets of foundational building blocks such as Interactable Object, Button, and Bounding Box. In this process, I helped the team to achieve visual and interaction quality that matches the Hololens OS shell through design direction, design integration, prototyping, and validation. I created example scenes to help our team experience and test out the features.
- Helped the team to define and prioritize the list of the building blocks for the developers
- Worked with technical designers to build extensible and modular building blocks that people can easily customize and infuse own brand identity
- Created example scenes that demonstrate how to use the building blocks
- Polished the visual and interaction design of the controls
- Reviewed Pull Requests on GitHub for UX related items
- Created documents that explain technical details and design guidelines
- Created videos with Mixed Reality Capture to promote MRTK’s features
- Engaged with the developer community through multiple channels including GitHub, Slack, and Mixed Reality Academy
- Drove re-branding effort from HoloToolkit to MRTK
Common UI controls
When we build the UI controls, we made sure to make them modular and extensible so that developers can easily customize and bring their own brand identity into their app experiences.
Publication on GitHub
We created the design team’s open-source repository ‘Mixed Reality Design Labs‘ on GitHub. This allowed us to publish experimental components and iterate fast.
To demonstrate the use of common UI controls and interaction building blocks, we created sample apps with a complete end-to-end experience.
This app demonstrates the use of Interactable Objects and Object Collections. It shows a good example of Microsoft’s Fluent design elements such as spatial typographic layout using depth, motion, and materials.
Example of using two-handed gesture and manipulation. The app shows how to provide UX flow for spatial mapping room scanning.
Responses from the community
MR community appreciated easy-to-use building blocks for frequently used crucial interactions and UI. With published open-source UI controls and sample apps, developers and creators started sharing their own creations and blog posts.
Migration to MRTK
With positive responses and requests from the community, the team decided to migrate the building blocks to MRTK v1 (HoloToolkit). We polished and ported UX building blocks from Mixed Reality Design Labs to MRTK v1 (HoloToolkit). During this process, we collaborated with the MRTK engineering team and community members.
Pull request examples:
- Button, Interactable Object, Object Collection
- Bounding Box and App Bar
- Dialog & Progress Control
Medium articles on MRTK’s building blocks
To introduce new building blocks ported to MRTK, I wrote articles on Medium. To make it easier to understand, I created visual assets including videos, screenshots, and diagrams. This Medium post became one of the most popular articles on mixed reality, in the AR/VR designer/developer community. Eventually, I presented the content at Microsoft Build conference, AWE, and Unity Unite Berlin 2018.