Mixed Reality Toolkit (MRTK) v2 for HoloLens 2

Apr 2018 – Present, Mixed Reality | Microsoft

Mixed Reality Toolkit-Unity is an open-source project that provides foundational components and features to accelerate Mixed Reality app development in Unity. It provides building blocks for crucial spatial interactions and UI. MRTK v2 supports HoloLens 2’s new Hand Tracking and Eye Tracking input.

Microsoft HoloLens 2

Background

To support HoloLens 2’s new input methods, such as hand-tracking and eye-tracking, MRTK had to be updated dramatically because of its limited flexibility and architecture. While building a new version of MRTK, the entire team collaborated to discover, experiment, and improve hand-tracking and eye-tracking input interactions. Based on the ‘1st party == 3rd party’ spirit, the team tried to achieve HoloLens 2 shell-parity quality in MRTK’s building blocks.

Role

I led the overall design of the MRTK’s UX building blocks. It includes the design of interaction behavior and UI controls, prefabs, example scenes, assets, and documentation. In this process, I closely worked with interaction designers and the engineering team to integrate visual and interaction details that match the HoloLens system’s UX so that customers can achieve the same user experience quality easily and quickly.

In addition to creating shell-parity UX building blocks, I made sure the building blocks were extensible and customizable, allowing customers to infuse their own brand identity. I collaborated with the engineers and program managers to identify and prioritize essential UX items, implement building blocks, and triage bugs based on internal/external customers’ needs.

Through various projects with HoloLens partner companies, including Philips, Bosch, Ford, Lowes, and NASA, I helped the team aggregate and extract common UX patterns and turn them into MRTK’s building blocks. This process involved identifying the common patterns for spatial UX design challenges, prototyping, working with user research studies, and publishing on GitHub.

UX Building Blocks & Example Scenes

  • Led the design of MRTK’s UX building blocks
  • Helped the team to validate hand tracking APIs by building prototypes
  • Updated existing UI controls and interactions to support hand tracking interactions
  • Created various example scenes to accelerate the design and development of the HoloLens 2 applications
  • Worked with the engineering team to ensure MRTK’s building blocks are extensible and customizable
  • Collaborated with user researchers to create hand tracking prototypes for the UR studies
  • Collaborated with technical artists to polish the visual and interaction details of the building blocks
  • Submitted & reviewed Pull Requests on Visual Studio Online and GitHub
  • Aggregated common UX patterns from HoloLens projects in various types of industries
  • Extracted crucial common UX patterns that customers need

Documentation

  • Authored technical and conceptual documentations
  • Wrote design guidelines for UX building blocks
  • Created visual assets for documents – image, diagrams, videos
  • Designed documentation layout template for visual consistency and streamlined browsing experience
  • Collaborated with the content team to author and publish tutorials and guidelines

Branding & Promotion Content

  • Created videos with Mixed Reality Capture to promote new releases
  • Produced MRTK brand assets and guidelines for coherent communication and promotion

Evangelism & Design Consulting

  • Engaged with the developer community through multiple channels, including GitHub, Slack, and Mixed Reality Academy
  • Created MRTK’s UX Building Blocks deep dive deck and presented at multiple AR/VR conferences, including Microsoft Build, Unity Unite, and Global XR Bootcamp
  • Worked with multiple partner companies and helped them to update HoloLens v1 apps to support new interactions
  • Helped partner companies onboard MRTK v2 by presenting UX building blocks details and providing design recommendations

Outcomes

Building Blocks – Spatial Interactions and UI

Here are some of the Pull Request examples on GitHub:

Bounding Box
Pressable Button
Manipulation Handler
Slate
Hand Menu
Near Menu
Integration of the visual and interaction details on HoloLens 2 Button prefabs

Example Scenes

Created example scenes where developers and creators can experience, play, and learn about HoloLens 2’s new hand tracking input interactions. All building blocks still support HoloLens v1’s GGV(Gaze/Gesture/Voice) input and Windows Mixed Reality immersive headset’s motion controller input.

Here are some of the Pull Request examples on GitHub:

Various types of example scenes that demonstrate the building blocks of MRTK

Documentation

Technical writing for MRTK’s building blocks and functionalities. Writing on Design/Dev guidelines. Design of the document layout and modules. Visual graphical elements such as diagrams, Mixed Reality Capture images, videos, etc.

Here are some of the Pull Request examples on GitHub:

Example of diagram designs for technical documentation:

UX Building Blocks Cheat Sheet – A poster graphic work to help developers and creators overview frequently used MRTK’s UX Building Blocks in a single page.

Branding

Created MRTK branding guidelines and assets for coherent communication. The branding assets were used for various types of endpoints such as presentations, official blog posts, event swags, promo videos, and app splash screens.

Evangelism

Created and presented HoloLens 2 & MRTK’s UX Building Blocks deep dive content in multiple conferences and AR/VR/MR events. Since the first presentation in Microsoft Build 2018, I got session requests for various types of conferences, workshops, and seminars.

See MR Community Engagement page and Talks page for more information

Articles

Wrote and published articles on medium to share Mixed Reality app design & development stories using MRTK.

Promotion

Shared MRTK updates on social media through visual content – Mixed Reality Capture videos, screens, examples, and sample apps.

Linked In

Twitter

MRTK v2 (Release Candidate 1) was released on 4/5/2019.

Release (v2.0.0)

MRTK v2 (Official Release) was released on 8/21/2019.

MRTK Examples Hub

Based on customer feedback and requests, I created and published MRTK Examples Hub, allowing developers to easily experience various example scenes by downloading the app from Microsoft Store.

MRTK Examples Hub on Microsoft Store

Next Reality
https://hololens.reality.news/news/magic-microsofts-hololens-2-hand-interaction-display-mixed-reality-toolkit-v2-demo-video-0195899/

Impact

Since its release in 2019, MRTK was quickly embraced by the Mixed Reality community and became an industry-leading toolkit for building experiences for AR/VR/MR. It is widely used by Mixed Reality industry partners and developers. In 2021, MRTK won AWE 2021 Auggie Award – Best Developer Tool.

GitHub Contribution

Made the most contributions to MRTK GitHub as a designer

https://github.com/Microsoft/MixedRealityToolkit-Unity

Copyright © Dong Yoon Park 박동윤 All Rights Reserved.