Mixed Reality Toolkit (MRTK) v2 for HoloLens 2

Apr 2018 – Present, Mixed Reality | Microsoft

Mixed Reality Toolkit-Unity is an open-source project that provides foundational components and features to accelerate Mixed Reality app development in Unity. It provides building blocks for crucial spatial interactions and UI. MRTK v2 supports HoloLens 2’s new Hand Tracking and Eye Tracking input.

Microsoft HoloLens 2

Background

To support new input methods on HoloLens 2, such as hand and eye tracking, MRTK required substantial updates due to its prior limitations in flexibility and architecture. Throughout the development of a new MRTK version, the team collaborated to explore, experiment with, and refine hand- and eye-tracking interactions. Following the philosophy of “1st party == 3rd party,” they aimed to match HoloLens 2 shell-parity standards within MRTK’s foundational elements.

Role

I led the overall design of MRTK’s UX building blocks, encompassing interaction behaviors, UI controls, prefabs, example scenes, assets, and documentation. This work involved close collaboration with interaction designers and engineers to ensure that visual and interaction elements aligned seamlessly with the HoloLens system UX, enabling customers to achieve high-quality user experiences quickly and easily.

Beyond creating shell-parity UX building blocks, I ensured these components were extensible and customizable, allowing customers to incorporate their own brand identity. Working alongside engineers and program managers, I helped identify and prioritize essential UX elements, implement core building blocks, and triage bugs based on feedback from internal and external customers.

Through partnerships with HoloLens clients like Philips, Bosch, Ford, Lowe’s, and NASA, I supported the team in gathering and distilling common UX patterns into MRTK building blocks. This involved identifying spatial UX design challenges, prototyping solutions, collaborating on user research studies, and publishing resources on GitHub.

UX Building Blocks & Example Scenes

  • Led the design of MRTK’s UX building blocks.
  • Supported team validation of hand-tracking APIs by developing prototypes.
  • Enhanced existing UI controls and interactions to support hand-tracking capabilities.
  • Created example scenes to accelerate HoloLens 2 application design and development.
  • Collaborated with engineering to ensure MRTK building blocks are extensible and customizable.
  • Worked with user researchers to develop hand-tracking prototypes for user research studies.
  • Partnered with technical artists to refine visual and interaction details of the building blocks.
  • Submitted and reviewed pull requests on Visual Studio Online and GitHub.
  • Aggregated common UX patterns from HoloLens projects across various industries.
  • Identified essential UX patterns to meet customer needs.

Documentation

  • Authored both technical and conceptual documentation.
  • Developed design guidelines for UX building blocks.
  • Created visual assets for documentation, including images, diagrams, and videos.
  • Designed a documentation layout template to ensure visual consistency and streamline the browsing experience.
  • Collaborated with the content team to author and publish tutorials and guidelines.

Branding & Promotion Content

  • Created promotional videos using Mixed Reality Capture to showcase new releases.
  • Developed MRTK brand assets and guidelines to ensure consistent communication and promotion.

Evangelism & Design Consulting

  • Engaged with the developer community across multiple channels, including GitHub, Slack, and Mixed Reality Academy.
  • Created a deep-dive presentation on MRTK’s UX Building Blocks and presented it at AR/VR conferences, including Microsoft Build, Unity Unite, and Global XR Bootcamp.
  • Collaborated with partner companies to update HoloLens v1 apps, adding support for new interaction methods.
  • Assisted partner companies in adopting MRTK v2 by presenting details on UX building blocks and providing design recommendations.

Outcomes

Building Blocks – Spatial Interactions and UI

Here are some of the Pull Request examples on GitHub:

Bounding Box
Pressable Button
Manipulation Handler
Slate
Hand Menu
Near Menu
Integration of the visual and interaction details on HoloLens 2 Button prefabs

Example Scenes

Created example scenes where developers and creators can experience, play, and learn about HoloLens 2’s new hand tracking input interactions. All building blocks still support HoloLens v1’s GGV(Gaze/Gesture/Voice) input and Windows Mixed Reality immersive headset’s motion controller input.

Here are some of the Pull Request examples on GitHub:

Various types of example scenes that demonstrate the building blocks of MRTK

Documentation

Authored technical documentation on MRTK’s building blocks and functionalities, including design and development guidelines. Designed document layouts and modular structures for clarity and consistency. Created visual elements, such as diagrams, Mixed Reality Capture images, and videos, to enhance documentation.

Here are some of the Pull Request examples on GitHub:

Example of diagram designs for technical documentation:

UX Building Blocks Cheat Sheet – A poster graphic work to help developers and creators overview frequently used MRTK’s UX Building Blocks in a single page.

Branding

Developed MRTK branding guidelines and assets to ensure consistent communication. These branding assets were utilized across various channels, including presentations, official blog posts, event swag, promotional videos, and app splash screens.

    Evangelism

    Created and presented HoloLens 2 & MRTK’s UX Building Blocks deep dive content in multiple conferences and AR/VR/MR events. Since the first presentation in Microsoft Build 2018, I got session requests for various types of conferences, workshops, and seminars.

    See MR Community Engagement page and Talks page for more information

    Articles

    Wrote and published articles on medium to share Mixed Reality app design & development stories using MRTK.

    Promotion

    Shared MRTK updates on social media through visual content – Mixed Reality Capture videos, screens, examples, and sample apps.

    Linked In

    Twitter

    https://twitter.com/cre8ivepark/status/1399896603025043462
    https://twitter.com/cre8ivepark/status/1399896603025043462?s=20
    https://twitter.com/cre8ivepark/status/1339067499246047237?s=20

    MRTK v2 (Release Candidate 1) was released on 4/5/2019.

    https://twitter.com/cre8ivepark/status/1114630131841036288

    Release (v2.0.0)

    MRTK v2 (Official Release) was released on 8/21/2019.

    MRTK Examples Hub

    Based on customer feedback and requests, I created and published MRTK Examples Hub, allowing developers to easily experience various example scenes by downloading the app from Microsoft Store.

    MRTK Examples Hub on Microsoft Store

    Next Reality
    https://hololens.reality.news/news/magic-microsofts-hololens-2-hand-interaction-display-mixed-reality-toolkit-v2-demo-video-0195899/

    Impact

    Since its release in 2019, MRTK has been quickly embraced by the Mixed Reality community, becoming an industry-leading toolkit for developing AR, VR, and MR experiences. It is widely utilized by Mixed Reality industry partners and developers alike. In 2021, MRTK won the AWE 2021 Auggie Award – Best Developer Tool.

    GitHub Contribution

    Made the most significant contributions to the MRTK GitHub repository as a designer

    https://github.com/Microsoft/MixedRealityToolkit-Unity

    Copyright © Dong Yoon Park 박동윤 All Rights Reserved.