Mixed Reality Toolkit (MRTK) v2 for HoloLens 2

Apr 2018 – 2022, Mixed Reality | Microsoft

Mixed Reality Toolkit for Unity (MRTK) is an open-source framework that provides foundational components to accelerate Mixed Reality application development in Unity. It offers a set of building blocks for core spatial interactions and UI, with MRTK v2 adding support for HoloLens 2’s hand tracking and eye tracking input.

Context

With the introduction of HoloLens 2, the platform shifted from indirect interaction models to direct, articulated hand tracking and eye tracking. While earlier versions of MRTK established solid foundations, they were not architected to fully support these new input modalities or the level of UX quality required to match the system experience.

This transition created an opportunity and a challenge: evolve MRTK into a toolkit that could express modern spatial interactions while remaining approachable, extensible, and production-ready for developers.

Problem

MRTK v1 and early iterations of v2 were constrained by assumptions rooted in head-gaze and air-tap gesture-based input. As new interaction capabilities emerged, several limitations became clear:

Existing interaction patterns did not translate naturally to direct hand input
Visual and behavioral inconsistencies made it difficult to achieve system-level UX quality
Developers lacked clear, reusable patterns for emerging spatial interactions
The toolkit needed to support new input methods without breaking existing workflows

The core challenge was not only adding new features, but redefining a coherent spatial interaction language that developers could trust, adopt, and extend.

Role

As Principal Designer, I led the design vision for MRTK’s UX building blocks and developer experience. My responsibility was to define how spatial interactions should behave and feel on HoloLens 2, and to translate system-level interaction principles into reusable patterns that could scale across applications and industries.

I worked closely with engineers, technical artists, program managers, and user researchers to align design intent, architectural decisions, and implementation quality.

Leadership in Action – What I Led

I led the design vision for MRTK’s UX building blocks, shaping the foundational interaction and UI system that enabled high-quality mixed reality experiences on HoloLens 2. My responsibility went beyond individual components—I focused on defining a coherent, scalable spatial UX language that could serve both the platform and the developer ecosystem.

As new input modalities such as hand tracking and eye tracking emerged, I translated evolving system behaviors into clear, human-centered interaction patterns. I worked at the intersection of design, engineering, and product to align on quality bars, establish shell-parity standards, and ensure that MRTK reflected the same level of clarity, consistency, and intentionality as the HoloLens system experience.

A core part of my role was making complexity approachable. By identifying essential spatial UX patterns and turning them into reusable, extensible building blocks, I helped developers move faster while maintaining experience quality. Focused on extensibility and adaptability—allowing teams and partners to express their own brand identity without sacrificing usability or coherence.

Throughout the project, I acted as a connector across disciplines and organizations, guiding discussions, synthesizing insights from real-world deployments, and ensuring that design decisions were grounded in both user empathy and practical constraints.

Design Leadership in Practice

Spatial UX System & Building Blocks

Led the end-to-end design of MRTK’s UX building blocks, including interaction behaviors, UI controls, prefabs, example scenes, assets, and documentation.
Defined shell-parity interaction standards for hand- and eye-tracking to align MRTK experiences with the HoloLens system UX.
Identified and prioritized essential spatial UX patterns by synthesizing customer feedback, partner needs, and internal research.

Cross-Functional Collaboration

Partnered closely with engineering to ensure MRTK components were extensible, customizable, and production-ready.
Collaborated with interaction designers and technical artists to refine visual quality, motion, and interaction nuance.
Worked with user researchers to prototype and validate hand-tracking and eye-tracking interactions through iterative studies.

Developer Experience & Enablement

Designed example scenes and reference implementations to accelerate developer onboarding and experimentation.
Aggregated recurring UX patterns from enterprise deployments across healthcare, automotive, retail, and aerospace.
Contributed directly to the open-source project through design reviews and pull requests on GitHub and internal tooling.

Documentation & Communication

Authored technical and conceptual documentation, including design guidelines, interaction principles, and best practices.
Designed a modular documentation layout system to ensure clarity, consistency, and scalability.
Created visual assets—diagrams, imagery, and videos—to clearly communicate complex spatial concepts.

Evangelism & Partner Enablement

Represented MRTK design at global developer events, including Microsoft Build, Unity Unite, and Global XR Bootcamp.
Supported partners in migrating existing apps to MRTK v2, providing hands-on design guidance for new interaction paradigms.
Engaged directly with the developer community across GitHub, Slack, and training programs to gather feedback and drive adoption.

Outcomes

Building Blocks – Spatial Interactions and UI

Delivered shell-parity, production-ready UX building blocks that translate HoloLens 2 system behaviors into reusable, extensible components for developers.

Key contributions included:

Advanced hand menu interactions with gaze activation, voice commands (accessibility), and new reference examples
Achieved shell-parity polish across core controls (Buttons, Bounding Boxes, Toggles), including proximity lighting, distance response, and motion effects
Refined material systems and visual fit-and-finish to align with HoloLens 2 system UX
Improved contextual feedback (e.g., manipulation cursors, speech confirmation labels) to enhance clarity and confidence

Selected GitHub PR examples (representative):

Integration of the visual and interaction details on HoloLens 2 Button prefabs

Example Scenes

Designed and shipped example scenes that allow developers to experience, explore, and learn HoloLens 2’s hand-tracking interactions in context—while maintaining backward compatibility with HoloLens v1 (Gaze/Gesture/Voice) and motion controllers on Windows Mixed Reality immersive headsets.

Highlights:

Curated end-to-end interaction scenarios using MRTK building block
Polished reference scenes to demonstrate best practices and real-world usage
Introduced experimental visuals (e.g., pulse shaders) to communicate spatial feedback

Selected PR examples:

Various types of example scenes that demonstrate the building blocks of MRTK

Documentation

Authored and shaped comprehensive documentation to support both learning and production use, spanning conceptual guidance and technical implementation.

Mixed Reality Documentation – http://aka.ms/mrdocs
MRTK Technical Documentation – http://aka.ms/mrtkdocs

Selected PR examples:

Example of diagram designs for technical documentation:

Branding

Developed MRTK branding guidelines and assets to ensure consistent communication across channels, including presentations, official blog posts, event materials, promotional videos, and app splash screens.

Evangelism

Led developer education and advocacy efforts by creating and presenting deep-dive content on HoloLens 2 and MRTK UX building blocks at global conferences and events.

See MR Community Engagement page and Talks page for more information

Articles

Authored and published articles sharing real-world design and development insights using MRTK.

MRTK Examples Hub

Based on customer feedback and requests, I created and published MRTK Examples Hub, allowing developers to easily experience various example scenes by downloading the app from Microsoft Store.

MRTK Examples Hub on Microsoft Store

Impact

Since its release in 2019, MRTK has been widely adopted by the Mixed Reality community and industry partners, becoming a leading toolkit for AR, VR, and MR development. It is widely utilized by Mixed Reality industry partners and developers alike. In 2021, MRTK won the AWE 2021 Auggie Award – Best Developer Tool, recognizing its impact on the XR industry.

Open-Source Contribution

As a designer, made the most significant contributions to the MRTK GitHub repository, shaping not only the UX direction but also the quality, consistency, and adoption of the open-source toolkit.

https://github.com/Microsoft/MixedRealityToolkit-Unity