Mixed Reality Toolkit (MRTK) v2 for HoloLens 2

Apr 2018 – 2022, Mixed Reality | Microsoft

MRTK is an open-source spatial interaction framework for Unity. I led the UX system for MRTK v2, translating HoloLens 2 hand and eye tracking into reusable, production-ready components that developers and partners could adopt across products and industries.

Context

HoloLens 2 shifted spatial computing from indirect gaze and gesture input toward articulated hands, eyes, and direct manipulation. MRTK needed a new interaction model and a higher quality bar to make those capabilities usable beyond Microsoft’s first-party experiences.

The opportunity was to turn emerging system behaviors into an approachable, extensible, and production-ready platform for the broader developer ecosystem.

---

Problem

MRTK v1 and early v2 architectures reflected head-gaze and air-tap assumptions. The move to multimodal, direct interaction exposed several system-level gaps:

• Existing interaction patterns did not translate naturally to direct hand input
• Visual and behavioral inconsistencies made it difficult to achieve system-level UX quality
• Developers lacked clear, reusable patterns for emerging spatial interactions
• The toolkit needed to support new input methods without breaking existing workflows

The challenge was not simply to add features. It was to define a coherent spatial interaction language that developers could trust, adopt, customize, and extend.

---

Strategy & System Leadership

As Principal Designer, I set the design direction for MRTK’s interaction model, UX building blocks, and developer experience. I translated HoloLens 2 system principles into reusable patterns that could scale across products, industries, and partner ecosystems.

I aligned engineering, technical art, program management, research, documentation, and partner teams around shared interaction principles, architectural priorities, and implementation quality.

---

Operating Model & Organizational Influence

I treated MRTK as a platform system, not a collection of components. The work connected interaction principles, reusable controls, examples, documentation, tooling, and partner guidance into one developer experience.

I established shell-parity standards and shared quality bars so open-source components reflected the clarity, consistency, and interaction fidelity of the HoloLens system experience.

I prioritized the smallest set of foundational patterns that could unlock many product scenarios. Each building block was designed for extensibility, allowing teams to express their brands without sacrificing usability or system coherence.

I connected disciplines and organizations, using research, customer deployments, community feedback, and implementation constraints to guide platform decisions and investment.

---

Design Leadership in Practice

Spatial UX System & Building Blocks

• Led the end-to-end design of MRTK’s UX building blocks, including interaction behaviors, UI controls, prefabs, example scenes, assets, and documentation.
• Defined shell-parity interaction standards for hand- and eye-tracking to align MRTK experiences with the HoloLens system UX.
• Identified and prioritized essential spatial UX patterns by synthesizing customer feedback, partner needs, and internal research.

Cross-Functional Collaboration

• Partnered closely with engineering to ensure MRTK components were extensible, customizable, and production-ready.
• Collaborated with interaction designers and technical artists to refine visual quality, motion, and interaction nuance.
• Worked with user researchers to prototype and validate hand-tracking and eye-tracking interactions through iterative studies.

Developer Experience & Enablement

• Designed example scenes and reference implementations to accelerate developer onboarding and experimentation.
• Aggregated recurring UX patterns from enterprise deployments across healthcare, automotive, retail, and aerospace.
• Contributed directly to the open-source project through design reviews and pull requests on GitHub and internal tooling.

Documentation & Communication

• Authored technical and conceptual documentation, including design guidelines, interaction principles, and best practices.
• Designed a modular documentation layout system to ensure clarity, consistency, and scalability.
• Created visual assets, including diagrams, imagery, and videos, to clearly communicate complex spatial concepts.

Evangelism & Partner Enablement

• Represented MRTK design at global developer events, including Microsoft Build, Unity Unite, and Global XR Bootcamp.
• Supported partners in migrating existing apps to MRTK v2, providing hands-on design guidance for new interaction paradigms.
• Engaged directly with the developer community across GitHub, Slack, and training programs to gather feedback and drive adoption.

---

Productization & Developer Enablement

Building Blocks – Spatial Interactions and UI

Delivered a production-ready system of reusable controls and interaction patterns that brought HoloLens 2 quality into developer applications while remaining extensible for partner needs.

Key contributions included:

• Advanced hand menu interactions with gaze activation, voice commands (accessibility), and new reference examples
• Achieved shell-parity polish across core controls (Buttons, Bounding Boxes, Toggles), including proximity lighting, distance response, and motion effects
• Refined material systems and visual fit-and-finish to align with HoloLens 2 system UX
• Improved contextual feedback (e.g., manipulation cursors, speech confirmation labels) to enhance clarity and confidence

Selected GitHub PR examples (representative):

• Hand menu improvements – added gaze activation option, new examples
• [Shell Parity] Adding HoloLens 2 shell-parity Toggle button
• [UX] Hand menu improvement – voice command (accessibility)
• HoloLens 2 Button material update

Example Scenes

Designed and shipped reference scenes that made new hand-tracking patterns understandable in context while preserving compatibility with HoloLens v1 and Windows Mixed Reality input workflows.

Highlights:

• Curated end-to-end interaction scenarios using MRTK building block
• Polished reference scenes to demonstrate best practices and real-world usage
• Introduced experimental visuals (e.g., pulse shaders) to communicate spatial feedback

Selected PR examples:

• Merge updates from MRTK Examples Hub 2.6.0
• Scrolling Object Collection – Example scene polish
• [Experimental] Pulse shader for hand mesh and spatial mesh visualization

Documentation

Created a documentation system that connected interaction principles, design guidance, implementation details, and production best practices.

• Mixed Reality Documentation – http://aka.ms/mrdocs
• MRTK Technical Documentation – http://aka.ms/mrtkdocs

Selected PR examples:

• [Docs] Bringing the landing page updates from dev branch to the release branch
• [Docs] Object Collection documentation update

Example of diagram designs for technical documentation:

Branding

Established MRTK’s visual identity and communication system across documentation, presentations, events, videos, and shipped applications.

Evangelism

Led developer education and partner enablement through technical design content, global events, community programs, and direct product guidance.

See MR Community Engagement page and Talks page for more information

Articles

Authored and published articles sharing real-world design and development insights using MRTK.

MRTK Examples Hub

Created and shipped MRTK Examples Hub as a productized learning surface, giving developers direct access to reference interactions and example scenes through the Microsoft Store.

MRTK Examples Hub on Microsoft Store

Platform & Ecosystem Impact

Since its release in 2019, MRTK has been widely adopted by the Mixed Reality community and industry partners, becoming a leading toolkit for AR, VR, and MR development. It is widely utilized by Mixed Reality industry partners and developers alike. In 2021, MRTK won the AWE 2021 Auggie Award – Best Developer Tool, recognizing its impact on the XR industry.

Open-Source Leadership

I contributed directly to the open-source codebase through design implementation, reviews, examples, and documentation. This hands-on model helped connect design intent to shipped quality and made design leadership visible in the same workflow as engineering.

https://github.com/Microsoft/MixedRealityToolkit-Unity