Mixed Reality Toolkit (MRTK) v2 for HoloLens 2

Apr 2018 – Present, Mixed Reality | Microsoft

Mixed Reality Toolkit-Unity is an open-source project which provides a set of foundational components and features to accelerate Mixed Reality app development in Unity. MRTK provides building blocks for crucial spatial interactions and UI. The latest release of MRTK v2 supports HoloLens 2’s new Hand Tracking and Eye Tracking input.

Microsoft HoloLens 2

I led the overall design of the MRTK’s UX building blocks. It includes the design of interaction behavior and UI controls, prefabs, example scenes, assets, and documentation. In this process, I closely worked with the engineering team to integrate visual and interaction details that match the HoloLens system’s UX so that customers can achieve the same user experience quality easily and quickly.

In addition to creating and providing shell-parity UX building blocks, I made sure the building blocks are extensible and customizable so that customers can infuse their own brand identity. I closely worked with the engineering team and program managers to prioritize UX items, implement building blocks, and triage bugs based on internal/external customer’s needs.

Through various types of projects with HoloLens partner companies including Philips, Bosch, Ford, Lowes, and NASA, I helped the team aggregate and extract common UX patterns and turn them into MRTK’s building blocks. This process involved identifying the common patterns for spatial UX design challenges, prototyping, working with user research studies, and publishing on GitHub.

Role

UX Building Blocks & Example Scenes

  • Led the design of MRTK’s UX building blocks
  • Helped the team to validate hand tracking APIs by building prototypes
  • Updated existing UI controls and interactions to support near interaction with hand tracking & far interaction with hand-ray
  • Created various example scenes to accelerate the design and development of the HoloLens 2 applications
  • Worked with the engineering team to ensure MRTK’s building blocks are extensible, modular and people can easily customize and infuse their own brand identity
  • Created test scenes for the near interaction user research studies
  • Polished the visual and interaction design of the controls
  • Submitted & reviewed Pull Requests on Visual Studio Online and GitHub
  • Aggregated common UX patterns from HoloLens projects in various types of industries
  • Extract crucial common UX patterns that customers need, prototype, build, and publish based on the customers’ needs and requests

Documentation

  • Authored technical and conceptual documentations
  • Wrote design guidelines for UX building blocks
  • Created visual assets for documents – image, diagrams, videos

Branding & Promotion Content

  • Created videos with Mixed Reality Capture to promote new releases
  • Created MRTK brand guidelines and assets for coherent communication

Evangelism

  • Engaged with the developer community through multiple channels including GitHub, Slack, and Mixed Reality Academy
  • Prepared MRTK’s UX Building Blocks deep dive deck and presented at multiple AR/VR conferences including Microsoft Build, Unity Unite, and Global XR Bootcamp
  • Worked with multiple partner companies and helped them to update HoloLens v1 apps to support new interactions

Building Blocks

Bounding Box
Pressable Button
Manipulation Handler
Slate
Hand Menu
Near Menu
HoloLens 2 Pressable Button Structure
Integration of the visual and interaction details on HoloLens 2 Button prefabs

Example Scenes

Created example scenes where developers and creators can experience, play, and learn about HoloLens 2’s new hand tracking input interactions. All building blocks still support HoloLens v1’s GGV(Gaze/Gesture/Voice) input and Windows Mixed Reality immersive headset’s motion controller input.

Various types of example scenes that demonstrate the building blocks of MRTK

Documentation

Technical writing for MRTK’s building blocks and functionalities. Writing on Design/Dev guidelines. Design of the document layout and modules. Visual graphical elements such as diagrams, Mixed Reality Capture images, videos, etc.

Mixed Reality Documentation – http://aka.ms/mrdocs
MRTK Technical Documentation – http://aka.ms/mrtkdocs

Branding

Created MRTK branding guidelines and assets for coherent communication.
MRTK branding guidelines PDF file

Evangelism

Created and presented HoloLens 2 & MRTK’s UX Building Blocks deep dive content in multiple conferences and AR/VR/MR events.
See Talks page for more information.

Articles

Wrote and published articles on medium to share Mixed Reality app design & development story using MRTK.

Promotion

Shared MRTK updates on social media through visual content – Mixed Reality Capture videos, screens, examples, and sample apps.

Linked In

Twitter

MRTK v2 (Release Candidate 1) has been released on 4/5/2019.

Release (v2.0.0)

MRTK v2 (Official Release) has been released on 8/21/2019.

MRTK Examples Hub

Created and released MRTK Examples Hub which allows developers to easily experience various example scenes in MRTK.

Next Reality
https://hololens.reality.news/news/magic-microsofts-hololens-2-hand-interaction-display-mixed-reality-toolkit-v2-demo-video-0195899/

GitHub

https://github.com/Microsoft/MixedRealityToolkit-Unity