May 2017, Mixed Reality | Microsoft

HoloSketch is an on-device spatial layout and UX sketching tool for HoloLens to help build holographic experiences. HoloSketch works with a paired Bluetooth keyboard and mouse as well as gesture and voice commands. The purpose of HoloSketch is to provide a simple UX layout tool for quick visualization and iteration.

A simple UX layout tool for quick visualization and iteration


Primitives for quick studies and scale-prototyping

Using primitive shapes

Use primitive shapes for quick massing studies and scale-prototyping.

Import objects through OneDrive

importing objects

Import PNG/JPG images or 3D FBX objects(requires packaging in Unity) into the mixed reality space through OneDrive.

Manipulate objects

manipulating objects

Manipulate objects (move/rotate/scale) with a familiar 3D object interface.

Bluetooth, mouse/keyboard, gestures and voice commands

supports Bluetooth

HoloSketch supports Bluetooth mouse/keyboard, gestures, and voice commands.


Importance of experiencing your design in the device

When you design something for HoloLens, it is important to experience your design in the device. One of the biggest challenges in mixed reality app design is that it is hard to get a sense of scale, position, and depth, especially through traditional 2D sketches.

Cost of 2D based communication

To effectively communicate UX flows and scenarios to others, a designer may end up spending a lot of time creating assets using traditional 2D tools such as Illustrator, Photoshop, and PowerPoint. These 2D designs often require additional effort to convert them into 3D. Some ideas are lost in this translation from 2D to 3D.

Complex deployment process

Since mixed reality is a new canvas for us, it involves a lot of design iteration and trial and error by its nature. For designers who are not familiar with tools such as Unity and Visual Studio, it is not easy to put something together in HoloLens. Typically you have to go through the process below to see your 2D/3D artwork in the device. This was a big barrier for designers iterating on ideas and scenarios quickly.

Complex deployment process
Deployment process

Simplified process with HoloSketch

With HoloSketch, we wanted to simplify this process without involving development tools and device portal pairing. Using OneDrive, users can easily put 2D/3D assets into HoloLens.

Simplified process with HoloSketch
Simplified process with HoloSketch

Encouraging three-dimensional design thinking and solutions

We thought this tool would give designers an opportunity to explore solutions in a truly three-dimensional space and not be stuck in 2D. They don’t have to think about creating a 3D background for their UI since the background is the real world in the case of HoloLens. HoloSketch opens up a way for designers to easily explore 3D designs on HoloLens.


Overall UX design, including the visual/interaction design of the UI and app publication assets. Collaborated with technical designer/engineer to iterate the ideas and implementations.

Concept Sketch



Published the app to the Microsoft Store in HoloLens. In the app, the user can load assets from OneDrive and place them in the physical space. The app was adapted by several ISV partner companies for the early-stage prototyping purpose.

Importing images and 3D objects through OneDrive:

Importing 2D images

Polishing the asset packaging flow:

Adjust collider dimension if it does not match the object.

Add files to Files/Documents/HoloSketch/ folder You will see added 3D object in HoloSketch's OneDrive menu

Interaction design and instructions

HoloSketch supports the traditional interface that is widely used in 3D software. The user can use the move, rotate, scale the objects with gestures and a mouse. Switch between different modes with voice or keyboard.

Object manipulation modes

How to manipulate the objects

Contextual and Tool Belt menus

The contextual menu can be invoked through double air-tap.

Menu items:

  • Layout Surface: 3D grid system where multiple objects can be laid out and manage them as a group.
  • Primitives: Use cubes, spheres, cylinders and cones for massing studies.
  • OneDrive: Open the OneDrive menu to import objects.
  • Help: Displays help screen.

Contextual menu

Keyboard, gestures and voice commands