Within smart factories, there is an increasing interest to involve Augmented and Virtual Reality. The rising market of AR and VR promises to offer new interfaces for industrial robots and the connected plant infrastructure, which provide intuitive status access or even control to the plant. Especially for close cooperation between humans and robots (human-robot coproduction) it is necessary that the intent of the robot is communicated, so that the human knows what the robot is about to do. This technology would also be handy for applications which require co-location of a remote person, like an expert involved in remote support, maintenance, planning or optimization.

The game engine Unity has evolved to one of the most used visualization platforms for AR / VR devices at the moment – one reason for Siemens to build the ROS-Sharp module (https://github.com/siemens/ros-sharp). The goal of this project is to bring this attempt even one step further and enable a visualization platform for AR / VR devices, especially for the Microsoft Hololens. This platform should not only ensure a stable and reliable communication of sensor data, but also should enable users to build their own interfaces for their robots.

Objectives:

The proposed software module should provide the following functionality after the end of the funding period:

• Communication

The software module should provide a stable communication link between the Unity application running on AR/VR gear (Hololens) and ROS, enabling to send and receive live data.

• Support of AR/VR gear

The module should already support current off-the-shelf AR/VR gear. We start to integrate the Microsoft Hololens, but plan also to use HTC Vive and Oculus Rift.

• Demonstration

We want to demonstrate the capabilities of AR/VR interfaces for a ROSin running robot by documenting a showcase and providing tutorials for setup and use.

• Interface toolkit

The software should include a toolkit for building AR/VR interfaces to the robot (roughly in the direction of rviz for AR/VR)

• Multi-User support

The software should be able to let multiple users see the same virtual content, enabling them to use this information for discussions.

We see the following challenges and also describe how we are going to meet them:

• We want of course build upon existing ROS modules. But ROSSharp does not support UWP yet, so either we need to solve this, build a workaround or use own infrastructure.
• The interfaces for AR/VR hardware (for example Hololens Dev Packages) still change continuously. Furthermore, new gear will come on the market so maintenance of the module will be crucial.
• Specific visualization needs for clients can exceed the capabilities of the toolkit, but can be inserted later in the maintained module.