# Smartphone-Assisted Virtual Reality

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1641636836355/fXngLna9V.jpeg)

Photo by [Lux Interaction](https://unsplash.com/@luxinteraction?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)

To conclude my bachelor in computer science, I wrote my thesis about using a Smartphone as an interaction device for Virtual Reality. This article is basically a summary of my thesis as a blog. Enjoy!

### Introduction

Virtual Reality is an emerging medium that enables presence and interactivity in a three-dimensional space. Conventional input devices like a mouse or a keyboard are made for two-dimensional environments. They require complex movements to complete tasks in a three-dimensional environment.

![](Uploading...)
VR controllers by L. Yang on [Steam Community](https://steamcommunity.com/games/250820/announcements/detail/1697188096865619876)

This is why a variety of different input devices, specially made for Virtual Reality, exist. But most of them are expensive and need complex tracking systems.

Most people already own a smartphone that they use on a daily basis. Such phones have a variety of different sensors already built-in, feature wireless capabilities, and are able to run custom software. This makes them affordable general-purpose devices.

### Experiments

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1641636837812/6xct2Fn4i.png)

Screenshot of the laser pointer experiment

Three experiments were implemented to demonstrate how the smartphone can help with common interactions when using Virtual Reality applications:

**Model Viewer:** An application to view and rotate three-dimensional models. The rotation of the 3D model displayed in VR is synchronized with the Smartphone. To rotate the model, the phone has to be rotated in the wanted direction.

**Laser Pointer:** A method to select objects or UI elements. A virtual representation of the Smartphone is shown in VR. The rotation is again synced with the real Smartphone. Out of the front of the phone comes a red line (the “laser”) which is used to point at user interface (UI) elements.

**Virtual Keyboard:** An application to write text on a virtual keyboard. The virtual keyboard is shown in VR. The buttons on it are mapped to the Smartphones’s touch screen. A blue circle is used to see where the location, touched on the display, maps to the keyboard in VR.

### Implementation

To get platform independence, I decided to use a web-based stack. All experiments were implemented in WebVR using [Three.js](https://threejs.org/) and Javascript.

The networking framework [Ubi-Interact](https://wiki.tum.de/display/infar/Ubi-Interact), developed at my university, is used to make the proposed experiments reusable and abstracted from device-specific environments. It connects the PC, running the VR experience, and the Smartphone together. Most of the experiment-specific interaction logic is abstracted into so-called “Ubii Interactions”, which run on a server.

The client running on the Smartphone was also developed using Javascript.

### Evaluation

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1641636839457/neuauK1wm.png)

Screenshot of the virtual keyboard experiment

In order to test the usability of the smartphone as an assistant device for VR, a user study was conducted. In the study, participants had to complete tasks with the three experiments. Also, a [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) (SUS) user study was performed to get feedback from the users.

A SUS questionnaire asks the participants to rate the following statements regarding the usability of a system from one to five:

1.  I think that I would like to use this system frequently.
2.  I found the system unnecessarily complex.
3.  I thought the system was easy to use.
4.  I think that I would need the support of a technical person to be able to use this system.
5.  I found the various functions in this system were well integrated.
6.  I thought there was too much inconsistency in this system.
7.  I would imagine that most people would learn to use this system very quickly.
8.  I found the system very cumbersome to use.
9.  I felt very confident using the system.
10.  I needed to learn a lot of things before I could get going with this system.

#### To sum up:

**Model Viewer:** Most participants agreed that this input method is very intuitive and effective to operate.

![Model Viewer Experiment: SUS Results](https://cdn.hashnode.com/res/hashnode/image/upload/v1641636841080/LLZ3RGcn0.png)

**Laser Pointer:** This experiment scored the highest amongst the ones presented in this research.

![Laser Pointer Experiment: SUS Results](https://cdn.hashnode.com/res/hashnode/image/upload/v1641636842743/c1okFmCs3.png)

**Virtual Keyboard:** While the model viewer and the laser pointer scenario reached a very high score, the virtual keyboard scored slightly lower but still demonstrates high usability despite its complexity.

![Virtual Keyboard Experiment: SUS Results](https://cdn.hashnode.com/res/hashnode/image/upload/v1641636844287/18dH7aPEY.png)

### Conclusion

A lot of feedback was collected during the user study, which can be used to improve these implementations further. Since all experiments are considered “acceptable” according to the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html), the Smartphone can indeed be considered a working input device for Virtual Reality.

If you want to read the full thesis, check out [this link](https://wiki.tum.de/display/infar/BA%3A+Smartphone-Assisted+Virtual+Reality+Using+Ubi-Interact).