AR Multimodal Interface

From HIT Lab NZ

Researchers:	Andreas Duenser , Scott Green , Sylvia Irawati
Date:	October, 2005 - Ongoing
Funding:

---

Contents 1 Publications 2 Additional Content 2.1 Image Gallery 2.2 Videos

We want to study how multimodal interfaces can improve the user interaction in an AR environment. We developed a multimodal interaction system that uses speech and paddle based gestures as input into an AR system. Our system is a modified version of the VOMAR application for tangible manipulation of virtual furniture in an AR setting. The goal is to develop a system for intuitively arranging AR content using a natural mixture of speech and gesture input. We have conducted a user study to evaluate the multimodal AR interface.

The Interface uses the Ariadne spoken dialog system and Microsoft Speech API 5.1 as a speech recognition engine. The application shows a layout of virtual furniture in a room. The 3D virtual models appear exactly superimposed over the real paper. Looking at the large piece of paper the user sees an empty virtual room. The user can then copy and transfer objects from the menu to the virtual room using the paddle gestures only, speech and static paddle, or speech and continuous paddle gestures.

From a user study we found: Using combined speech and paddle, subjects can place objects more accurately than using paddle gestures only. This is due to the difficulty in translation and rotation using only paddle gestures. In the current application, speech did not improve the intuitiveness of the interaction although it improved the efficiency. Using paddle gestures the user can directly interact with the virtual objects which may cause the interaction to be more intuitive for the user.

In this application, speech can overcome the limitation of the current paddle gestures in object placement. Therefore, the user tasks can be done more easily and efficiently.

In the future, new gestures can be introduced to improve the current paddle gestures such as gestures for locking and unlocking an object. This multimodal interface can be extended to other application domains such as human robot collaboration research that uses speech and AR technology to control a virtual robot.

Publications

[2006-ICAT-ARUsingSpeechPaddleGestures.pdf An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures.]

Irawati, S., Green, S., Billinghurst, M., Dünser, A., Ko, H.

In Proceedings of the 16th International Conference on Artificial Reality and Telexistence (ICAT 2006), November 29th - December 1st, 2006, Hangzhou, China.

Project: [[{{{project}}}]]

[2006-ISMAR-MoveTheCouchWhere.pdf "Move the Couch Where?": Developing an Augmented Reality Multimodal Interface]

Irawati, S., Green, S., Billinghurst, M., Dünser, A., Ko, H.

In Proceedings of the Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2006), October 22nd - 25th, 2006 Santa Barbara, California.

Project: [[{{{project}}}]]

Additional Content

Image Gallery

There are currently no images uploaded...

Videos

There are currently no videos uploaded...