Implementing voice recognition and natural language processing in the NPSNET networked virtual environment

Abstract

Interfaces to military Virtual Reality (VR) systems, such as NPSNET IV.9, have been limited mainly to keyboard, mouse, and joystick devices. This presents two major problems; remembering how to access all the functionality of the system, and using the interface when the user is otherwise physically constrained. This can occur during the use of body-position tracking devices and Heads-Up-Displays (HUD). Voice recognition and Natural Language Processing (NLP) were used as a solution to both problems. The approach taken was to develop a networked Spoken Language System (SLS) using a Commercial-Off-The-Shelf (COTS) voice recognition and NLP system. The Nuance Speech Recognition System from Nuance Communications was chosen after analyzing the special requirements of NPSNET. Implementing the SLS occurred in four phases. First, vocabularies and grammars were developed to simulate the 108 keyboard commands, focusing on flexibility and decreased response latency. Second, new C++ classes were written to ease reuse of the Nuance API's. Third, a control panel was written to manage the voice processing, and fourth, the code was integrated into NPSNET. As a result of this effort, a new voice-enabled interface exists for NPSNET. In addition, C++ classes exist to ease future use of the Nuance API in other software systems. All of the 108 keyboard commands are executable through voice control with a 83.8% sentence understanding rate in a noisy background environment. :