This project is focused on the development of a distributed object system that facilitates the representation of three-dimensional, collaborative virtual environments. Furthermore, we are working to render these environments in real time on a wearable computer - allowing a user to access the virtual environment from any location in real space. In particular, we utilize a head tracking system (orientation is derived from a gyroscope; position is derived from GPS or through cameras located within a smart room) to move the viewpoint within the virtual world, thus creating an illusion of the user's movement within the virtual space. When the virtual environment is aligned with the real environment it becomes possible to overlay objects and animated characters on top of what the user sees in the real world. We are applying the these technologies to the visualization of real-time data collected by distributed sensor networks.
Amstutz, P. and Fagg, A. H. (2002), Real Time Visualization of Robot State with Mobile Virtual Reality submitted to the International Conference on Robotics and Automation (ICRA'02)
Supervised by: Andrew H. Fagg
Human beings are particularly capable of processing multiple sound sources simultaneously - as long as the individual sources are distributed spatially. We are developing techniques that will allow a wearable computer to artificially project a sound to an arbitrary location in space. This will enable multiple agents (e.g., news and email readers, or music players) to present audio information to the user at the same time. The user need only shift her attention to one of the available sources in order to absorb its full informational content.
Supervised by: Andrew H. Fagg
Humans communicate with one another using many different mechanisms. Often, the choice of mechanism (e.g. speech vs written vs hand gesture) depends greatly upon what each of the individuals is currently doing and the nature of the information being communicated. We are developing software techniques that will allow application programs to communicate with the user in a modality-independent manner. The computer's interface will then be able to make real-time choices as to when and how to present this information to the user in an as efficient manner possible (while minimizing the distraction that it might cause). For example, the interface might choose to whisper an email message into the user's ear while he is walking down the hall or driving a car, but choose to display the same information while the user is engaged in a conversation or in a business meeting.
Supervised by: Andrew H. Fagg
The main objective of the project is to design a system which will be able to detect speech or impulsive sound and locate the source of origin of the acoustic signal. The idea behind the source localization problem is to process the incoming signals from an arrangement of microphones (directional and omni directional) by applying basic signal processing techniques. The goal has been to develop a robust algorithm for source localization in which problems in localization due to multi-path propagation/reverberation are reduced. In continuing work, speaker identification and recognition will also be implemented.
Supervised by: Chandu Ravela
Neurons in a subregion of the premotor cortex exhibit both tactile and visual responses. Surprisingly, when a cell has a tactile receptive field on the arm, the visual receptive field of neuron will move depending upon the configuration of the arm. Within the parietal cortex (upstream from this premotor area), cells also exhibit multi-modal responses, but do not show such an alignment between the tactile and visual receptive fields. We are developing a model of associative learning in an effort to explain the development of this mapping.
Supervised by: Andrew H. Fagg
We are developing a tour guide system that combines an audio presentation with live, three-dimensional graphics rendering. In particular, we are focusing on the creation of virtual characters that will be able to apparently move around in the real space, describe the related research activities, and ultimately interact with the user.
Supervised by: Andrew H. Fagg
Primates are very capable of generating orienting movements (saccades) toward both auditory and visual stimuli. What is the process by which these sensory-motor maps are learned? In this project, we focus on the acquisition of auditory-driven saccades for the UMass Torso. Audio input is derived from the four microphones that are mounted on the head of the Torso. A set of high-level auditory features are derived from cross-correlation and fft operators. A linear function approximator is employed to estimate the appropriate saccade direction and magnitude in response to the incoming audio input. We use an approximate supervised learning approach to adjust this map. When an auditory saccade is followed by a saccade that is generated from a visual motion cue, this second saccade can be used as an error signal for the first.
Supervised by: Andrew H. Fagg