D4 - Remote Behavioral Assessment and Job Coaching via Video and Motion Technology

Task Leaders: Michael McCue, Ph.D. and Jessica Hodgins, Ph.D.

Co-Investigator: Edmund LoPresti, Ph.D.

Other participants: Robotics Institute, Carnegie Mellon University, PA Office of Vocational Rehabilitation: Hiram G. Andrews Rehabilitation Center, AT Sciences, Inc.

Objectives of task :

To use machine vision technology to ascertain task specific, qualitative behavioral data regarding individuals with cognitive/behavioral disabilities performing real tasks in natural settings.

1. To determine the reliability and validity of this technology for "in vivo" behavioral assessment.
2. To use the capacity of the technology to deliver specific task instructions (job coaching) based upon the behavioral assessment described above.
3. To determine the utility (validity, cost-effectiveness) of the technology for enhancing performance of specific work tasks in persons with significant cognitive disabilities.
4. To identify the specific environmental and client characteristics that indicate that such technology is likely to be effective.                                

Project update:

Individuals with cognitive disability resulting from conditions such as brain injury and autism present unique challenges to education and vocational rehabilitation. Persons with these cognitive disabilities experience a complex array of functional limitations that impact their ability to perform effectively in education, vocational training and employment settings.

This task is developing and testing a model of telerehabilitation-based “in vivo” supports with consumers who have cognitive disabilities that present obstacles to functioning in a work environment. We are implementing motion tracking using video and sensors to monitor specific pre-determined work behaviors and, ultimately, to deliver cues and instruction in response to occurrence of problem behaviors. We have used a motion database to train learning algorithms to reproduce high quality motion from low quality, inexpensive, and easily installed sensors. The setup for such a system is shown in Figure 1.

We are building on this insight to develop a mechanism for collecting qualitative data on task performances of subjects at work. The data will be used to assess the subject’s performance and the frequency and duration of symptomatic behavior (e.g., inattention, repetitive behaviors, task sequence problems). Once the reliability and validity of the assessment phase are determined, we will be able to remotely deliver specific task guidance and cues/prompts to the subject in response to the occurrence of specific problem behaviors (e.g., inattention -> auditory prompt via headphones to attend, or task sequence error -> task instruction via headphone to repeat step in proper sequence).

The specific aims of the project are as follows:

1. To use machine vision technology to ascertain task specific, qualitative behavioral data regarding individuals with cognitive/behavioral disabilities performing real tasks in natural settings.
2. To determine the reliability and validity of this technology for “in vivo” behavioral assessment.
3. To use the capacity of the technology to deliver specific task instructions (job coaching) based upon the behavioral assessment described above.
4. To determine the utility (validity, cost-effectiveness) of the technology for enhancing performance of specific work tasks in persons with significant cognitive disabilities.
5. To identify the specific environmental and client characteristics that indicate that such technology is likely to be effective.               

The goal of our research is to provide untrained users (rehabilitation providers) with a vision-based interface for interactively assessing and influencing complex human performance. Simple vision processing provides partial information about the user’s movements and domain knowledge from a previously captured motion database supplements the information from the interface to allow plausible movement to be inferred. The process can be performed interactively, with less than a second of delay between the capture of the video and the rendering of the animated motion.

The experimental scenario is an individual who must perform a multi-step food service task in a work setting (grilling hamburgers or making salads). We present a low cost camera-based system that allows an untrained user to “drive” the task assembly. The user need not be proficient at the task because the movements are based on previously captured sequences.  

http://graphics.cs.cmu.edu/people/jkh/JobCoaching/JobCoaching_Mpeg4.mov

Last Updated: 4-27-06