Active Vision Mechanisms for a Humanoid Robot


Currently the vision group is working in developing and revamping vision mechanisms to help the humanoid robot ISAC have human-like eye motion. The vision system for ISAC is biologically inspired, thus the goal is for ISAC to be able to recognize objects, fixate and track them, determine their depth, motion characteristics, etc.


Improve the current active vision system that demonstrates capabilities similar to that of the human visual system

Implement and test robust vision mechanisms such as saccades, smooth-pursuit, and vergence

Determine depth and motion characteristics of objects through different techniques

Combine this information with sensory information from other sources in the robot to develop cognitive behaviors for ISAC

Robots Used

ISAC Humanoid Robot

Figure 1. ISAC vision directed at Barney

Project Description

The goal of this project is to improve the current Active Vision Gaze Controller on the humanoid robot ISAC. The active vision controller was originally implemented on a camera head that consists of two color cameras and four degrees of freedom (pan, tilt, left verge, and right verge). The camera controls were first designed to mimic five human-like eye movements:

  • Saccades: are the ballistic movements of the eyes when they jump from one fixation point in space to another

  • Smooth-pursuit: maintains a fixation point of a target moving at moderate speeds on the fovea

  • Vergence adjusts the eyes so that the optical axes intersecting on the same target while depth varies. It ensures that both eyes fixate on the same point on the target

  • Vestibulo-ocular reflex (VOR) and opto-kinetic reflex (OKR) are mechanisms to stabilize the image of the target during head movements

In our revamping of the system, a new and more precise camera head will be used as illustrated in Figure 2. The latter consists of four degrees of freedom but these are different from the previous head, where each camera has two degrees of freedom: pan and tilt. For this type of system VOR and OKR are not needed. Thus the goal consists of obtaining more accurate results in the existing mechanisms and developing new capabilities in our system such as motion detection and accurate depth estimation. This is an extension of the work done by a former CIS student, Atit (JJ) Srikaew, named A Biologically Inspired Active Vision Gaze Controller.

Figure 2. Testbed platform for new ISAC pan-tilt system developed in CRL

Figure 3 shows several color swatches being tested on the new testbed. ISAC will be able to distinguish between colors, objects, etc. and the new pan-tilt will offer more degrees of freedom than are presently available.

Figure 3. Color swatches being tested on vision testbed developed for ISAC

Additionally, a greater shift towards cognitive robots is currently being pursued in the lab. Another goal is for the visual information to be combined with sensory information from other sources such as: audio queues, infra red data, torque values from the arm, and touch sensors from the hand. In attempting to implement cognitive behaviors, it is necessary to check for overlapping of sensory feedback, in that, if this information overlaps indicating activity occurring in the same location it will trigger a reach and grasp behavior; much like human baby behaviors.