SeeMos Project

Overview

The primary objective of the SeeMos Project is to propose a research platform dedicated to active vision and in particular to the early vision process. Thus, the SeeMos platform which constitutes an embedded system is composed of a modular hardware and a software design. The main feature of this platform is an architecture based on FPGA, CMOS imager, Inertial devices, High speed communication and a DSP-based Codesigned board.

Imaging device

This module consists of a CMOS imager manufactured by Neuricam. The imager allows full 2D addressing with a column bus and a row bus. It has a resolution of 640x480 (VGA) and provides a broad dynamic range (120db) due to the logarithmic response of its pixel structure.

Four digital-to-analog converters allow modification of the four analog voltages of the imager: analog signal offset, digital conversion range, voltage reference and a pixel precharge voltage. These four converters are used to optimize the conversion range. In effect, the CMOS imager has a logarithmic curve that enables the broad dynamic range (120dB).

FPGA Board

This is the core of the system. It was designed around a Stratix EP1S60 manufactured by Altera which is connected to 5x1MB SRAM and 64MB SDRAM. Of course, this component enables a high density of integration (57120 Logic Elements).

The need for strong parallelization was what led us to connect 5x2MB SRAM synchronous memory blocks. Each 2MB memory has private data and address buses. Consequently, in the FPGA, 5 processes (using 2MB each) can address all the memory at the same time and an SDRAM module socket provides an extension of the memory to 64 MB.

DSP Board

This module can be considered as a "co-processor" for the FPGA. The main idea consists in using the DSP like a part of the FPGA in order to carry out a particular task. In our design, the originality of the connection between the DSP and the FPGA will be to consider the set " DSP + algorithm " as a custom instruction.

Consequently, few custom instructions will be able to easily write in C or assembler and optimized for a such device (Filtering, FFT,...). Thus a set of compiled C routines (saved in a dedicated memory on the DSP board module) corresponds to a set of custom
instruction.

Place the mouse over the module for details

Communication

This board is connected to the main board and manages all communications with a host computer. The communication bus is actually high-speed USB2.0 and we develop a FireWire link.

Inertial device

The inertial set is composed of two 2D linear accelerometers ADXL 311 designed by Analog Devices and three gyrometers ENC03-M designed by Murata.

These sensors are soldered onto the imager PCB and aligned with the imager axis. A single 8-input analog-to-digital converter allows conversion of the different axis measurements. It is important to notice that a temperature sensor is included in this board to regulate the inertial sensors' deviations.

The Z gyrometer and accelerometer are soldered on a second board positioned at right angle with the imager board.

Architectural features

The main purpose of our architecture is to allow the implementation of early vision processes as in the human or primate visual system. In these systems it is well known that the first neural layers (in the retina) pre-filter the visual data flow in order to select only the conspicious information.
This Prefiltering step needs strong parallelization, on one hand to respect real-time constraints, and on the other hand because of the intrinsic characteristics of the algorithms. For exemple, some classical algorithms in an attention task used to elaborate an efficient saliency map are motion detection, Gabor filters, and color segmentation. However, the characteristics of particular visual tasks may require dedicated image processing and only an FPGA approach allows such flexibility. For architectures such as these, a Stratix EP1S60 from Altera has been chosen. The need for strong parallelization was what led us to connect 5x2MB SRAM synchronous memory blocks. In our approach, the high-level processing has to be performed on a host computer rather than on the embedded system.

Architecture Motivations: The active vision approach

An artificial active vision system uses observer-controlled input sensors. Its main goal must be to actively extract the requested information in order to solve a given task. In this spirit, an active vision system can be divided in 2 two main parts: Early vision step and Cognitive processes. In the design of our sensor, we only consider the Early vision step which can be divided into two successive tasks:

Attention: This is the initializing step of the process. Whole images are grabbed while waiting for the building of the saliency maps. These maps are built in parallel and represent/code conspicuity within the visual field along particular dimensions (e.g. color, orientation, or motion). The result of this step is a set of ROIs (Regions Of Interest).

Focusing: This step allows the generation of the geometry of an ROI (rectangular, tilted, foveal, circular) and the optimization of the Signal/Noise Ratio. Classical processings are Contrast optimization in an ROI, Tracking of an ROI in motion, ...

Tracking of a gray scale template (32x32) at 2000fr/s

Implementation approach

The implementation of such an approach requires management, sequential and concurrent execution of the routines previously described. Indeed, all task-oriented execution (attention, focusing) is controlled by supplied results and these layers possibly have to share areas of interest. Moreover, the information bottleneck located in the imager level should be continuously optimized to ensure an efficient performance. In our hardware architecture, these functions are carried out by what we term a "Sequencer'' (noted M0 on the figure below) and are performed on the NIOS soft core processor. This solution has two main advantages:

We benefit from software flexibility to define the routines' interactions, and
The soft core processor allows an efficient architectural matching with the other parts of the supervision unit.

An internal RAM (noted R0 on the figure below) is used to store the instruction sequences which define the sequencer behavior according to the task under consideration. The host computer which uses our embedded system communicates with it through a standard communication bus (actually USB 2.0 protocol) and sends requests in order to indicate to the sequencer to adopt the relevant behavior.

More precisely, the controls are accordingly (and potentially a set of parameters) stored in a dedicated stack, then the sequencer M0 chooses pre-established interactions between the modules (noted P4 on the figure below) which constitute a dedicated processing chain.

A number of these modules, due to environmental adaptation (Processing No. 1 to i), modify the pixel flow which is going to be used by the attention, focusing (Processing No. j to j+1). The different data flows (corrected windows of interest and
inertial measurements) can be used by these modules to perform computation. The set of results that are provided by these processing modules are collected in a buffer. This is how the sequencer selects results to send to the host computer. The sequencer is going to use a part of these results to perform visual feedback on sensing devices (noted S0 on the figure below) using dedicated control modules (noted P0, P1 and P3 in the figure below ). We note in Figure above the module P2 which works the external RAM R1. This module performs the Fixed Pattern Noise Correction which is absolutely essential for the image sensor technology we use. Lastly, the dedicated communication module P5 is a multiplexer that synchronizes the corrected pixels flow and the sequencer results flow for sending to the host computer.