Robotic teleoperation with an eye tracker

The MIRo Team obtained a first impressive result in the field of robotic teleoperation by controlling a mobile robot just by using the gaze. This is an ongoing internal project in collaboration with Xtensa and Robosense.

The result has been achieved by combining various technologies and knowledges in a system made of: an industrial PLC/PC, a robotic platform, an eye tracker system and a local robotic manager.

The system is just an early prototype but the results are very impressive since we managed to perform maneuvers with the same ease of a joystick.

In the video the second successful trial, on two tests (100% success rate), of Malvina that maneuvers the robot outside the lab.

Skeletonization with crutches, preliminary tests

The improvements in 3D technology that took place in the last 5 years led to the creation of innovative solutions in the most various and different application fields.
Last activity of the MIRo Lab was focused on the utilization of such technology for health care and well-being applications, in particular for the diseases and pathologies related to mobility impairment.

The exoskeleton represents a very promising technology for those persons who suffer such state, it could potentially improve their lifestyle and way of moving. However the technology is not intuitive and quite scary at the first contact. Commonly the user must attend a long training phase, always followed by medics and and technicians that help him to fully understand how the exoskeleton works. Such strategy is slow, expensive and in most of the cases not efficient nor effective. It is quite difficult to explain and to show how to properly use an exoscleleton unless beeing the same user: it is a matter of self consciusness, motion perception, balance etc.

A possible better solution could be represented by an innovative training structure, aimed at providing a dynamic feedback directly to the user while performing specific trainings, allowing him to see himself and to understand where or what he is performing correctly or not. With modern virtual reality tools and 3D technology it will be possible to create virtual/mixed/augmented reality optimized for such purposes, decreasing in this way the number of supervising persons while increasing the effectiveness of the training

Currently MIRo is considering the development of a measuring system able to track the posture and motion of the user wearing an exoskeleton (currently under submission for the PRIN 2015), first step for the creation of an avatar of the user and the associated virtual reality.

Possible issue for such application are the crutches, necessary for the use of the exoskeleton. MIRo performed some preliminary tests in order to verify the behaviour of the skeletonization algorithm with such operative condition.
Here a video example showing a the correct identification of the upper limbs using a the crutches:

Gamocap & Memscon

MEMSCON project aim is to test a new-generation monitoring system based on MEMS and wireless transmission. This is an integrated system consisting of a set of small acceleration and strain sensors, distributed within a concrete structure. This unit automatically records and interprets the data, and provides the user with a damage index representing the structural safety after a seismic event. To test these sensors a concrete structure, Figure 1, was stressed with a shock wave that replicate a real earthquake.

memscon concrete structure
Figure 1: Concrete structure on which MEMSCON is tested

To assess the reliability of these algorithms the output of the MEMS monitoring system were compared with those coming from other measuring systems, such as linear voltage displacement transducers (LVDT) and multi-stereo camera system, GaMoCap.

GaMoCap system was used to map the displacements of marker placed at the base of one pillar, Figure 2-left, and at the intersection of the pillar with the upper part of the concrete structure, Figure 2-left.

base and head of the pillar
Figure 2-left: Base of the pillar with markers Figure 2-right: Head of the pillar with markers

In Figure 3 is reported a draft of the GaMoCap configuration that display the disposition of the 4 cameras, the connection with the pc and the trigger device.

draft of Gamocap system
Figure 3: Disposition of cameras, pc and trigger device of GAMOCAP system

Images captured were saved directly on the RAM of a PC and at the end of the test transferred on an hard disk. The camera setup used provided images with a resolution of 1600x1200 pixel at 10 Hz. Trigger is given by an external source to assure synchronisation between the 4 images.

 In Figure 4 is reported 3D points and the mesh reconstructed for the base of the pillar and the marker at the head.

3d points reconstructed at the base and head of the concrete pillar
Figure 4: 3D points meshed and trajectory of points at the top of the grid of the base and head of the pillar

Below there is video of the 3D meshed-points reconstructed during one trial with a maximum displacement at the top equal to 34 mm and the trajectory in Y direction of the point at the head of the structure.

Displacement results measured with vision system GaMoCap were compared and validate also by the actuator displacement measurement system, Figure 5. 

comparison of displacements measured by Gamocap and the hydraulic actuator
Figure 5: Comparison of displacements measured by Gamocap and the hydraulic actuator

The measurement campaigns were conducted in the framework of the MEMSCON project headed as Task Manager and Dissemination Manager by Assistant Professor Ing. Zonta Daniele, with the help of Ph.D students Trapani Davide.

Gamocap & Induse

The INDUSE project is motivated by the need of safeguarding the integrity of industrial equipment steel structures (liquid storage tanks, pressure vessels and piping systems), under earthquake loading.

In order to characterize the dynamic behavior of piping systems, Univeristy of Trento performed some pseudo-dynamic tests with dynamic sub-structuring on a typical industrial piping network, Figure 1. This type of test is a novel hybrid method for the dynamic testing of structures or systems.

draft piping system
Figure 1-left: Draft of the piping system tested Figure 1-right: Photo of the piping system realized at
University of Trento

GaMoCap, the multi-camera vision system and a color-coded pattern attached to the structure, Figure 2, was used to evaluate the map of displacements of the pipe structure at elbow during a simulation of a earthquake.

pipe system with color-coded markers
Figure 2: Stereo camera system and color-coded marker attached on the elbow of the pipe.

With respect to commercial systems already available on the market the GaMoCap has a significant advantage because it can recover a high density (50  markers in 100 cm^2) measurement of displacements and deformation of the structure in analysis maintaining high accuracy with respect to dense vision reconstruction methods that can outperform wrt resolution but can achieve a lower level of accuracy making them not suitable for measurement systems.

Below is a video of the 3D points reconstructed during a simulation of a earthquake with color-coded displacements.

In Figure  3 are reported the key-frames of the video. It is possible to see how the pipe during the trial bend in the middle of the elbow.

key frames of measurement
Figure 3: Key frames of point reconstruction with colour-coded displacement respect time 0.

Maximum and minimum displacements are compatible with the one obtained from the analysis of the numerical model of the structure.

The measurement campaign were conducted in the framework of INDUSE projects headed as P.I. of Local Unity of INDUSE at Univeristy of Trento by Prof. Bursi Oreste, with the help to Ph.D students Abbiati Giuseppe and Md. Shahin Reza.

Gamocap & Lognature

Objective of LOGNATURE project was to characterize the response of an experimental wooden wall fixed to the ground to a lateral force.

To evaluate the response of the structure were executed two trials: one applying the force in only one direction, Figure 1 - Trial 1, the other with a periodical lateral one (push pull), Figure 1 - Trial 2. The two trials last respectively 21 and 83 minutes.

Figure 1: Wooden wall in trial 1 and trial 2

 A multi-camera vision system, GaMoCap, was used to evaluate the displacements of the surface of the wall and the relative displacements of the planks of wood that compose the surface of the wall.

The multi-camera system is composed by 4 industrial cameras mounted on a aluminum frame Figure 2-a, pointing at the surface to reconstruct Figure 2-b.

frame and cameras used FOV of cameras
Figure 2-a: Aluminum frame on which are mounted the 4 cameras Figure 2-b: FOV of the 4 cameras

Capture frequency was set equal to 2 Hz. Camera resolution was 1600 x 1200 pixels.

To evaluate the displacements of the structure were used color-coded pattern of circular marker, Figure 3-a that covers all the planks to analyze.

A chess marker Figure 3-b was also attached on the head of the actuator to compare displacement results calculated with vision system.

marker employed
Figure 3-a colour coded marker for the planks.
Figure 3-b marker attached to the head of the actuator

 In Figure 4 we can observe the 3D points reconstructed (a)  and the mesh build on them (b).

points 3d reconcstructed and mesh
Figure 4-a: 3D points reconstructed.  Figure 4-b: mesh build on the 3D points.

In Figure 5 we can see the displacement values relative to the first trial of the marker at the top of the grid in the 3 directions.

Trial 1 - traejctories of encircled point
Figure 5: Trial 1 - trajectories of encircled point in X Y and Z directions.

Below the video relative to the first trial. To be notice is the step of X coordinate at ~1200 secs due to the break of the metal bracket that holds down the wall.

In Figure 6 are reported the displacement in Y direction of 3 points, respectively located at the base (A), in the middle (B) and at the top of the grid C, for the trial of alternate trust. The point (M) is the point relative to the marker attached to the head of the actuator.

Lognature Trial 2
Figure 6: Trial 2 - Trajectories A, B, C and M in Y direction


The measurement campaigns were conducted in the framework of the projects LOGNATURE headed by Prof. Piazza Maurizio, with the help of Assistant Professor Ing. Roberto Tomasi, and the Ph.D student Paolo Grossi.

София EMSIEN-3