Methodology

Use case overall objective

Enhance (physically and functionally) the prosthesis with different sensors (high-density EMG, IMU) and AI-based algorithms to:

• Guarantee grasp stability with a variety of objects and conditions and inform the user about incipient fault conditions, thus reducing the cognitive burden for the user
• Reduce problems due to the misplacement of the EMG sensors guaranteeing a suitable level of performance by introducing general AI-based feature recognition
• Reduce the training time procedure of the AI by employing persistent learning capabilities
• Increase acceptability for amputees by leveraging on adaptive shared autonomy to set the autonomy threshold to a suitable level in the manipulation task structure, decreasing in turn the abandonment of prostheses by improving embodiment and reducing the phantom limb-related complaints
• Demonstrate the capability of transferring manipulation tasks knowledge from other use cases).

Development platform

Modular arm prosthesis Hannes Arm (adaptive to all levels of amputation).

Proposed solution approach

A prosthesis that can interact with the human and the environment in an intelligent and adaptive way through shared autonomy – a framework describing how human and prosthesis exchange information and interact with the environment based on an arbitration mechanism. Task control is assigned either to the human or the prosthesis, depending on capabilities and intentions. Feedback to human is performed of the through haptics. It is expected to leverage safety and reliability of movements and grasping through automatic failure detection and through intent detection algorithms.

Dynamic changes in role arbitration are meant to either increase the prosthesis level of autonomy at the expense of the human’s authority, or, conversely, increase the human’s control over the shared cooperative activity at the expense of the prosthesis autonomy. By shared autonomy, human and prosthesis divide the subtasks of a complex activity (co-activity, used when the human is unable to actuate the prosthesis and must rely on it carrying out intelligently the communicated intent). Arbitration’s supervisory role:

• Manage latency between amputee and prosthesis
• Increase effectiveness and efficiency of the task
• Improve embodiment and acceptability of the prosthesis.

IntelliMan prosthesis control strategy: environment, prosthesis, sensors communicate with the central part, which processes the commands to send to the prosthesis.

Demonstrators to this use case are available here.

Use case overall objective

Expand robotic manipulation capability in daily-life human-centred (kitchen) environment to perform (kitchen-aid) representative tasks:

• Understanding manipulation task structure of fragile objects and planning complex manipulation strategies with one or two coordinated arms, including in-hand manipulation
• Manipulating objects with movable parts and understanding the underlying kinematics
• Synthesizing different task-oriented grasps for objects with different shape, size and surface properties
• Picking objects in a cluttered scenario which may require physically interacting with the environment with non-deterministic results, ensuring reliability by focusing on the detection of feasibility and stability conditions and on relevant features need to accomplish the task and reusing previously learned methods
• Human-Robot Interaction in teaching of the manipulation actions and in robot-to-human handovers (i.e., understanding whether the human is available to receive the object – maybe he/she is doing other things -, where he/she wants to receive it – he/she may offer a hand, and for which part of the object – a knife must be offered with the handle free) or human-to-robot handovers (e.g., handing over an object impossible to reach because out of the robot workspace) 
• Demonstrate the capability of transferring manipulation tasks knowledge from and to other use cases (prosthetic aspects of UC1, manipulation of deformable objects from UC3 and manipulation of soft vegetables and fruits from UC4) and other tasks and applications.

Development platform

• Dual-Arm and Single-Arm TIAGo Mobile Manipulators
• Dual-Arm Universal Robots UR5 manipulator on a mobile platform
• Robot arms extended by parallel-finger grippers
• Tactile sensors for perception during grasping
• Kitchen Mockup equipped with OptiTrack tracking system and 3D cameras.

Defined scenario

Prepare breakfast in a mock-up kitchen by:

• Setting the table, retrieving plates and glasses from the dishwasher
• Retrieving cutlery from drawers and place them on the table
• Pouring cereal into a bowl from cupboard
• Handling soft and deformable objects by selecting pieces of fruit and serving them on the plates.

Demonstrators to this use case are available here.

Use case overall objectiv

Enabling the robotic assembly of connectors leveraging on AI-powered manipulation with industrial arms and parallel grippers by:

• Learning the manipulation and picking of small deformable objects with limited trails, such as the cable with metallic terminal located in a collector
• Learning the kinematic relationship and the salient manipulation features between the terminal and the connector case, to insert the terminal with proper orientation and position
• Checking the correct or incorrect execution by evaluating the force profile by means of tactile sensors
• Complete the insertion of all the cables in the connector case a semantic sequence description
• Check the result using both tactile and vision sensors
• Evaluated the capability of transferring the insertion tasks knowledge to other robotic systems.

Development platform

• Dual-arm Universal Robot UR5 collaborative robotic platform
• Robotic grippers: Robotiq Hand-E adaptive grippers and suitable sensorized fingers
• Sensors for cable grasping and evaluation of insertion phase correctness: tactile sensor and ToF proximity or multi-range sensors
• Vision system based on multiple cameras used to detect the location of the objects during the task execution: 2D camera for collecting multiple views for 3D shape reconstruction, 3D camera for depth information reconstruction, endoscopic camera to detect cable axial orientation to perform insertion in connector case.

Defined scenarios

1. Connector in a known position, wires grasped from known positions, insertion and checking with a single arm
2. Connector in a known position, wires grasped from a collector, estimation of wire pose, insertion and checking with a single arm
3. Connector grasped from a known position, wires grasped from a collector, estimation of wire pose, insertion and checking with two arms
4. Connector grasped from a collector, estimation of connector pose, wires grasped from a collector, estimation of wire pose, insertion and checking with two arms.

Demonstrators to this use case are available here.

Use case overall objective

Expand the capability to the robotic pick and place of loose fragile items like delicate fruits and vegetables:

• Study the use case for various objects with increased complexity of shape, fragility and clutter
• Enabled to picking from a dense environment – pile of fruits in the storage container
• Capable of packing in a punnet or a small container to achieve required fill or number of items
• Reduce damage during picking and discard already bruised items
• Learn and apply human picking and packing strategies.

Development platform

to be added soon

Proposed solution approach

to be added soon

Defined scenarios

to be added soon

Demonstrators to this use case are available here.