IntelliMan Pert Chart
WP3 – Interactive Learning and Perception
Objectives: data formats in robotics vary significantly across tasks, environments, users and platforms (different sensors and actuators, not only in formats but also in modalities and organizations)
- Ensure knowledge portability and facilitate knowledge transfer between different domains, platforms and systems
- Learn manipulation task structure and hierarchy from sensory measures and human cues that can be used in a wide range of tasks
- Reduce the need of new training through samples relying on only a few demonstrators or trials
- Guarantee performance, safety and fault detection.
Fusion of control and sensing data as a product of experts
Objective: creation of a framework to handle control and sensing data in a unified way, to easily combine information from both control actions and objectives described in the state space.
Solution: comprehensive and versatile framework based on a geometric algebra approach that combines and extends popular formalisms like screw theory, Lie algebra and dual quaternations. By providing a unified method for modelling cost functions in optimal control, which can be expressed uniformly across different geometric primitives, this framework has the potential to revolutionize the way robot manipulation tasks are treated. Due to the inherent nature of geometric algebra, a reduction in symbolic complexity of the resulting mathematical expressions involved in robot manipulation is achieved. Geometric algebra provides the same geometric representation for states and actions (both geometric objects and transformations use the same algebra) and affords a higher level of geometric intuitiveness in understanding and interpretation of the underlying mathematical concepts. It facilitates researchers and practitioners to reason about and analyse robot manipulation skills towards more effective and efficient solutions.
Publication: “Geometric Algebra for Optimal Control with Applications in Manipulation Tasks”, IEEE Transactions on Robotics, investigates manipulation tasks that consider the various constraints of the use cases UC1 – UC4.
Example: holding a tray for kitchen UC2, a cooperative dual-task space involving two manipulators.
Example: task-oriented stiffness, based on stiffness ellipsoids for choosing the best intrinsic stiffness for a variable stiffness actuator that guarantees the required force application capability while keeping the system as adaptable to uncertainties as possible.
The approach is tested in grasping applications, where the fingers can resist the weight of the object while being as compliant as possible to wrap or cage around the object shape.
Extraction of low dimensional manifolds with tensor factorization
Objective: investigation of (two) approaches for extracting low dimensional representations.
1. Tensor train methods for low-rank decomposition of tensor data, enabling efficient sampling and conditioning over decision variables.
2. Use of basis functions providing a compression of data as multivariate piecewise polynomial approximation, to encode signed distance fields (SDFs) in 3D space by utilizing a sparse set of weights that can be updated incrementally through recursive ridge regression.
Multiresolution shape encoding with GPIS representation
Objective: extension of the low dimensional representation by focusing on multiresolution and uncertainty modelling aspects.
Multiresolution provides shape estimates with different granularity, according to the resolution required by the ongoing situation – progressive resolution increase while approaching the object. The modelling of uncertainty provides information about the regions to explore and on how much the robot can trust the current model of shapes for control, planning and trustworthiness aspects. The encoding of shapes as implicit SDFs is utilized as starting point for inquiring how basis functions can provide the required multiresolution and uncertainty modelling capabilities. It is investigated how SDF representation can be used to model the whole body of an articulated robot as SDFs and how this representation can be extended to model coordinate fields and Riemannian metric fields for planning and control tasks.
Exploration/exploitation with probabilistic implicit surface representation
To be started soon
Portable Reinforcement Learning for Grasping and Manipulation
To be started soon
WP4 – Adaptive Shared Autonomy
Objectives: in order to successfully operate in dynamic environments as in case of robotic manipulation applications, robots must be capable to deal with the environment’s complexity, avoid critical failures and be able to adapt to novel task features when requested.
Shared autonomy involving human operators facilitates the overcoming of the system’s lack of knowledge through exploiting human-robot interaction: by adjusting the level of autonomy during execution of tasks, the capability to deal with dynamic conditions and safety requirements is improved. The adaptation is performed each time a novel critical situation is detected during task execution, either by automatic detection through the robot system using sensor information, or by the human operator temporally requesting control on some specific autonomy levels. Advanced human-robot interaction modalities will be investigated through the development of appropriate hierarchical shared autonomy models.
Hierarchical shared autonomy models for adaptive behaviours
Objective: development of a novel hierarchical shared-autonomy framework for regulating grasp strength based on Hidden Markov Models (HMMs).
The framework is employed to detect the various phases within the grasping process, enabling a more precise understanding of the task at hand. The probabilities associated to each phase, that are being provided by the HMM, have been leveraged to adjust the user’s shared autonomy level. This adaptive modulation allowed for a finer and more accurate human-in-the loop (HITL) regulation of grasp strength.
Preliminary successful tests involving repeated grasp-carry-release tasks that explicitly require different grasp strength regulation, minimum, medium and maximum (using three different object, namely paper cup, plastic bottle and rigid box), have been performed. The regulation of the three different required grasp strength levels were successfully regulated with statistical significant difference.
Advanced human-robot interaction modalities
Objective: evaluation of different intent detection and interaction strategies.
Two simultaneous/proportional myocontrol control strategies for upper-limb prostheses based on an incremental learning approach on able-bodied users as well as on users with below-elbow limb differences have been conducted. The interaction strategy provided the user with simultaneous and proportional control over 3 DOF.
Moreover, feasibility of trust and agency in an upper-limb prosthesis via eyes-tracking with artificial failures is being assessed. The study yields also a database of labelled sEMG, eye-tracking and kinematic data during prosthesis interaction and failures.
A vibrotactile somatosensory feedback bracelet with semi-sphere shapes featuring two vibration motors has been designed.
Furthermore, an algorithm (based on Independent Component Analysis – ICA) for Motor Unit (MU) extraction from (16-channel) sEMG signals for gesture classification setup has been developed. The algorithm has an offline phase, where MUs are extracted during isometric contractions, and an online phase, where MUs are tracked during natural gestures.
Human-to-Robot (H2R) and Robot-to-Human (R2H) handover strategies using high level Finite State Machines (FSMs), Extended Kalman Filters and multimodal approaches combining haptic and visual cues have been investigated.
Human intent detection for autonomy arbitration
The intent is automated recognition of faulty intent detection by merging human-side and robot-side probabilistic models to improve medium- and long-term stability of HITL grasp control via on-demand incremental updates. This is realized by exploiting Ridge Regression-based (RR) human intent estimation applied to sEMG signals to obtain a measurement of confidence of the prediction of the grasp to be controlled, combined with using on robot side probabilistic model based on HMMs applied to tactile sensors, to measure the confidence on grasp reliability and stability.
Algorithms for autonomy arbitration
WP5 – Grasping, Manipulation and Arm-Hand Coordination
Objectives: produce a suitable hardware EMG-based platform for acquiring relevant information to analyse and learn from human performance in the use cases, as well as planning the actions required for grasping and manipulation in the use cases.
The design of grippers will be tackled from AI perspective and considering its sensorization, such that suitable data acquisition and fusion of information for exploiting model- and learning-based approaches facilitate the creation of novel grasping and manipulation approaches.
Data Fusion and Sensing Technology
Objectives: creation of data fusion and sensing technologies as basis technologies for other tasks and for the use cases.
Multi-sensorized fingers with finger-nails-like structure for wire manipulation for parallel grippers and multi-sensorized fingertips with interchangeable modules for multi-fingered grippers have been developed: tactile module with 2×6 matrix of taxels, proximity sensor with up to four time-of-flight sensors oriented in different directions, six DoF Inertial Measurement Unit (IMU) with temperature sensor, all with ROS software integration.
Three modular fingertips with different sensor integration have been developed, with low maintenance effort and no need for different end-effectors:
1. Multi-sensory fingertip with 4 triaxal HAL tactile sensors (4 taxels) one IMU, 1 ToF proximity sensor
2. Fingertip with 16 triaxial HAL sensor-based Xela tactile sensors
3. Multi-modal grasp fingertip with suction cups and electromagnets.
A medium-density sEMG acquisition and processing platform has been developed: 16-channel custom armband with dry soft-pulse electrodes, a BioGAP acquisition board @4ksps and a GUI to ease the storage of the sEMG data and the implementation of EMG-based human-robot interfaces.
Furthermore, a perception framework for 6D object pose estimation has been developed. Synthetic data is generated (by employing BlenderProc), used to train a YOLO V7 for bounding box detection. Detected bounding boxes are further processed for instance segmentation for clutter scenarios. The 6D pose is estimated and refined with depth data.
Additionally, a data-driven framework for manipulation of deformable linear objects (DLOs) has been developed. It uses an analytical model of the DLO for offline training, while the trained neural network model is then used online for gradient-based optimization of model parameters.
AI-Oriented Manipulator Design
Objective: development of a manipulator hand.
The solution encompasses:
1. A novel concept of three-fingered compliant gripper embedding variable stiffness actuators
2. A novel monolithic 2 DoF compliant wrist
3. A concept of an under-actuated low-cost anthropomorphic hand for possible usage as prosthesis.
Understanding and Reasoning Manipulation Task Structures
Objective: tbd.
Development of a tree-based world model representing robot knowledge about its surroundings and on its current state. Topological level description of the relation between different objects are encoded. Information is shared by the primary robot activities through a “tell and ask” interface. A multimodal grasp planner for hybrid grippers to choose an appropriate grasp modality suitable for a specific object in a specific scenario has been developed.
A schema of an ontology-based manipulation framework structure based on ontologies to represent manipulation knowledge has been created, being able to:
- Reason on perception, planning and execution
- Obtain situation, domain and execution awareness
- Adapt to potential changes through smart monitoring.
A planning algorithm for combined task and motion planning, with a new interleaved approach for integrating the two dimensions of TAMP making use of Sketches, has been created. Sketches, a recent powerful language for expressing the decomposition of problems into sub problems, is a collection of hand-crafted rules over state features and expresses domain knowledge.
Experience- and model-based Grasp Synthesis and Manipulation
To be started soon