Learning-Based Multimodal Control for a Supernumerary Robotic System in Human-Robot Collaborative Sorting

In this letter, a multi-modal learning and control framework is proposed for the control of a supernumerary robotic limb (SRL). The SRL is a wearable robotic arm designed to enhance the manipulation capabilities of its human user and extend the workspace by reaching greater heights. The multi-modal learning interface incorporates a neural network to estimate the user's hand position in space and to replicate the human grasp force during object handovers. This is essential to ensure that the grasp forces applied by the user are accurately mimicked, thus achieving a reliable grip, which may vary depending on the object's characteristics, such as its weight. The interface also includes a Gaussian Mixture Model (GMM) based intention estimation algorithm to identify the target reaching locations by the SRL. Electromyography and IMU information collected from a single armband are the only input information provided to the multi-modal learning interface. The execution of the target locations and grasp forces is achieved through the trajectory planner and the controller of the SRL. To evaluate the system and its control framework, we initially compute the coefficient of determination for grasping force regression, yielding a high value of 0.993. In addition, the hand position prediction error is determined to be 1.99 cm ± 1.39cm. Next, the overall SRL is tested in a human-robot collaborative sorting task through a user study with 7 participants. The results show the effectiveness of the proposed SRL control framework.