Blind Robotic Grasp Stability Estimation Based on Tactile Measurements and Natural Language Prompts

We design and train a composition of neural network modules that predicts robotic grasp success based on tactile sensor measurements and natural language prompts identifying the object. We use a Franka Emika Panda robot arm equipped with two DIGIT sensors for grasping and language descriptions generated by chatGPT. Our short-term goal is to utilize this approach to improve the accuracy of a grasp stability estimator.The longer-term goal of this work is to enhance haptically driven robot control with language-based context, i.e. task-relevant information which might not be robustly inferred from vision.