Improving Instrument Detection for a Robotic Scrub Nurse Using a Multi-Frame Voting Scheme

A fundamental task of a robotic scrub nurse is surgical instrument detection. Errors in the detection process can severely interrupt the surgical workflow of a medical procedure, and therefore, near-perfect performance is required. Despite constituting powerful potential solutions, deep learning methods are subject to imperfections. Even if a scene remains unchanged, the predictions of a trained model can vary from one frame to the next. Moreover, for the instrument detection task, the presence of similar-looking instruments and changing lighting conditions promote misclassifications. In this work, we rely on an RGB-D camera, mounted on a robot manipulator in an eye-in-hand configuration, and we propose a multi-frame voting scheme for instrument detection using Mask R-CNN [1]. The predictions on multiple individual frames of a given scene are used as votes and employed for the determination of a final prediction. We consider different robot poses with the goal of capturing the important visual features of the instruments in the scene. From every pose, the predicted data of the instruments are projected onto a common plane, where the predictions are matched for our voting scheme. With this strategy, we expect to significantly improve the performance of a previously trained model and contribute toward the goal of error-free instrument detection.