Capturing fine-grained details for video-based automation of suturing skills assessment

With current medical interest in the minimally invasive surgery paradigm, there has been palpable interest in the nascent field of machine learning in robot-assisted surgery. Consequently, several attempts have been made to train neural networks that extract intelligible data from robotically controlled surgical procedures.

These attempts have focused on single data points (e.g. action recognition or surgical skills assessment), and have barely reached model training thresholds adequate enough to be deemed useful in the high-stakes surgical domain.

In this paper, we propose a neural network training regime that accounts for both surgical action recognition and surgeon skills assessment while also training above prior validation accuracy benchmarks. More specifically, we use an attention mechanism that mimics the visual perception attention mechanism humans use to solve domain specific tasks.

To incorporate an attention mechanism in the action recognition and skills assessment processes, our attention implementation simultaneously recognizes three information benchmarks: the visual information in each frame, knowledge of the ongoing task(s), and the spatial attention in previous frames. Our implementation resulted in a 20 percentage-point-average increase in top-1 validation accuracy of all surgical action recognition and skills assessment tasks.