Timezone: »

 
Poster
ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment
Joseph DelPreto · Chao Liu · Yiyue Luo · Michael Foshey · Yunzhu Li · Antonio Torralba · Wojciech Matusik · Daniela Rus

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #1025
in Poster Session 1 »

This paper introduces ActionSense, a multimodal dataset and recording framework with an emphasis on wearable sensing in a kitchen environment. It provides rich, synchronized data streams along with ground truth data to facilitate learning pipelines that could extract insights about how humans interact with the physical world during activities of daily living, and help lead to more capable and collaborative robot assistants. The wearable sensing suite captures motion, force, and attention information; it includes eye tracking with a first-person camera, forearm muscle activity sensors, a body-tracking system using 17 inertial sensors, finger-tracking gloves, and custom tactile sensors on the hands that use a matrix of conductive threads. This is coupled with activity labels and with externally-captured data from multiple RGB cameras, a depth camera, and microphones. The specific tasks recorded in ActionSense are designed to highlight lower-level physical skills and higher-level scene reasoning or action planning. They include simple object manipulations (e.g., stacking plates), dexterous actions (e.g., peeling or cutting vegetables), and complex action sequences (e.g., setting a table or loading a dishwasher). The resulting dataset and underlying experiment framework are available at https://action-sense.csail.mit.edu. Preliminary networks and analyses explore modality subsets and cross-modal correlations. ActionSense aims to support applications including learning from demonstrations, dexterous robot control, cross-modal predictions, and fine-grained action segmentation. It could also help inform the next generation of smart textiles that may one day unobtrusively send rich data streams to in-home collaborative or autonomous robot assistants.

Keywords: Neural Networks · machine learning · experimental design · Attention · Video · kitchen activities · activities of daily living · tactile sensing · muscle activity · cameras · Wearable sensors · eye tracking · multimodal dataset · microphones · motion tracking · learning pipelines · robot assistants · EMG · gaze · body tracking · multimodal recording · human subjects · RGBD · recording software · open-source · audio · joint angles · depth
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Joseph DelPreto (MIT)
Joseph DelPreto

Joseph is a Postdoc in the Distributed Robotics Lab at MIT, where he works with Daniela Rus to develop wearable systems for improving our understanding of human actions or enabling more natural human-robot collaboration. He obtained a PhD from MIT in Electrical Engineering and Computer Science, focusing on using biosignals to supervise, control, or teach robots. He previously obtained a Master of Science degree from MIT in Electrical Engineering and Computer Science with a minor in Finance and Business, and a Bachelor of Science degree from Columbia University in Electrical Engineering with minors in Mechanical Engineering and Psychology.

Chao Liu (Computer Science and Artificial Intelligence Laboratory, MIT)
Yiyue Luo (Computer Science and Artificial Intelligence Laboratory, Electrical Engineering & Computer Science)
Michael Foshey (Massachusetts Institute of Technology)
Yunzhu Li (Stanford University)
Yunzhu Li

Yunzhu Li is a Postdoctoral Scholar at Stanford University working with Prof. Fei-Fei Li and Prof. Jiajun Wu. He received his Ph.D. from MIT and will join the Department of Computer Science at the University of Illinois at Urbana-Champaign (UIUC) as an Assistant Professor in Fall 2023. His work stands at the intersection of robotics, computer vision, and machine learning, with the goal of helping robots perceive and interact with the physical world as dexterously and effectively as humans do. Yunzhu received the Adobe Research Fellowship and was selected as the First Place Recipient of the Ernst A. Guillemin Master's Thesis Award in Artificial Intelligence and Decision Making at MIT. His research has been published in top journals and conferences, including Nature, NeurIPS, CVPR, and RSS, and featured by major media outlets, including CNN, BBC, The Wall Street Journal, Forbes, The Economist, and MIT Technology Review. Before coming to MIT, he received a B.S. Degree from Peking University. He has also spent time at the NVIDIA Robotics Research Lab.

Antonio Torralba (MIT)
Wojciech Matusik (MIT)
Daniela Rus (Massachusetts Institute of Technology)

More from the Same Authors