Moderator: Kumar Garg, Managing Director and Head of Partnerships, Schmidt Futures

Panelists include: Steve Ritter, Founder & Chief Scientist, Carnegie Learning Heejae Lim, Founder & CEO, TalkingPoints
Jeremy Roschelle, Executive Director , Digital Promise

Moderator: Kumar Garg, Managing Director and Head of Partnerships, Schmidt Futures

Panelists: John Whitmer, Former Senior Director of Data Science & Analytics, ACTnext Aigner Piccou, Program Director, The Learning Agency Lab

In 2007, Professor Fei-Fei Li started assembling a massive dataset of 14 million pictures, labeled with the objects that appeared in those images. This dataset, dubbed ImageNet, spurred dramatic progress over the next decade in computer vision, the field of artificial intelligence that trains computers to understand images and videos. Such datasets can serve as “benchmark” challenges that researchers compete on, and incentivize advancements in fundamental and domain-specific fields.

We sent solicited ideas for a potential dataset that could drive a similarly transformative impact in education. Applicants submitted 300 word abstracts, as we selected a few to showcase.

Please use this time to listen to the recordings at the bottom of the schedule to learn more about the benchmark data set ideas.

Moderator: Kumar Garg

Panelists: Richard Tang, Student, University of California, Berkeley Ajoy Vase, COO, the Learning Collider at Teachers College, Columbia University Ken Koedinger, Professor of Human Computer Interaction and Psychology, Carnegie Mellon University

Q&A to follow

Moderator: Neil Heffernan, William Smith Dean's Professor of Computer Science at Worcester Polytechnic Institute, and Co-Founder of ASSISTments

Panelists: Osonde Osoba, Senior Information Scientist, RAND Corporation Emma Brunskill, Assistant Professor in the Computer Science Department, Stanford University Kathi Fisler, Research Professor, Brown University

Will be followed by a 10 minutes Q+A

Abstract: In this breakout session, we propose the idea of an "ImageNet for Teaching Computer Science." The proposed idea involves collecting a large set of labeled programming datasets from classrooms, using a shared format, and developing a set of benchmarks and challenges that will facilitate research for K-20 computing education. This data would benefit a growing research community at the intersection of computing education and learning analytics, with implications for students across many fields that teach computing.

Rationale: Programming data is ideal for learning analytics/edu data mining, since it is rich, capturing students' every state and action as they work, and the data is structured by syntax rules. Advances in programming analysis techniques for open-ended, sequential, and semi-structured data will have broad applications across educational domains. However, recent advances in deep learning require larger datasets and more meaningful labels than those typically available from individual classrooms, necessitating cross-institutional data collection and labeling efforts.

Background and Progress: A series of workshops from CS-SPLICE (https://cssplice.github.io/) and CSEDM (http://go.ncsu.edu/csedm2020) have brought together the research community to develop infrastructure and analysis techniques for programming data. The community has developed the shared ProgSnap2 format for programming log data (https://go.ncsu.edu/progsnap2), which is already used by 10+ datasets, comprising 750,000+ program snapshots in various languages (many of the datasets can be found on https://pslcdatashop.web.cmu.edu/). Researchers have used this data to develop automated support (e.g. hints, feedback, curated examples), predict student success, and personalize interventions. The CSEDM Data Challenge (https://go.ncsu.edu/csedm-dc) is a recurring data mining competition (held 2019, planned 2021) to gain insight from classroom programming data, which has helped to define shared machine learning benchmarks on common datasets.

Next Steps: The key challenges will be collecting diverse existing datasets, and creating infrastructure to support collecting and labeling new data. This will allow us to tackle novel research challenges, such as generalizing algorithms and labels across problems -- for example detecting knowledge components, strategies, or misconceptions on one problem using data from others. The CS-SPLICE and CSEDM communities include developers of many widely-used educational programming platforms and will be important stakeholders in driving the work forward.

Acknowledgements: This reflects joint work by Thomas Price, Tiffany Barnes, Min Chi, Samiha Marwan, Yang Shi, Preya Shabrina, and Ye Mao at NC State University. It presents and builds on ideas and foundational work by the CS-SPLICE and CSEDM teams.

Authors: Zac Hancock, Michael Chifala, Callie Federer, Jiamin He, & Quinn N Lathrop

One of the best ways to learn and practice math is by hand on paper. Digital math applications can take advantage of this natural interaction by including a handwriting recognition capability. We introduce a dataset that can be used to create such models to bridge math learners and digital applications. Given the importance of mathematical expressions across all scientific branches, including physics, engineering, and economics, this dataset can become an important resource for advancing the use of machine learning for the benefit of education.

Our dataset (available at www.kaggle.com/aidapearson/ocr-data) consists of 100,000 images of handwritten math expressions within calculus. The images are synthetically generated which affords 100% correct pixel-level tagging and results in realistic images capable of training models whose performance generalize to real images. It has a very permissive license, the full collaboration tools of Kaggle, and standard data formats that increase generalizability and usability. The dataset offers something to all levels, from beginners building simple character recognition models to experts who wish to predict pixel-by-pixel masks with object detection models and decode the complex structure of math expressions.

The most similar dataset is CROHME, which provides digital ink with stroke data. Our dataset differs in that it focuses on images of math and covers a targeted scope of limits expressions. Also, because our dataset is generated, this scope of math could be changed as needed and the size of the dataset is limited only by practicalities.

The ease of use and richness of the dataset will hopefully excite ML researchers within education and draw new ML researchers to the field. Applications beyond handwriting recognition include translating students’ math to pdfs and automated grading for instructors. ML capabilities built using this dataset would benefit many educational institutions by helping to connect the natural mode of math learners and digital educational applications