Paper 32: Conversational Programming with LLM-Powered Interactive Support in an Introductory Computer Science Course

Chatbot interfaces for LLMs enable students to get immediate, interactive help on homework assignments, but doing so naively may not serve pedagogical goals. In this workshop paper, we report on the development and preliminary deployment of a GPT-4-based interactive homework assistant for students in a large introductory computer science course. Our assistant offers both a “Get Help” button within a popular code editor, as well as a “get feedback” feature within our command-line autograder, wrapping student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. We have found that our assistant can identify students' conceptual struggles and offer suggestions, plans, and template code in pedagogically appropriate ways---but sometimes inappropriately labels correct student code as incorrect, or pushes students to use correct-but-lesson-inappropriate approaches, among other failures, sometimes sending students down long frustrating paths. We report on a number of development and deployment challenges and conclude with next steps.