Schema-Constrained LLM-Based Entity Extraction for Geometry Word Problems

Schema-Constrained LLM-Based Entity Extraction for Geometry Word Problems An human being solving a geometry word problem can identify the geometry properties with naked eye such as shape, angle , sides , relationship between geometric entities and what needs to be found.For a computer, to recognize all this needs information extraction from text as well as diagram. Geometry problems also contains symbols whose meaning changes with context. For example, FGH may refer to a triangle in the problem text “triangle ABC,” but the same letters may represent an angle in “angle FGH.” This miss-identification of geometry entities makes it difficult to automate this system. Earlier work on geometry word problems introduced a domain-specific Named Entity Recognition (NER) scheme to identify geometric information before converting the problem into a machine-readable form using NER Annotator [1]. Eleven entity categories are defined: FIND, SHAPE, SHAPE VARIABLE, SEGMENT VARIABLE, SEGMENT VALUE, ANGLE VARIABLE, ANGLE VALUE, POINT, TYPE, GEO TERM, and OTHER VALUE. A supervised spaCy model is trained on manually annotated geometry questions to recognize these entities. However, the NER component is used mainly as an intermediate step for predicate generation, and its performance was not evaluated independently [1]. Later work incorporated the same geometric tags into a sequence-to-sequence attention-based model for generating structured logic forms [2]. That study also observed that errors during entity recognition can result into the generated representation; for example, a point may be incorrectly interpreted as an angle-related entity. In this work, we focus directly on the extracting the geometric entities form the problem text of geomtey focusing whether large language models (LLMs) can recognize geometry-specific entities without requiring a separately trained NER model. We propose GeoNER-LX, a structured extraction approach implemented using LangExtract [3]. Instead of training the LLM on hundreds of labelled examples, we describe the eleven geometry entity classes in natural language and ask the model to extract only information belonging to these classes. We study the approach under different levels of guidance. In the zero-shot setting, the LLM receives only descriptions of the geometry entity classes. In the few-shot setting, a small number of correctly annotated geometry questions are also provided as examples. We further study a schema-constrained few-shot setting in which the model is restricted to the predefined geometry labels. These settings allow us to examine whether providing a small number of examples helps the LLM understand geometric entities. The evaluation uses a manually annotated collection of more than 500+ elementary and middle-school geometry word problems covering triangles, circles, rectangles, squares, rhombuses, parallelograms, and trapeziums. We will compare the LangExtract-based methods with the supervised spaCy NER model using standard evaluation measures such as precision, recall, F1-score, and class-wise F1-score. The experiments are still ongoing.This study is done to identify whether LLM-based extraction improves recognition of context-dependent entities such as SHAPE VARIABLE, ANGLE VARIABLE, and SEGMENT VARIABLE, and whether it can correctly distinguish numerical information such as segment, angle, and other geometric values. References [1] A. Boob, P. Bodakhe, M. A. Radke, and U. A. Deshpande. “Extracting Structured Information from the Textual Description of Geometry Word Problems.” In 2023 7th International Conference NLPIR 2023, December 15–17, 2023, Seoul, Republic of Korea. ACM, New York, NY, USA. https://doi.org/10.1145/3639233.3639255 [2] Boob, A., Landa, T., Devi, D.R. et al. Extraction of structured logic forms from geometry problem descriptions. Knowl Inf Syst 68, 163 (2026). https://doi.org/10.1007/s10115-026-02763-9 [3]A. Goel, “LangExtract,” Version 1.6.0, Google, 2026. doi: 10.5281/zenodo.21126643..