Semantic Parsing and Grounding of Assembly Instructions With Ontologies

Embodied agents need learned perception and language understanding together with symbolic models that expose goals, preconditions, and action effects. Assembly instructions are a demanding testbed for this integration: actions must be grounded in multimodal observations while respecting a changing configuration of parts. Existing datasets often encode actions as verb--object labels, leaving the structural constraints that determine valid execution implicit. We treat these labels as surface realizations of a richer model of objects, states, and state-changing activities. Our framework maps candidate instruction groundings to first-order representations shared by a domain ontology, the Process Specification Language (PSL), and a domain process ontology. We instantiate the approach for \textit{attach} and \textit{detach} and implement temporal projection in Prolog. Two demonstrations show symbolic simulation of a drawer-installation sequence and rule-based rejection of disconnected candidate subassemblies. The logical forms provide an interpretable neuro-symbolic interface between multimodal annotations, action-model verification, and assembly planning.