Building and Evaluating an Intent-to-Code Agent for Enterprise Software Customization

Historically, business stakeholders had to explain their intents to professional software engineers which would then develop needed customizations for Enterprise Software. With the increase in coding capabilities of LLMs, having the stakeholders directly explaining their business intent to an agent able to codify and configure customizations is not far-fetched anymore. We develop an Intent-to-Code agent capable of authoring customizations in our Enterprise solution and benchmark a selection of models in their ability of accomplishing this task. We contribute several findings that will be useful for groups exploring coding applications in Enterprise environments, as well as more generally applications where sequential tool calls have to be accurately executed in order.