When Tools Look Alike: Evaluating Semantic Confusion in LLM Tool Selection

Large language models (LLMs) are increasingly used as agents that interact with external tools such as search engines, databases, calculators, and application programming interfaces. A fundamental component of these systems is tool selection: given a user request and a collection of available tools, the model must identify the tool capable of performing the requested operation. While existing work has shown that LLMs can be sensitive to tool descriptions and candidate presentation, less is understood about how the semantic similarity between available tools affects selection reliability when the underlying task and number of candidate tools are held constant. We study how semantic similarity within a tool library affects LLM tool-selection reliability through a controlled evaluation framework. Each user query has a known target tool and exactly five candidate tools. We vary only the relationship between the target tool and four distractors. In a low-confusion condition, distractors perform unrelated functions; in a medium-confusion condition, they operate in the same application domain but perform distinct operations; and in a high-confusion condition, they expose closely overlapping capabilities. By holding candidate-set size and the user query fixed, this design isolates semantic ambiguity from effects caused by increasing context length, toolset size, or task difficulty. In an initial evaluation on 50 tool-selection tasks, we observe that tool-selection accuracy decreases as candidate tools become more semantically similar to the correct tool. Because the query, target tool, and number of available actions remain unchanged across conditions, this preliminary finding suggests that the structure of the available action space itself can contribute to tool-selection failures. Building on this result, we examine whether tool documentation can mitigate semantic confusion. We compare standard tool descriptions, length-matched detailed descriptions, and contrastive boundary descriptions that explicitly distinguish neighboring tools by specifying when a tool should and should not be selected. Comparing detailed and contrastive descriptions allows us to investigate whether explicitly communicating decision boundaries provides benefits beyond simply supplying additional descriptive information. We additionally examine selection stability by randomly permuting otherwise identical candidate lists and recording whether model choices change. Our evaluation considers target-tool accuracy, confusion between semantically neighboring tools, and consistency across candidate permutations and LLMs of different scales. By controlling semantic similarity, documentation style, candidate-set size, and presentation order, this work provides a framework for distinguishing errors caused by ambiguity among available actions from those caused by the user request itself. Our preliminary results suggest that increasing semantic similarity among candidate tools can increase tool-selection error, motivating lightweight approaches to improving tool documentation without fine-tuning or modifying the underlying language model.