Can LLMs Reliably Detect Subtle Sexism? Bias, Confidence, and Failure Modes in Multi-Category Sexism Detection for Online Political Discourse

LLMs are increasingly proposed as low-cost annotators and content moderators, yet their reliability in socially sensitive, high-stakes tasks remains poorly understood. We present an empirical study investigating whether open-source LLMs can reliably detect multiple categories of sexism in online political discourse — distinguishing overt hostile abuse from subtle forms such as benevolent, implicit, and modern sexism that evade keyword-based detection. Using 1,343 tweets directed at female UK politicians, we evaluate four open-source LLMs (FlanT5, Mistral, Gemma, Llama) across three prompting strategies (roleplay, zero-shot, few-shot), measuring classification performance and model uncertainty via perplexity-based confidence estimation. A feature-level error analysis using bootstrapped logistic regression and random forest importance scores identifies the linguistic drivers of false positives and false negatives. Our findings reveal three consistent failure modes: (1) models over-rely on surface features (word count, readability) producing overconfident false positives; (2) subtle sexism categories — implicit, benevolent, internalized — are systematically under-detected, with models showing high uncertainty when they do engage; and (3) perplexity-based confidence is a useful but insufficient safeguard: models remain confidently wrong at rates unsafe for unassisted deployment. These results have direct implications for trustworthy AI design in content moderation and argue for interpretability-focused evaluation alongside accuracy metrics.