Interpretable vs. Black-Box: Automated CEFR Assessment for Ukrainian Texts

Automatic assessment of language proficiency is an important task for language education and linguistic research, yet Ukrainian remains considerably under-resourced compared to high-resource languages. We present Fluencymeter, a web-based service for automatic CEFR-level (A1–B2) assessment of Ukrainian texts, built on a comparative evaluation of feature-based machine learning, transformer-based models, and LLM-based few-shot classification. Using a curated dataset of ~400 educational texts digitized from Ukrainian-as-a-foreign-language textbooks, we show that a Random Forest classifier trained on handcrafted linguistic features (lexical, morphological, syntactic, and cohesion-based) achieves a macro-F1 of 0.576, marginally outperforming fine-tuned XLM-RoBERTa (0.574) and clearly ahead of GPT-5.5 few-shot prompting (0.564) and GPT-4.1 (0.498). Rather than exposing only a black-box label, Fluencymeter surfaces this linguistic evidence to users – showing which lexical, syntactic, and morphological features drive a text's estimated level. The system further integrates lexical resources from the PULSE project and collocation/construction data from CoCoCo, allowing users to inspect vocabulary and phraseological complexity alongside the automatic score. Our results demonstrate that explicit linguistic features remain a robust, interpretable alternative to purely neural approaches for morphologically rich, low-resource languages – an important finding as the field increasingly defaults to LLM-based evaluation. We discuss the trade-off between neural performance and pedagogical interpretability, and present a live demonstration of Fluencymeter alongside the opportunities and challenges of building explainable language-assessment tools for Ukrainian. Fluencymeter is designed for researchers, language teachers, and learners working with Ukrainian, supporting both the analysis of existing texts and the development of level-appropriate educational materials. We will demonstrate Fluencymeter live at the workshop, allowing attendees to submit their own Ukrainian texts and explore the linguistic evidence behind the assessed CEFR level.