Interpretable Stylometric Detection of LLM-Generated Code Under Distribution Shift

Large language models are increasingly used to generate source code, making reliable detection of machine-generated code important in settings such as programming education, technical hiring, and software development. This problem is particularly challenging because source code is strongly constrained by syntax and semantics, reducing many of the stylistic differences commonly exploited for machine-generated text detection. Moreover, practical detection systems must generalize across programming languages, application domains, and model families that may not be represented during training. We study machine-generated code detection in the setting of SemEval-2026 Task 13, which requires binary classification of human-written and LLM-generated code under substantial distribution shift. The training data contains 500K samples, predominantly Python, C++, and Java, while evaluation additionally covers unseen languages including Go, PHP, C#, C, and JavaScript, as well as unseen research and production-code domains. Rather than relying solely on high-capacity pretrained code models, we investigate whether lightweight and interpretable stylometric signals can generalize across these settings. Our approach is based on length-normalized ratio features capturing properties of code descriptiveness and structure. In particular, we measure the proportion of comment lines and the frequency of verbs within comments, motivated by the observation that LLM-generated code often contains unusually descriptive comments and natural-language artifacts. To extract these signals robustly across languages, we use Tree-Sitter parsing together with a lightweight programming-language identification model based on character n-gram TF-IDF features. We additionally construct a line-level code-versus-text dataset and train a classifier for detecting natural-language segments embedded within source-code samples. This enables a further text-like ratio feature that captures cases where generated outputs contain explanatory text rather than code alone. We compare these handcrafted representations with pretrained code encoders including GraphCodeBERT, CodeT5+, and CodeRankLLM. Although the pretrained representations achieve high performance on validation data, their performance drops substantially under the multilingual and cross-domain test distribution, suggesting sensitivity to language-specific cues. In contrast, a small subset of interpretable stylometric features remains useful across evaluation settings. Our final system uses only two primary features, comment ratio and verb-comment ratio, with a depth-2 decision tree and lightweight heuristics derived from systematic artifacts observed in generated code. One such artifact is the tendency of some models to emit Markdown-style programming-language markers before a code snippet. The resulting system achieves a macro-F1 of 67.35 on the official test set and ranks within the top 15% of SemEval-2026 Task 13 submissions. The feature pipeline requires only CPU resources and supports near-instant inference. These results show that simple, interpretable properties of generated code can remain competitive under substantial distribution shift, while providing a computationally lightweight alternative to large pretrained detectors.