Beyond Channel Labels: Structured Label Noise in Telegram Affiliation Classification

Large-scale social-media datasets frequently use account-, community-, or channel-level metadata as a proxy for labels on individual messages. This form of label propagation can substantially reduce annotation costs, but it assumes that content within each source is sufficiently homogeneous for an aggregate label to represent individual instances. When this assumption fails, weak supervision may introduce not merely random noise, but systematic class-dependent errors that distort downstream machine-learning evaluation. We empirically audit this assumption using 1,372 manually annotated Telegram posts from 72 channels, each assigned one of three political-affiliation labels: Neutral, pro-Russian, or pro-Ukrainian. Two annotators achieve 83.5% raw agreement and Cohen’s kappa=0.65; subsequent analyses use adjudicated post-level labels. We define each channel’s dominant affiliation from its annotated posts and quantify within-channel heterogeneity using purity, entropy, post-to-channel mismatch rates, mutual information, and leave-one-out channel-based label propagation. Channels exhibit substantial internal heterogeneity. Only 33.3% of channels are completely homogeneous, while 43.1% have purity below 75%. Among channels with a unique dominant affiliation, 23.3% of posts disagree with their channel-level label, and this rate remains approximately 24% when restricting analysis to channels containing at least 5, 10, or 20 annotated posts. Crucially, this disagreement is highly class-dependent. Only 7.0% of Neutral posts occur outside Neutral-dominant channels, compared with 37.9% of pro-Ukrainian posts and 80.1% of pro-Russian posts. Pro-Russian posts account for 66.3% of all observed mismatches and occur predominantly inside channels whose majority affiliation is Neutral. Thus, propagating a channel label to its posts would not introduce symmetric annotation noise: it would systematically suppress one minority affiliation. Channel identity nevertheless contains meaningful predictive information. Mutual information analysis shows that channel identity retains approximately 35.9% of post-level affiliation information, significantly exceeding a permutation-based null distribution (p=.001). Similarly, leave-one-out channel propagation improves macro-F1 from 0.282 to 0.528 over a global-majority baseline. However, aggregate performance conceals severe class-specific failure: despite achieving 73.6% overall accuracy, channel propagation recovers only 14% of pro-Russian posts, compared with 91% recall for Neutral and 54% for pro-Ukrainian posts. Cluster bootstrap analyses further support the stability of the main estimates across channels. These results identify a consequential source of structured label noise in weakly supervised social-media classification. Channel-level affiliation is informative contextual metadata, but it is not interchangeable with post-level annotation. More broadly, our findings show how seemingly reasonable aggregation assumptions can create asymmetric errors that are hidden by overall accuracy and disproportionately affect minority classes. We recommend preserving source- and instance-level labels as distinct variables, reporting class-specific propagation error, and treating channel metadata as weak supervision rather than deterministic ground truth when constructing political and conflict-related NLP datasets.