Perceptual Crowdsourcing Under Bias: A Reinforcement Learning Framework for Urban Image Comparisons

Crowdsourced urban-perception datasets such as PlacePulse ask anonymous online raters to compare pairs of street-level images along dimensions like safety and wealth, and most prior work aggregates these pairwise judgments while ignoring who cast each vote. We present a systematic, largely negative-results-driven study of annotator bias in PlacePulse~2's safety category (503{,}290 votes, 48{,}358 users, 108{,}411 images). We first show that the dataset is too sparse for classical inter-annotator agreement statistics (only 0.39\% of image pairs are judged more than once) and instead diagnose bias with a graph-based transitivity measure, consensus-residual analysis, an Annotator Influence Score, and an effective-annotator-count metric. We then test three increasingly expressive ways of correcting for annotator bias: a 32-dimensional per-user embedding learned via matrix factorization, a single shrinkage-regularized positional-bias term fit jointly with a Bradley-Terry consensus, and a contextual-bandit reinforcement-learning agent that learns a per-vote trust weight from seven diagnostic features. Only the single-scalar model generalizes to held-out data (+0.16 accuracy points over an unweighted baseline); the 32-dimensional embedding overfits, and the RL agent -- despite two independently debugged implementations and 1{,}000 training rounds -- never outperforms either the unweighted baseline or the single-scalar heuristic. In contrast, two content-agnostic analyses turn up strong, statistically robust bias signal: cross-continent safety comparisons show South America losing to every other continent by 12--39 percentage points (permutation $p \approx 0$ for 14 of 15 continent pairs), and safety and wealth consensus scores correlate at $r = 0.35$, a relationship the positional-bias correction barely touches. Together these results argue that, under realistic crowdsourcing sparsity, simple regularized statistical corrections can be competitive with or superior to considerably more complex learned corrections, and that geographic and cross-category bias -- not purely annotator-level idiosyncrasy -- may be a larger source of distortion in this class of dataset than the literature's focus on per-annotator reliability models suggests.