Curriculum Design for Low-Resource Dense Retrieval: Staged English Scaffolding in a Compact Multilingual Encoder

Curriculum Design for Low-Resource Dense Retrieval: Staged English Scaffolding in a Compact Multilingual Encoder Dense retrieval models are tuned for a handful of high-resource languages, so semantic search and retrieval-augmented generation degrade sharply for the billions who query in low-resource languages (LRLs). This abstract is proposing a curriculum recipe rather than a new architecture. Our central question: at a fixed parameter budget, does staging English supervision before LRL supervision beat mixing the two? Across five strategies the answer is yes. Setup: The backbone is tiny-aya-global [1], a 3.35B-parameter decoder covering 70 languages, used as an encoder with last-token pooling (it is a causal LM, so last-token pooling matches its attention pattern) and LoRA [2] (r=16, α=32) on query/value projections — ≈3M trainable parameters (≈0.1%), bfloat16, max length 256. Training is language-balanced contrastive learning on (query, positive, hard negative) triplets [3] with a cached multiplenegatives ranking loss, L = −log [es(q,p+)/τ /(es(q,p+)/τ + P p− es(q,p−)/τ )], where s is cosine similarity and p− includes both the explicit hard negative and in-batch negatives; effective batch 32 via gradient caching on one A100, 1 epoch. English scaffolding comes from RLHN-680K [4] (LLM-relabelled hard negatives); LRL supervision from IndicSQuAD [5] in 10 Indic languages (bn, gu, hi, kn, ml, mr, or, pa, ta, te), plus MahaSQuAD [6] for the Marathi run, with hard negatives constructed from a different paragraph of the same article. Multi-stage runs use merge-then-retrain: the prior stage’s LoRA is merged into the base and a fresh adapter attached, so earlier knowledge becomes frozen base. Primary targets: Bharat-NanoBEIR [7] (NanoBEIR [8] in Indian languages) and per-language passage retrieval (P@1/P@5); English NanoBEIR only checks the scaffold survives. Bharat-NanoBEIR Passage retr. NanoBEIR (en) Strategy (training recipe) nDCG@10 P@1 P@5 nDCG@10 Base model (zero-shot, no tuning) 0.044 5.2% 11.0% 0.189 A1 English-only (En 10K) — — — 0.539 A2 Single-LRL∗ (En 10K→Marathi 20K) 0.358 71.4%∗ 100%∗ 0.560 A3 Joint (En 10K + Indic) 0.337 70.1% 89.2% 0.564 A4 Scaled, wider LoRA (En 100K + Indic; full attn + MLP) 0.334 69.7% 86.9% 0.584 A5 Two-stage (En 200K→En 100K + Indic) 0.366 70.7% 90.3% 0.609 multilingual-e5-base [9] (reference model) 0.406 85.7% 97.9% 0.572 Indic = 10K per language × 10. ∗A2 is Marathi-only: its LRL columns are not a 10-language mean. Results: Contrastive tuning alone lifts English retrieval from 0.189 to 0.539 nDCG@10, so the base checkpoint is merely misaligned for dense retrieval, not incapable (A1). LRL supervision then lifts Bharat-NanoBEIR from 0.044 to 0.337–0.366 (≈8×) and per-language P@1 from 5.2% to ≈70%, via cross-lingual in-batch negatives. Two findings are the crux. First, staging beats mixing without trading English against LRL: A5 is best on LRL (0.366 Bharat, 90.3% P@5) and on English (0.609, above every other run and the reference), so building a stable English IR scaffold before the Indic mixture reduces interference between “learn English IR” and “learn ten Indic retrievers”. Second, wider adapters did not transfer: A4 doubles English exposure and widens LoRA to full attention + MLP, helping English (0.584) while LRL stays flat (0.334 vs. 0.337—within noise at this scale). Scheduling, not adapter width, moved LRL here. Figure 1: Left: mean nDCG@10. Right: P@1 per language. Analysis of the residual gap: A purpose-built retrieval encoder like multilingual-e5-base [9] stays ahead on LRL (0.406 Bharat, 85.7% P@1) even though our best run overtakes it on English, and the per-language view shows this deficit is uneven: Malayalam trails by 37 points (38% vs. 75% P@1), Kannada and Tamil by 19–21, but Bengali, Gujarati and Hindi by only 6–10 — despite every language receiving an identical 10K samples. Equal data does not buy equal transfer, suggesting sampling each language in proportion to its validation loss, rather than uniform balance, as a candidate next step. Caveats: single seed, 1 epoch, no Indic eval for A1, A2 is Marathi-only. References. [1] Salamanca et al. Tiny Aya. arXiv:2603.11510, 2026. [2] Hu et al. LoRA. arXiv:2106.09685, 2021. [3] Reimers & Gurevych. Sentence-BERT. EMNLP 2019. [4] Thakur et al. Hard Negatives, Hard Lessons (RLHN-680K). arXiv:2505.16967, 2025. [5] Endait et al. IndicSQuAD. arXiv:2505.03688, 2025. [6] Ghatage et al. MahaSQuAD. ICON 2023. [7] Bharat-NanoBEIR (HuggingFace, carlfeynman). [8] Thakur et al. BEIR. NeurIPS 2021. [9] Wang et al. Multilingual E5. arXiv:2402.05672.