Timezone: »

 
Poster
On the (Non-)existence of Convex, Calibrated Surrogate Losses for Ranking
Clément Calauzènes · Nicolas Usunier · Patrick Gallinari

Wed Dec 05 07:00 PM -- 12:00 AM (PST) @ Harrah’s Special Events Center 2nd Floor
in Poster Session »

We study surrogate losses for learning to rank, in a framework where the rankings are induced by scores and the task is to learn the scoring function. We focus on the calibration of surrogate losses with respect to a ranking evaluation metric, where the calibration is equivalent to the guarantee that near-optimal values of the surrogate risk imply near-optimal values of the risk defined by the evaluation metric. We prove that if a surrogate loss is a convex function of the scores, then it is not calibrated with respect to two evaluation metrics widely used for search engine evaluation, namely the Average Precision and the Expected Reciprocal Rank. We also show that such convex surrogate losses cannot be calibrated with respect to the Pairwise Disagreement, an evaluation metric used when learning from pairwise preferences. Our results cast lights on the intrinsic difficulty of some ranking problems, as well as on the limitations of learning-to-rank algorithms based on the minimization of a convex surrogate risk.

[ PDF] 

Author Information

Clément Calauzènes (Criteo AI Lab)
Nicolas Usunier (Université Pierre et Marie Curie)
Patrick Gallinari (Sorbonne Universite, Criteo AI Lab)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors