Timezone: »

 
Poster
Rankmax: An Adaptive Projection Alternative to the Softmax Function
Weiwei Kong · Walid Krichene · Nicolas E Mayoraz · Steffen Rendle · Li Zhang

Mon Dec 07 09:00 PM -- 11:00 PM (PST) @ Poster Session 0 #0
in Poster Session 1 »

Several machine learning models involve mapping a score vector to a probability vector. Usually, this is done by projecting the score vector onto a probability simplex, and such projections are often characterized as Lipschitz continuous approximations of the argmax function, whose Lipschitz constant is controlled by a parameter that is similar to a softmax temperature. The aforementioned parameter has been observed to affect the quality of these models and is typically either treated as a constant or decayed over time. In this work, we propose a method that adapts this parameter to individual training examples. The resulting method exhibits desirable properties, such as sparsity of its support and numerically efficient implementation, and we find that it can significantly outperform competing non-adaptive projection methods.

Keywords: Algorithms -> Unsupervised Learning · Theory -> Learning Theory

Author Information

Weiwei Kong (Georgia Institute of Technology)
Walid Krichene (Google)
Nic E Mayoraz (Google, Inc.)
Steffen Rendle (Google)
Li Zhang (Google)

More from the Same Authors