Performance metric elicitation is a type of inverse decision problem where the goal is to learn a loss function for a classification problem using expert comparisons between candidate classifiers. However, for many practical tasks, such an expert can be noisy. We present an approach for learning performance metrics in this setting that can handle general noise models. Our approach takes advantage of the problem's similarity to probabilistic bisection search and uses pairwise comparisons to update a pseudo-belief distribution for the performance metric. Our theoretical results guarantee convergence in practical settings and extend beyond previous results to include multi-expert elicitation. Quantitative comparisons against prior work demonstrate the superiority of our approach.