A Methodology for Assessing the Risk of Metric Failure in LLMs Within the Financial Domain.
Will Flanagan ⋅ Mukunda Das ⋅ Rajitha Ramanayake ⋅ Swanuja Maslekar ⋅ Meghana Mangipudi ⋅ Jeel Shah ⋅ Joong Choi ⋅ Shruti Nair ⋅ Shambhavi Bhushan ⋅ Sanjana Dulam ⋅ Mouni Pendharkar ⋅ Nidhi Singh ⋅ Vashishth Doshi ⋅ Sachi Shah
Abstract
As Generative Artificial Intelligence is adopted across the financial services indus- try, a significant barrier to adoption and usage is measuring model performance. Historical machine learning metrics can oftentimes fail to generalize to GenAI workloads and are often supplemented using Subject Matter Expert Evaluation. Even in this combination, many projects fail to account for various unique risks present in choosing specific metrics. Additionally, many widespread benchmarks created by foundational research labs and educational institutions fail to generalize to industrial use. This paper explains these challenges and provides a Risk As- sessment Framework to allow for better application of SME and machine learning Metrics.
Chat is not available.
Successful Page Load