Timezone: »

 
Oral
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Boxin Wang · Weixin Chen · Hengzhi Pei · Chulin Xie · Mintong Kang · Chenhui Zhang · Chejian Xu · Zidi Xiong · Ritik Dutta · Rylan Schaeffer · Sang Truong · Simran Arora · Mantas Mazeika · Dan Hendrycks · Zinan Lin · Yu Cheng · Sanmi Koyejo · Dawn Song · Bo Li

Tue Dec 12 08:30 AM -- 08:45 AM (PST) @
in Oral 1B »

Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications to healthcare and finance – where mistakes can be costly. To this end, this work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5, considering diverse perspectives – including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Based on our evaluations, we discover previously unpublished vulnerabilities to trustworthiness threats. For instance, we find that GPT models can be easily misled to generate toxic and biased outputs and leak private information in both training data and conversation history. We also find that although GPT-4 is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more vulnerable given jailbreaking system or user prompts, potentially due to the reason that GPT-4 follows the (misleading) instructions more precisely. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps. Our benchmark is publicly available at https://decodingtrust.github.io/.

Author Information

Boxin Wang (Department of Computer Science, University of Illinois, Urbana Champaign)
Weixin Chen (Tsinghua University)
Hengzhi Pei (University of Illinois, Urbana Champaign)
Chulin Xie (University of Illinois at Urbana-Champaign)
Mintong Kang (University of Illinois at Urbana-Champaign)
Chenhui Zhang (Massachusetts Institute of Technology)
Chejian Xu (University of Illinois at Urbana-Champaign)
Zidi Xiong (UIUC)
Ritik Dutta (IIT Gandhinagar, Dhirubhai Ambani Institute Of Information and Communication Technology)
Rylan Schaeffer (Stanford University)
Sang Truong (Stanford University)
Simran Arora (Stanford)
Mantas Mazeika (University of Illinois Urbana-Champaign)
Dan Hendrycks (Center for AI Safety)
Zinan Lin (Microsoft Research)
Yu Cheng (Microsoft Research)
Sanmi Koyejo (Stanford University / Google)
Dawn Song (UC Berkeley)
Bo Li (UChicago/UIUC)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors