Skip to yearly menu bar Skip to main content


Poster

Reasons and solutions for the decline in model performance after editing

Xiusheng Huang · Jiaxiang Liu · Yequan Wang · Kang Liu

[ ]
Thu 12 Dec 4:30 p.m. PST — 7:30 p.m. PST

Abstract:

Knowledge editing techniques are used to update incorrect or outdated information in large-scale language models. Recent studies have found that edited models exhibit varying degrees of performance decline. However, the reasons behind this phenomenon and potential solutions have not been provided. To investigate the causes of performance degradation in edited models and to optimize them, we conducted experiments from two perspectives: data and model. Specifically, to clarify the impact of data on the performance of edited models, we first evaluated how editing different types of data affects model performance. Then, we constructed a Multi-Question Dataset (MQD) and identified that the performance of the edited models is primarily influenced by the diversity of the editing objectives and the length of the tokens. Secondly, we explored the factors that affect model performance from a model perspective. Experiments revealed a strong correlation between the L1 norm of the edited model layers and the editing accuracy, and identified an editing quantity bottleneck. To enhance the performance of edited models, we proposed a Dump for sequence (D4C) method that effectively improves the performance of edited models and overcomes the previous editing bottleneck issue. This method allows for multiple effective edits with minimal impact on model performance.

Live content is unavailable. Log in and register to view live content