The Model Agreed, But Didn’t Learn: Diagnosing Surface Compliance in Large Language Models
Summary: arXiv:2604.05995v1 Announce Type: cross
Abstract
Large Language Models (LLMs) internalize vast world knowledge as parametric memory, yet inevitably inherit the staleness and errors of their source corpora. Consequently, ensuring the reliability and malleability of these internal representations is imperative for trustworthy real-world deployment. Knowledge editing offers a pivotal paradigm for surgically modifying memory without retraining. However, while recent editors demonstrate high success rates on standard benchmarks, it remains questionable whether current evaluation frameworks that rely on assessing output under specific prompting conditions can reliably authenticate genuine memory modification.
In this work, we introduce a simple diagnostic framework that subjects models to discriminative self-assessment under in-context learning (ICL) settings that better reflect real-world application environments, specifically designed to scrutinize the subtle behavioral nuances induced by memory modifications. This probing reveals a pervasive phenomenon of Surface Compliance, where editors achieve high benchmark scores by merely mimicking target outputs without structurally overwriting internal beliefs.
Key Findings
- Surface Compliance: The phenomenon where models appear to perform well on benchmarks while failing to genuinely modify their internal representations.
- Cognitive Instability: Recursive modifications can lead to an accumulation of representational residues, impacting the model’s long-term memory state.
- Reversibility Issues: The study underscores the permanent diminishment of the model’s ability to return to a previous memory state after modifications.
Implications for Future Research
These insights underscore the risks of current editing paradigms and highlight the pivotal role of robust memory modification in building trustworthy, long-term sustainable LLM systems.
Conclusion
As we advance the capabilities of AI models, it is crucial to develop evaluation frameworks that not only assess immediate output but also ensure the underlying memory structures are genuinely modified. This work lays the groundwork for future explorations into memory editing and compliance in large language models, paving the way for more reliable and adaptable AI systems.
For those interested, the code related to this research is available at https://github.com/XiaojieGu/SA-MCQ.