What to Remember, What to Reveal: Privacy-Aware Memory for Conversational Agents

Long-term memory enables personalized conversational agents to retain user information across sessions. However, existing memory architectures primarily optimize for utility but neglect the risks of storing and reusing private attributes such as personally identifiable information (PII) unnecessarily. Dealing with privacy risk in personalized memory is challenging as simply removing sensitive values would undermine the utility of the memory system. Therefore, privacy protection for memory agents must govern the full life-cycle of sensitive values rather than just sanitizing individual records. To fill this research gap, we introduce $\textbf{S}$anitized $\textbf{P}$rivacy-$\textbf{M}$apped M$\textbf{em}$ory (SP-Mem), a privacy-aware memory architecture that decouples memory utility from exact private-value exposure. SP-Mem provides full life-cycle privacy-related design including determining how to identify and separate sensitive information from raw user inputs, how to store sanitized content and exact private values in isolated structures, and how to selectively retrieve values based on the task requirement and user consent. We further introduce a privacy-aware memory benchmark that jointly assesses response quality, privacy behavior, and inference cost. Extensive experiments across multiple LLM-based agents show that SP-Mem achieves stronger personalization while reducing unnecessary privacy exposure. Code and data are available at https://anonymous.4open.science/r/SP-Mem-0CAE/.