Paper Accepted at COLM 2026! LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning

I am happy to share that our paper, "LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning," has been accepted at COLM 2026, the Third Annual Conference on Language Modeling!

You can find the full paper on arXiv here: arXiv:2607.02513

LLMs memorize sensitive training data, including personally identifiable information (PII), which creates a pressing need for reliable post hoc removal methods. Existing unlearning benchmarks, however, only measure behavior at the output level: they tell us whether a model *says* the sensitive information, not whether the information has actually been removed from the weights.

LACUNA is the first unlearning testbed with ground-truth parameter-level localization. We inject synthetic PII into specific model weights through masked continual pretraining of OLMo models at 1B and 7B scale, which lets us measure directly whether an unlearning method edits the parameters that hold the target knowledge.

Using this setup, we find that state-of-the-art unlearning methods achieve strong output-level performance while being highly imprecise at the parameter level, and remain vulnerable to attacks that resurface the supposedly erased information. We also show the flip side: when localization is accurate, even a simple gradient-based method achieves better erasure and greater robustness. Getting the *where* right matters as much as the *how*.

This is joint work with Thibault Rousset, Siva Reddy, Marius Mosbach, and Verna Dankers.