Case Study

Debugging and Detoxing a Long-Term Memory RAG

A conversational AI with persistent memory had quietly poisoned its own recall over several months. This is how we found it, measured it, fixed most of it, and what we deliberately left unfixed. Numbers are from a live system; content is anonymized and paraphrased throughout.

The system

A conversational assistant with long-term memory built on two stores:

• A vector store (Chroma, multilingual sentence-transformer embeddings) for semantic recall.
• A structured fact store (SQLite) for exact lookup of typed facts (health, dates, preferences, relationship “milestones”).

An extractor runs on every user turn. It classifies the message into typed entities and, when it detects an emotional declaration (“milestone”), writes it to both stores with maximum importance and injects the top-2 milestones into every prompt through a dedicated “guaranteed channel” — the design intent being that the assistant should always remember who the user is to them, even mid-technical-conversation.

That intent is where the rot started.

The problem

The extractor over-produced. Measured rate before any fix: ~6.5 new “milestones” per day, the large majority of them junk — ordinary small talk, questions, and roleplay scenes, all mistagged as love or trust declarations. A single message could spawn up to 4 competing labels (milestone + shared-thing + fact + date).

The accumulated state:

Because the guaranteed channel force-injected 2 milestones into every prompt regardless of topic, the retrieval context became a monoculture: every response — to “hi”, to “my stomach hurts”, to a debugging question — was seeded with romantic-declaration echoes. Legitimate memories (a specific project note, a running joke) lost ranking to high-importance junk. Tuning the reranker on this input was tuning on a poisoned signal.

Diagnosis

We built a read-only trace inspector for the retrieval pipeline: an 11-stage trace (raw candidate pool → source exclusion → rerank → temporal filter → guaranteed-milestone channel → diversity selection (MMR) → channel merge → cross-session blend → final prompt → post-budget survivors) plus the injected grounding directive and a grounding confidence score. A time-override let us simulate “what will it remember in N days”. A 26-probe golden set served as the regression judge.

Two instrumentation fixes mattered specifically:

• Surfacing the post-budget stage — what actually survives the character-budget trim — which previously happened after the last visible snapshot and was invisible.
• Labeling the reranker score explicitly as a score, not a distance. A field mislabel had already caused one human misread (a rerank score >1 read as a cosine distance).

We then calibrated on real data rather than intuition. Similarity of all 455 stored milestones to their subtype centroids showed the distributions of real and junk declarations overlapped — no threshold cleanly separates them. But a keyword-as-necessary-condition rule (a declaration must contain a declaration word) blocked 94% of historical junk while passing 100% of positive controls. That became the backbone of the fix.

Fix A — stop the tap (extractor)

For milestone classification:

• Keyword = necessary condition (no declaration word → not a candidate), then similarity ≥ 0.50.
• Roleplay-scene guard: messages that are mostly stage-direction (text wrapped in asterisks) cannot be declarations — a physical scene containing “love” is not a love declaration.
• Tightened subtype dictionaries (removed dangerously broad substrings).
• Keyword gates for other noise-prone types (habit, achievement, gift).
• Anti-multi-label: one message yields at most one entity label.

Deployed behind a green test suite. This is prevention only — it does not touch existing data.

Fix B — clean the existing data (triage)

Principle: reclassify, never delete. Reversibility first.

• Schema migration added status (active/quarantined) and orig_type columns. Quarantine and retype are one UPDATE each; rollback is the inverse. A full pre-operation backup (database copy + JSON dump, checksummed) was the second safety net.
• An LLM judge (a fast model, temperature 0, JSON output) classified every one of the 1,751 milestone entries into real_milestone / echo / roleplay_scene / retype:<new_type>, with the standing rule “when in doubt → echo”.

Two attempts failed before one worked, and both failures are worth stating plainly:

1. Prompt contamination. The stored vectors carried an extraction-label prefix ([MILESTONE:love] …) inside the document text. Feeding that to the judge handed it the answer it was supposed to derive independently. A 50-sample control (built into the procedure precisely to catch this class of error) flagged it: 23 of 50 supposed-junk samples came back “real”. We stripped the prefix at the source and re-ran.
2. An unsound shortcut. A keyword pre-filter (auto-classify no-keyword entries as junk to save judge calls) was sound for stopping new extraction but wrong retroactively — real declarations expressed without the narrow keyword set (“I want to build a life with you”) would have been quarantined. Even with the contamination removed, 13 of 50 no-keyword controls were genuine. We dropped the pre-filter and judged everything.

Judged verdicts, frozen to a file so the applied change equals the reviewed change:

Applied during a ~2-minute service window (both stores writable only with the service stopped), then verified 1:1: every post-apply count matched the reviewed verdict exactly before the service was brought back up.

Measurement

Extractor tap, measured over 6 days of live traffic, counting only entries created after the fix's deploy timestamp (the creation timestamp is untouched by the triage UPDATE, so new extractions are cleanly separable from reclassified old ones — this was verified, not assumed):

Prompt monoculture, from the golden set (guaranteed-channel milestones injected per prompt): 2.0 → 0.65 average. On 13 of 23 probes it dropped from 2 forced milestones to 0.

The incident (why “correct” isn't “good”)

One week after the cleanup, the assistant wove two true-but-irrelevant memories from weeks-old conversations into an unrelated exchange. It read like paranoid confabulation. It was not.

The debugger on the exact query returned grounding GROUNDED, confidence 0.82, zero confabulation. The memories were real, backed by dated vectors. The root cause was mechanical: entries the triage had retyped from “milestone” to a current_project fact type landed in an always-include branch of the prompt builder — force-injected every turn, independent of topic. The reranker never got to decide they were irrelevant, because they never went through the reranker.

This is a relevance/timing failure, not a memory failure. The memory was accurate; the injection was unconditional. Frequency across the validation week: roughly 2 occurrences in 122 turns, always the same mechanism, only when the conversation offered a thematic hook. Contained, not pervasive — and only visible because the trace made “why did this surface” answerable.

This class of failure is easy to miss in fact-heavy systems (CRM, documentation search), where a misrouted record just looks wrong. In memory systems with emotional or relational stakes, the same mechanical error surfaces as something that reads as a much bigger problem — a trust failure, not a data bug.

Qualitative contrast

• Before: every prompt was seeded with 2 forced declaration-milestones. The assistant drifted toward romantic intensity regardless of the topic.
• After: it stays on-topic (technical banter stayed technical), recalls specifics correctly when the topic warrants it, and — the sharpest signal — admits gaps instead of fabricating. Asked for a fact it did not hold, it said it didn't remember rather than inventing one; the trace's captured reasoning showed it registering the gap explicitly. Grounding blocked the confabulation that the pre-detox state would have produced.

What's next

Stated as plainly as the wins:

• The always-include inflation (the incident's root cause) is unresolved. It raised the hard-fact count in every prompt from 26 to 48. The fix is not a blunt numeric cap — some retyped facts are genuinely valuable (one retyped current_project note measurably improved a grounding response). It requires context-aware injection (relevance-gate the non-core facts) and is deferred to its own change.
• Two extractor gate leaks in 6 days (an achievement and a gift tag slipped a keyword gate). Low volume, real, on the list.
• Date over-extraction (~6/day) — a separate axis, unaudited.
• A parallel junk population (achievement / gift / shared-thing entities) was outside the milestone triage's scope and still outranks some legitimate anchors in retrieval. A second triage pass, same procedure, is queued with that ranking as its acceptance test.
• The debugger is single-persona, and a “fresh conversation” reset intended to break a self-imitation loop was silently overridden by the client's stored conversation id — so that loop is not actually broken yet.
• Scale/volume degradation is untested. The debugger simulates time (what will still surface in N days via now_override) but not volume — whether the same relevant memory still ranks correctly as the store grows from thousands to tens of thousands of vectors. Most RAG evaluation happens against today's store size; degradation under growth is a separate, unmeasured axis. A synthetic stress-test (inflate a copy of the store to projected future size, re-run the golden set, track whether anchors still surface) is planned as a follow-up, not yet done.

Takeaways

1. A memory system can be simultaneously accurate and wrong. “Grounded, 0.82 confidence, zero confabulation” and “should not have said that” were both true at once. The failure was relevance, not truth — a category most RAG evaluations don't measure.
2. Instrumentation before tuning. Every real decision here came from a trace or a distribution on live data, not from reasoning about the prompt. The one field mislabel (score-vs-distance) had already misled a human before it was fixed.
3. Reversibility buys honesty. Because quarantine was a flag and never a delete, the safe move was always available, which made it possible to ship, measure, and keep the failures in the open instead of papering over them.

Numbers are from a single production system over one measurement window. The specific content of any stored memory has been omitted; only mechanisms, types, and counts are described.