Memory that keeps its receipts
Extracted facts answered 84.4% of LongMemEval-S. Linking each fact back to the conversation it came from raised that to 89.6%, for 20% more storage.
Most agent memory works the same way. A model reads a conversation, writes down the facts it thinks matter, and stores those facts. Later, the agent searches the facts and answers from them.
Facts are small and fast to search. They also lose things. We measured how much, and what it costs to get the detail back. This post shows the numbers from Lore, our open-source memory engine.
The short version
- Facts alone answered 84.4% of the 500 LongMemEval-S questions.
- Search often found the right fact, but the fact lacked the detail. Reading the conversation that fact came from fixed most of the gap: 89.6%.
- This needs no extra search index. It needs the original text and one link per fact. Storage grows by about 20%.
- Reading the source only when the facts are not enough keeps most of the gain for a fraction of the tokens: 87.8% at 1.37× the reader input.
Where facts fail
We ran LongMemEval-S, a public benchmark of 500 questions over long chat histories. Each question comes with about 50 past conversations. Lore stored the facts extracted from them, searched them, and gave the top 10 to a reader model.
We then looked at the questions Memory got wrong. About nine in ten had already found the right conversation. The search was not the problem. The fact was too short.
The clearest case is what the assistant said. Say a user asks which chord progression the assistant suggested for their song. The stored fact says the user is writing a song. It does not keep the chords. Facts answered 42 of 56 such questions. With the source conversation attached, the reader answered 54.
What we tried
We kept the Memory results fixed and changed only what raw text the reader also saw, inside the same 24,000-character budget. Memory evidence was never cut. We ran eleven variants on all 500 questions, and paired each one against Memory alone on the same questions.
| What the reader sees | Correct | Reader input |
|---|---|---|
| Facts only | 84.4% | 1.0× |
| Facts + a raw search with the original question | 85.0% | 1.0× |
| Facts + raw keyword search | 88.0% | 2.1× |
| Facts + raw hybrid search (keyword and semantic) | 89.4% | 2.0× |
| Facts + each fact's own source conversation | 89.6% | 2.4× |
| Facts + own source + raw hybrid search | 91.8% | 2.5× |
| Own source, read only when the facts fall short | 87.8% | 1.37× |
84.4% · 1.0×
87.8% · 1.37×
89.6% · 2.4×
91.8% · 2.5×
Three things stood out.
A plain raw search barely helps. A standard full-text query asks one chunk to contain every word of the question. Only 85 of 500 questions got any raw text that way. The gain was 0.6 points.
Depth beats breadth. At the same input size, whole conversations for the top one or two facts (89.6%) beat short windows around many facts (88.0%). The answer often sits several turns away from the words that match.
The link does most of the work. Following each fact to its own conversation gave +5.2 points: 32 questions fixed, 6 broken. Adding a hybrid search over all raw text reached 91.8%. Its semantic half beat keywords alone (p = 0.027). When we judged the answers a second time, that edge was no longer significant (p = 0.15).
What it costs
Tokens, per 1,000 questions, at the reader model's list prices on 2026-10-13:
| Strategy | Cost |
|---|---|
| Facts only | $4.45 |
| Facts + own source, always | $7.19 |
| Facts + own source + hybrid raw search | $7.78 |
Storage is the other cost. We measured it per 1,000 conversations.
The original text is small: about 6 KB in Postgres for a conversation of about 10,000 characters. Indexes are the expensive part. A full-text index of the raw text was 8× the size of the text. It added no correct answers over reading the linked conversations. With vectors for the raw text as well, storage reached 7.6× that of the facts alone.
What we are building
The numbers point to a simple design.
- Keep the original text. Lore already stores raw conversations as immutable evidence, separate from Memory.
- Link each fact to its source. A fact records which conversation it came from. Reading the source is a lookup, not a search. It took 4 ms here.
- Answer from facts first. The agent reads the source only when the facts do not answer the question. In our run the reader said so on 76 of 500 questions.
- Keep raw indexes optional. Search over raw text stays available on request. It is not built by default.
Two caveats matter. First, raw text has a cost beyond tokens. With conversations attached, the reader answered 2 of 30 trick questions it should have refused. We plan to label raw text clearly as conversation, not memory, and measure that fix on its own. Second, the gate only helps when the reader admits it does not know. Of 78 wrong answers from facts alone, 37 were confident. No gate on the answer can catch those.
How we measured
- Dataset: the cleaned LongMemEval-S split, all 500 questions, including the 30 that should be refused.
- Reader:
gemini-3.5-flash, temperature 0. Judge: GPT-5.5, with the five official LongMemEval rubric prompts unchanged. Scores from a different judge are not directly comparable to ours. - Statistics: every variant is paired against facts alone on the same questions, with an exact McNemar test and a paired bootstrap. We judged the four strongest variants a second time, independently. The two passes agreed on 99.3–99.6% of verdicts.
- Isolation: each question ran in its own workspace under Postgres row-level security, with a private record from another user planted in every one. No search returned it.
- The benchmark's metadata told us which conversation each fact came from. Lore does not store that link yet. Building it is the next step, and we will measure again on the real path.
The full report, the code, and the commands to reproduce it are in the Lore repository.