Agent memory · LongMemEval · June 2026
What we learned benchmarking agent memory on LongMemEval
Raw conversation evidence and extracted facts were more useful together than either was alone. On our run, the combined Redis strategy reached 86.1% task-averaged accuracy—with qualifications that matter.
Before the numbers
Notes on the comparison
The Redis run. The headline result covers all 500 questions in the LongMemEval small split, using gpt-4o for answers and judging.
Other systems. Some were run through our harness; others are published figures shown for context. Their benchmark split or answer model may differ, and the chart notes those cases.
Cost estimates. Dollar estimates cover measured LLM token usage at list prices. They exclude infrastructure, hosting, and any hidden server-side ingest spend.
The problem
Long histories are costly to include in every prompt
Keeping every prior conversation in the prompt becomes expensive and slow as history grows. An external memory system separates that work into two paths: ingest a session once, then retrieve a small amount of relevant context for each question.
LongMemEval tests whether that retrieved context is sufficient. Its small split contains 500 questions over multi-session chat histories, covering user and assistant facts, preferences, knowledge updates, evidence spread across sessions, and temporal reasoning.
Three Redis strategies
Keep the evidence, curate the facts, or do both
Instruct
LLM-extracted memory
An extractor sees each session and the current memory store, then creates, updates, or deletes granular facts. It consolidates changes well, but omitted details cannot be recovered later.
Remis
Hybrid semantic RAG
Sessions are split at topic shifts and indexed with dense and BM25 signals. Neighboring chunks expand the retrieved context. Exact details survive, though the answer model sees more noise.
Remis + Instruct
Combined context
Both stores are queried, then one answer call receives two labeled sections: conversation excerpts and extracted memories. The answer model can draw from either source when responding.
What we measured
Combining both improved accuracy
The primary metric gives each of the six task types equal weight. This avoids letting the larger categories dominate the result.
Redis strategy accuracy
Task-averaged accuracy on LongMemEval small (%)
Competitive landscape
Task-averaged accuracy on LongMemEval small (%)
Interpretation
The useful finding is complementarity—not that one retrieval pattern won
Extracted memories help consolidate information spread across sessions and resolve changes over time. Raw chunks preserve quotes, names, dates, and numbers that an extractor may compress away. In the combined setup, the answer model can use either representation in one call.
The experiment supports that explanation for this benchmark and configuration. It does not show that every application needs two stores, or that the same trade-off will hold for different histories, models, prompts, or latency budgets.
Cost and latency
Accuracy is only one part of a production memory system
We estimated a holistic per-session LLM cost as one ingest plus 5.13 queries, the average number of user turns per session in this benchmark split.
Measured accuracy vs. estimated LLM cost
Total memory cost per session (USD, log scale)
Remis + Instruct
Mastra OM cost
for the combined run
Why rerun benchmarks
Published figures did not reliably predict our runs
Reproduced scores moved in both directions. That does not establish that a published number is wrong: products, prompts, defaults, and answer models differ. It does show why results should be measured under the conditions that matter to your application.
Published figure vs. our measurement
LongMemEval accuracy (%)
Limits
What this evaluation does not establish
-
01
One benchmark shape. LongMemEval small has a fixed question mix and roughly 24 sessions per example on average. Production histories can be longer, noisier, and more dynamic.
-
02
One answer backbone for most reproduced runs. The findings may shift with newer or smaller answer and extraction models.
-
03
An LLM judge. The official binary judge improves consistency, but it is not error-free and does not measure answer style, calibration, or user satisfaction.
Bottom line
Use the result as a design clue, then test your own workload
On this evaluation, preserving raw evidence alongside curated facts improved task-averaged accuracy and produced a favorable measured cost/accuracy trade-off. The result is strong enough to motivate the architecture, not broad enough to settle agent memory.