R&D prototype · no official leaderboard placement · two certified category leads, cross-reader replicated · what that means
Context Swarm Memory (CSM) is open-source agent memory whose retrieval cost per query stays flat — about 36–38K input tokens all-in (the small, fully-cited packet the answer model sees, plus CSM's own probe/recall/synthesize calls) — whether the underlying memory holds 100K or 10M tokens. That is a property of the architecture, measured across a 76× range — and it is a claim about query cost specifically, since CSM’s write-time stage has an ingest cost that is not flat. Accuracy is the live question, and this page reports both the leads that survived falsification and the levers that did not.
The flat per-query figure comes from the unmodified public AMB runner across all four BEAM tiers (their CLI, scoring and judge path), single-trial, on a configuration that had no write-time stage — today’s default does, and its ingest cost is disclosed separately. The certified leads come from a calibrated head-to-head instrument where one reader and one judge serve both arms and only the retrieved context differs. Numbers from the two instruments are reported separately and never pooled. CSM has no official leaderboard placement and does not claim SOTA.
Where this actually stands
The most useful thing this page can do is tell you exactly which claims are load-bearing and which are history. CSM has never had a leaderboard placement. Its one upstream submission was author-closed unmerged on 2026-06-22. What it does have is a cost property measured across a 100× range, two category-level wins that survived a deliberate falsification attempt, and a published record of every lever that was measured and died.
Per-query input stays at 36–38K tokens whether the per-unit haystack is 154K or 11.7M. A token count does not depend on the prompt or judge, which is why this is the one figure from the June ladder still worth quoting — with two corrections attached below.
It measured a CSM that no longer exists (before the hybrid router, ID repair, the batched probe and the fact fold), and the upstream runner has since changed its prompt, judge and temperature. Pre-change scores cannot be set beside post-change ones, on either side.
Config frozen, eight pre-flight blockers found and fixed, launch is one command. Blocked on API credit and an upstream 10M loader fix — procurement problems, not engineering ones.
| Category @ 1M | June 2026 ladder | Today's defaults |
|---|---|---|
knowledge_update | −0.435 | −0.111 |
event_ordering | −0.216 | −0.023 |
abstention | +0.086 | +0.193 — certified |
Evidence STATUS.md PREFLIGHT_OFFICIAL_LADDER.md
The measured result
Measured on a free apples-to-apples instrument: one answer model and one judge serve both arms, with the same queries and prompts, and the retrieved context is the only variable. Hindsight's arm is replayed from its own published BEAM contexts, so its system never has to be re-run. The judge was calibrated before it was trusted — holdout ρ 0.864 / MAE 0.077 against the official Gemini judge — and it reproduced the official tie at 100K and loss at 1M.
The delta must exceed its minimum detectable effect (MDE) and its confidence interval must exclude zero, at n=70 — the entire category, with no further sampling possible. Both results below were then re-run end to end on a second, independent reader as a deliberate attempt to break them, and survived. A delta smaller than its MDE is reported as directional only and is never called a lead. This project retracted a +0.068 result for failing that bar; the rule is not decorative.
| Tier | Category | n | CSM | Hindsight | Delta | MDE | W/L/T | Reader 2 |
|---|---|---|---|---|---|---|---|---|
| 500K | knowledge_update | 70 | 0.758 | 0.376 | +0.382 | 0.173 | 37/6/27 | +0.326 (n=67) |
| 1M | abstention | 70 | 0.679 | 0.486 | +0.193 | 0.167 | 23/8/39 | +0.185 (n=69) |
At 500K, contradiction_resolution (+0.096, 38W/21L) and preference_following
(+0.075, 18W/10L) are positive at full n and replicate across both readers — and both sit
under their MDE, so neither is claimed as a lead. This is the honest ceiling of an
unpaid instrument: n=70 is already the entire category, and the MDE is set by the reader and judge's
per-query variance, not by sample size. A ~0.09 effect is unresolvable here at maximum n.
Settling it needs the lower-variance official path, which is credit-blocked.
500K and 1M each certify one. On the June official ladder 100K led 7 categories and 10M led 4 — but that is a different instrument measuring an older configuration, so those counts are reported separately and never summed into the total. Stating it any other way would be an instrument error.
Kept because the cost column is still true and still the point. Read the scores as history: they measured an older CSM, on a runner that has since changed.
| BEAM tier | CSM score | Hindsight score | CSM answer-ctx | Hindsight answer-ctx | CSM all-in input |
|---|---|---|---|---|---|
| 100K | 0.7367 | 0.7337 | 27.0K | 17.7K | 35.8K |
| 500K | 0.6589 | 0.7112 | 26.6K | 20.5K | 36.2K |
| 1M | 0.5693 | 0.7386 | 28.2K | 23.9K | 38.1K |
| 10M | 0.5616 | 0.6408 | 32.5K | 27.3K | 35.9K |
CSM's internal pipeline is about 25% of the token count but runs on models roughly 10× cheaper than the answer model, so it is about 7% of the dollars. We print the sum rather than reporting the halves separately. On the apples-to-apples answer slice Hindsight is leaner (17.7–27.3K vs 26–33K) — that is a disclosed trade, not a footnote. Neither column includes either system's ingest-time cost; CSM's is disclosed below. No gold answers, rubrics or query IDs ever reach retrieval.
Evidence
EXP-category-leadership-2026-08.md
AMB_BEAM_LADDER_2026_06_18.md
npm run verify:published
The cost claim, precisely
“The cost stays flat” is the line this project is most quoted on, so it is worth stating exactly what it does and does not cover. It is a claim about query cost, and it was measured on a configuration with no write-time stage. Today’s default config has one, and its ingest cost is O(corpus). Both halves belong on the same page.
| Cost | Flat? | Figure |
|---|---|---|
| Per-query retrieval (answer packet + internal pipeline) | Yes | ~36–38K input tokens across a 76× haystack range (avg per-unit 154,431 → 11,707,222; census-sourced) |
| Per-query, with the fact fold on | Yes | token-neutral at answer time: 7,106 → 7,080 (abstention), 7,010 → 7,074 (knowledge_update) |
| Ingest (write-time fact extraction) | No — O(corpus) | the registry chunks each unit at 100K tokens and reads every chunk |
| Tier | Per-unit haystack (avg) | Units | First-ingest input | Queries | Amortized/query |
|---|---|---|---|---|---|
| 100K | 154,431 | 20 | 3,088,625 | 400 | ~7.7K |
| 500K | 560,086 | 35 | 19,602,995 | 700 | ~28K |
| 1M | 1,155,117 | 35 | 40,429,107 | 700 | ~58K |
| 10M | 11,707,222 | 10 | 117,072,219 | 200 | ~585K |
split|user|model|promptVersion; re-runs pay nothing, and it runs on the cheap model tier
In a real deployment memory arrives incrementally, so facts are extracted per turn as it lands — the whole-corpus read is a backfill, not a recurring cost. That is the honest defense of the ingest number above. It is also precisely Hindsight’s defense. Earlier versions of this page argued that because Hindsight distills at ingest and discloses no internal figure, CSM’s accounting was the complete one. CSM now distills at ingest too, so that comparison was no longer true and has been removed. Both systems have an ingest cost; CSM’s is on this page, Hindsight’s is not. That, and only that, is the difference worth stating.
Evidence STATUS.md § the cost claim amb-fact-registry.ts Token anatomy
Architecture
Memory is a swarm of bounded, immutable shards behind a read-only manifest. CSM spends context only after a zero-LLM router finds plausible shards; the answer arrives as one compact, cited packet.
Hybrid lexical + local-embedding scorer over the shard directory. No LLM call, no vector DB required at index time. The router reports when a ranking is degenerate rather than returning a confident-looking arbitrary order.
One batched relevance pass over the candidate shards — “is this memory worth recalling from?” Every shard still gets its own verdict; batching the question cut internal input tokens 21% with no score change.
Structured, citation-bearing extraction from only the shards that passed the probe.
Merge, dedupe, flag conflicts — emit a compact MemoryPacket for the agent.
Answer + key claims, every line cited to shard, snapshot, and event IDs.
ask() never appends events, writes snapshots, or mutates the chronicle.
Enforced in CI by SHA-256 file hashes (tests/mutationSafety.test.ts).
Durable memory changes only through appendEventAndSnapshot or an explicit
Committer decision. Snapshots are immutable and versioned; the storage layer refuses overwrites.
Routing is lexical plus a local all-MiniLM-L6-v2 embedding leg —
no LLM-generated index is ever built, so adding memory costs no API tokens.
Summary/ordering/temporal queries get a date-ordered, fully-cited timeline assembled without extra LLM calls. Date arithmetic is computed, never delegated to the model.
Every CSM_* read goes through one module that throws on a value it
does not recognise. The rule exists because a silent fallback once let
CSM_ROUTER_HYBRID=off turn the router on for weeks.
Distilled facts are prepended into the evidence capsule rather than added as a document, because a fixed return-K means an added document evicts real evidence. On point queries where no capsule exists, the artifact rides as one standalone document rather than being dropped — so the rule is conditional, not absolute. The fold-vs-append +0.068 measurement itself was retracted as a render artifact.
Evidence
ARCHITECTURE.md
mutationSafety.test.ts
npm test
The falsification record
This section is the point, not an appendix. A memory system that only publishes its wins is indistinguishable from one that never checked. Every entry below cost real money to discover and is documented in the repo with its artifacts, its statistics, and the rule it forced.
The winning arm was graded on a context that could not be reproduced byte-for-byte: the capsule had rendered as a placeholder string. Rule adopted: never grade an arm on a context you cannot re-render. Synthesized text is now persisted per run.
A reranker gained +11.6 on retrieval coverage and lost answers on
event_ordering (4W/13L, p=0.049). Order-changing levers are now gated on the answer
metric only — a proxy that moves the wrong way is worse than no proxy.
A router component benchmark predicted +0.24 retrieval coverage; wired into the real pipeline the same change delivered −0.12 coverage, with retrieved evidence down 55% because every downstream stage is calibrated in units of shards, not events. Both figures are coverage, not answer score — see F1 for why those are not interchangeable. Rule adopted: benchmark the unit production conserves, not the unit that is convenient to isolate.
The needle net worked at 500K and did not replicate at 1M. Lean-return K=16 shipped and was reverted
the same day for a 500K information_extraction regression. Every default flip now
requires cross-tier evidence.
A projected 40–60% cost cut died on a measured 4,096-token implicit-cache floor: every CSM call sits below it, so the pipeline gets exactly zero caching. The observability shipped anyway and surfaced ~$2.00/run of previously invisible thinking spend.
A local pre-gate that skipped witnesses before the LLM saw them lost knowledge_update
across a 3-point dose-response and was killed. Cheapening the question worked instead: the
batched probe keeps every witness at −21% input tokens, score-neutral, and shipped.
n=25 is a pointer, not a verdict — abstention read +0.040 at n=25
and +0.193 at n=70, and the re-measurement noise at n=25 is ±0.05, the size of most “leads”.
Re-measure the same arm before believing a delta — re-scoring identical contexts
moved one arm 0.06, which is how a retracted lead was caught.
A delta below its MDE is not an effect, stated at birth and never softened afterwards.
And Mem0 and HippoRAG are documented as blocked, not beaten: they could not be run on
available hardware, which is a gap in the evidence, never a win.
| Lever | Default | Evidence |
|---|---|---|
| Hybrid router + descriptors | ON | +0.365 answer at 1M, 26W/5L — the strongest single effect measured on this project. Caveat: measured on sidecar arm r1mC before the render-gap fix, on a harness that dropped the evidence capsule from both arms; the delta is within-harness valid, the absolutes are not, and it has not been re-measured with the capsule rendered. Every post-fix certified result runs with it ON — indirect support, not a re-measurement |
| ID repair | ON | ~0.20 |
| Batched probe (hosted providers only) | ON | −21% internal input by arithmetic (one shared scaffold replaces 8); score delta +0.032, below its 0.079 MDE. Local providers stay OFF — the evidence is from one hosted model family |
| Fact fold (write-time registry) | ON | 500K knowledge_update certified on two readers; 1M paired +0.114 with CI above zero; token-neutral at answer time |
| Preference profile | OFF | composed with the fold it measured −0.036 (4W/9L) versus the fold alone |
Coverage mode + chronicle CSM_COVERAGE | ON | deterministic cited timeline for summary/ordering/temporal queries; default since 2026-06-10 |
Lean return, needle net, session digests, ordered capsule, local probe gate, probe shrink, coverage reranker CSM_AMB_COVERAGE_RERANK (the +11.6-proxy one), cross-encoder reranker CSM_HYBRID_RERANK (unrelated), virtual shards | OFF | each measured negative, non-replicating, or a wash |
Evidence Category campaign Token efficiency PERF_BREAKDOWN.md RD_PORTFOLIO_2026_06.md
Direct answers
These mirror the structured FAQ data in the page head, so humans and answer engines quote the same hedged claims — including the ones that are unflattering.
Not overall — and the honest answer has two halves. On the June 2026 official-runner ladder
CSM led only at 100K (0.7367 vs 0.7337, inside single-trial noise) and trailed at 500K/1M/10M.
That ladder is now stale twice over: it measured a configuration that no longer exists, and the
upstream runner has since changed its prompt, judge and temperature, so pre-change scores are not
comparable to post-change ones on either side. On a calibrated head-to-head instrument
with today's configuration, CSM holds two certified category-level leads, each
replicated on a second independent reader: knowledge_update at 500K
(+0.382) and abstention at 1M (+0.193). Two
categories is not the whole benchmark.
No. CSM has no official leaderboard placement and has never had one. The single upstream submission (vectorize-io/agent-memory-benchmark#19) was author-closed on 2026-06-22, unmerged. A future submission would be a new PR against the post-#20 runner, with the comparison system re-scored on the same pinned commit. Nothing here is claimed as state of the art.
A pre-registered statistical bar: the delta exceeds its minimum detectable effect (MDE) and its confidence interval excludes zero, at n=70 — the entire category, with no further sampling possible. Both certified results were then re-run end to end on a second, independent reader as a deliberate falsification attempt, and survived. A delta below its MDE is reported as directional only and never called a lead. This project retracted a +0.068 result for failing that bar.
It is fully staged — frozen config, eight pre-flight blockers found and fixed, launch is one command — and blocked on two things outside the code: API credit (est. $200–300 for three tiers) and an upstream 10M loader fix. The free instrument was built precisely so that measurement did not stop while that blocker stood, but it cannot substitute for the official path: a ~0.09 effect is below its resolving power even at maximum sample size.
No. Retrieval receives only the ingested documents, the query, user id, and timestamp — no gold answers, rubrics, or query IDs. There is no benchmark-specific vocabulary in the retrieval path, and an import-graph leakage-firewall test enforces that boundary in CI.
An open-source LLM memory system that treats memory as bounded, immutable, read-only shards. A Memory Manager routes a query to candidate shards, probes them cheaply, recalls from only the useful ones, and synthesizes a compact answer cited to shard, snapshot, and event IDs. Querying memory never mutates it; durable changes go through an explicit Committer protocol.
On the apples-to-apples answer-context slice Hindsight is leaner (17.7–27.3K vs CSM's 26–33K). CSM publishes a per-query all-in figure instead — ~36–38K input tokens — which adds its own probe/recall/synthesize calls to the answer packet; Hindsight discloses no internal figure, so the two all-in numbers are not directly comparable. CSM's internal tokens run on models ~10× cheaper: ~25% of the token count, ~7% of the dollars. Neither figure includes ingest-time cost, and both systems now distill at ingest — CSM's ingest cost is disclosed, Hindsight's is not.
A +0.068 displacement result was retracted as a harness artifact. A coverage reranker gained +11.6 on its proxy and lost answers. A router component benchmark predicted +0.24 and the assembled system gave −0.12. A projected 40–60% caching saving was falsified by a 4,096-token cache floor. Several levers worked at one corpus size and did not replicate at another. Mem0 and HippoRAG could not be run on available hardware and are recorded as blocked, not beaten. The full list is in the falsification record.
Reproducibility
The test suite runs offline against a deterministic mock provider — no API keys.
The published-evidence verifier hashes the committed result rows and recomputes the
headline counts, citation F1, and McNemar checks from results.jsonl.
Hindsight's side of every comparison re-derives from its own published artifacts, so
you never have to take our word for the other arm either.
Go deeper STATUS.md REPRODUCING.md REPLICATION_KIT.md BENCHMARK_METHODOLOGY.md
# offline — no API keys required
$ npm install
$ npm test # 563 tests, MockProvider
$ npm run verify:published # re-hash + recompute claims
# the read path cannot write — enforced by SHA-256 hashes
$ npx vitest run tests/mutationSafety.test.ts