EVIDENCE / BENCHMARKS

What has Semantix actually measured?

This page separates repository behavior from claims that still need real-session or production evidence.

Current evidence level: E1 · repository tests. This is first-party engineering evidence, not an independent production benchmark.

Run record and page review by radianceded. Last reviewed 2026-08-13.

Why this run is public

We published the run after the website audit found that architecture claims were easier to find than raw outcomes. The aim is narrower than a performance benchmark: a reader should be able to see the command, environment, passing packages, and adverse result without trusting a product summary.

The Windows permission failures are kept in the record because removing them would overstate portability. They do not invalidate BM25 or extraction tests, but they do show that Unix mode-bit assertions need a platform-specific interpretation.

Capability status

JSONL event parsingE1Repository tests

Semantic slice extractionE1Repository fixtures

BM25 retrievalE1Repository tests

Stable cross-session injectionE1Repository tests

Real-session relevanceE0Not yet measured

Production cost reductionE0Not yet independently benchmarked

Latest reproducible run

Download JSON ↗

Run: Windows package run

Date: 2026-08-12

Environment: Go 1.26.5 · Windows/amd64

Command: go test -count=1 ./...

Core retrieval, cache, ingestion, injection, evolve, usage, and zone packages passed. Two Unix-style file permission assertions failed on Windows.

Passed packages

semantix/kernel/bm25
semantix/kernel/cache
semantix/kernel/ingest
semantix/kernel/inject
semantix/kernel/evolve
semantix/kernel/usage
semantix/kernel/zone

Known failures

semantix/kernel/slice: store file perm = 666, want 600
semantix/cmd/semantix: evolve state perms = 666, want 600

Commit: e93668ef9199 ↗

Reproducible retrieval artifact

E2 · reproducible synthetic run

Single-threshold versus three-region retrieval

Compare a single BM25 reuse threshold with the conservative hit/grey/miss policy on the repository's labeled oracle fixture.

Input: cmd/semantix/testdata/eval-greyzone.tsv · SHA-256 83963A04C27E3949A53FCA5C5690D46242017D2C64658848C67032773B0F1E08
Command: go run ./cmd/semantix eval --set cmd/semantix/testdata/eval-greyzone.tsv --train-frac 0.7 --single-tau 1.0 --grey-target 30
Observed: 15 rows · 10 train / 5 replay · single reuse 100.0% with 20.0% error · three-region reuse 60.0% with 0.0% error · verdict PASS · grey ratio 40.0% (above the 30.0% alarm)

On this small fixture, the three-region policy trades reuse volume for zero observed false reuse. The grey-zone alarm remains visible and is not presented as a tuned production result.

· The rows are a repository fixture, not a real-session corpus.
· Five replay rows are too few for a general relevance claim.
· The command reports an alarm because the fixture's grey ratio exceeds the default target.

View fixture source ↗

What this does not prove

· Repository fixtures are not an independent production dataset.
· The run does not establish real-session relevance or cost reduction.
· Windows ACL behavior is not equivalent to Unix file mode bits.

Next evaluation

The next useful run is a frozen set of real, consented sessions with relevance labels written before threshold tuning. It should publish per-query decisions, rejected slices, downstream task outcomes, and a no-memory baseline. Until that exists, this page supports implementation claims only.

Read the FAQ on current project status →