release · v0.2.1-rc2 · 6 complete · 2 partial · 0 valid negative / 8 lanes

Antfly Circus — v0.2.1-rc2

Fresh v0.2.1-rc2 honed evidence measured from the product-equivalent b4a46545 tree; the final tag adds only release-workflow fixes at 4be5a0e9.

Reading contract

Each selected outcome binds its Antfly evidence to v0.2.1-rc2. Machines, dates, and protocols may differ.

2 of 2 lanes measured

VectorDBBench

Qualified vector search quality, throughput, serial latency, serving demand, and database volume across operating points.

vector-vdbbench-50k

complete

50K

Fresh three-lifecycle Antfly and same-machine peer evidence for the 1536d/50K vector comparison.

Methodology scope · protocol · interpretation

VectorDBBench methodology

Antfly subject and execution lane

Circus accepts an explicit subject manifest containing the asserted release tag, commit, and Linux/arm64 Antfly binary. It records the binary path, SHA-256, architecture, and observed --version output, while treating the supplied tag and commit as release metadata. The honed matrix mounts that binary read-only into Antfly’s Linux container. Every comparison engine runs in the same private Docker network; no native macOS database is mixed into this benchmark.

The public benchmarks/VectorDBBench checkout remains clean. Circus-owned parameter selection, compatibility fixes, query partitioning, and diagnostics live in benchmarks/VectorDBBench-harness and are loaded as a runtime overlay.

Workloads and query protocol

The publication matrix contains six engines on both upstream workloads:

Performance1536D50K: 50,000 vectors with 1,536 dimensions.
Performance768D1M: 1,000,000 vectors with 768 dimensions.

For each workload, the 1,000 upstream query/ground-truth pairs are ranked by a fixed SHA-256 seed and split into 200 calibration queries and 800 publication queries. Their exact overlap is zero; both partition fingerprints are retained.

Each engine/workload trial has one fresh service and one fixed index lifecycle:

Start lifecycle memory sampling before indexing.
Load and build the index once using fixed construction parameters.
Sweep only the query-time parameter on the 200-query calibration partition.
Freeze the lowest setting reaching recall 0.955. If none qualifies, retain the highest-recall setting and mark it below the qualification floor.
Without rebuilding, run the selected setting on the untouched 800-query partition at 1, 5, 10, 20, and 30 concurrent clients for 30 seconds each, followed by the upstream serial quality and latency pass.
Capture one diagnostic post-workload storage observation, gracefully stop and verify the service, capture final storage, retain the artifacts, and only then reclaim the engine’s data.

The 50K and 1M settings are selected independently because the larger graph may need a higher search setting. Publication results never influence selection. A row below the 0.950 publication floor remains visible and is not silently retuned or omitted. No process-restart measurement is published.

Tuned and fixed parameters

Engine	Fixed index/build parameters	Query-time selection
Antfly	AKNN, one shard, packed float32 queries	`search_effort` sweep
Milvus	HNSW `M=16`, `efConstruction=256`	`ef_search` sweep
Chroma	HNSW `M=16`, `efConstruction=256`	fixed `ef_search=256` qualification
Weaviate	HNSW `maxConnections=16`, `efConstruction=256`	`ef` sweep
pgvector	HNSW `m=16`, `ef_construction=128`	`ef_search` sweep
Elasticsearch	HNSW `m=16`, `ef_construction=100`	`num_candidates` sweep

The final report prints the selected value for every workload and engine. Weaviate and Elasticsearch are local containers; their upstream adapter class names (WeaviateCloud and ElasticCloud) do not describe the deployment.

Memory

The table reports both memory views. Demand working set is the maximum sampled cgroup-v2 anon + shmem from before indexing through readiness, calibration, and publication queries; it excludes reclaimable file cache and follows MEASURING_ANTFLY.md. Cache-inclusive peak is the maximum concurrently sampled sum of memory.current across the complete service stack and includes file cache and cgroup-charged kernel memory. Sampling runs every 200 ms and retains the timeline. Kernel memory.peak is also retained as an exact peak for a single cgroup, or a conservative sum of independent peaks for a multi-container stack.

Honed publication requires three fresh lifecycles per engine/workload. The headline table shows each arithmetic mean; detail artifacts retain every observation, the mean, and the individual-trial range. A trial is rejected if the canonical sampler reports a failure, misses any container in the service stack, starts after indexing, stops before publication finishes, or leaves a timeline gap above five seconds. Peak throughput, its concurrency, C=30 throughput and concurrent p99, recall, nDCG, serial latency, and memory are arithmetic means of the three compatible lifecycle observations. Raw curves and ranges remain retained. The selected setting and protocol identity must match across all three trials.

The complete Antfly standalone container is measured, so all Antfly processes and threads in that cgroup are included. Termite is not part of this workload because VectorDBBench supplies precomputed vectors.

Persistent database volume

Every publication engine receives one newly created, run-labeled Docker local volume at its complete persistence root. The digest-pinned Alpine probe captures whole-volume allocated and apparent bytes once at empty, ready, and post_workload, then at the verified post-stop final barrier. Host-path/no-network Antfly is rejected for comparative storage. Pre-stop background compaction remains diagnostic and does not gate the post-stop metric.

Final allocated bytes are the headline. The same three construction lifecycles define performance, memory, and storage aggregates. Every value, arithmetic mean, range, receipt, and barrier sample remains under raw/storage.

Internal Antfly scale diagnostic

make vdbbench-scale-fast and make vdbbench-scale-big run an Antfly-only diagnostic beside the cross-engine publication matrix. The fast profile uses 20K, 40K, 60K, 80K, and 100K vectors; the big profile uses 200K, 400K, 600K, 800K, and 1M vectors. make vdbbench-scale selects both. All points use 768 dimensions and the same 200-query calibration partition, disjoint 800-query held-out partition, serial pass, and 1/5/10/20/30-client 30-second capped sweep described above.

The diagnostic has exactly two fresh index lifecycles per point. Trial one runs from the smallest to largest selected corpus; trial two reverses that order. This controls simple order effects without claiming the three-lifecycle publication cohort. The aggregate and every row set publication_eligible: false, retain both raw values plus their arithmetic mean and range, and state the two-lifecycle caveat. Scale results must not be merged into the honed cross-engine cohort.

The source corpus is the pinned upstream Cohere 768-dimensional 1M-vector fixture. Circus verifies its immutable source files, computes exact cosine top-k ground truth against deterministic prefix sizes, and writes explicit custom-dataset manifests. A streaming iterator is bounded by the manifest’s declared prefix size; a smaller point cannot consume rows belonging to a larger shared Parquet file.

Each point/trial is an immutable publication-cache cell with workload-specific dataset identity. A cache miss receives one fresh retry, never an in-place resume. Only a complete positive lifecycle or a classified held-out capacity/rejection result can be promoted. The canonical diagnostic artifact retains every cell’s receipt/cache reference, query-partition digest, selected search effort, request-error evidence, memory timeline, final allocated volume, and insert_duration + optimize_duration build time.

Output and retention

The canonical artifact contains report.md, normalized result.json, provenance, a receipt, selected settings, the raw capped-concurrency JSON/CSV diagnostic, memory rows and timelines, logs, supervisor/status evidence, and upstream result JSON. Standard publication runs VectorDBBench’s increasing concurrency progression at 1, 5, 10, 20, and 30 clients. Normalized rows report peak QPS and its concurrency as well as qps_c30 and p99_c30_ms.

A lifecycle that completes the selected-setting, held-out capped sweep but has explicit nonzero request rejections is retained as a classified capacity terminal negative. It is reusable only with the exact concurrency schedule, query partition, selected setting, full memory timeline, quiet-host assertion, logs, and supervisor evidence. Setup/index/configuration failures, crashes, timeouts, partial lifecycles, and unknown failures are never reusable publication cells. Database volumes are deleted only after the verified final capture unless an operator explicitly opts into retaining them for debugging. The retained storage.json receipt and complete optional-trial evidence are results; the live volume is not.

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: vector-vdbbench-50k
Domain: vector
Source: https://github.com/antflydb/VectorDBBench
Upstream: https://github.com/zilliztech/VectorDBBench
Known Antfly versions: v0.1.3, v0.2.*
Expected duration: 480 min
Enabled: yes

Benchmark parameters

{"concurrency_duration_seconds":30,"concurrency_schedule":[1,5,10,20,30],"query_count":800,"query_partition_sha256":"7be5c5f24dfa31b66c5dfc3280c8f1bc5c1c8db3a35046a1a1ab9d50a0d232f7","search_parameter":"search_effort","search_value":0.425}

Trial variation

[{"engine":"Antfly","ranges":{"demand_working_set":{"maximum":6361.6,"mean":5274.5,"minimum":4430.3,"unit":"MB"},"final_database_volume":{"maximum":352.804864,"mean":352.582315,"minimum":352.239616,"unit":"MB"},"p99_at_30_concurrent":{"maximum":64.7,"mean":63.1,"minimum":61.8,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Chroma","ranges":{"demand_working_set":{"maximum":466.7,"mean":463.3,"minimum":460.9,"unit":"MB"},"final_database_volume":{"maximum":332.488704,"mean":332.485973,"minimum":332.480512,"unit":"MB"},"p99_at_30_concurrent":{"maximum":179,"mean":177.6,"minimum":175.3,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Elasticsearch","ranges":{"demand_working_set":{"maximum":1952.7,"mean":1941,"minimum":1934.8,"unit":"MB"},"final_database_volume":{"maximum":312.082432,"mean":312.067413,"minimum":312.049664,"unit":"MB"},"p99_at_30_concurrent":{"maximum":15.1,"mean":14.467,"minimum":14,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Milvus 2.6.22","ranges":{"demand_working_set":{"maximum":2766.6,"mean":2666.2,"minimum":2563.6,"unit":"MB"},"final_database_volume":{"maximum":2393.47712,"mean":2393.459371,"minimum":2393.432064,"unit":"MB"},"p99_at_30_concurrent":{"maximum":11.4,"mean":11.233,"minimum":11.1,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"pgvector","ranges":{"demand_working_set":{"maximum":2303.8,"mean":2303.7,"minimum":2303.7,"unit":"MB"},"final_database_volume":{"maximum":1523.970048,"mean":1523.967317,"minimum":1523.965952,"unit":"MB"},"p99_at_30_concurrent":{"maximum":33.1,"mean":32.033,"minimum":31.4,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Weaviate","ranges":{"demand_working_set":{"maximum":977.9,"mean":953.8,"minimum":936.6,"unit":"MB"},"final_database_volume":{"maximum":391.55712,"mean":391.514795,"minimum":391.491584,"unit":"MB"},"p99_at_30_concurrent":{"maximum":89.6,"mean":86.967,"minimum":85.3,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Milvus 3.0.0","ranges":{"demand_working_set":{"maximum":2949.6,"mean":2819,"minimum":2719.7,"unit":"MB"},"final_database_volume":{"maximum":2489.167872,"mean":2489.156949,"minimum":2489.143296,"unit":"MB"},"p99_at_30_concurrent":{"maximum":12,"mean":11.667,"minimum":11.4,"unit":"ms"}},"workload":"1536d-50K"}]

Peer versions

Chroma1.5.9
Elasticsearch9.5.2
Milvus 2.6.222.6.22
pgvector0.8.6 (PostgreSQL 16)
Weaviate1.39.2
Milvus 3.0.03.0.0

Evidence digests

Receipt: df2b34cd8e28c7e146dfa2b30855b4453b05cf85dc7485fa392b11eb7e044a94
Aggregate: d181a0979e402f0da423d0580b3049608fa8a0fc6ed8b683d38d6b382c753b3d
Provenance: 02a2f7a98d27d2f41121a3f78755fada002b5afadc81b4c38940e065de92ed3d
Harness commit: ad4635e78a54618f3217a49489324853ca8548e3

Peer cohort evidence · 1536d-50K

Completed: 28 Aug 2026 · 14:56 UTC
Fresh lifecycles: 18
Peer variants: 6
Receipt: 782e8e470765fae9914945e67c9da3f3e82a1b38571ebe6e5940228a8248601c
Aggregate: 0eb49aba01e15e065ec68b1667beb2c41a580cde5dab2cb928b6f8c01c16daf4
Provenance: 700d49bf35039da97e9b5366d4a4e6ec4c417fdfdb5fd720b980349a538d68c3

Selected measurement

Completed: 31 Aug 2026 · 01:48 UTC
Antfly: v0.2.1-rc2 · 29 August 2026 · 21:46 UTC
Runtime: 3h 15m
Platform: linux · arm64
Machine: Apple M4 Max
Build: ReleaseFast

Peer cohort · 1536d-50K

Completed: 28 Aug 2026 · 14:56 UTC
Variants: 6 independently executed peers
Lifecycles: 18 fresh
Compatibility: Same machine · protocol, input, and environment identity-bound

Peak throughput

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 1536d-50K · three fresh lifecycles · Higher is better

Serial query p95

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 1536d-50K · three fresh lifecycles · Lower is better

Demand working set

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 1536d-50K · three fresh lifecycles · Lower is better

Final database volume

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 1536d-50K · three fresh lifecycles · Lower is better

Performance, quality, and resource evidence

Antfly v0.2.1-rc2 · b4a46545

Engine	Workload	Recall	nDCG	Peak QPS queries/s	p95 ms	Demand working set MB	Cache-inclusive peak MB	Final database volume MB	Trials
Antfly	1536d-50K	0.9580	0.9630	1,565.2	2.433	5,274.5 MB	5,643.4 MB	352.6 MB	3
Chroma	1536d-50K	0.9870	0.9890	180.9	8.067	463.3 MB	812.9 MB	332.5 MB	3
Elasticsearch	1536d-50K	0.9560	0.9640	4,434.6	2.033	1,941.0 MB	4,385.1 MB	312.1 MB	3
Milvus 2.6.22	1536d-50K	0.9630	0.9700	5,293.1	1.600	2,666.2 MB	5,426.1 MB	2,393.5 MB	3
pgvector	1536d-50K	0.9690	0.9750	2,721.0	4.167	2,303.7 MB	3,533.5 MB	1,524.0 MB	3
Weaviate	1536d-50K	0.9550	0.9630	1,008.8	6.300	953.8 MB	1,511.3 MB	391.5 MB	3
Milvus 3.0.0	1536d-50K	0.9630	0.9700	5,019.7	1.700	2,819.0 MB	5,557.7 MB	2,489.2 MB	3

vector-vdbbench-1m

complete

1M

Fresh three-lifecycle Antfly and same-machine peer evidence for the 768d/1M vector comparison.

Methodology scope · protocol · interpretation

VectorDBBench methodology

Antfly subject and execution lane

Workloads and query protocol

The publication matrix contains six engines on both upstream workloads:

Performance1536D50K: 50,000 vectors with 1,536 dimensions.
Performance768D1M: 1,000,000 vectors with 768 dimensions.

Each engine/workload trial has one fresh service and one fixed index lifecycle:

Start lifecycle memory sampling before indexing.
Load and build the index once using fixed construction parameters.
Sweep only the query-time parameter on the 200-query calibration partition.
Freeze the lowest setting reaching recall 0.955. If none qualifies, retain the highest-recall setting and mark it below the qualification floor.
Without rebuilding, run the selected setting on the untouched 800-query partition at 1, 5, 10, 20, and 30 concurrent clients for 30 seconds each, followed by the upstream serial quality and latency pass.
Capture one diagnostic post-workload storage observation, gracefully stop and verify the service, capture final storage, retain the artifacts, and only then reclaim the engine’s data.

Tuned and fixed parameters

Engine	Fixed index/build parameters	Query-time selection
Antfly	AKNN, one shard, packed float32 queries	`search_effort` sweep
Milvus	HNSW `M=16`, `efConstruction=256`	`ef_search` sweep
Chroma	HNSW `M=16`, `efConstruction=256`	fixed `ef_search=256` qualification
Weaviate	HNSW `maxConnections=16`, `efConstruction=256`	`ef` sweep
pgvector	HNSW `m=16`, `ef_construction=128`	`ef_search` sweep
Elasticsearch	HNSW `m=16`, `ef_construction=100`	`num_candidates` sweep

Memory

Persistent database volume

Internal Antfly scale diagnostic

Output and retention

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: vector-vdbbench-1m
Domain: vector
Source: https://github.com/antflydb/VectorDBBench
Upstream: https://github.com/zilliztech/VectorDBBench
Known Antfly versions: v0.1.3, v0.2.*
Expected duration: 480 min
Enabled: yes

Benchmark parameters

{"concurrency_duration_seconds":30,"concurrency_schedule":[1,5,10,20,30],"query_count":800,"query_partition_sha256":"7be5c5f24dfa31b66c5dfc3280c8f1bc5c1c8db3a35046a1a1ab9d50a0d232f7","search_parameter":"search_effort","search_value":0.45}

Trial variation

[{"engine":"Antfly","ranges":{"demand_working_set":{"maximum":17706.7,"mean":16000.4,"minimum":15096.8,"unit":"MB"},"final_database_volume":{"maximum":3821.907968,"mean":3816.0384,"minimum":3805.806592,"unit":"MB"},"p99_at_30_concurrent":{"maximum":760.7,"mean":709.933,"minimum":656.2,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Chroma","ranges":{"demand_working_set":{"maximum":3699.1,"mean":3693.6,"minimum":3688.8,"unit":"MB"},"final_database_volume":{"maximum":3460.194304,"mean":3459.731456,"minimum":3458.838528,"unit":"MB"},"p99_at_30_concurrent":{"maximum":136.1,"mean":135.567,"minimum":134.6,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Elasticsearch","ranges":{"demand_working_set":{"maximum":2135.8,"mean":2095.2,"minimum":2029.8,"unit":"MB"},"final_database_volume":{"maximum":3148.750848,"mean":3148.500992,"minimum":3148.361728,"unit":"MB"},"p99_at_30_concurrent":{"maximum":21.6,"mean":20.967,"minimum":19.7,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Milvus 2.6.22","ranges":{"demand_working_set":{"maximum":13999.4,"mean":13920.6,"minimum":13837.1,"unit":"MB"},"final_database_volume":{"maximum":25669.378048,"mean":25642.164224,"minimum":25626.836992,"unit":"MB"},"p99_at_30_concurrent":{"maximum":13.7,"mean":13.633,"minimum":13.5,"unit":"ms"}},"workload":"768d-1M"},{"engine":"pgvector","ranges":{"demand_working_set":{"maximum":2318.6,"mean":2317.3,"minimum":2316.6,"unit":"MB"},"final_database_volume":{"maximum":9329.364992,"mean":9329.318571,"minimum":9329.27488,"unit":"MB"},"p99_at_30_concurrent":{"maximum":76.6,"mean":72.867,"minimum":69.9,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Weaviate","ranges":{"demand_working_set":{"maximum":7370.7,"mean":7345.1,"minimum":7331,"unit":"MB"},"final_database_volume":{"maximum":3955.232768,"mean":3947.47904,"minimum":3942.985728,"unit":"MB"},"p99_at_30_concurrent":{"maximum":68.3,"mean":67.2,"minimum":65.7,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Milvus 3.0.0","ranges":{"demand_working_set":{"maximum":13551.8,"mean":13262.6,"minimum":13016.7,"unit":"MB"},"final_database_volume":{"maximum":35112.308736,"mean":33718.551893,"minimum":33003.88864,"unit":"MB"},"p99_at_30_concurrent":{"maximum":14.4,"mean":14.233,"minimum":14.1,"unit":"ms"}},"workload":"768d-1M"}]

Peer versions

Chroma1.5.9
Elasticsearch9.5.2
Milvus 2.6.222.6.22
pgvector0.8.6 (PostgreSQL 16)
Weaviate1.39.2
Milvus 3.0.03.0.0

Evidence digests

Receipt: c425f822721a24813a04eda8930fc70f9ea947ca62a40f1599d993ffa4719f91
Aggregate: e7fb11418f05a238ffe44d83afcf0ff909a4f4e7864b0682c11f38d784a930dc
Provenance: f9374bb86bf82b33fbf72bec33388c146713111143f0f6b85dfc849077fa02de
Harness commit: 27ef68801c65cb9274290131ed49b6020233e9ef

Peer cohort evidence · 768d-1M

Completed: 28 Aug 2026 · 15:20 UTC
Fresh lifecycles: 18
Peer variants: 6
Receipt: 4366569aa9cfe5f7e796c5eadc30a6d2ae610cde650543eb3546ddcb856dfebf
Aggregate: 828e2d97f299926c3bb1f39866db983ef61895793243ec02a569fe955507a197
Provenance: 4af875e079c7c2538ed378027ac14be10294f30a7761e8fe0300f885ea3a1ea4

Selected measurement

Completed: 31 Aug 2026 · 13:32 UTC
Antfly: v0.2.1-rc2 · 29 August 2026 · 21:46 UTC
Runtime: 8h 01m
Platform: linux · arm64
Machine: Apple M4 Max
Build: ReleaseFast

Peer cohort · 768d-1M

Completed: 28 Aug 2026 · 15:20 UTC
Variants: 6 independently executed peers
Lifecycles: 18 fresh
Compatibility: Same machine · protocol, input, and environment identity-bound

Peak throughput

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 768d-1M · three fresh lifecycles · Higher is better

Serial query p95

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 768d-1M · three fresh lifecycles · Lower is better

Demand working set

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 768d-1M · three fresh lifecycles · Lower is better

Final database volume

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · 768d-1M · three fresh lifecycles · Lower is better

Performance, quality, and resource evidence

Antfly v0.2.1-rc2 · b4a46545

Engine	Workload	Recall	nDCG	Peak QPS queries/s	p95 ms	Demand working set MB	Cache-inclusive peak MB	Final database volume MB	Trials
Antfly	768d-1M	0.9670	0.9700	70.9	54.967	16,000.4 MB	22,875.1 MB	3,816.0 MB	3
Chroma	768d-1M	0.9560	0.9620	238.7	6.600	3,693.6 MB	7,289.8 MB	3,459.7 MB	3
Elasticsearch	768d-1M	0.9590	0.9610	3,110.8	3.233	2,095.2 MB	11,426.8 MB	3,148.5 MB	3
Milvus 2.6.22	768d-1M	0.9610	0.9660	4,085.9	2.200	13,920.6 MB	21,605.9 MB	25,642.2 MB	3
pgvector	768d-1M	0.9660	0.9700	955.8	10.767	2,317.3 MB	11,974.9 MB	9,329.3 MB	3
Weaviate	768d-1M	0.9560	0.9620	1,300.8	5.700	7,345.1 MB	13,791.8 MB	3,947.5 MB	3
Milvus 3.0.0	768d-1M	0.9610	0.9670	3,898.5	2.367	13,262.6 MB	21,631.0 MB	33,718.6 MB	3

vector-filtered-ann

complete

Filtered ANN

Fresh three-lifecycle rc2 Antfly tenant-scoped filtered ANN evidence with the retained peer comparison.

Methodology scope · protocol · interpretation

Filtered ANN Methodology

This benchmark is intentionally separate from the unfiltered VectorDBBench matrix. The honed publication workload measures equality-filtered, tenant-scoped vector search; the broader portable matrix and gauntlet retain numeric-filter diagnostics. All use exact filter-aware ground truth and the same database adapters and service topology as the regular vector benchmark.

Portable 50K matrix

The Circus-owned portable matrix loads each engine once with the vector, integer ID, and a deterministic categorical bucket, then runs six scenarios:

numeric tails retaining 10%, 2%, 1%, and 0.2% of the corpus;
a fixed categorical bucket retaining 1% / 500 scattered rows; and
a rotating categorical bucket, changed per query, also retaining 1%.

Rows use bucket_{id % 100}. The fixed case selects bucket_037; the rotating case selects (query_index * 37 + 11) % 100, which visits every bucket ten times over the 1,000-query test set. Each scenario has exact cosine top-100 ground truth. The 0.2% numeric point deliberately leaves exactly 100 eligible rows and is the minimum valid selectivity for top-100.

Each database performs three complete search sweeps, with scenario order deterministically shuffled per database and repetition. Reports use the median with min-max whiskers and show Recall@100, peak QPS, and p99 at the concurrency producing peak QPS. Performance execution fails closed while unrelated benchmark processes or containers own the shared host.

Honed profile

Dataset: Small OpenAI, 50,000 vectors, 1,536 dimensions, cosine distance
Tenant model: 100 equal buckets defined by bucket_{id % 100}, with 500 vectors per tenant
Filter: categorical equality on the tenant bucket selected for each query
Quality schedule: (source_query_index * 37 + 11) % 100, which visits every tenant and remains stable across the deterministic calibration/publication partition
Throughput schedule: every worker rotates across tenant buckets so the measurement is not a repeated hot-filter cache lookup
Top-k: 100
Engines: Antfly, Weaviate, Elasticsearch, Milvus, pgvector, and Chroma. This keeps the honed comparison focused on the source benchmark target, a mainstream hybrid-search engine, a vector-native engine, and the ubiquitous Postgres extension baseline, while retaining the popular embedded/developer-oriented vector store.
Lifecycle: three fresh indexes per engine. Each index is tuned only on the deterministic 200-query calibration partition, then measured on the disjoint 800-query holdout. The same three lifecycles define performance, memory, and storage aggregates.
Storage: each lifecycle uses one newly labeled Docker local volume at the engine’s complete persistence root. Capture whole-volume allocated/apparent bytes once at empty, ready, and post_workload, then at the verified post-stop final barrier before reclamation. Host paths are ineligible. Retain every value and report the arithmetic mean and range across all three lifecycles.
Search mode: serial latency/recall plus QPS at concurrency 1, 8, and 32 for 10 seconds each.

Antfly calibration-only topology qualification

Before the three publication lifecycles, Antfly alone qualifies shard counts 1, 2, and 4 on fresh indexes using only the same deterministic 200-query calibration partition. Each topology sweeps the preregistered search-effort ladder 0, 0.025, 0.05, …, 0.35; the lowest effort reaching recall 0.955 is selected. Holdout queries, publication concurrency, and publication metrics are forbidden inputs to this topology decision.

When all three topology candidates qualify, the selector minimizes calibration (p95, p99, shard count) in that order. If any candidate is quality-negative or operationally incomplete after one retry, publication uses the preregistered one-shard control rather than selecting from a partial successful set. The chosen shard count is frozen across all three Antfly publication lifecycles and included in their cache identity.

Gauntlet profile

Dataset: Medium OpenAI, 500,000 vectors, 1,536 dimensions, cosine distance
Filter: id >= 495,000, leaving 1% / 5,000 candidates
Engines: the same six-engine set. Qdrant is not part of either profile. Quickwit is a lexical/log-search engine in this suite and does not expose a comparable VectorDBBench filtered-ANN adapter.
Metrics: same as the honed profile, with the exhaustive concurrency 1, 5, 10, 20, 30, 40, 60, and 80 sweep at 30 seconds per point
Peak-memory sampling is enabled by the enclosing gauntlet benchmark profile

Correctness and fairness

The honed adapters implement categorical equality against the same labels field and load the same 100-bucket assignment. Its exact cosine top-100 ground truth uses the tenant selected for each source query. The query schedule is applied before the deterministic 200/800 partition so each retained vector is paired with the same tenant in calibration, publication, and independent reproduction.

The gauntlet’s numeric adapters implement inclusive >= semantics. VectorDBBench compares each result to neighbors computed within the surviving candidate set, so unfiltered or post-filtered top-k results lose recall rather than appearing artificially fast. Results report recall, nDCG, peak concurrent QPS, the concurrency and p99 latency at that peak, p95/p99 serial latency, insert time, and total load time. Index and query knobs are recorded in the command logs.

Filtered services use their own container names, data roots, result labels, logs, status rows, and host ports. Antfly defaults to port 28081 so it cannot attach to or stop the regular/full-text Antfly service on port 28080.

Canonical output retains every storage.json, barrier sample, declared volume, trial value, arithmetic mean, and range under raw/storage. Final allocated bytes are the headline; apparent bytes and engine-reported index sizes are diagnostics. Filtered peer storage appears in the comparative review only when the existing review-inclusion gate passes, while canonical/raw evidence is always retained.

The portable matrix still does not constitute a complete claim about filtered search. Planned follow-ups add vector-correlated categories, skewed hot/cold filter churn, conjunctions/exclusions, concurrent-ingest interference, matched-recall tuning, and 500K/1M datasets.

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: vector-filtered-ann
Domain: vector/filtered-ann
Source: https://github.com/antflydb/VectorDBBench
Upstream: https://github.com/zilliztech/VectorDBBench
Known Antfly versions: v0.2.*
Expected duration: 270 min
Enabled: yes

Benchmark parameters

{"num_shards":1,"quality_floor_passed":true,"selected_settings":[{"num_shards":1,"parameter":"search_effort","trial":1,"value":0.35},{"num_shards":1,"parameter":"search_effort","trial":2,"value":0.35},{"num_shards":1,"parameter":"search_effort","trial":3,"value":0.35}],"workload":"tenant-scoped-1536d-50K-1pct"}

Peer versions

Peer versions were not recorded in the published result fields.

Evidence digests

Receipt: d5cc3d67deb8e29e3074152aae7aa24da199e6bda19be8baaf320f7fee2d3627
Aggregate: 8dccfeeb4e817ac53f72ebc440a5515553e06c9bc92f4f766d22b48ce00e5ccb
Provenance: c16174ae09ab1ab76d9b701fbd38de78d56e162664e861c9be901ade1c8cd28f
Harness commit: 27ef68801c65cb9274290131ed49b6020233e9ef

Selected measurement

Completed: 31 Aug 2026 · 16:11 UTC
Antfly: v0.2.1-rc2 · 29 August 2026 · 21:46 UTC
Runtime: 2h 34m
Platform: linux · arm64
Machine: Apple M4 Max
Build: ReleaseFast

Peak filtered throughput

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · tenant-scoped 1536d/50K · 3 fresh lifecycles · Higher is better

Peak-concurrency p99

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · tenant-scoped 1536d/50K · 3 fresh lifecycles · Lower is better

Demand working set

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · tenant-scoped 1536d/50K · 3 fresh lifecycles · Lower is better

Tenant-scoped filtered ANN

Antfly v0.2.1-rc2 · b4a46545

Engine	Recall@100	nDCG	Peak QPS queries/s	Peak p99 ms	Serial p99 ms	Demand MB	Final volume MB	Trials
Antfly	1.0000	1.0000	3,941.4	28.86	2.03	5,092.8	353.8	3

III

fulltext-miracl-scale

partial

MIRACL full-text scale

MIRACL English lexical scaling across a 32.9M-passage source corpus, using 799 public development queries and 8,350 human relevance judgments.

Methodology scope · protocol · interpretation

Full-text MIRACL scale methodology

Scope

fulltext-miracl-scale measures full-text operational cost with retrieval-quality guardrails as a judged English passage field grows. Antfly, Elasticsearch, Quickwit, and PostgreSQL built-in full-text search index identical nested fixtures and answer the same MIRACL English development queries. The lane reports complete service memory demand, final allocated database-volume bytes, index duration, query p95/QPS, and judged ranking metrics.

This lane supplies million-scale lexical operational evidence. Search Benchmark Game owns query-class throughput, BEIR owns native-hybrid uplift, and STaRK owns structured retrieval. No service-count, round-trip consolidation, or official MIRACL leaderboard claim follows from this lane.

Source and license

The corpus is the English portion of MIRACL v1.0: 32,893,221 passages from 5,758,285 English Wikipedia articles. The public development panel contains 799 queries and 8,350 human positive and negative judgments. Corpus repository revision d921ec7e349ce0d28daf30b2da9da5ee698bef0d and judgment repository revision 5be20db9509754dadad47689368639fcec739c00 are immutable cache identities. Every corpus shard is pinned by its Hugging Face LFS SHA-256; topics and qrels are pinned by observed SHA-256.

The MIRACL dataset cards declare Apache-2.0. The corpus is derived from English Wikipedia, so applicable Wikipedia attribution and share-alike obligations are retained rather than treating the text as newly permissive content.

Citation:

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. “MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages.” Transactions of the Association for Computational Linguistics 11, 2023. DOI 10.1162/tacl_a_00595.

Deterministic nested fixtures

Each source passage becomes one canonical UTF-8 JSONL record with MIRACL docid as id and title + U+000A + text as the searchable text. Source strings are not normalized, truncated, or analyzer-normalized.

Every document appearing in a development qrel is placed first, ordered by UTF-8 document ID. Natural distractors are ordered by SHA-256 of the UTF-8 document ID. The implementation scans every pinned corpus shard, retains low SHA-256 prefix buckets, externally sorts only the retained candidate set under LC_ALL=C, and writes exact nested prefixes of 250,000, 1,000,000, 2,000,000, 3,000,000, and 5,000,000 passages. Every judged document must occur in every prefix. Selection does not inspect relevance grade.

Queries are exact MIRACL development topic text in UTF-8 query-ID order. Qrels are canonical four-column TREC rows in query-ID/document-ID order. Preparation rejects duplicate judgments, missing positive queries, missing judged documents, malformed rows, source count drift, and digest drift. preparation-receipt.json binds every source asset, full-corpus canonical digest, prefix identity, query order, qrels, grade histogram, judged-document counts, and selection algorithm.

Engine and query protocol

All engines receive one text field with product-default English lexical analysis. Antfly retains bounded ingest retries, a 900-second request timeout required by observed 5M-pass compaction stalls, and its searchable-readiness gate. Elasticsearch uses one primary shard, no replicas, a 1 GiB heap, refresh, and force merge. Quickwit uses its default tokenizer with positions and field norms; startup ingest retries tolerate the temporary no-shard gate, and publication waits up to 900 seconds for the exact indexed count. PostgreSQL 16 uses a generated English tsvector, a GIN index, and ts_rank_cd, with otherwise default server settings.

The runner streams the declared fixture and requires the indexed count to match. The first ten ordered queries are warmups. The measured pass includes all 799 queries once at concurrency one with top_k=100. Per-query latency spans request dispatch through complete response decoding; QPS is query count divided by measured wall time. Raw evidence contains only query ID, one-based ranked IDs, latency, and normalized error status.

Quality metrics

MIRACL English dev judgments are binary. The lane macro-averages:

nDCG@10;
MRR@10;
Recall@100;
Judged@100, the fraction of returned unique IDs having either a positive or negative human judgment.

MIRACL uses pooled judgments: an unjudged passage is not proven irrelevant. Judged@100 exposes assessment coverage, while nDCG, MRR, and Recall act as consistent cross-engine guardrails. Verification recomputes every metric from retained ranked IDs and rejects request errors, unknown IDs, duplicates, more than 100 hits, query-order drift, and summary disagreement.

Resource lifecycle and failures

The canonical publication cohort is four engines by four common prefix sizes (250K, 1M, 2M, and 3M) by three fresh trials. The retained 5M fixture is a diagnostic boundary: exploratory evidence showed PostgreSQL ranking would consume roughly another half day and Antfly entered sustained full-text merge backpressure, while Elasticsearch and Quickwit completed. Odd canonical trials run Antfly, Elasticsearch, Quickwit, PostgreSQL; even trials reverse the order. Cgroup v2 sampling begins before indexing and continues through all queries at 300 ms. Every engine is measured as one complete service container, and PostgreSQL shmem is included. Headline demand is peak concurrent anon + shmem; memory.current and kernel peaks remain diagnostics.

Fresh labeled Docker volumes capture empty, ready, post_workload, and stopped final barriers. post_workload follows a fixed 30-second settlement. Final allocated and apparent bytes use arithmetic mean plus every value/range; demand, index duration, p95, QPS, and judged metrics use median plus every value/range.

Every planned cell writes normalized status and a digest-bound receipt. All cells are attempted once. Aggregate status is ok for complete success, failed for a repeated uniform terminal gate across a complete engine/size group, and partial for every other fully accounted mix. Unaccounted cells invalidate finalization.

Circus packages presentation-neutral aggregate, source receipt, summary, methodology, input manifest, and preparation receipt through evidence_handoff.py; the site repository owns evidence selection and presentation.

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: fulltext-miracl-scale
Domain: full-text/retrieval
Source: https://huggingface.co/datasets/miracl
Known Antfly versions: v0.2.1-rc2
Enabled: yes

Benchmark parameters

{"common_complete_through_passages":3000000,"corpus_passages":32893221,"corpus_sizes":[250000,1000000,2000000,3000000,5000000],"dataset":"MIRACL v1.0 English development split","evidence_class":"1-trial descriptive scaling evidence","fresh_lifecycles_per_point":1,"judgments":8350,"memory_metric":"cgroup v2 anon + shmem","omissions":[{"engine":"postgresql","passages":5000000,"reason":"The complete 3M query pass took 4.45 hours at 0.0499 QPS. A full 5M pass was projected to consume substantially more than the remaining useful overnight window; the query panel was not narrowed and no timeout value was presented as latency."}],"queries":799,"storage_metric":"final allocated database-volume bytes"}

Peer versions

Peer versions were not recorded in the published result fields.

Evidence digests

Receipt: 0e9849866c690064980eef699805cb5123c4b19ea3979f02fd20995036488c95
Aggregate: 4069aa330155db2450d71df331764658fb9e7bdacfd3c4c8962e5c0fb105e162
Provenance: b89529140dde5c5556f0e7c32839ec62b59266fcde0e20a7d664c52636aef0ee
Harness commit: d400ffc1eb4e70beca97983aaec2abeccba9c7a3

Selected partial evidence

Completed: 2 Sep 2026 · 18:00 UTC
Antfly: v0.2.1-rc2 · 30 August 2026 · 23:16 UTC
Runtime: 12h 11m
Platform: linux · arm64
Machine: Apple M4 Max
Build: ReleaseFast

Query p95

Antfly v0.2.1-rc2 (4be5a0e9, ReleaseFast) · Linux/arm64 containers on Apple M4 Max · MIRACL English dev · 1 fresh lifecycles planned per point · Log scale · lower is better

Demand working set

Antfly v0.2.1-rc2 (4be5a0e9, ReleaseFast) · Linux/arm64 containers on Apple M4 Max · MIRACL English dev · 1 fresh lifecycles planned per point · Lower is better

Final database volume

Antfly v0.2.1-rc2 (4be5a0e9, ReleaseFast) · Linux/arm64 containers on Apple M4 Max · MIRACL English dev · 1 fresh lifecycles planned per point · Lower is better

Working set and complete post-stop volume

Antfly v0.2.1-rc2 · 4be5a0e9

Engine	Passages	Demand MB	Cache + memory MB	Final volume MB	Valid / total
Antfly	250,000	469.5 MB	795.9 MB	157.4 MB	1 / 1
Elasticsearch	250,000	1,712.7 MB	2,305.5 MB	123.3 MB	1 / 1
PostgreSQL	250,000	207.6 MB	901.3 MB	665.9 MB	1 / 1
Quickwit	250,000	547.4 MB	767.7 MB	202.7 MB	1 / 1
Antfly	1,000,000	1,811.9 MB	2,669.0 MB	596.3 MB	1 / 1
Elasticsearch	1,000,000	1,801.2 MB	2,886.6 MB	477.4 MB	1 / 1
PostgreSQL	1,000,000	293.1 MB	2,758.3 MB	2,322.6 MB	1 / 1
Quickwit	1,000,000	1,102.4 MB	1,725.0 MB	390.6 MB	1 / 1
Antfly	2,000,000	5,207.8 MB	6,747.4 MB	1,137.8 MB	1 / 1
Elasticsearch	2,000,000	1,881.6 MB	5,001.3 MB	947.0 MB	1 / 1
PostgreSQL	2,000,000	327.0 MB	3,936.5 MB	3,495.2 MB	1 / 1
Quickwit	2,000,000	1,827.8 MB	2,936.2 MB	773.4 MB	1 / 1
Antfly	3,000,000	8,279.2 MB	10,501.1 MB	1,721.4 MB	1 / 1
Elasticsearch	3,000,000	1,880.9 MB	4,903.8 MB	1,416.2 MB	1 / 1
PostgreSQL	3,000,000	321.3 MB	5,114.4 MB	4,645.2 MB	1 / 1
Quickwit	3,000,000	2,210.9 MB	3,818.9 MB	1,029.1 MB	1 / 1
Elasticsearch	5,000,000	1,881.5 MB	7,694.0 MB	2,352.8 MB	1 / 1
Quickwit	5,000,000	2,154.3 MB	4,785.5 MB	2,212.5 MB	1 / 1

Query and judged retrieval evidence

Antfly v0.2.1-rc2 · 4be5a0e9

Engine	Passages	QPS	p95 ms	nDCG@10	Recall@100	Judged@100
Antfly	250,000	788.289	2.28	0.576	0.942	0.098
Elasticsearch	250,000	363.665	3.44	0.552	0.930	0.094
PostgreSQL	250,000	2.719	1,081.78	0.239	0.791	0.083
Quickwit	250,000	47.024	22.80	0.555	0.939	0.095
Antfly	1,000,000	405.904	5.92	0.532	0.928	0.094
Elasticsearch	1,000,000	302.605	4.36	0.493	0.892	0.090
PostgreSQL	1,000,000	0.350	12,385.26	0.143	0.630	0.061
Quickwit	1,000,000	43.014	25.66	0.498	0.903	0.091
Antfly	2,000,000	249.998	10.42	0.459	0.844	0.086
Elasticsearch	2,000,000	258.106	5.62	0.453	0.866	0.087
PostgreSQL	2,000,000	0.091	37,382.05	0.097	0.508	0.048
Quickwit	2,000,000	40.320	27.82	0.458	0.877	0.088
Antfly	3,000,000	182.201	15.01	0.462	0.859	0.087
Elasticsearch	3,000,000	195.861	7.15	0.425	0.849	0.085
PostgreSQL	3,000,000	0.050	61,861.82	0.076	0.432	0.041
Quickwit	3,000,000	38.805	29.29	0.427	0.856	0.086
Elasticsearch	5,000,000	168.856	8.73	0.390	0.825	0.082
Quickwit	5,000,000	27.452	46.30	0.391	0.829	0.083

deployment-artifacts

complete

Deployment artifacts

Verified Linux/arm64 deployment artifact metadata for the exact measured rc2 subject.

Methodology scope · protocol · interpretation

Deployment artifacts

This lane establishes packaging and transfer facts for the exact Linux/arm64 artifacts represented by the honed benchmark review. It does not start a database, measure runtime resource use, or assign a composite score.

Cohort and identity

The cohort is Antfly plus the five peers in the honed VectorDBBench publication matrix: Weaviate, pgvector, Chroma, Milvus, and Elasticsearch. Antfly is bound to the exact archive and executable in the honed subject manifest. A published subject resolves its archive through the GitHub Releases API. An operator-built subject instead names its retained local archive explicitly; the collector verifies the file name and SHA-256 before measuring it. Each peer is bound to the exact digest-pinned image reference declared by this lane and uses the same image environment overrides as VectorDBBench.

Every engine/artifact pair is one portable publication-cache cell. A cache hit requires the same subject archive identity or peer image reference, platform, collector protocol, and workload identity. Cache objects retain normalized facts, source metadata, logs, and a lifecycle receipt. Cache identity excludes timing because the lane performs no timed work.

Download payload

Antfly’s download payload is the byte size of the exact Linux/arm64 subject archive. For a published release, the collector requires the GitHub Releases API asset digest to match the archive SHA-256 in the honed subject manifest. For an operator build, it requires an explicit retained archive path and verifies the local file’s name and SHA-256 against the same manifest fields. The result identifies whether the archive came from an official release or an operator build. It separately retains the executable byte size, executable SHA-256, and static-linkage identification from that subject manifest.

A peer’s download payload is the sum of the compressed layer sizes in the one os=linux, architecture=arm64 child selected from docker manifest inspect --verbose. The collector retains the configured index reference, resolved child digest, selected descriptor, config digest, layer count, and selected manifest. Missing or ambiguous platform selection fails the lane.

The image total sums every compressed layer in the selected manifest without credit for layers already cached on an operator’s machine. A machine with shared layers may therefore transfer less. Registry transfer overhead and expanded filesystem size are excluded. This is a reproducible artifact comparison, not an estimate of every installation path.

Deployment vocabulary

“Separately managed prerequisites” means software an operator must install, launch, upgrade, and coordinate separately from the displayed server artifact. None excludes operating-system kernel, storage, and networking. Peer container rows therefore name an OCI-compatible container runtime. Components packaged inside an image, such as Elasticsearch’s JVM or Milvus’s embedded etcd/local-storage topology, are bundle components rather than separately managed prerequisites.

The compact review table describes only the exact artifacts used by the publication matrix. It does not claim that these are the only installation forms supported by each project.

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: deployment-artifacts
Domain: deployment/artifacts
Source: https://github.com/antflydb/antfly/releases
Known Antfly versions: v0.1.3, v0.2.*
Expected duration: 2 min
Enabled: yes

Benchmark parameters

{"artifacts":[{"archive_name":"antfly_0.2.1-rc2_Linux_arm64.tar.gz","archive_sha256":"444029d7996d841808680141ae2730dd3b258a71b21dfda6312ac3bad016840c","bundle_components":[],"engine":"antfly","executable_sha256":"4e9440e5c19f7b5b41514190f8b8d692860c6a0262dca87a8755b32f30e0a88e","linkage":"static","prerequisites":[]}],"platform":"linux-arm64"}

Peer versions

Weaviate1.39.2
pgvector0.8.6
Chroma1.5.9
Milvus2.6.22
Elasticsearch9.5.2

Evidence digests

Receipt: 3e457b4e74ad4909a358ae34eeee6873a2f9ff15be0f66a3f4e4312d83816a49
Aggregate: 8fbbac537ac3190a4d4661c0102ef34a107d13356dec57299e5e1b6aec7f12a0
Provenance: 1150254466ab00aee1885e1b181ceae00b86696f4e983de75de834e6394912c0
Harness commit: 27ef68801c65cb9274290131ed49b6020233e9ef

Selected measurement

Completed: 31 Aug 2026 · 05:31 UTC
Antfly: v0.2.1-rc2 · 29 August 2026 · 21:46 UTC
Runtime: 20.4s
Platform: linux · arm64
Machine: Apple M4 Max
Build: ReleaseFast

Linux/arm64 deployable artifacts

Antfly v0.2.1-rc2 · b4a46545

Engine	Deployable form	Artifact origin	Download MB	Executable MB	Status
antfly	Static executable	operator supplied archive	52.7	121.1	ok

Retained peer artifact status

Antfly v0.2.1-rc2 · b4a46545

Engine	Status
weaviate	ok
pgvector	ok
chroma	ok
milvus	ok
elasticsearch	ok

2 of 2 lanes measured

STaRK

Official retrieval quality over textual and relational knowledge bases, with exact candidate, query, metric, and lifecycle provenance.

retrieval-stark-mag

complete

MAG

Official 2,665-query STaRK-MAG retrieval quality over 700,244 paper candidates, with one fresh rc2 Antfly container-footprint observation.

Methodology scope · protocol · interpretation

STaRK-MAG in Antfly Circus

This lane evaluates paper retrieval over 700,244 official candidates and the complete official 2,665-query test split. Candidate identity uses stable official numeric paper candidate IDs; quality uses the official STaRK evaluator with official text-embedding-ada-002 vectors.

Terms

Lexical: full-text ranking over serialized paper text.
Dense: nearest-neighbor search over the official embeddings.
RRF: reciprocal-rank fusion of lexical and dense rankings.
Relation-expanded: paper text with flattened graph-relation context.
Recall@20: mean fraction of gold paper IDs returned in the first 20 results.

Evidence scope

Quality covers the complete official split. Published reference rows retain their reported metric coverage and execution setting; LLM-assisted systems are not database-native equivalents. Memory and qualified Docker storage are individual observations from one fresh lifecycle; no variance, ratio, or comparative footprint claim is selected. Latency is a local diagnostic.

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: retrieval-stark-mag
Domain: retrieval/knowledge-base
Source: https://github.com/snap-stanford/stark
Known Antfly versions: v0.2.1-rc2
Enabled: yes

Benchmark parameters

{"candidate_identity_sha256":"584ec028e9c92616f44e0a462f0c2d088af138a0e40ff7f3c50f6deab8eb060b","dataset":"snap-stanford/stark","dataset_revision":"88269e23e90587f99476c5dd74e235a0877e69be","queries":2665,"records":700244,"vector_dimensions":1536,"vector_model":"text-embedding-ada-002","vector_score":"inner_product"}

Retrieval systems

Antfly lexical · default text Native lexical retrieval 2,665 queries 373.79 ms mean
Antfly dense · official vectors Native dense retrieval 2,665 queries 106.89 ms mean
Antfly built-in RRF · default text One native hybrid request 2,665 queries 576.41 ms mean
Antfly lexical · relation-expanded text Native lexical retrieval 2,665 queries 442.60 ms mean
Antfly built-in RRF · relation-expanded text One native hybrid request 2,665 queries 729.27 ms mean
Upstream exact VSS Exact official-vector baseline 2,665 queries 21.20 ms mean
Upstream BM25 Pinned upstream lexical baseline 2,665 queries 4.40 ms mean

Published comparison sources

AF-Retriever full synthetic test; Recall@20 not reported zero-shot LLM, graph retrieval, vector search, pairwise reranking Source
GraphRAFT full synthetic test; Recall@20 not reported dataset-trained LLMs, Cypher templates, graph retrieval, reranking Source
HybGRAG full synthetic test; Recall@20 not reported LLM-assisted hybrid graph and text retrieval Source
multi-ada-002 full synthetic test original STaRK multivector baseline Source
voyage-large-2-instruct full synthetic test original STaRK dense baseline Source

Peer versions

Peer versions were not recorded in the published result fields.

Evidence digests

Receipt: 6fe666a90ec3a2fbdc6ff4100fcec2fe97b8b35fca27309ab3e01a30ffa29d2f
Aggregate: e0134c6b4b67c8e540bc0d4eb6561af0fbfdc0e27fc4281e5932b65ac7478498
Provenance: a858d3d5fa807f829772d390593a208923a4b59fa5cfe29505ad2a3964ee7353
Harness commit: 598cdd0483de5b51ea562f8ff11dc9ca2ae9d4a3

Selected measurement

Completed: 1 Sep 2026 · 07:54 UTC
Antfly: v0.2.1-rc2 · 30 August 2026 · 21:48 UTC
Runtime: 263.4 min
Platform: Linux · ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped
Machine: Apple M5 Pro · 8 CPU · 25.2 GB
Build: ReleaseFast · Zig antfly 0.2.1-rc2 (zig runtime)

STaRK-MAG Recall@20

Official full synthetic test · Antfly rc2 and original-paper full-test baselines · Higher is better

STaRK-MAG retrieval quality

Antfly v0.2.1-rc2 · 4be5a0e9

Method	Hit@1 %	Hit@5 %	Recall@20 %	MRR %
Antfly lexical · relation-expanded text	52.76%	72.61%	72.18%	61.69%
Antfly built-in RRF · relation-expanded text	42.89%	68.82%	70.38%	54.44%
Antfly built-in RRF · default text	32.53%	52.83%	52.76%	41.72%
Published · multi-ada-002	25.92%	50.43%	50.80%	36.94%
Published · voyage-large-2-instruct	30.06%	50.58%	50.49%	39.66%
Upstream exact VSS	29.08%	49.61%	48.36%	38.23%
Antfly dense · official vectors	27.73%	48.89%	48.36%	37.26%
Antfly lexical · default text	27.92%	45.55%	46.11%	35.91%
Upstream BM25	25.85%	45.25%	45.69%	34.56%
Published · AF-Retriever	78.60%	91.40%	—	84.00%
Published · GraphRAFT	71.10%	85.30%	—	76.90%
Published · HybGRAG	65.40%	75.30%	—	69.80%

Container footprint observation

Antfly v0.2.1-rc2 · 4be5a0e9

Engine	Lifecycle evidence	Demand memory GiB	Cache-inclusive GiB	Final allocated storage GiB
Antfly	One fresh lifecycle · descriptive observation	17.018 GiB	21.264 GiB	12.080 GiB

retrieval-stark-amazon

partial

Amazon

Complete 1,642-query STaRK-Amazon default-text adapter study over 957,192 product candidates, with relation-expanded construction retained as unavailable scale evidence.

Methodology scope · protocol · interpretation

STaRK-Amazon default-text adapter study

Scope

This exploratory lane evaluates six Antfly query formulations over 957,192 official product candidates and all 1,642 synthesized test queries. Default documents contain upstream product title, description, review, and customer Q&A text. Every lexical row is one Antfly request; hybrid rows add the official ada-002 query vector in the same request and use built-in RRF.

The all-term formulation retains every unique lowercase ASCII-alphanumeric query term and joins them by OR without the legacy stopword, minimum-length, or first-20 filters. IDF top-20 selects terms using document frequency without qrels. Metrics are recomputed from returned top-20 IDs and official qrels.

Evidence quality

Default text has six of six valid rows from one complete lifecycle. Memory and storage are descriptive single observations. Latency is a local diagnostic and is kept in Run details. Published BM25 and ada-002 values are external Table 6 references, not same-run controls.

Relation-expanded coverage is zero of six: No relation-expanded quality row was accepted because no attempted lifecycle ingested all official candidates and reached the final visibility barrier. No partial relation index was scored. Relation expansion means STaRK’s application-side brand, color, category, also-bought, and also-viewed graph context flattened into text; it is not native graph traversal.

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: retrieval-stark-amazon
Domain: retrieval/knowledge-base
Source: https://github.com/snap-stanford/stark
Known Antfly versions: v0.2.1-rc2
Enabled: yes

Benchmark parameters

{"candidate_id_sha256":"c15dbc912e6494145295c9178e2848374b23b2291bf201c961b03f7b36edef8b","candidates":957192,"corpus_sha256":"aa52c91035b4aef826bf1e9ab70c3c9d264b0d949e08e349423bcab56996b832","dataset_revision":"88269e23e90587f99476c5dd74e235a0877e69be","name":"amazon","queries":1642}

Peer versions

Peer versions were not recorded in the published result fields.

Evidence digests

Receipt: 00d56bdfef2a36b31d0c256d4201210c63da0ed9a46a7fd6bae42d881d37656f
Aggregate: a2c587a79c8b1e60dd18677a33d3c4512d3970d69cdcabb2a1ea8184e45b4436
Provenance: 9536848b7e88b248ef41fee2bc710fdc41423eafff0b6eb76a8274cc128806f8
Harness commit: 3d85a0a6ab985ad3f6a6abf9281c927cc8277ee6aa986cabc55ed3dcc1f514e0

Selected partial evidence

Completed: 3 Sep 2026 · 07:32 UTC
Antfly: v0.2.1-rc2 · 30 August 2026 · 21:48 UTC
Runtime: 4.1 h
Platform: darwin · arm64
Machine: Apple M5 Pro · 8 CPU · 35.2 GiB Docker VM
Build: ReleaseFast · Zig antfly 0.2.1-rc2 (zig runtime)

STaRK-Amazon default-text Recall@20

Antfly v0.2.1-rc2 · Docker Desktop Linux/arm64 on Apple M5 Pro · 957,192 products · 1,642 test queries · Higher is better

Default-text query formulation

Antfly v0.2.1-rc2 · 4be5a0e9

Method	Hit@1 %	Hit@5 %	Recall@20 %	MRR@20 %	Official queries
Built-in RRF · all-term OR	49.70%	74.06%	60.75%	60.10%	1,642 / 1,642
Built-in RRF · first-20 OR	48.54%	72.90%	59.95%	59.26%	1,642 / 1,642
Lexical · all-term OR, uncapped	47.93%	68.70%	55.85%	57.28%	1,642 / 1,642
Lexical · IDF top-20 OR	48.23%	68.94%	55.48%	57.38%	1,642 / 1,642
Lexical · filtered OR, uncapped	46.53%	68.27%	54.96%	56.00%	1,642 / 1,642
Lexical · first-20 filtered OR	46.35%	67.66%	54.44%	55.81%	1,642 / 1,642
Published · BM25	44.94%	67.42%	53.77%	55.30%	Published full test
Published · ada-002	39.16%	62.73%	53.29%	50.35%	Published full test

Representation coverage and default footprint

Antfly v0.2.1-rc2 · 4be5a0e9

Representation	Valid rows	Evidence	Demand memory GiB	Cache-inclusive peak GiB	Final allocated storage GiB
Default text	6 / 6 valid	One complete lifecycle · descriptive	22.985 GiB	32.069 GiB	8.689 GiB
Relation-expanded	0 / 6 valid	Unavailable · construction did not reach full visibility	—	—	—

graph-benchmark

complete

Graph

Graph query performance, Pokec-small cross-engine diagnostics, official LSQB capability, and bounded SNB Interactive qualification.

Methodology scope · protocol · interpretation

Graph Database Benchmark v2 Methodology

Scope and provenance

This benchmark has two lanes and does not calculate a composite winner. LSQB-derived is an Antfly-owned anchored microbenchmark over LSQB data. The Pokec suite is a clean-room implementation informed by the public categories and workload ratios described by Memgraph’s Benchgraph material. It reuses the public Pokec import files, not Benchgraph implementation code. Neither lane is an audited LDBC result.

The scorecard keeps these measurements separate:

Lane	Scorecard metrics	Direction
LSQB-derived F2	warm serial p50, warm serial p99	latency lower
Pokec W1	completed QPS, peak process RSS	QPS higher; RSS lower

Full statistics, errors, run ranges, and raw samples remain mandatory even when the scorecard is used.

Engines and equal client work

The comparison is Antfly, Neo4j 5.26.29, and Memgraph 3.12.0. Kuzu is excluded because it is not exercised through a comparable network service. Every timed throughput cell uses the same Go executable, deterministic schedule, quantile implementation, response-consumption rule, worker count, and wall-clock. Antfly is accessed over persistent HTTP/1.1; both peers use the same official Neo4j Go Bolt driver. Transport is part of observed client latency and is reported, not hidden.

Only equivalent User(id) and benchmark-namespace ID indexes are created. There is no engine-specific query tuning. Each run starts from a fresh database and performs complete correctness checks before timing. Mutating cells use a reserved BenchUser namespace so source reads cannot change; that namespace is restored to its 1,024-node ring baseline after every mutating cell. Timed requests are never retried. Errors are retained and exclude a run from the publication gate.

LSQB-derived lane

The pinned source, data loading, parameter selection, BLAKE3 result oracle, and N1/P2/F2 definitions are specified in graph-lsqb-derived.md. The public LSQB-derived metric is warmed serial latency for this exact endpoint-filtered lookup, returning matching Post IDs:

(Forum)-[:CONTAINER_OF]->(Post)-[:HAS_TAG]->(Tag {id: $tag_id})

Unfiltered P2 C12 throughput remains a retained diagnostic. It is not part of the graph scorecard or public filtered-lookup claim. Default, medium, and full use LSQB SF0.003, SF0.1, and SF1 respectively.

Pokec data and parameter oracle

Profiles use the public Memgraph Pokec import files:

Profile	Dataset	Users	Directed `FRIEND` edges	Anchors
quick	Pokec-small	10,000	121,716	12
medium	Pokec-medium	100,000	1,768,515	48
full	Pokec-large	1,632,803	30,622,564	128

The harness records the downloaded source SHA-256 and refuses unexpected node or edge counts. Anchors are selected deterministically across out-degree deciles with seed 0xB37C4. Every selected anchor must return its complete result below 900 rows through four hops; this stays below rc18’s 1,000 pattern match ceiling rather than comparing truncated results.

Expansion semantics are one endpoint result per directed walk. Antfly uses a chain of fixed one-hop pattern steps. Cypher uses one fixed MATCH clause per hop, avoiding vendor differences in variable-length path uniqueness. The offline SQLite oracle stores the exact result count and canonical BLAKE3 digest including multiplicity for every parameter. Aggregates and shortest path have exact scalar oracles. All 101 quick-profile checks per engine/run must pass before timing. Writes are executed and verified in the isolated benchmark namespace.

Pokec operations

ID	Operation
R1	exact indexed node lookup
WN	insert a new benchmark node
U1	increment one benchmark-node property
WE	insert one directed benchmark edge
AG1	age buckets and counts
AG2	age count/min/max/sum/average
E1–E4	exact fixed 1-, 2-, 3-, or 4-hop directed expansion
E2F	exact 2-hop expansion with endpoint `age >= 18`
N2	concatenated exact 1-hop and exact 2-hop neighborhood
SP	directed unweighted shortest path, maximum depth 8

Every operation has an unwarmed serial observation and a prewarmed serial distribution. Isolated throughput uses 12 clients. Each non-mutating operation also receives a 70% operation / 30% write cell, split evenly between node and edge insert. The deterministic realistic workloads are:

W1: 20% analytical, 40% point read, 10% update, 30% write;
W2: 70% point read, 30% write;
W3: 50% point read, 50% write;
W4: 30% point read, 70% write.

The quick profile uses one-second isolated/mixed cells and two-second W1–W4 cells. Medium uses 5/10 seconds; full uses 15/30 seconds. Warmup is untimed. Every successful timed request is retained in JSONL. Reports contain completed QPS, error counts, p50/p95/p99, per-operation statistics, and peak PID RSS.

Persistent database volume

The honed LSQB SF0.1 F2 lane creates a new, run-labeled Docker local volume for every engine and repetition. Only the complete writable database root is measured (/data for Antfly and Neo4j, /var/lib/memgraph for Memgraph); read-only dataset/tool volumes and client scratch are excluded. A digest-pinned Alpine probe captures whole-volume allocated and apparent bytes once at empty, service-ready-before-ingestion, and post-workload, then at the verified post-stop final barrier. Final allocated bytes are the headline.

All three storage values, their arithmetic mean and range, apparent-byte diagnostics, barrier samples, volume manifests, and lifecycle receipts are retained. A provisional stop timeout remains visible but cannot support a comparative ratio.

Runs, ordering, and reporting

Each profile requires three clean process runs. Engine order rotates by run and uses a different rotation between lanes. Public serial point estimates are the medians of the three run-level p50, p95, and p99 values. Every result row also retains the three run-level percentile records, their min–max ranges, pooled sample percentiles as diagnostics, and all raw samples. Load/index readiness is diagnostic and is never included in query metrics.

summary.json is the machine-readable source of truth. The Markdown report contains the full cell table; scorecard.svg is only the compact two-panel view. There is deliberately no geometric mean or composite rank.

The honed LSQB SF0.1 F2 lane caches only the complete three-engine cohort. Promotion requires Antfly, Neo4j, and Memgraph to each retain three fresh database constructions and process runs, 15,000 timed samples, all three run-level percentile records and ranges, three complete storage receipts and their allocated-byte mean/range, 1,000/1,000 correctness, a valid canonical receipt, and the quiet-host record. The cache identity binds the Antfly subject, both peer image digests, the Linux client and storage-probe image, dataset manifest, harness sources, and protocol. A hit materializes the entire cohort; per-engine cache mixing is not permitted.

Internal fresh-service F2 readiness and convergence

The honed and Circus profiles retain a fresh-service diagnostic that measures how quickly a preloaded graph service becomes useful and settles into its warm filtered-two-hop behavior. It is graph-specific: readiness is followed immediately by the same exact LSQB SF0.1 F2 request used by the warm latency lane. It is not a general process-startup or system-wide Antfly claim.

This cohort is deliberately internal. It is attached to the canonical graph run report and structured result, but the public benchmark-page renderer does not render its table. It is never eligible for the honed publication cache: each honed or gauntlet graph invocation collects it afresh with make graph-f2-warmup-run. The retained diagnostic contract requires:

the same Antfly binary, pinned Neo4j and Memgraph images, official LSQB SF0.1 dataset, and Linux/arm64 Docker environment as the primary graph lane;
a recorded physical-host state, including whether the honed quiet-host assertion was active, without treating a diagnostic run as publication data;
one database construction followed by five fresh service lifecycles per engine, with balanced engine ordering;
restart-to-ready measured from immediately before service launch through the first successful engine-native probe: Antfly GET /readyz, Neo4j cypher-shell RETURN 1 plus the pinned-version check, and Memgraph autocommit SHOW VERSION over Bolt;
the first F2 latency and first-100 p50/p95 from a fixed 100-anchor panel selected by SHA-256 rank independently of result cardinality;
convergence at the first point where three consecutive adjacent 100-query window p50 changes are each within 5%, with a 10,000-query ceiling;
steady-state p50/p95 from five deterministic sweeps over all 1,000 anchors after convergence;
every timed result checked against the independent count+BLAKE3 oracle, with all 1,000 anchors covered in every engine/lifecycle; and
median-of-five run metrics with complete run ranges, raw timestamped samples, lifecycle logs, host evidence, exact schedules, and a canonical receipt.

These seven values are an indivisible internal reporting family: restart-to-ready, first-query, first-100 p50, first-100 p95, queries-to-stable-p50, steady p50, and steady p95. The retained report must show all seven for all three engines; it must not headline convergence while omitting the steady-state latency guardrail or Memgraph.

Qualification and publication

The default command is a small, complete optimization loop with a target wall time below 15 minutes. Medium and full are explicit diagnostic commands. A mixed native-macOS/container run is qualification-only because its process RSS views are not comparable across engines.

The release-facing honed F2 lane is a separate, environment-qualified publication protocol. Antfly, Neo4j, and Memgraph all run sequentially inside the same Docker Desktop Linux/arm64 VM on an explicitly quiescent Apple Silicon host. The published claim must name that environment and is limited to warm serial filtered two-hop latency; it is not a bare-metal Linux result or a claim about general graph-database performance. Publication requires:

the exact v0.2.0 Linux/arm64 binary and expected SHA-256;
the pinned Neo4j 5.26.29, Memgraph 3.12.0, and Linux client image digests;
official LSQB SF0.1 with 432,235 nodes and 1,827,523 relationships;
three fresh database constructions and process runs per engine, each on a newly created labeled database volume;
five warmed serial sweeps per run over 1,000 deterministic anchors;
1,000/1,000 count+BLAKE3 correctness in every engine/run and zero timed errors;
median run-level p50/p95/p99 with all run ranges, pooled diagnostics, raw samples, lifecycle logs, and the physical-host record retained; and
a canonical receipt for the indivisible three-engine cohort.

The broader graph-v2/Pokec publication gate remains native-Linux-only and is not satisfied by this bounded F2 artifact.

Run evidence parameters · variation · provenance

Benchmark metadata

Benchmark ID: graph-benchmark
Domain: graph benchmark
Source: https://github.com/ldbc/lsqb
Known Antfly versions: v0.1.3, v0.2.*
Expected duration: 100 min
Enabled: yes

Benchmark parameters

{"peers_not_run":["neo4j","memgraph"],"query":"F2","runs":3,"samples":15000}

Trial variation

[{"engine":"memgraph","ranges":{"final_database_volume":{"maximum":311.906304,"mean":311.906304,"minimum":311.906304,"unit":"MB"},"query_p95":{"maximum":0.14466905,"mean":0.1440441,"minimum":0.14354615,"unit":"ms"}}},{"engine":"neo4j","ranges":{"final_database_volume":{"maximum":635.670528,"mean":635.663701,"minimum":635.650048,"unit":"MB"},"query_p95":{"maximum":0.6908785500000003,"mean":0.6795417000000001,"minimum":0.6769125000000001,"unit":"ms"}}}]

Peer versions

Memgraph3.12.0
Neo4j5.26.29

Evidence digests

Receipt: 0eb64fb994a14490b27e7444216ed8307f563a4479f67ecdef20a22647f38571
Aggregate: 0e9fee28f227f3cb9afc8b7c032803c2daec41608c07cf184491a1aabd8dccec
Provenance: e60cc69cc71d710d3628494114c51f9b722fb1b6e51fb5df7b40bf1f42c6d17f
Harness commit: ad4635e78a54618f3217a49489324853ca8548e3

Selected measurement

Completed: 30 Aug 2026 · 22:28 UTC
Antfly: v0.2.1-rc2 · 29 August 2026 · 21:46 UTC
Runtime: 12m 27s
Platform: linux · arm64
Machine: Apple M4 Max
Build: ReleaseFast

Warm serial p95

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · LSQB-derived SF0.1 F2 · 15,000 samples / three runs · Lower is better

Demand working set

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · LSQB-derived SF0.1 F2 · 15,000 samples / three runs · Lower is better

Final database volume

Antfly v0.2.1-rc2 (b4a46545, ReleaseFast/full) · linux-arm64 · LSQB-derived SF0.1 F2 · 15,000 samples / three runs · Lower is better

Latency, resources, and served samples

Antfly v0.2.1-rc2 · b4a46545

Engine	Version	Query	p50 ms	p95 ms	p99 ms	Demand working set MB	Final database volume MB	Served samples
antfly	antfly 0.2.1-rc2 (zig runtime)	F2	0.268	0.393	0.602	1,989.3 MB	182.4 MB	15,000
memgraph	3.12.0	F2	0.113	0.144	0.175	Not recorded MB	311.9 MB	15,000
neo4j	5.26.29	F2	0.376	0.680	1.076	Not recorded MB	635.7 MB	15,000