2 of 2 lanes measured
VectorDBBench
Qualified vector search quality, throughput, serial latency, serving demand, and database volume across operating points.
vector-vdbbench-50k
complete50K
Three independent rc.5 lifecycles on 50K 1536-dimensional vectors; historical peer references.
Methodology scope · protocol · interpretation
Subject and execution
Official v0.2.1-rc.5, commit 43fda0ba4684163a3ee563f18fd4ad61849003cf, unchanged Linux ARM64 release executable. Runtime: rootless Podman 6.1.1 on Apple M4 Max, 10 VM CPUs and 24 GiB VM memory. The build is identified as the official release; compiler optimization flags are not independently asserted.
Only Antfly was measured in this run. Retained peer rows have their original versions and evidence dates; they are historical references, not a matched runtime cohort or evidence of a release-to-release speedup.
Vector protocol
Three independently built indexes per workload. Search parameters are selected only on the deterministic calibration partition; publication queries use the disjoint holdout without rebuilding. All three selected search values are retained. Serial latency and peak throughput are different operating points; peak QPS is the best measured concurrency point, not serial QPS. Final rows have zero request errors.
One completed same-rc.5 lifecycle per workload was reused through validated cache materialization; four additional fresh lifecycles completed the matrix. No rc.4 measurements were reused. An earlier 1M attempt cell was overwritten by the old retry loop; its combined client output remains retained and the attempt is excluded from final metrics.
Resources and retained evidence
Demand is container cgroup-v2 anon plus shmem, excluding reclaimable file cache. Cache-inclusive peaks are reported separately. Resource point estimates are arithmetic means of three lifecycle peaks, with retained min–max ranges. Final database volume is complete writable database-root allocated bytes at the verified stopped-final barrier, not archive or executable size. Packages retain hashes of the original result, provenance and checksum receipt; source files were verified against those receipts before packaging.
Run evidence parameters · variation · provenance
Benchmark parameters
{"concurrency_duration_seconds":30,"concurrency_schedule":[1,5,10,20,30],"execution_environment":{"container_runtime":"rootless Podman 6.1.1","processor":"Apple M4 Max","vm_cpus":10,"vm_memory_gib":24},"lifecycle_search_values":[0.425,0.425,0.425],"query_count":800,"query_partition_sha256":"7be5c5f24dfa31b66c5dfc3280c8f1bc5c1c8db3a35046a1a1ab9d50a0d232f7","search_parameter":"search_effort"}Trial variation
[{"engine":"Antfly","ranges":{"demand_working_set":{"maximum":5681.8,"mean":5298.5,"minimum":4541.8,"unit":"MB"},"final_database_volume":{"maximum":354.213888,"mean":353.924437,"minimum":353.419264,"unit":"MB"},"p99_at_30_concurrent":{"maximum":84.8,"mean":84.333,"minimum":84,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Chroma","ranges":{"demand_working_set":{"maximum":466.7,"mean":463.3,"minimum":460.9,"unit":"MB"},"final_database_volume":{"maximum":332.488704,"mean":332.485973,"minimum":332.480512,"unit":"MB"},"p99_at_30_concurrent":{"maximum":179,"mean":177.6,"minimum":175.3,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Elasticsearch","ranges":{"demand_working_set":{"maximum":1952.7,"mean":1941,"minimum":1934.8,"unit":"MB"},"final_database_volume":{"maximum":312.082432,"mean":312.067413,"minimum":312.049664,"unit":"MB"},"p99_at_30_concurrent":{"maximum":15.1,"mean":14.467,"minimum":14,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Milvus 2.6.22","ranges":{"demand_working_set":{"maximum":2766.6,"mean":2666.2,"minimum":2563.6,"unit":"MB"},"final_database_volume":{"maximum":2393.47712,"mean":2393.459371,"minimum":2393.432064,"unit":"MB"},"p99_at_30_concurrent":{"maximum":11.4,"mean":11.233,"minimum":11.1,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"pgvector","ranges":{"demand_working_set":{"maximum":2303.8,"mean":2303.7,"minimum":2303.7,"unit":"MB"},"final_database_volume":{"maximum":1523.970048,"mean":1523.967317,"minimum":1523.965952,"unit":"MB"},"p99_at_30_concurrent":{"maximum":33.1,"mean":32.033,"minimum":31.4,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Weaviate","ranges":{"demand_working_set":{"maximum":977.9,"mean":953.8,"minimum":936.6,"unit":"MB"},"final_database_volume":{"maximum":391.55712,"mean":391.514795,"minimum":391.491584,"unit":"MB"},"p99_at_30_concurrent":{"maximum":89.6,"mean":86.967,"minimum":85.3,"unit":"ms"}},"workload":"1536d-50K"},{"engine":"Milvus 3.0.0","ranges":{"demand_working_set":{"maximum":2949.6,"mean":2819,"minimum":2719.7,"unit":"MB"},"final_database_volume":{"maximum":2489.167872,"mean":2489.156949,"minimum":2489.143296,"unit":"MB"},"p99_at_30_concurrent":{"maximum":12,"mean":11.667,"minimum":11.4,"unit":"ms"}},"workload":"1536d-50K"}]Peer versions
- Chroma1.5.9
- Elasticsearch9.5.2
- Milvus 2.6.222.6.22
- pgvector0.8.6 (PostgreSQL 16)
- Weaviate1.39.2
- Milvus 3.0.03.0.0
Evidence digests
- Receipt
5d0855e3ecced09eb42842fc6d366f207b78eff0b1df415e62f5a3a8cbe77b98- Aggregate
8cea655d41f469ca7ae54c208751bd88090225677d4b889dbf73a69b8b5c416a- Provenance
8cea655d41f469ca7ae54c208751bd88090225677d4b889dbf73a69b8b5c416a- Harness commit
Peer cohort evidence · 1536d-50K
- Completed
- Fresh lifecycles
- 18
- Peer variants
- 6
- Receipt
782e8e470765fae9914945e67c9da3f3e82a1b38571ebe6e5940228a8248601c- Aggregate
0eb49aba01e15e065ec68b1667beb2c41a580cde5dab2cb928b6f8c01c16daf4- Provenance
700d49bf35039da97e9b5366d4a4e6ec4c417fdfdb5fd720b980349a538d68c3
- Completed
- Antfly
v0.2.1-rc.5·- Runtime
- 2h 16m
- Platform
- linux · arm64
- Machine
- Apple M4 Max
- Build
- official release
Peer cohort · 1536d-50K
- Completed
- Variants
- 6 independently executed peers
- Lifecycles
- 18 fresh
- Compatibility
- Historical reference · not a matched runtime
Peak throughput
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 1536d-50K · three fresh lifecycles · Higher is better · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Serial query p95
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 1536d-50K · three fresh lifecycles · Lower is better · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Demand working set
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 1536d-50K · three fresh lifecycles · Lower is better · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Final database volume
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 1536d-50K · three fresh lifecycles · Lower is better · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Performance, quality, and resource evidence
Antfly v0.2.1-rc.5 · 43fda0ba
| Engine | Workload | Recall | nDCG | Peak QPS queries/s | p95 ms | Demand working set MB | Cache-inclusive peak MB | Final database volume MB | Trials |
|---|---|---|---|---|---|---|---|---|---|
| Antfly | 1536d-50K | 0.9570 | 0.9630 | 990.1 | 3.533 | 5,298.5 MB | 5,556.2 MB | 353.9 MB | 3 |
| Chroma | 1536d-50K | 0.9870 | 0.9890 | 180.9 | 8.067 | 463.3 MB | 812.9 MB | 332.5 MB | 3 |
| Elasticsearch | 1536d-50K | 0.9560 | 0.9640 | 4,434.6 | 2.033 | 1,941.0 MB | 4,385.1 MB | 312.1 MB | 3 |
| Milvus 2.6.22 | 1536d-50K | 0.9630 | 0.9700 | 5,293.1 | 1.600 | 2,666.2 MB | 5,426.1 MB | 2,393.5 MB | 3 |
| pgvector | 1536d-50K | 0.9690 | 0.9750 | 2,721.0 | 4.167 | 2,303.7 MB | 3,533.5 MB | 1,524.0 MB | 3 |
| Weaviate | 1536d-50K | 0.9550 | 0.9630 | 1,008.8 | 6.300 | 953.8 MB | 1,511.3 MB | 391.5 MB | 3 |
| Milvus 3.0.0 | 1536d-50K | 0.9630 | 0.9700 | 5,019.7 | 1.700 | 2,819.0 MB | 5,557.7 MB | 2,489.2 MB | 3 |
vector-vdbbench-1m
complete1M
Three independent rc.5 lifecycles on 1M 768-dimensional vectors; historical peer references.
Methodology scope · protocol · interpretation
Subject and execution
Official v0.2.1-rc.5, commit 43fda0ba4684163a3ee563f18fd4ad61849003cf, unchanged Linux ARM64 release executable. Runtime: rootless Podman 6.1.1 on Apple M4 Max, 10 VM CPUs and 24 GiB VM memory. The build is identified as the official release; compiler optimization flags are not independently asserted.
Only Antfly was measured in this run. Retained peer rows have their original versions and evidence dates; they are historical references, not a matched runtime cohort or evidence of a release-to-release speedup.
Vector protocol
Three independently built indexes per workload. Search parameters are selected only on the deterministic calibration partition; publication queries use the disjoint holdout without rebuilding. All three selected search values are retained. Serial latency and peak throughput are different operating points; peak QPS is the best measured concurrency point, not serial QPS. Final rows have zero request errors.
One completed same-rc.5 lifecycle per workload was reused through validated cache materialization; four additional fresh lifecycles completed the matrix. No rc.4 measurements were reused. An earlier 1M attempt cell was overwritten by the old retry loop; its combined client output remains retained and the attempt is excluded from final metrics.
Resources and retained evidence
Demand is container cgroup-v2 anon plus shmem, excluding reclaimable file cache. Cache-inclusive peaks are reported separately. Resource point estimates are arithmetic means of three lifecycle peaks, with retained min–max ranges. Final database volume is complete writable database-root allocated bytes at the verified stopped-final barrier, not archive or executable size. Packages retain hashes of the original result, provenance and checksum receipt; source files were verified against those receipts before packaging.
Run evidence parameters · variation · provenance
Benchmark parameters
{"concurrency_duration_seconds":30,"concurrency_schedule":[1,5,10,20,30],"execution_environment":{"container_runtime":"rootless Podman 6.1.1","processor":"Apple M4 Max","vm_cpus":10,"vm_memory_gib":24},"lifecycle_search_values":[0.45,0.425,0.425],"query_count":800,"query_partition_sha256":"7be5c5f24dfa31b66c5dfc3280c8f1bc5c1c8db3a35046a1a1ab9d50a0d232f7","search_parameter":"search_effort","storage_status":{"reasons":[{"detail":"service did not exit cleanly: {\"container\": \"antfly_bench_vdbb_antfly_t1_768d-1M\", \"exit_code\": 1, \"forced\": false, \"oom_killed\": false, \"stop_command_error\": null}","kind":"abnormal_stop","last_observation_retained":true,"limit_seconds":60,"stage":"final","trial":"1"},{"detail":"service did not exit cleanly: {\"container\": \"antfly_bench_vdbb_antfly_t2_768d-1M\", \"exit_code\": 1, \"forced\": false, \"oom_killed\": false, \"stop_command_error\": null}","kind":"abnormal_stop","last_observation_retained":true,"limit_seconds":60,"stage":"final","trial":"2"},{"detail":"service did not exit cleanly: {\"container\": \"antfly_bench_vdbb_antfly_t3_768d-1M\", \"exit_code\": 1, \"forced\": false, \"oom_killed\": false, \"stop_command_error\": null}","kind":"abnormal_stop","last_observation_retained":true,"limit_seconds":60,"stage":"final","trial":"3"}],"status":"provisional"}}Trial variation
[{"engine":"Antfly","ranges":{"demand_working_set":{"maximum":17095.3,"mean":16418.2,"minimum":15382.5,"unit":"MB"},"final_database_volume":{"maximum":3854.37696,"mean":3834.426709,"minimum":3824.070656,"unit":"MB"},"p99_at_30_concurrent":{"maximum":773.1,"mean":649.733,"minimum":537.8,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Chroma","ranges":{"demand_working_set":{"maximum":3699.1,"mean":3693.6,"minimum":3688.8,"unit":"MB"},"final_database_volume":{"maximum":3460.194304,"mean":3459.731456,"minimum":3458.838528,"unit":"MB"},"p99_at_30_concurrent":{"maximum":136.1,"mean":135.567,"minimum":134.6,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Elasticsearch","ranges":{"demand_working_set":{"maximum":2135.8,"mean":2095.2,"minimum":2029.8,"unit":"MB"},"final_database_volume":{"maximum":3148.750848,"mean":3148.500992,"minimum":3148.361728,"unit":"MB"},"p99_at_30_concurrent":{"maximum":21.6,"mean":20.967,"minimum":19.7,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Milvus 2.6.22","ranges":{"demand_working_set":{"maximum":13999.4,"mean":13920.6,"minimum":13837.1,"unit":"MB"},"final_database_volume":{"maximum":25669.378048,"mean":25642.164224,"minimum":25626.836992,"unit":"MB"},"p99_at_30_concurrent":{"maximum":13.7,"mean":13.633,"minimum":13.5,"unit":"ms"}},"workload":"768d-1M"},{"engine":"pgvector","ranges":{"demand_working_set":{"maximum":2318.6,"mean":2317.3,"minimum":2316.6,"unit":"MB"},"final_database_volume":{"maximum":9329.364992,"mean":9329.318571,"minimum":9329.27488,"unit":"MB"},"p99_at_30_concurrent":{"maximum":76.6,"mean":72.867,"minimum":69.9,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Weaviate","ranges":{"demand_working_set":{"maximum":7370.7,"mean":7345.1,"minimum":7331,"unit":"MB"},"final_database_volume":{"maximum":3955.232768,"mean":3947.47904,"minimum":3942.985728,"unit":"MB"},"p99_at_30_concurrent":{"maximum":68.3,"mean":67.2,"minimum":65.7,"unit":"ms"}},"workload":"768d-1M"},{"engine":"Milvus 3.0.0","ranges":{"demand_working_set":{"maximum":13551.8,"mean":13262.6,"minimum":13016.7,"unit":"MB"},"final_database_volume":{"maximum":35112.308736,"mean":33718.551893,"minimum":33003.88864,"unit":"MB"},"p99_at_30_concurrent":{"maximum":14.4,"mean":14.233,"minimum":14.1,"unit":"ms"}},"workload":"768d-1M"}]Peer versions
- Chroma1.5.9
- Elasticsearch9.5.2
- Milvus 2.6.222.6.22
- pgvector0.8.6 (PostgreSQL 16)
- Weaviate1.39.2
- Milvus 3.0.03.0.0
Evidence digests
- Receipt
70a68072957d88709559844b1fcd41eb6c916a488bbf5d12a137bb86fed8397f- Aggregate
6269f2229ccd1a895f3f8ff913c13269cc55df384c65601d490386141b0792d0- Provenance
6269f2229ccd1a895f3f8ff913c13269cc55df384c65601d490386141b0792d0- Harness commit
Peer cohort evidence · 768d-1M
- Completed
- Fresh lifecycles
- 18
- Peer variants
- 6
- Receipt
4366569aa9cfe5f7e796c5eadc30a6d2ae610cde650543eb3546ddcb856dfebf- Aggregate
828e2d97f299926c3bb1f39866db983ef61895793243ec02a569fe955507a197- Provenance
4af875e079c7c2538ed378027ac14be10294f30a7761e8fe0300f885ea3a1ea4
- Completed
- Antfly
v0.2.1-rc.5·- Runtime
- 2h 16m
- Platform
- linux · arm64
- Machine
- Apple M4 Max
- Build
- official release
Peer cohort · 768d-1M
- Completed
- Variants
- 6 independently executed peers
- Lifecycles
- 18 fresh
- Compatibility
- Historical reference · not a matched runtime
Peak throughput
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 768d-1M · three fresh lifecycles · Higher is better · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Serial query p95
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 768d-1M · three fresh lifecycles · Lower is better · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Demand working set
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 768d-1M · three fresh lifecycles · Lower is better · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Final database volume
Antfly v0.2.1-rc.5 (43fda0ba, official release) · linux-arm64 · 768d-1M · three fresh lifecycles · Antfly storage provisional; measured value retained in the table · Apple M4 Max · rootless Podman 6.1.1 · 10 VM CPUs / 24 GiB · Historical peers, not a matched run
Performance, quality, and resource evidence
Antfly v0.2.1-rc.5 · 43fda0ba
| Engine | Workload | Recall | nDCG | Peak QPS queries/s | p95 ms | Demand working set MB | Cache-inclusive peak MB | Final database volume MB | Trials | Clean stops / total |
|---|---|---|---|---|---|---|---|---|---|---|
| Antfly | 768d-1M | 0.9600 | 0.9640 | 75.4 | 39.667 | 16,418.2 MB | 19,736.8 MB | 3,834.4 MB (provisional) | 3 | 0 / 3 |
| Chroma | 768d-1M | 0.9560 | 0.9620 | 238.7 | 6.600 | 3,693.6 MB | 7,289.8 MB | 3,459.7 MB | 3 | 3 / 3 |
| Elasticsearch | 768d-1M | 0.9590 | 0.9610 | 3,110.8 | 3.233 | 2,095.2 MB | 11,426.8 MB | 3,148.5 MB | 3 | 3 / 3 |
| Milvus 2.6.22 | 768d-1M | 0.9610 | 0.9660 | 4,085.9 | 2.200 | 13,920.6 MB | 21,605.9 MB | 25,642.2 MB | 3 | 3 / 3 |
| pgvector | 768d-1M | 0.9660 | 0.9700 | 955.8 | 10.767 | 2,317.3 MB | 11,974.9 MB | 9,329.3 MB | 3 | 3 / 3 |
| Weaviate | 768d-1M | 0.9560 | 0.9620 | 1,300.8 | 5.700 | 7,345.1 MB | 13,791.8 MB | 3,947.5 MB | 3 | 3 / 3 |
| Milvus 3.0.0 | 768d-1M | 0.9610 | 0.9670 | 3,898.5 | 2.367 | 13,262.6 MB | 21,631.0 MB | 33,718.6 MB | 3 | 3 / 3 |