Filtered ANN Methodology

This benchmark is intentionally separate from the unfiltered VectorDBBench matrix. The honed publication workload measures equality-filtered, tenant-scoped vector search; the broader portable matrix and gauntlet retain numeric-filter diagnostics. All use exact filter-aware ground truth and the same database adapters and service topology as the regular vector benchmark.

Portable 50K matrix

The Circus-owned portable matrix loads each engine once with the vector, integer ID, and a deterministic categorical bucket, then runs six scenarios:

• numeric tails retaining 10%, 2%, 1%, and 0.2% of the corpus;
• a fixed categorical bucket retaining 1% / 500 scattered rows; and
• a rotating categorical bucket, changed per query, also retaining 1%.

Rows use bucket_{id % 100}. The fixed case selects bucket_037; the rotating case selects (query_index * 37 + 11) % 100, which visits every bucket ten times over the 1,000-query test set. Each scenario has exact cosine top-100 ground truth. The 0.2% numeric point deliberately leaves exactly 100 eligible rows and is the minimum valid selectivity for top-100.

Each database performs three complete search sweeps, with scenario order deterministically shuffled per database and repetition. Reports use the median with min-max whiskers and show Recall@100, peak QPS, and p99 at the concurrency producing peak QPS. Performance execution fails closed while unrelated benchmark processes or containers own the shared host.

Honed profile

• Dataset: Small OpenAI, 50,000 vectors, 1,536 dimensions, cosine distance
• Tenant model: 100 equal buckets defined by bucket_{id % 100}, with 500 vectors per tenant
• Filter: categorical equality on the tenant bucket selected for each query
• Quality schedule: (source_query_index * 37 + 11) % 100, which visits every tenant and remains stable across the deterministic calibration/publication partition
• Throughput schedule: every worker rotates across tenant buckets so the measurement is not a repeated hot-filter cache lookup
• Top-k: 100
• Engines: Antfly, Weaviate, Elasticsearch, Milvus, pgvector, and Chroma. This keeps the honed comparison focused on the source benchmark target, a mainstream hybrid-search engine, a vector-native engine, and the ubiquitous Postgres extension baseline, while retaining the popular embedded/developer-oriented vector store.
• Lifecycle: three fresh indexes per engine. Each index is tuned only on the deterministic 200-query calibration partition, then measured on the disjoint 800-query holdout. The same three lifecycles define performance, memory, and storage aggregates.
• Storage: each lifecycle uses one newly labeled Docker local volume at the engine’s complete persistence root. Capture whole-volume allocated/apparent bytes once at empty, ready, and post_workload, then at the verified post-stop final barrier before reclamation. Host paths are ineligible. Retain every value and report the arithmetic mean and range across all three lifecycles.
• Search mode: serial latency/recall plus QPS at concurrency 1, 8, and 32 for 10 seconds each.

Antfly calibration-only topology qualification

Before the three publication lifecycles, Antfly alone qualifies shard counts 1, 2, and 4 on fresh indexes using only the same deterministic 200-query calibration partition. Each topology sweeps the preregistered search-effort ladder 0, 0.025, 0.05, …, 0.35; the lowest effort reaching recall 0.955 is selected. Holdout queries, publication concurrency, and publication metrics are forbidden inputs to this topology decision.

When all three topology candidates qualify, the selector minimizes calibration (p95, p99, shard count) in that order. If any candidate is quality-negative or operationally incomplete after one retry, publication uses the preregistered one-shard control rather than selecting from a partial successful set. The chosen shard count is frozen across all three Antfly publication lifecycles and included in their cache identity.

Gauntlet profile

• Dataset: Medium OpenAI, 500,000 vectors, 1,536 dimensions, cosine distance
• Filter: id >= 495,000, leaving 1% / 5,000 candidates
• Engines: the same six-engine set. Qdrant is not part of either profile. Quickwit is a lexical/log-search engine in this suite and does not expose a comparable VectorDBBench filtered-ANN adapter.
• Metrics: same as the honed profile, with the exhaustive concurrency 1, 5, 10, 20, 30, 40, 60, and 80 sweep at 30 seconds per point
• Peak-memory sampling is enabled by the enclosing gauntlet benchmark profile

Correctness and fairness

The honed adapters implement categorical equality against the same labels field and load the same 100-bucket assignment. Its exact cosine top-100 ground truth uses the tenant selected for each source query. The query schedule is applied before the deterministic 200/800 partition so each retained vector is paired with the same tenant in calibration, publication, and independent reproduction.

The gauntlet’s numeric adapters implement inclusive >= semantics. VectorDBBench compares each result to neighbors computed within the surviving candidate set, so unfiltered or post-filtered top-k results lose recall rather than appearing artificially fast. Results report recall, nDCG, peak concurrent QPS, the concurrency and p99 latency at that peak, p95/p99 serial latency, insert time, and total load time. Index and query knobs are recorded in the command logs.

Filtered services use their own container names, data roots, result labels, logs, status rows, and host ports. Antfly defaults to port 28081 so it cannot attach to or stop the regular/full-text Antfly service on port 28080.

Canonical output retains every storage.json, barrier sample, declared volume, trial value, arithmetic mean, and range under raw/storage. Final allocated bytes are the headline; apparent bytes and engine-reported index sizes are diagnostics. Filtered peer storage appears in the comparative review only when the existing review-inclusion gate passes, while canonical/raw evidence is always retained.

The portable matrix still does not constitute a complete claim about filtered search. Planned follow-ups add vector-correlated categories, skewed hot/cold filter churn, conjunctions/exclusions, concurrent-ingest interference, matched-recall tuning, and 500K/1M datasets.

Search Benchmark Game Methodology

What this lane measures

This is an end-to-end, self-hosted database comparison over the first 250,000 documents in the Search Benchmark Game Wikipedia corpus. Every timed adapter uses a persistent database connection and requests ranked document IDs without stored document bodies. Tantivy is deliberately absent: it is an embedded search library, while every measured row here is a database service.

The run uses the 902 upstream term, union, intersection, and phrase queries. The 60 mixed required/optional and negated Lucene-syntax queries remain excluded because Antfly cannot represent their matching and scoring semantics directly. If the remaining workload proves too Lucene-oriented for database comparison, that will motivate a separate benchmark rather than further changing this one.

Database matrix and capability policy

• Antfly 0.2.0-rc.18: term, union, intersection, phrase
• Elasticsearch 8.15.3: term, union, intersection, phrase
• Weaviate 1.38.2: term, union, intersection; exact phrase unsupported
• Quickwit 0.8.2: term, union, intersection, phrase
• PostgreSQL 16 from the pgvector image: term, union, intersection, phrase; pgvector is not involved in lexical execution
• Milvus 2.5.8: retained as unsupported because its BM25 raw-text input is a VARCHAR capped at 65,535 bytes, while 20 corpus documents exceed that limit and the maximum is 202,798 bytes
• Chroma self-hosted: retained as unsupported because its text surface provides containment/regex filtering rather than ranked lexical top-k retrieval; the richer Chroma Search API is cloud-only

Missing capabilities are never emulated client-side. Public output uses —* for a product limitation and failed† for a setup or execution failure, with the reason preserved in JSON and rendered beside the table.

Execution protocol

The benchmark runs TOP_10 and TOP_100 at concurrency 1 and 8. Each cell has a five-second warmup followed by two measured rounds. Concurrency uses one persistent connection per worker; individual request latency and aggregate QPS are both recorded. Only one database is timed at once. Indexing is sequential, uses the unchanged corpus, and uses native bulk paths (COPY for PostgreSQL, bulk APIs elsewhere).

Antfly uses a prebuilt release binary; the benchmark never builds Zig. It runs one shard and sends fields: []. Elasticsearch uses one primary shard, zero replicas, a 1 GiB heap, and a one-segment force merge. Weaviate runs one node with vectorization disabled. Quickwit runs one node with a positional, fieldnorm-enabled text index and BM25 score ordering; only external IDs are stored. PostgreSQL uses a stored tsvector, GIN index, and ts_rank_cd ordering.

Where configurable, adapters approximate Antfly’s fixed Bleve analyzer with standard/word tokenization, lowercase normalization, and English stopwords. Quickwit’s built-in default tokenizer lowercases but does not expose a matching stop-word filter. Residual analyzer and scoring differences are reported through an untimed top-10 ID/count diagnostic; exact cross-engine ranking is not a pass condition.

Interpretation

The headline table shows TOP_10 concurrency-8 p95 latency by query class so partial capability is visible. Full artifacts retain p50/p95/p99, QPS, sample counts, status, and reasons for TOP_10/TOP_100 at both concurrency levels. This benchmark has no relevance judgments, so it supports performance and capability claims only—not claims that one engine ranks documents more correctly.

OmniDocBench Methodology

benchmarks/OmniDocBench is the raw benchmark checkout and stays pointed at the Antfly-owned OmniDocBench origin. This harness directory contains antfly-circus-specific adapter code, methodology notes, generated prediction directories, and OmniDocBench evaluator outputs.

Measurement surface

OmniDocBench measures document parsing quality, not plain OCR alone. Each engine must produce one markdown prediction per page; OmniDocBench then scores text recognition, table reconstruction, formula rendering, and reading order.

Antfly is invoked through the central Bastille handoff: antfly-circus passes the exact benchmark binary as ANTFLY_BIN, and the adapter only consumes that binary. It does not discover or substitute another Antfly executable.

Default smoke profile

• Dataset: OmniDocBench demo data
• Prediction format: one markdown file per page
• Evaluation: OmniDocBench end2end config shape
• Smoke behavior: verify adapter wiring and output paths without claiming full OmniDocBench leaderboard coverage

Full benchmark profile

The default full profile runs --engine antfly --protocol page-image over the bundled OmniDocBench demo subset: 18 page images. It writes one Antfly-generated markdown prediction per page, evaluates with OmniDocBench, and reports quality metrics first: Overall, text edit distance, table TEDS, formula CDM, and reading-order edit distance. Prediction throughput is reported separately as pages/s.

Set ANTFLY_DOC_MODEL_DIR to the explicit reader model directory and ANTFLY_BIN to the exact Antfly binary under test. The adapter intentionally does not substitute bundled demo predictions for missing Antfly output.

Protocols and peers

• page-image: runnable with the bundled demo data. Engines receive raster page images and must emit markdown files whose names match the OmniDocBench image stems.
• native-pdf: dataset/API-gated. It is blocked unless PDF inputs and a compatible native-PDF engine path are configured. The adapter does not silently rasterize PDFs and call that native PDF parsing.

Open-source peer rows are wrapper-based. Configure command templates with OMNIDOCBENCH_MINERU_CMD, OMNIDOCBENCH_MARKER_CMD, OMNIDOCBENCH_DOCLING_CMD, or OMNIDOCBENCH_OCR_BASELINE_CMD. Templates may use {input}, {output}, {output_dir}, and {stem} placeholders. The OCR-only baseline can also use TESSERACT_BIN or tesseract on PATH.

Record the exact image/PDF input set, command template or Antfly command, model directory/backend selected, protocol, worker counts, whether CDM is enabled, and all generated OmniDocBench result JSON files under benchmarks/OmniDocBench-harness/result. The full 1651-page OmniDocBench dataset is not bundled in this checkout; pull it from the upstream Hugging Face/OpenDataLab dataset before reporting full benchmark coverage.

OHR-Bench Antfly adapter

This adapter runs a curated internal qualification, a deterministic 10% sample, and the complete OHR-Bench corpus against Antfly. It does not modify the upstream clone.

The curated qualification contains 11 documents across all seven domains. It includes every PDF regression found during the first full-corpus attempt, an OCR-required scan, the largest corpus PDF, an encrypted PDF, a filename that exercises URL encoding, and QA-backed representatives. Antfly owns PDF retrieval, extraction, OCR fallback, chunking, embedding, and indexing. The adapter then evaluates lexical and hybrid retrieval with OHR-Bench’s official correct-document/correct-page gate and word-level evidence LCS.

The PDF dataset has conflicting surface metadata: its Hugging Face card says CC-BY-4.0, while the upstream README restricts it to research and non-commercial use. This harness follows the stricter statement. Do not publish or use these artifacts as product/marketing results without a separate review.

Commands

Run these from the antfly-circus repository root. Plain make shows them.

make ohr-bench-setup
make ohr-bench-smoke
make ohr-bench-download
make ohr-bench-qualify
make ohr-bench-report
make ohr-bench-medium
make ohr-bench-medium-report
make ohr-bench-full
make ohr-bench-full-report

ohr-bench-download downloads the public pdfs.zip (about 1.52 GB), verifies its SHA-256 digest, and extracts it below ignored .data/ohr-bench/. Setup also clones the OHR-Bench annotations at the pinned upstream commit into ignored .repos/OHR-Bench/; the upstream repository is not vendored or modified.

ohr-bench-qualify uses ~/.local/bin/antfly-pdf-ocr by default, starts it on port 28580 with an isolated data directory, and writes artifacts to results/ohr-bench/qualification/. Each source row contains a URL such as http://127.0.0.1:18085/law/example.pdf; Antfly’s artifact backfill fetches that URL and performs all parsing itself. The harness starts a loopback-only static PDF byte origin with no directory listing or non-PDF access. It is not a parser and does no OCR, embedding, or transformation. DocsAF and inference sidecars are not used. Override the binary with OHR_ANTFLY_BIN=/absolute/path/to/antfly.

ohr-bench-medium deterministically samples 10% of each domain (129 PDFs) while forcing all curated fixtures into the sample. ohr-bench-full ingests all 1,261 PDFs into seven domain-scoped tables and evaluates all 8,498 questions in lexical and hybrid lanes. Run the qualification before either larger cold ingest.

Antfly’s embedded inference registry loads antflydb/Florence-2-base for OCR and BAAI/bge-small-en-v1.5 for 384-dimensional embeddings from ~/.antfly/inference/models. Override that directory with OHR_MODELS_DIR=/absolute/path. The harness deliberately has no configuration for external inference URLs.

Every load uses the public linear-merge API with sync_level: full_index. Before evaluation, the harness fetches document_units_v1 for every source and requires its unit count to equal an independent pypdf page count and its chunk count to be nonzero. It also reconciles every source URL with the byte origin’s access log, rejects inline/non-HTTP sources and artifact errors, and requires linear-merge cleanup to converge. The OCR-required fixture must show a successful OCR selection. Per-document evidence is retained in coverage.jsonl, including ingest wall time and OCR attempted/selected/retained/ failed counts; the aggregate structural gate is coverage-summary.json.

Artifact-level extraction failures do not stop evaluation. The adapter scores all questions, records zeros for unavailable evidence, produces summary.json and summary.md, and fails only at the final verification step. Infrastructure failures that prevent the run from continuing still stop immediately and write failure.json. A report whose structural gate failed labels its retrieval metrics diagnostic rather than benchmark-comparable.

The curated qualification has an additional content-quality gate. Returned hits may not equal a known normalized OCR prompt echo, and every selected QA-backed document must have at least one hybrid top-5 evidence-page result with positive word-LCS. Medium and full runs reject prompt echoes but do not turn benchmark score magnitude into a pass/fail threshold. Reports also retain unique trivial-text hits, per-document positive-LCS counts, artifact failures, and the slowest ingest documents.

A stopped run preserves failure.json, checkpoints, coverage collected so far, the run fingerprint, and antfly.log. Resume only with the same corpus manifest, binary digest/source SHA, models, and extraction configuration:

make ohr-bench-medium OHR_RESUME=1

The harness never builds Antfly.

Metrics

• Primary: top-1 correct evidence page and mean word-level evidence LCS.
• Diagnostic: top-5 correct evidence page and query latency.
• Setup: cold PDF extraction/OCR/chunk/embed/index wall time.

OHR evidence pages are zero-based. Antfly PDF provenance is one-based, and the adapter tests and applies that conversion explicitly. Queries are isolated by OHR domain, matching the upstream retriever’s per-domain search behavior.

MTCB Antfly Benchmark

MTCB evaluates retrieval quality for chunked RAG corpora. The upstream harness chunks documents, embeds chunks and questions, retrieves top-k chunks, and scores whether the gold passage appears in the retrieved chunk set.

Antfly Surface

This adapter keeps MTCB’s evaluator and metric calculations, but replaces the in-memory vector store with Antfly external-vector retrieval:

• one Antfly table per MTCB dataset evaluator;
• one external aknn_v0 cosine vector index named vec;
• chunk text is stored as table metadata and the vector is inserted in _embeddings.vec;
• queries use /api/v1/query with embeddings.vec and the requested limit.

The adapter uses local deterministic lexical hashing embeddings by default. This keeps the benchmark self-contained and avoids depending on hosted embedding APIs. The result is therefore a benchmark of Antfly storage/index/query behavior inside MTCB’s scoring loop, not a claim about a production embedding model.

Profiles

• smoke: one nano dataset, currently nano-macha, intended to validate adapter wiring and artifact shape.
• nano / honed: three nano datasets, currently nano-macha, nano-sencha, and nano-hojicha, intended to produce a small but real retrieval result.

The adapter reports Recall@k, Precision@k, MRR@k, and NDCG@k for each dataset, plus mean metrics across datasets. It also recomputes mean Recall@k from the per-dataset rows and marks the run failed if the recomputed value diverges from MTCB’s BenchmarkResult.

Runtime Pin

The benchmark should be run against Antfly 0.2.0-rc.17 for the current Bastille result. The run artifact records the Antfly URL and best-effort runtime version endpoint response. The local harness can use ANTFLY_BIN to point at a specific prebuilt runtime; no Zig build should be triggered by this benchmark.

Caveats

MTCB’s public examples usually use neural embedding models such as Voyage. The current Antfly adapter deliberately uses local hashing embeddings to make the benchmark repeatable without credentials or model downloads. Before making a leaderboard-style quality claim, rerun the adapter with a documented production embedding model and compare against MTCB’s original in-memory vector store using the same chunks and embeddings.

Agent Retrieval Cost harness

Measures what it actually costs an agent to answer questions over a large fixed corpus with different retrieval tooling. Quality (answer accuracy) is reported alongside cost; cost is the story: total LLM tokens, USD, wall time, and tool calls per query.

BrowseComp-Plus native-hybrid honed lane

bcp_native_hybrid/ is the publication lane for the four existing products that expose a genuinely native hybrid request: Antfly, Elasticsearch, Weaviate, and Milvus. PgVector and Chroma are excluded because their current harness paths are dense-only; adding those paths would change the retrieval contract. The separate lexical/TREC adapter is diagnostic and is not consumed by the publication page.

The checked bcp_native_hybrid/panel.json freezes 50 seed-17 development queries and their disjoint 780-query complement. The development panel is used only to warm memory lifecycles. Luna, through pinned pi 0.73.1, answers every held-out query against every backend: 3,120 evaluated query cells. Each cell uses the official Qwen3-Embedding-8B vectors and query prefix, the upstream QUERY_TEMPLATE_NO_GET_DOCUMENT, one search(query) tool, top five results, 512 Qwen3-0.6B snippet tokens, and the checked turn/token ceilings. Harbor supervises the Luna process without adding shell, filesystem, browser, or document-fetch tools.

Query publication evidence is resumed one backend/query cell at a time through the generic Circus publication cache. An exact hit includes the normalized trajectory, raw pi trajectory and stderr, Luna final record, Harbor result, complete Qwen3-32B judgement, original timestamps, and receipt. Corpus, vector, prompt, client, model, judge, backend, and Antfly-subject identities are part of the key. Incorrect answers and complete no-search outcomes are measured evidence; no-search always forces protocol correctness false. Crashes, partial trajectories, missing or unparseable judgements, and setup failures never hit. Database volumes, indexes, downloads, and memory measurements are never cached. First qualifying cache evidence wins under the generic cache policy. Fresh promotion also requires the operator’s explicit CIRCUS_QUIET_HOST_READY=1 assertion, retained as quiet-host.json.

The checked four-shard Latin-square schedule gives each backend every execution position once while still running all 780 held-out queries per backend. Only one backend stack is resident at a time. Every backend also receives exactly three deliberately fresh cgroup-v2 whole-stack memory lifecycles. Sampling starts before launch and covers prelaunch, service-launch, ingest, readiness, warm-query, and post-warm-idle. Publication memory is the warm-query anon + shmem peak mean and range; lifecycle demand, memory.current, kernel peaks, disk, events, images, logs, and raw timelines remain retained diagnostics. Swap, failed samples, incomplete phases, reused storage, or fewer than three trials rejects the report.

The report publishes per-backend correct count, accuracy with Wilson 95% interval, evidence recall, search/input/total-token distributions, completion, zero-search, and cached/fresh counts. Antfly-minus-peer paired accuracy and evidence-recall differences use the named paired-score mean ± 1.959963984540054 standard-error interval. These intervals are descriptive; no comparative acceptance threshold or automatic editorial selection is applied. Agent and judge wall time is network-bound diagnostic telemetry, not a latency claim.

From the repository root:

make browsecomp-native-hybrid-prepare
make browsecomp-native-hybrid-smoke
make browsecomp-native-hybrid-honed
make browsecomp-native-hybrid-report

The honed command writes strict summary.json, summary.md, detailed.json, raw runs/evals, cache references, memory timelines, and receipt.json to $BASTILLE_OUT/data/rag-browsecomp-plus/.

Design: trust through off-the-shelf parts

• Agent: pi, an independent off-the-shelf coding agent (tools: read/write/edit/bash), run headless.
• Orchestration: Harbor (the Terminal-Bench 2.0 harness, pinned version in the root Makefile) using Harbor’s first-party pi agent adapter (harbor run -a pi).
• Antfly authors only the environments and the report. There is no bespoke agent loop; per-query tokens and USD cost come from pi’s own session accounting (message_end usage events in /logs/agent/pi.txt), parsed the same way Harbor itself does, not from our instrumentation.

One Harbor task per (tooling x question); one pi session per question; session totals are the per-query metrics.

Dataset

BrowseComp-Plus (Tevatron/browsecomp-plus): 100,195-document corpus, 830 queries with gold answers, cached at .data/browsecomp-plus/ by the BrowseComp-Plus harness. Questions are deliberately hard deep-research puzzles; absolute accuracy is expected to be low for every tooling. v1 applies no quality bar — accuracy is reported next to cost.

Fairness rules

• Same questions for every tooling: one seed-17 pool of 200 query ids, sampled once; every run size is a prefix (probe 10 ⊂ default 50 ⊂ publish 200).
• Same corpus surface: identical document text, capped at ARC_DOC_CHARS (default 20,000 chars/doc), in every store.
• Same agent, same model (default openai-codex/gpt-5.6-luna, run on a ChatGPT subscription), same instruction contract; only the tooling cheat-sheet section differs.
• Setup is performed by the harness, deterministically, before agent runs; setup wall time is reported in its own table and never billed to per-query cost. (Agent-performed setup is the separate agent-effort suite.)
• The judge (default openai/gpt-4.1, same prompt as the BrowseComp-Plus answer eval) grades answers offline; judge spend is reported separately and never added to a row’s cost. Failed/unanswered queries count as incorrect.
• Egress is audited, not firewalled: pi needs openrouter.ai, so containers have network access. The instruction forbids internet use, pi has no web tool, and the harvester scans every bash tool call for non-local hosts and flags suspect_egress per record (surfaced in the run-health table). A proxy allowlist is future work.

Toolings

Running

Everything is driven from the repo root Makefile:

make agent-retrieval-cost-setup    # venv + harbor patch + base images + ingest (timed)
make agent-retrieval-cost-smoke    # free, offline: templates + parser fixture + report
make agent-retrieval-cost-probe    # paid: 10 questions x grep,sqlite,antfly
make agent-retrieval-cost-honed    # paid: ARC_SAMPLE (50) x ARC_TOOLINGS
make agent-retrieval-cost-judge    # paid: offline judge pass over harvested records
make agent-retrieval-cost-report   # free: agent-summary.json + summary.md

Knobs: ARC_MODEL, ARC_JUDGE_MODEL, ARC_SAMPLE, ARC_TOOLINGS, ARC_CONCURRENCY, ARC_ROW_CONCURRENCY, ARC_DOC_CHARS, ARC_AGENT_TIMEOUT (uniform per-task agent timeout, default 1800 s — the same for every tooling so wall/cost comparisons stay fair), ARC_RUN_ID. The OpenRouter key (used by the offline judge) is read from the process environment, the legacy agent_retrieval_cost.env, or pi’s private ~/.pi/agent/auth.json. The key is never copied into the repository.

Full 830-question checkpointed run

The full lane uses the seven validated surfaces (grep, sqlite, antfly, elasticsearch, pgvector, chroma, and antfly_hybrid) and the pinned v0.2.0-rc.20 ReleaseFast subject manifest. It is cohort-driven: 33 cohorts contain 25 questions and the final cohort contains five. Every cohort is paired across all seven rows, and one command invocation runs at most one cohort.

make agent-retrieval-cost-full-prepare  # dataset, stores, immutable manifest
make agent-retrieval-cost-full-canary   # one question x seven rows
make agent-retrieval-cost-full-next     # one 25-question paired cohort
make agent-retrieval-cost-full-drain    # repeat cohorts; stop on first safety pause
make agent-retrieval-cost-full-status   # no model calls

full-drain is only a checkpoint supervisor: it invokes the same one-cohort lane repeatedly, reruns every preflight before advancing, and exits immediately if an attempt is paused or fails. It does not weaken the per-cohort cost, auth, disk, service, or retry protections.

full-run-manifest.json freezes question order, cohort boundaries, models, toolings, and subject hashes. full-run-ledger.json records immutable attempts. Harvest merges by (tooling, query_id), refuses to overwrite a final cell, and only replaces explicit authentication, rate-limit, or infrastructure failures. Completed cells therefore survive an outage and are skipped on resume.

The preflight requires all stores to report 100,195 indexed documents, a ready and fully query-visible Antfly vector index, at least 30 GB free, and at least two hours of OAuth validity. It warns at $550 and refuses to launch a new cohort at $650 of API-equivalent Luna usage. The incremental OpenRouter judge has a separate $5 actual-spend hard cap. Subscription agent spend remains $0 incremental; equivalent cost is an analysis yardstick, not a cash charge.

Partial reports contain only fully paired questions. Reports include the accuracy-vs-budget curve with bootstrap intervals, a cost-vs-quality scatter/Pareto chart, cost per correct answer, payload volume, command errors, and evidence-to-first-hit diagnostics. Raw de-obfuscated questions and document content remain local and must not be pasted into public summaries.

Rows run through a bounded scheduler. The default is two Harbor rows in parallel (ARC_ROW_CONCURRENCY=2), with four concurrent trials in each row (ARC_CONCURRENCY=4): at most eight subscription agents at once. The 50×7 RF335 run’s observed row durations project to roughly 95 minutes of inference at this setting instead of 172 minutes sequentially. Parallel rows share host resources, so wall-time comparisons should be treated as contention-sensitive; token, cost, and accuracy metrics remain per isolated trial. Set ARC_ROW_CONCURRENCY=1 to reproduce the old sequential schedule.

Model auth

The agent model defaults to openai-codex/gpt-5.6-luna — GPT-5.6 Luna run on a ChatGPT Plus/Pro subscription via pi’s Codex OAuth provider. One-time setup: run pi interactively, /login, and pick “ChatGPT Plus/Pro (Codex)” to complete the browser OAuth. That writes ~/.pi/agent/auth.json, which the harness bind-mounts read-only into every trial container (--mounts). The run preflight refuses to start if that file is empty.

Because a subscription is flat-rate, pi reports token counts but $0 cost, so $/query is computed from token counts at published Luna rates ($1/$6/$0.10 per 1M in/out/cached-read) and each record is tagged cost_source: price_table_estimate. See arc/pricing.py. Pass a different ARC_MODEL (e.g. openai/gpt-5.6-luna with OPENAI_API_KEY, or openrouter/...) to use real per-token billing instead; pi’s own cost_usd is then used and tagged pi_usage.

Note: Codex OAuth access tokens expire in hours. For long matrix runs, log in (or refresh) shortly before starting; the read-only mount can’t refresh mid-run.

Outputs

Under $BASTILLE_OUT/data/agent-retrieval-cost/:

• <tooling>/records.jsonl — one record per (tooling, query): status, answer, gold, tokens {input, output, cache_read, cache_write, total}, cost_usd (pi’s own accounting), wall_seconds (first-to-last session event; excludes image build and pi install), tool_calls, suspect_egress, judge verdict.
• agent-summary.json, setup-summary.json, judge-summary.json, summary.md (per-query means, setup table, cumulative cost, run health).

Caveats

• Wall/query depends on host load and provider latency at run time; treat it as indicative, not a latency benchmark.
• Timed-out and provider-errored trials (e.g. OpenRouter routes occasionally return content_filter mid-run) count as incorrect and stay in the denominator; the run-health table reports them per row.
• pi is installed per trial by Harbor’s adapter (npm install -g ...@latest); the base image pre-installs node+pi so this is fast, but pi’s version floats with the run date and is recorded in /etc/arc-pi-version in the image.
• Harbor’s old upload_dir docker-cp bug (verifier tests landing at /tests/tests/) is fixed upstream in the pinned version; patches/apply_harbor_upload_dir_patch.py verifies this and can patch the legacy layout if the pin ever moves backwards.
• Vector rows share one embedding model and code path (ingest == query-time); engines’ native embedding pipelines are intentionally not used.

Agent Retrieval Cost harness

Measures what it actually costs an agent to answer questions over a large fixed corpus with different retrieval tooling. Quality (answer accuracy) is reported alongside cost; cost is the story: total LLM tokens, USD, wall time, and tool calls per query.

BrowseComp-Plus native-hybrid honed lane

bcp_native_hybrid/ is the publication lane for the four existing products that expose a genuinely native hybrid request: Antfly, Elasticsearch, Weaviate, and Milvus. PgVector and Chroma are excluded because their current harness paths are dense-only; adding those paths would change the retrieval contract. The separate lexical/TREC adapter is diagnostic and is not consumed by the publication page.

The checked bcp_native_hybrid/panel.json freezes 50 seed-17 development queries and their disjoint 780-query complement. The development panel is used only to warm memory lifecycles. Luna, through pinned pi 0.73.1, answers every held-out query against every backend: 3,120 evaluated query cells. Each cell uses the official Qwen3-Embedding-8B vectors and query prefix, the upstream QUERY_TEMPLATE_NO_GET_DOCUMENT, one search(query) tool, top five results, 512 Qwen3-0.6B snippet tokens, and the checked turn/token ceilings. Harbor supervises the Luna process without adding shell, filesystem, browser, or document-fetch tools.

Query publication evidence is resumed one backend/query cell at a time through the generic Circus publication cache. An exact hit includes the normalized trajectory, raw pi trajectory and stderr, Luna final record, Harbor result, complete Qwen3-32B judgement, original timestamps, and receipt. Corpus, vector, prompt, client, model, judge, backend, and Antfly-subject identities are part of the key. Incorrect answers and complete no-search outcomes are measured evidence; no-search always forces protocol correctness false. Crashes, partial trajectories, missing or unparseable judgements, and setup failures never hit. Database volumes, indexes, downloads, and memory measurements are never cached. First qualifying cache evidence wins under the generic cache policy. Fresh promotion also requires the operator’s explicit CIRCUS_QUIET_HOST_READY=1 assertion, retained as quiet-host.json.

The checked four-shard Latin-square schedule gives each backend every execution position once while still running all 780 held-out queries per backend. Only one backend stack is resident at a time. Every backend also receives exactly three deliberately fresh cgroup-v2 whole-stack memory lifecycles. Sampling starts before launch and covers prelaunch, service-launch, ingest, readiness, warm-query, and post-warm-idle. Publication memory is the warm-query anon + shmem peak mean and range; lifecycle demand, memory.current, kernel peaks, disk, events, images, logs, and raw timelines remain retained diagnostics. Swap, failed samples, incomplete phases, reused storage, or fewer than three trials rejects the report.

The report publishes per-backend correct count, accuracy with Wilson 95% interval, evidence recall, search/input/total-token distributions, completion, zero-search, and cached/fresh counts. Antfly-minus-peer paired accuracy and evidence-recall differences use the named paired-score mean ± 1.959963984540054 standard-error interval. These intervals are descriptive; no comparative acceptance threshold or automatic editorial selection is applied. Agent and judge wall time is network-bound diagnostic telemetry, not a latency claim.

From the repository root:

make browsecomp-native-hybrid-prepare
make browsecomp-native-hybrid-smoke
make browsecomp-native-hybrid-honed
make browsecomp-native-hybrid-report

The honed command writes strict summary.json, summary.md, detailed.json, raw runs/evals, cache references, memory timelines, and receipt.json to $BASTILLE_OUT/data/rag-browsecomp-plus/.

Design: trust through off-the-shelf parts

• Agent: pi, an independent off-the-shelf coding agent (tools: read/write/edit/bash), run headless.
• Orchestration: Harbor (the Terminal-Bench 2.0 harness, pinned version in the root Makefile) using Harbor’s first-party pi agent adapter (harbor run -a pi).
• Antfly authors only the environments and the report. There is no bespoke agent loop; per-query tokens and USD cost come from pi’s own session accounting (message_end usage events in /logs/agent/pi.txt), parsed the same way Harbor itself does, not from our instrumentation.

One Harbor task per (tooling x question); one pi session per question; session totals are the per-query metrics.

Dataset

BrowseComp-Plus (Tevatron/browsecomp-plus): 100,195-document corpus, 830 queries with gold answers, cached at .data/browsecomp-plus/ by the BrowseComp-Plus harness. Questions are deliberately hard deep-research puzzles; absolute accuracy is expected to be low for every tooling. v1 applies no quality bar — accuracy is reported next to cost.

Fairness rules

• Same questions for every tooling: one seed-17 pool of 200 query ids, sampled once; every run size is a prefix (probe 10 ⊂ default 50 ⊂ publish 200).
• Same corpus surface: identical document text, capped at ARC_DOC_CHARS (default 20,000 chars/doc), in every store.
• Same agent, same model (default openai-codex/gpt-5.6-luna, run on a ChatGPT subscription), same instruction contract; only the tooling cheat-sheet section differs.
• Setup is performed by the harness, deterministically, before agent runs; setup wall time is reported in its own table and never billed to per-query cost. (Agent-performed setup is the separate agent-effort suite.)
• The judge (default openai/gpt-4.1, same prompt as the BrowseComp-Plus answer eval) grades answers offline; judge spend is reported separately and never added to a row’s cost. Failed/unanswered queries count as incorrect.
• Egress is audited, not firewalled: pi needs openrouter.ai, so containers have network access. The instruction forbids internet use, pi has no web tool, and the harvester scans every bash tool call for non-local hosts and flags suspect_egress per record (surfaced in the run-health table). A proxy allowlist is future work.

Toolings

Running

Everything is driven from the repo root Makefile:

make agent-retrieval-cost-setup    # venv + harbor patch + base images + ingest (timed)
make agent-retrieval-cost-smoke    # free, offline: templates + parser fixture + report
make agent-retrieval-cost-probe    # paid: 10 questions x grep,sqlite,antfly
make agent-retrieval-cost-honed    # paid: ARC_SAMPLE (50) x ARC_TOOLINGS
make agent-retrieval-cost-judge    # paid: offline judge pass over harvested records
make agent-retrieval-cost-report   # free: agent-summary.json + summary.md

Knobs: ARC_MODEL, ARC_JUDGE_MODEL, ARC_SAMPLE, ARC_TOOLINGS, ARC_CONCURRENCY, ARC_ROW_CONCURRENCY, ARC_DOC_CHARS, ARC_AGENT_TIMEOUT (uniform per-task agent timeout, default 1800 s — the same for every tooling so wall/cost comparisons stay fair), ARC_RUN_ID. The OpenRouter key (used by the offline judge) is read from the process environment, the legacy agent_retrieval_cost.env, or pi’s private ~/.pi/agent/auth.json. The key is never copied into the repository.

Full 830-question checkpointed run

The full lane uses the seven validated surfaces (grep, sqlite, antfly, elasticsearch, pgvector, chroma, and antfly_hybrid) and the pinned v0.2.0-rc.20 ReleaseFast subject manifest. It is cohort-driven: 33 cohorts contain 25 questions and the final cohort contains five. Every cohort is paired across all seven rows, and one command invocation runs at most one cohort.

make agent-retrieval-cost-full-prepare  # dataset, stores, immutable manifest
make agent-retrieval-cost-full-canary   # one question x seven rows
make agent-retrieval-cost-full-next     # one 25-question paired cohort
make agent-retrieval-cost-full-drain    # repeat cohorts; stop on first safety pause
make agent-retrieval-cost-full-status   # no model calls

full-drain is only a checkpoint supervisor: it invokes the same one-cohort lane repeatedly, reruns every preflight before advancing, and exits immediately if an attempt is paused or fails. It does not weaken the per-cohort cost, auth, disk, service, or retry protections.

full-run-manifest.json freezes question order, cohort boundaries, models, toolings, and subject hashes. full-run-ledger.json records immutable attempts. Harvest merges by (tooling, query_id), refuses to overwrite a final cell, and only replaces explicit authentication, rate-limit, or infrastructure failures. Completed cells therefore survive an outage and are skipped on resume.

The preflight requires all stores to report 100,195 indexed documents, a ready and fully query-visible Antfly vector index, at least 30 GB free, and at least two hours of OAuth validity. It warns at $550 and refuses to launch a new cohort at $650 of API-equivalent Luna usage. The incremental OpenRouter judge has a separate $5 actual-spend hard cap. Subscription agent spend remains $0 incremental; equivalent cost is an analysis yardstick, not a cash charge.

Partial reports contain only fully paired questions. Reports include the accuracy-vs-budget curve with bootstrap intervals, a cost-vs-quality scatter/Pareto chart, cost per correct answer, payload volume, command errors, and evidence-to-first-hit diagnostics. Raw de-obfuscated questions and document content remain local and must not be pasted into public summaries.

Rows run through a bounded scheduler. The default is two Harbor rows in parallel (ARC_ROW_CONCURRENCY=2), with four concurrent trials in each row (ARC_CONCURRENCY=4): at most eight subscription agents at once. The 50×7 RF335 run’s observed row durations project to roughly 95 minutes of inference at this setting instead of 172 minutes sequentially. Parallel rows share host resources, so wall-time comparisons should be treated as contention-sensitive; token, cost, and accuracy metrics remain per isolated trial. Set ARC_ROW_CONCURRENCY=1 to reproduce the old sequential schedule.

Model auth

The agent model defaults to openai-codex/gpt-5.6-luna — GPT-5.6 Luna run on a ChatGPT Plus/Pro subscription via pi’s Codex OAuth provider. One-time setup: run pi interactively, /login, and pick “ChatGPT Plus/Pro (Codex)” to complete the browser OAuth. That writes ~/.pi/agent/auth.json, which the harness bind-mounts read-only into every trial container (--mounts). The run preflight refuses to start if that file is empty.

Because a subscription is flat-rate, pi reports token counts but $0 cost, so $/query is computed from token counts at published Luna rates ($1/$6/$0.10 per 1M in/out/cached-read) and each record is tagged cost_source: price_table_estimate. See arc/pricing.py. Pass a different ARC_MODEL (e.g. openai/gpt-5.6-luna with OPENAI_API_KEY, or openrouter/...) to use real per-token billing instead; pi’s own cost_usd is then used and tagged pi_usage.

Note: Codex OAuth access tokens expire in hours. For long matrix runs, log in (or refresh) shortly before starting; the read-only mount can’t refresh mid-run.

Outputs

Under $BASTILLE_OUT/data/agent-retrieval-cost/:

• <tooling>/records.jsonl — one record per (tooling, query): status, answer, gold, tokens {input, output, cache_read, cache_write, total}, cost_usd (pi’s own accounting), wall_seconds (first-to-last session event; excludes image build and pi install), tool_calls, suspect_egress, judge verdict.
• agent-summary.json, setup-summary.json, judge-summary.json, summary.md (per-query means, setup table, cumulative cost, run health).

Caveats

• Wall/query depends on host load and provider latency at run time; treat it as indicative, not a latency benchmark.
• Timed-out and provider-errored trials (e.g. OpenRouter routes occasionally return content_filter mid-run) count as incorrect and stay in the denominator; the run-health table reports them per row.
• pi is installed per trial by Harbor’s adapter (npm install -g ...@latest); the base image pre-installs node+pi so this is fast, but pi’s version floats with the run date and is recorded in /etc/arc-pi-version in the image.
• Harbor’s old upload_dir docker-cp bug (verifier tests landing at /tests/tests/) is fixed upstream in the pinned version; patches/apply_harbor_upload_dir_patch.py verifies this and can patch the legacy layout if the pin ever moves backwards.
• Vector rows share one embedding model and code path (ingest == query-time); engines’ native embedding pipelines are intentionally not used.

OfficeQA-Pro raw-PDF benchmark

This harness measures the complete path from 697 Treasury Bulletin PDFs to an agent’s final answer. The headline comparison is ingestion complexity and cold setup time; answer accuracy and agent cost are measured on the same seeded OfficeQA-Pro questions after each index is ready.

The default matrix is 133 questions per row with seed 17:

The three Docling rows share one extraction pass. The oracle files never enter a raw-PDF row. Questions given to agents contain neither source_files hints nor gold answers.

Prerequisites

• Docker, uv, Go, and enough disk for the 4 GB corpus plus derived indexes.
• Hugging Face access accepted for the gated databricks/officeqa dataset and HF_TOKEN exported for the explicit download step.
• pi logged into ChatGPT (pi, then /login) for the default openai-codex/gpt-5.6-luna agent.
• A ReleaseFast Antfly binary. Override the suite default with ANTFLY_BIN=/absolute/path/to/antfly.

The Antfly qualification deliberately includes one scanned and one born-digital PDF. Run it before committing to the corpus-wide ingest. Current Antfly source needs to render the requested PDF page to PNG before sending it to the local Florence reader; passing the PDF URL directly is not supported by that reader. docsaf sync --help also needs to show the --ocr* options.

Only rebuild Antfly after checking those help texts and qualifying the binary. On this machine a full Zig build may run out of memory, so warn and get explicit approval first, then constrain parallelism:

cd /path/to/antfly/zig
zig build lib-pdf-test -j1
zig build lib-db-test -j1 -- --test-filter 'document OCR media parts'
zig build install -Dedition=full -Doptimize=ReleaseFast -j1

Run sequence

make targets live at the antfly-circus repository root.

make office-qa-pro-smoke
make office-qa-pro-setup
make office-qa-pro-download

# Before the large ingest:
ANTFLY_BIN=/path/to/antfly make office-qa-pro-qualify-antfly

# Paid 10-question, three-row end-to-end probe:
ANTFLY_BIN=/path/to/antfly make office-qa-pro-probe

# Paid Pro-133, five-row matrix:
ANTFLY_BIN=/path/to/antfly make office-qa-pro-honed

office-qa-pro-honed refuses to download data implicitly. It starts only the services selected by OQA_TOOLINGS, performs a cold setup, generates identical seeded tasks, runs Harbor/pi at OQA_CONCURRENCY=4, harvests, deterministically scores, and writes the report. Useful overrides include:

make office-qa-pro-honed \
  OQA_SAMPLE=20 \
  OQA_TOOLINGS=docling_pgvector,docling_sqlite \
  OQA_CONCURRENCY=2

Measurement and scoring

Downloads and model-cache warmup are excluded from setup timing. Setup begins with local source files and cached models; each recorded stage and total wall time is written to setup-summary.json. The raw-PDF rows include extraction, OCR, chunking, embedding where applicable, and indexing.

The scorer is reward.py pinned to OfficeQA commit 86753108d69e149cc28abd346bb8c3ca1cbfc7cf. It is applied to the final answer and every running best guess at exact, 0.1%, 1%, and 5% relative tolerance. The primary metric is accuracy at 1%; no LLM judge is used.

Agent accounting is inherited from the ARC harness: input/cache/output tokens, price-table-equivalent USD, wall time, tool calls, answer trajectory, and cost to the first answer that remains correct. Results land under results/benchmark-page/manual/data/office-qa-pro/ by default.

Planning estimate

The full default run is 665 trials (133 questions × 5 rows). Existing local ARC Luna trials average about $0.42 price-table-equivalent and 138 seconds of total trial wall time. That gives a lower-bound query phase of roughly $280 and 6.4 hours at concurrency four. OfficeQA is likely to require longer investigation, so budget approximately $300–$600 equivalent and 7–12 hours for agent trials. The default model uses a ChatGPT subscription, so this is benchmark accounting, not necessarily an incremental API invoice, but it will consume subscription capacity.

The larger uncertainty is cold ingestion of about 89,000 PDF pages with CPU/local ML OCR. Until the two-PDF qualification and a small corpus timing are measured, allow roughly 2–6 elapsed days end to end, of which most is Antfly and Docling extraction/OCR. The 10-question probe should be used to replace this range with measured throughput before launching Pro-133.