Quickstart time to first retrieval
This lane measures how much agent work each tagged Antfly quickstart
demands before its first successful retrieval, and retains a
low-overhead milestone timeline for product acquisition, loading,
retrieval, and vector readiness. Protocol 2 covers v0.1.0
(728c39fd…), v0.1.3 (1b070b5d…), v0.2.0 (56a1b1a5…), and
v0.2.1-rc0 (efea2f96…), with three fresh Pi/Codex-subscription
trials per release and complete retained trajectories. Each fresh
container receives one uniform tagged package: the target guide, its
Downloads page, and a benchmark-verified version-resolution reference.
No container receives an Antfly binary or cached archive; product,
model, and dataset acquisition all count inside measured agent time.
The lane stands apart from Agent access effort: that benchmark
excludes software installation and live quickstart UX, while this one
includes both on purpose.
Layout
versions.json reviewed SemVer-ordered subject/acquisition registry
quickstart_ttfr.py CLI: prepare | smoke | run | collect | handoff
ttfr/
core.py matrix validation, fixtures, task generation, Harbor command
milestones.py passive Pi-event parsing, classification, egress audit
collector.py normalization, phase cohorts, statistics, source receipts
handoff.py presentation-neutral aggregate and evidence package
runner.py preflight, sequential trials, infrastructure-only reruns
selfcheck.py deterministic fake-clock drivers (smoke + tests share them)
templates/ committed Harbor task templates (task.toml, Dockerfile, instruction)
task-code/ code copied into each task image:
marker.py the sole qualifying retrieval command
quickstart-first-retrieval shim that execs marker.py
watch_vector_index.py detached post-retrieval index watcher
verify.py terminal verifier
test.sh Harbor verifier entry
tests/ deterministic fake-clock, history, and protocol suites
Generated artifacts never commit: prepared data lands under
.data/quickstart-first-retrieval/, run evidence under
CAMPAIGN_DIR/raw/TAG/trial-NN/attempt-NN/, and acquisition
preflights, source summaries, source receipts, and handoff packages under the
matching ignored campaign result directory.
Commands
python3 benchmarks/quickstart-first-retrieval-harness/quickstart_ttfr.py prepare # verify tagged packages and release archives
python3 benchmarks/quickstart-first-retrieval-harness/quickstart_ttfr.py smoke # offline deterministic checks
python3 benchmarks/quickstart-first-retrieval-harness/quickstart_ttfr.py run \
--campaign 2026-08-25-antfly-quickstart-history --trials 3 \
--releases v0.1.3,v0.1.0
python3 benchmarks/quickstart-first-retrieval-harness/quickstart_ttfr.py collect
python3 benchmarks/quickstart-first-retrieval-harness/quickstart_ttfr.py handoff \
--campaign 2026-08-25-antfly-quickstart-history \
--out HANDOFF_PACKAGE
The make targets pass CAMPAIGN, TRIALS, comma-separated RELEASES,
ARC_PI_AUTH, and MODEL; the handoff target additionally requires
HANDOFF_OUT.
Measurement contract
- Task. Ubuntu 24.04, native linux/amd64, 4 CPU, 8 GiB RAM, 30 GiB
storage, 5,400 s agent timeout, 1,860 s verifier timeout. Only
ca-certificates curl jq python3 procps plus archive utilities install.
The image carries the three-file tagged package and benchmark code,
never an Antfly binary, dataset, OpenAPI file, SDK, or extra docs. - First retrieval. The agent invokes
MARKER_COMMAND instead of the
guide’s shown query command. The wrapper issues the frozen QUERY_ROUTE
(body:"Korea" full-text, title/url fields) first, captures wall-clock
nanoseconds the instant a qualifying hit arrives, never polls, then — before
persisting — requires exact antfly --version equality, a live engine, the
prepared dataset SHA-256 at DATASET_FILE, and ≥9,000 documents in
full_text_index_v0. Tagged response shapes include both direct hits
and responses[].hits.hits[]._source. The title_body semantic index
must exist with the tagged provider/model before marker persistence;
semantic completion remains a post-retrieval diagnostic and never
changes the full-text timestamp. A failed gate discards the capture;
the first valid marker stays immutable. - Vector readiness. Only after a valid marker persists does one
detached watcher sample the semantic index endpoint every 5 s. The
verifier waits through the original 1,800 s budget plus poll/probe
grace, never takes ownership while a live watcher PID remains, and
resumes only after a failed spawn or dead owner. Completion accepts
indexed counts at or above the dataset size plus a semantic hit.
Timeouts retain the last successful incomplete sample as a
right-censored lower bound; error-only evidence remains unavailable.
- Milestones. A fixed map of 16 milestones; each entry carries
seconds_from_agent_start, lower_bound_seconds,
upper_bound_seconds, source, scope, and resolution_seconds,
with null wherever direct observation stays absent. Classification is
passive, from direct Pi Bash calls against frozen commands/URLs:
opaque scripts classify as nothing, a piped installer earns
download_and_install scope (never a split into invented sub-events),
and antfly load retries select the final successful call before the
marker. - Egress. Acquisition endpoints, model hosts, the dataset CDN,
localhost, package registries, and Pi provider traffic stay allowed;
extra Antfly documentation or source marks
protocol_violation;
anything unclassifiable stays retained as unclassified_egress and
contributes no measured aggregate value until manual review. - Aggregation. Handoff metric summaries retain
n, arithmetic mean,
sample variance, relative variance (100 × variance / mean²), minimum,
and maximum without presentation thresholds. The consuming site decides
when to display means and relative variance and how to warn about incomplete
valid/total results.
Phase statistics use the identical phase-valid cohort described below; rows
with an absent or out-of-order boundary retain their headline TTFR but
contribute no phase split. - Cost. Codex OAuth charges no per-token fee, so
actual_incremental_spend_usd records 0.0; api_equivalent_cost_usd
prices the token snapshot at the first retrieval through ARC’s
published-rate table with cost_source: price_table. Neither figure
claims observed spend. - Evidence. Every attempt directory keeps the Harbor trial, raw
logs/agent/pi.txt, marker, watcher samples, verifier outputs,
generated task, normalized attempt.json, and provenance; sibling
trial.json records the selected and superseded attempts with digests.
Attempts never get overwritten, and
--rerun-infrastructure-invalid stays the sole rerun mode — it demands
a prior terminal infrastructure_invalid classification. Product,
agent, verifier, and protocol failures remain final retained evidence.
Phase transition detection
The phase clock starts at Harbor’s recorded agent start. Passive parsing then
correlates each direct Pi Bash tool call with its tool-result timestamp. Opaque
scripts contribute no inferred boundary.
Phase aggregation requires these five finite, ordered points:
antfly_acquisition_started: start of the first successful direct
Antfly acquisition call.antfly_acquisition_finished: finish of the last successful direct
Antfly acquisition call.data_load_started: start of the first direct antfly load call targeting
the wikipedia table.data_load_finished: finish of the final matching load call, before the
marker, whose Pi tool result carries no error.first_retrieval: response timestamp captured by MARKER_COMMAND after
one frozen full-text request returns at least one projected title and
url hit. Post-checks gate marker persistence but do not move the captured
response timestamp.
The five displayed intervals follow directly:
Pre-acquisition = acquisition start - agent start
Antfly acquisition = acquisition finish - acquisition start
Setup before load = first load start - acquisition finish
Load window = final successful load finish - first load start
Post-load retrieval = marker response - final successful load finish
A successful load boundary means only that the matching Bash tool call
completed without a Pi tool error. It does not prove full-text index readiness
or successful querying. The load window may include failed load attempts and
intervening remediation when several matching calls occur.
Post-load retrieval therefore measures elapsed agent work after the load
command, not server query latency. It may include background full-text
indexing, agent reasoning, polling, failed marker attempts, sleeps,
diagnostics, engine restarts, and compatibility work before the accepted
marker response. Vector readiness starts after marker persistence and never
enters TTFR or these five phases.
Reproducing checks offline
The tests/ suites run entirely on fake clocks, scripted HTTP routers,
fake process runners, and synthetic Pi trajectories. Coverage includes
dynamic SemVer and acquisition references, release subsets and runtime
sizing, marker identity/index gates, single watcher ownership,
right-censored bounds, phase cohorts and adjacent deltas, receipt binding,
presentation-neutral aggregate safety, deterministic handoff packaging, and
rerun semantics.
make quickstart-first-retrieval-test
smoke re-runs the same fake-clock drivers plus matrix/template/task
generation and negative schema fixtures, and probes Docker/Harbor/Pi
advisorily. Neither command builds Antfly, launches a model session, or
touches the network.