Graph Database Benchmark v2 Methodology

Scope and provenance

This benchmark has two lanes and does not calculate a composite winner. LSQB-derived is an Antfly-owned anchored microbenchmark over LSQB data. The Pokec suite is a clean-room implementation informed by the public categories and workload ratios described by Memgraph’s Benchgraph material. It reuses the public Pokec import files, not Benchgraph implementation code. Neither lane is an audited LDBC result.

The scorecard keeps these measurements separate:

Full statistics, errors, run ranges, and raw samples remain mandatory even when the scorecard is used.

Engines and equal client work

The comparison is Antfly, Neo4j 5.26.29, and Memgraph 3.12.0. Kuzu is excluded because it is not exercised through a comparable network service. Every timed throughput cell uses the same Go executable, deterministic schedule, quantile implementation, response-consumption rule, worker count, and wall-clock. Antfly is accessed over persistent HTTP/1.1; both peers use the same official Neo4j Go Bolt driver. Transport is part of observed client latency and is reported, not hidden.

Only equivalent User(id) and benchmark-namespace ID indexes are created. There is no engine-specific query tuning. Each run starts from a fresh database and performs complete correctness checks before timing. Mutating cells use a reserved BenchUser namespace so source reads cannot change; that namespace is restored to its 1,024-node ring baseline after every mutating cell. Timed requests are never retried. Errors are retained and exclude a run from the publication gate.

LSQB-derived lane

The pinned source, data loading, parameter selection, BLAKE3 result oracle, and N1/P2/F2 definitions are specified in graph-lsqb-derived.md. The public LSQB-derived metric is warmed serial latency for this exact endpoint-filtered lookup, returning matching Post IDs:

(Forum)-[:CONTAINER_OF]->(Post)-[:HAS_TAG]->(Tag {id: $tag_id})

Unfiltered P2 C12 throughput remains a retained diagnostic. It is not part of the graph scorecard or public filtered-lookup claim. Default, medium, and full use LSQB SF0.003, SF0.1, and SF1 respectively.

Pokec data and parameter oracle

Profiles use the public Memgraph Pokec import files:

The harness records the downloaded source SHA-256 and refuses unexpected node or edge counts. Anchors are selected deterministically across out-degree deciles with seed 0xB37C4. Every selected anchor must return its complete result below 900 rows through four hops; this stays below rc18’s 1,000 pattern match ceiling rather than comparing truncated results.

Expansion semantics are one endpoint result per directed walk. Antfly uses a chain of fixed one-hop pattern steps. Cypher uses one fixed MATCH clause per hop, avoiding vendor differences in variable-length path uniqueness. The offline SQLite oracle stores the exact result count and canonical BLAKE3 digest including multiplicity for every parameter. Aggregates and shortest path have exact scalar oracles. All 101 quick-profile checks per engine/run must pass before timing. Writes are executed and verified in the isolated benchmark namespace.

Pokec operations

Every operation has an unwarmed serial observation and a prewarmed serial distribution. Isolated throughput uses 12 clients. Each non-mutating operation also receives a 70% operation / 30% write cell, split evenly between node and edge insert. The deterministic realistic workloads are:

• W1: 20% analytical, 40% point read, 10% update, 30% write;
• W2: 70% point read, 30% write;
• W3: 50% point read, 50% write;
• W4: 30% point read, 70% write.

The quick profile uses one-second isolated/mixed cells and two-second W1–W4 cells. Medium uses 5/10 seconds; full uses 15/30 seconds. Warmup is untimed. Every successful timed request is retained in JSONL. Reports contain completed QPS, error counts, p50/p95/p99, per-operation statistics, and peak PID RSS.

Persistent database volume

The honed LSQB SF0.1 F2 lane creates a new, run-labeled Docker local volume for every engine and repetition. Only the complete writable database root is measured (/data for Antfly and Neo4j, /var/lib/memgraph for Memgraph); read-only dataset/tool volumes and client scratch are excluded. A digest-pinned Alpine probe captures whole-volume allocated and apparent bytes once at empty, service-ready-before-ingestion, and post-workload, then at the verified post-stop final barrier. Final allocated bytes are the headline.

All three storage values, their arithmetic mean and range, apparent-byte diagnostics, barrier samples, volume manifests, and lifecycle receipts are retained. A provisional stop timeout remains visible but cannot support a comparative ratio.

Runs, ordering, and reporting

Each profile requires three clean process runs. Engine order rotates by run and uses a different rotation between lanes. Public serial point estimates are the medians of the three run-level p50, p95, and p99 values. Every result row also retains the three run-level percentile records, their min–max ranges, pooled sample percentiles as diagnostics, and all raw samples. Load/index readiness is diagnostic and is never included in query metrics.

summary.json is the machine-readable source of truth. The Markdown report contains the full cell table; scorecard.svg is only the compact two-panel view. There is deliberately no geometric mean or composite rank.

The honed LSQB SF0.1 F2 lane caches only the complete three-engine cohort. Promotion requires Antfly, Neo4j, and Memgraph to each retain three fresh database constructions and process runs, 15,000 timed samples, all three run-level percentile records and ranges, three complete storage receipts and their allocated-byte mean/range, 1,000/1,000 correctness, a valid canonical receipt, and the quiet-host record. The cache identity binds the Antfly subject, both peer image digests, the Linux client and storage-probe image, dataset manifest, harness sources, and protocol. A hit materializes the entire cohort; per-engine cache mixing is not permitted.

Internal fresh-service F2 readiness and convergence

The honed and Circus profiles retain a fresh-service diagnostic that measures how quickly a preloaded graph service becomes useful and settles into its warm filtered-two-hop behavior. It is graph-specific: readiness is followed immediately by the same exact LSQB SF0.1 F2 request used by the warm latency lane. It is not a general process-startup or system-wide Antfly claim.

This cohort is deliberately internal. It is attached to the canonical graph run report and structured result, but the public benchmark-page renderer does not render its table. It is never eligible for the honed publication cache: each honed or gauntlet graph invocation collects it afresh with make graph-f2-warmup-run. The retained diagnostic contract requires:

• the same Antfly binary, pinned Neo4j and Memgraph images, official LSQB SF0.1 dataset, and Linux/arm64 Docker environment as the primary graph lane;
• a recorded physical-host state, including whether the honed quiet-host assertion was active, without treating a diagnostic run as publication data;
• one database construction followed by five fresh service lifecycles per engine, with balanced engine ordering;
• restart-to-ready measured from immediately before service launch through the first successful engine-native probe: Antfly GET /readyz, Neo4j cypher-shell RETURN 1 plus the pinned-version check, and Memgraph autocommit SHOW VERSION over Bolt;
• the first F2 latency and first-100 p50/p95 from a fixed 100-anchor panel selected by SHA-256 rank independently of result cardinality;
• convergence at the first point where three consecutive adjacent 100-query window p50 changes are each within 5%, with a 10,000-query ceiling;
• steady-state p50/p95 from five deterministic sweeps over all 1,000 anchors after convergence;
• every timed result checked against the independent count+BLAKE3 oracle, with all 1,000 anchors covered in every engine/lifecycle; and
• median-of-five run metrics with complete run ranges, raw timestamped samples, lifecycle logs, host evidence, exact schedules, and a canonical receipt.

These seven values are an indivisible internal reporting family: restart-to-ready, first-query, first-100 p50, first-100 p95, queries-to-stable-p50, steady p50, and steady p95. The retained report must show all seven for all three engines; it must not headline convergence while omitting the steady-state latency guardrail or Memgraph.

Qualification and publication

The default command is a small, complete optimization loop with a target wall time below 15 minutes. Medium and full are explicit diagnostic commands. A mixed native-macOS/container run is qualification-only because its process RSS views are not comparable across engines.

The release-facing honed F2 lane is a separate, environment-qualified publication protocol. Antfly, Neo4j, and Memgraph all run sequentially inside the same Docker Desktop Linux/arm64 VM on an explicitly quiescent Apple Silicon host. The published claim must name that environment and is limited to warm serial filtered two-hop latency; it is not a bare-metal Linux result or a claim about general graph-database performance. Publication requires:

• the exact v0.2.0 Linux/arm64 binary and expected SHA-256;
• the pinned Neo4j 5.26.29, Memgraph 3.12.0, and Linux client image digests;
• official LSQB SF0.1 with 432,235 nodes and 1,827,523 relationships;
• three fresh database constructions and process runs per engine, each on a newly created labeled database volume;
• five warmed serial sweeps per run over 1,000 deterministic anchors;
• 1,000/1,000 count+BLAKE3 correctness in every engine/run and zero timed errors;
• median run-level p50/p95/p99 with all run ranges, pooled diagnostics, raw samples, lifecycle logs, and the physical-host record retained; and
• a canonical receipt for the indivisible three-engine cohort.

The broader graph-v2/Pokec publication gate remains native-Linux-only and is not satisfied by this bounded F2 artifact.