Parse Atlas · ParseBench · retained Antfly 0.2.1-rc1 evaluation

ParseBench

Original pages, extracted text, broken rules, and evaluator failures for 2,553 evaluation records. Evaluator execution status and metric outcome remain separate fields.

Metric accounting

Outcome distribution

zero + partial + perfect = metric count; metric count + unavailable = total examples. Evaluation execution counts reconcile separately.

Default metric outcome12.50%

Charts

Layout

Tables

Content

Formatting

ParseBench execution and default-metric accounting by dimension
Dimension / metric	Score	Total	Successful	Failed	Card skipped	Planner skipped	Zero	Partial	Perfect	Unavailable
Charts `Rule pass rate`	0.00%	568	568	0	0	0	568	0	0	0
Layout `Layout element pass rate`	0.00%	500	416	24	60	0	352	0	0	148
Tables `GTRM composite`	0.00%	503	503	0	0	0	503	0	0	0
Content `Content faithfulness`	61.67%	506	506	0	0	1	87	410	9	0
Formatting `Semantic formatting`	0.81%	476	476	0	0	31	445	11	1	19

successful means the evaluator completed. It does not mean the benchmark metric passed. Card skips participate in total examples; planner skips count unmatched shared inference records outside that denominator. Generated 29 August 2026 · 02:03 UTC.

Data notes

successful counts evaluator completion, never a passing benchmark result.
failure_examples cap at 6 and success_examples at 3 per card; markdown excerpts cap at 240 characters and explanations at 280 characters; omitted_failure_examples and omitted_success_examples record the dropped entries.
outcome thresholds: zero score == 0; partial 0 < score < 1; perfect score >= 1; unavailable means no default-metric score (evaluation failed, the evaluator skipped it, or the metric did not run for the card).
layout rule accounting uses synthetic checks layout.localization, layout.classification and layout.attribution derived per ground-truth element; attribution counts only elements where it applies.
ocr_attempted_count and ocr_failed_count come from each retained result manifest; this run disabled OCR fallback, so all values equal 0.
source URLs pin Hugging Face dataset llamaindex/ParseBench at revision 2805a1d940f95a203e0ae4b88be9934f7765b3fc.
report-level skipped counts exceed per-card skips for text_content:1, text_formatting:31 (planner-level skips that produced no evaluation record and no card).

Failure taxonomy

Failure classes

Classes overlap. Counts use either records or failed rules; definitions follow the table.

Category	Count	Unit	Charts	Layout	Tables	Content	Formatting
Missing required word	33896	rule	0	0	0	33896	0
Layout classification	11809	rule	0	11809	0	0	0
Layout localization	11225	rule	0	11225	0	0	0
Missing required sentence	10978	rule	0	0	0	10978	0
Layout attribution	10647	rule	0	10647	0	0	0
Reading order	6065	rule	0	273	0	5792	0
Chart point	4864	rule	4864	0	0	0	0
Bold	2063	rule	0	0	0	0	2063
Zero default metric	1955	record	568	352	503	87	445
Title	1870	rule	0	0	0	0	1870
Italic	553	rule	0	0	0	0	553
No predicted table	503	record	0	0	503	0	0
Missing sentences	491	rule	0	0	0	491	0
Unexpected sentences	454	rule	0	0	0	454	0
Missing words	445	rule	0	0	0	445	0
No layout items	420	record	22	87	34	145	132
Underline	405	rule	0	0	0	0	405
Unexpected words	403	rule	0	0	0	403	0
Title hierarchy	402	rule	0	0	0	0	402
Superscript	318	rule	0	0	0	0	318
Footer	307	rule	0	0	0	307	0
Header	278	rule	0	0	0	278	0
Blank output	260	record	16	67	6	90	81
Repeated words	247	rule	0	0	0	247	0
Repeated sentences	231	rule	0	0	0	231	0
Digit mismatch	216	rule	0	0	0	216	0
LaTeX	112	rule	0	0	0	0	112
Highlight	88	rule	0	0	0	0	88
No layout data	60	record	0	60	0	0	0
Strikethrough	44	rule	0	0	0	0	44
Evaluator error	24	record	0	24	0	0	0
Subscript	13	rule	0	0	0	0	13
Code block	10	rule	0	0	0	0	10

Missing required word rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Layout classification rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Layout localization rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Missing required sentence rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Layout attribution rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Reading order rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Chart point rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Bold rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Zero default metric record: The dimension's default leaderboard metric evaluated to exactly zero.
Title rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Italic rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
No predicted table record: The table evaluator found ground-truth tables and no predicted table structure.
Missing sentences rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Unexpected sentences rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Missing words rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
No layout items record: The normalized output contains no layout items for source attribution.
Underline rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Unexpected words rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Title hierarchy rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Superscript rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Footer rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Header rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Blank output record: The normalized output contains at least one page with no retained text.
Repeated words rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Repeated sentences rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Digit mismatch rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
LaTeX rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Highlight rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
No layout data record: Cross-evaluation skip: the provider produced no layout predictions for the requested page, so the evaluator did not score the case.
Strikethrough rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Evaluator error record: The evaluation worker raised before producing a score. Retained error text remains; the parse output stays unaffected.
Subscript rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.
Code block rule: Benchmark rule instances of this type that did not pass across all evaluated cards; counts come from per-rule results in the retained evaluation reports.

Rule accounting

Rule-type ledger

Rows sort by pass rate, highest first. Each row reconciles as passed + failed = total.

Dimension	Rule type	Total	Passed	Failed	Pass rate
text_content	missing_specific_word	105369	71473	33896	67.83%
layout	order	637	364	273	57.14%
text_content	order	13087	7295	5792	55.74%
text_content	bag_of_digit_percent	486	270	216	55.56%
text_content	too_many_sentence_occurence_percent	503	272	231	54.08%
text_content	too_many_word_occurence_percent	506	259	247	51.19%
text_content	missing_specific_sentence	18768	7790	10978	41.51%
text_content	unexpected_word_percent	506	103	403	20.36%
text_formatting	is_italic	655	102	553	15.57%
text_content	missing_word_percent	506	61	445	12.06%
text_content	unexpected_sentence_percent	503	49	454	9.74%
text_formatting	is_latex	123	11	112	8.94%
layout	layout.localization	12141	916	11225	7.54%
text_formatting	is_sub	14	1	13	7.14%
layout	layout.classification	12141	332	11809	2.73%
text_content	missing_sentence_percent	503	12	491	2.39%
text_formatting	is_bold	2066	3	2063	0.15%
text_formatting	is_title	1872	2	1870	0.11%
chart	chart_data_point	4864	0	4864	0.00%
layout	layout.attribution	10647	0	10647	0.00%
text_content	is_footer	307	0	307	0.00%
text_content	is_header	278	0	278	0.00%
text_formatting	is_code_block	10	0	10	0.00%
text_formatting	is_mark	88	0	88	0.00%
text_formatting	is_strikeout	44	0	44	0.00%
text_formatting	is_sup	318	0	318	0.00%
text_formatting	is_underline	405	0	405	0.00%
text_formatting	title_hierarchy_percent	402	0	402	0.00%

Deterministic sample

Sampled nonzero and perfect records

Deterministic top scores from dimensions with nonzero default metrics. Sampling ignores the current browser state.

text_contentscore 99.60%

text/text_misc__6slidesnonzero content_faithfulness score

text_contentscore 100.00%

text/text_misc__lightblueperfect content_faithfulness score

text_contentscore 100.00%

text/text_misc__templatedperfect content_faithfulness score

text_contentscore 99.74%

text/text_multicolumns__2col_fund_summarynonzero content_faithfulness score

text_contentscore 100.00%

text/text_multicolumns__gridofnumbersperfect content_faithfulness score

text_contentscore 99.73%

text/text_multicolumns__radioprogramnonzero content_faithfulness score

text_contentscore 100.00%

text/text_simple__financialReturnCoverperfect content_faithfulness score

text_contentscore 100.00%

text/text_simple__missionperfect content_faithfulness score

text_contentscore 100.00%

text/text_sparse__agendaperfect content_faithfulness score

text_formattingscore 9.09%

text/text_misc__edit2nonzero semantic_formatting score

text_formattingscore 16.67%

text/text_multicolumns__encarnonzero semantic_formatting score

text_formattingscore 52.78%

text/text_ocr__boldnonzero semantic_formatting score

text_formattingscore 50.00%

text/text_ocr__supnonzero semantic_formatting score

text_formattingscore 16.67%

text/text_simple__billnonzero semantic_formatting score

text_formattingscore 100.00%

text/text_simple__linenumperfect semantic_formatting score

text_formattingscore 16.67%

text/text_simple__redactednonzero semantic_formatting score

text_formattingscore 27.78%

text/text_simple__solvaynonzero semantic_formatting score

text_formattingscore 62.50%

text/text_sparse__agendanonzero semantic_formatting score

Retained evidence

Evaluation records

One dimension chunk loads initially. Selecting all dimensions loads the remaining chunks and keeps the full 2,553-card set available to the filters.

DimensionMetric outcomeEvaluation statusRule typeInput kindSample kindText search

Loading the selected dimension chunk…