Define three separate tests
A fictional invoice says that its total is 125. A model response of {"total":152} may be valid JSON and may satisfy a schema requiring a number, while still being factually wrong. Report syntax validity, schema validity and extraction accuracy separately. This guide replaces an unsupported 99.8% case-study claim with a reproducible evaluation method; it does not assert a measured client outcome.
Constrain the output
Define required fields, allowed types and the handling of absent values. For example, require invoice_id as a string and total as a number or null, with no unexpected properties. Validate outside the model before using the response. A well-structured prompt helps communicate the task, but it does not replace runtime validation or prove that the extracted value matches the document.
source: invoice INV-7, total 125.00
valid extraction: {"invoice_id":"INV-7","total":125.00}
schema-valid but wrong: {"invoice_id":"INV-7","total":152.00}
abstention case: unreadable amount -> total:null with review routingBuild a labelled fixture
Use documents you are authorized to process and record the expected values independently of the model output. Include unreadable amounts, multiple currencies, missing identifiers and instructions embedded in document text. Treat those embedded instructions as data. Keep a held-out set so a prompt revision is not evaluated only on examples used to write it. Preserve the model configuration and evaluation rules with each result.
Handle failure without hiding it
A schema rejection can trigger a bounded repair attempt, but retain the initial failure in evaluation results. Do not keep retrying until one plausible response appears and then report only the successful attempt. Route uncertain or conflicting values to review. Compare cost, latency, abstention and factual errors together before allowing a downstream system to act on an extraction.
