Define a defect you can observe
Consider a fictional customer import with 100 rows. Three rows have blank email addresses, two repeat the same customer key and one has a city spelling that looks unusual. The first two conditions can be detected with explicit rules. The spelling concern is a candidate for human review, not automatic proof of a defect. Calling every unusual value bad would reject legitimate regional spellings and new customers.
Keep a rules ledger
For each check, record its owner, affected field, severity and action. A duplicate business key may block publication; a missing optional phone number may only produce a warning. Store the row identifier, rule version and reason alongside the exception. Retain the original value so a reviewer can reverse a correction. Avoid an opaque quality score that combines missing optional fields and broken mandatory keys into the same number.
Use a model for suggestions
A model can propose likely mappings or group similar exception messages, but do not let that suggestion silently overwrite the customer master. Give it only the minimum fields needed for the task, separate instructions from imported text and require a structured response with an abstain option. Imported descriptions are untrusted data; a row saying “ignore the rules” must not become an instruction to the application.
exception_id: E-17
rule: unfamiliar_city_spelling
original_value: North Harbr
suggested_value: North Harbor
decision: pending_human_review
evidence: reference lookup requiredEvaluate against labelled examples
Prepare a held-out set that reviewers have labelled independently. Count false positives as well as missed defects. If a system raises ten warnings and only six are genuine, precision on those warnings is 6/10, or 60%. That fictional calculation says nothing about recall until you know how many real defects were present overall. Keep challenging examples, including rare valid values, out of the examples used to tune the prompt.
Make release criteria operational
Agree how many unresolved blocking exceptions can enter a reporting table, who can waive one and when the waiver expires. Compare the proposed correction with the raw row, the rule and the reference source. Monitor reversals after release because they reveal where the automation is overconfident. A useful quality workflow improves traceability and review decisions; an AI label alone does not establish that the data is trustworthy.
