Define the unit that can be replayed
Imagine a fictional nightly import of invoices for one business date. Give the run a batch identifier and retain the source file checksum. A checkpoint saying “download complete” is different from one saying “validated and published.” If those states are collapsed into a single success flag, a retry can skip work that never reached the reporting table. Model the states before adding a scheduler.
Use a stable key at the destination
Choose an invoice business key that survives a repeated download. Loading the same batch twice should not double the invoice count. A generated row identifier alone will not deduplicate the same invoice arriving again. Decide how amended invoices are recognized, how deletes or reversals are represented and which version wins. Keep that logic separate from the transport-level decision to retry a failed network call.
Classify failures
A brief connection interruption may be transient. An expired credential, missing required column or violated accounting rule generally needs investigation rather than repeated attempts with identical inputs. Use bounded backoff for retryable operations and surface the terminal failure clearly. Avoid nested retries at every layer, which can multiply the number of requests sent to a service that is already struggling.
received -> staged -> validated -> published
on temporary download failure: retry within a bounded budget
on schema or reconciliation failure: quarantine and stop publication
on retry after commit: detect the existing batch before writing againPublish only after reconciliation
For a source batch with 20 invoices and a hypothetical total of 4,200, compare count, unique keys and amount before switching consumers to the new batch. Do not advance a source watermark until the durable publication step succeeds. If a row is rejected, determine whether partial publication is acceptable for this business process. Financial totals often require a deliberate all-or-nothing decision rather than an implicit partial success.
Exercise the failure boundaries
Inject failures after staging, after validation and immediately after the destination commit but before the caller receives confirmation. Replay each run and check for duplicate invoices, incorrect watermarks and repeated notifications. Retrying a database write safely does not automatically make sending an email safe. Record a recovery procedure for the final failure state, including who can authorize a replay and how the previous attempt is identified.
