Integrations2 min readBy Ry
Your integrations should fail loudly
The most expensive integration failure is the one nobody notices. A plain-English guide to alerting, idempotency and replayable syncs for business owners.
Every integration fails eventually. APIs time out, tokens expire, someone renames a field, a platform has an outage. The question is not whether yours will fail, but whether anyone will know when it does.
A sync that crashes with an error message is annoying. A sync that quietly stops, or quietly writes half the data, is expensive, because you discover it weeks later from a customer, an auditor or a stockout.
Four properties worth asking for
You do not need to know how to build these to ask whoever builds your integrations whether they are there.
1. Loud failure
When something goes wrong, a named person is told, in a place they will see (Slack, email, a ticket) with enough context to act: what failed, for which record, and what was already done. Successful runs can be silent.
2. Idempotency
An idempotent operation can safely be run twice. If the order sync is retried after a timeout, you should not get two orders. This is usually achieved by giving every operation a unique key and checking it before acting. Ask: “What happens if this runs twice?”
3. Replay
Incoming events should be stored before they are processed. If processing fails, or a bug is found later, the stored events can be replayed through the fixed code. Without that, recovering means asking people to re-enter data.
4. Validation before publishing
Before writing to a system of record or sending a report, check the data looks right: expected fields present, totals in a plausible range, row counts not suddenly zero. Refusing to publish bad data is far cheaper than publishing it and cleaning up.
What good monitoring looks like
- A heartbeat: if a scheduled job has not run when expected, someone is alerted
- Error rates tracked over time, not just individual failures
- A dead-letter queue: a list of failed items waiting for a human, visible to the team
- A short runbook explaining the common failures and how to fix them
None of this is exotic. It is standard practice in software teams, and it should be standard in the integrations a business depends on too. It is built into every integration project from the start, because adding it later is always harder.
Written by Ry, Evenbuilt · Los Angeles
All posts