Architecture
Designing Webhooks Other Developers Can Rely On
Updated June 16, 2020By the CalliArc team
Key takeaway
Webhooks are at-least-once by nature, so consumers will receive duplicates and out-of-order events. Give every event a stable id and a timestamp, sign the payload, retry with backoff, and provide a way to list and replay past events — that last feature is what stops integration support tickets.
If your product notifies other systems when something happens, you are running a distributed messaging system on behalf of people you've never met. The quality of that design determines how much of their integration debugging lands in your support queue.
Payload and delivery basics
- Every event carries a unique id, an event type, a timestamp, and a version. The id is what lets consumers deduplicate.
- Keep payloads small and include a link to fetch the full resource. Large payloads age badly and leak data into logs you don't control.
- Deliver at least once, and say so explicitly in your documentation. Promising exactly-once is a promise you cannot keep.
- Events can arrive out of order. Include a sequence or timestamp so consumers can discard stale updates.
Security
- Sign each payload with a shared secret and include the signature and timestamp in headers, so consumers can verify authenticity and reject replays.
- Support secret rotation with an overlap window, or every rotation is an outage for someone.
- Deliver only over HTTPS, and don't follow redirects.
- Never put credentials or full personal data in the payload; assume it will be logged somewhere insecure.
Retries and failure
- Retry with exponential backoff over hours, not minutes — endpoints go down for deployments.
- Treat any 2xx as success, retry on 5xx and timeouts, and don't retry on 4xx other than 429.
- Disable an endpoint after sustained failure and notify its owner, rather than retrying into the void forever.
- Enforce a short delivery timeout and require consumers to acknowledge quickly and process asynchronously.
The features that reduce support load
- A delivery log in your dashboard showing each attempt, response code, and body. Integrators can then debug themselves.
- Manual replay of an individual event, and bulk replay over a time range after an outage on their side.
- A test event button, so a new integration can be verified before real data flows.
- Clear documentation of every event type with an example payload — and a changelog when you add fields.