Webhook Patterns

A webhook is an HTTP call made by one service to another when an event occurs. Sender and receiver do not need to be available at the same time, so delivery guarantees, signatures, retries, duplicate handling, and observability are part of the contract.

Quick Decision

Need	Approach	Watch out
Notify an external system	Signed webhook	Keep receiver timeouts short
Delivery guarantee	Persistent delivery record + retry	Do not retry forever
Repeated event	Event ID + idempotent consumer	Do not rely on timestamps alone
Long-running result	Webhook + status endpoint	Do not fit the result into one request

Production Checklist

• Problem: Which events and delivery level does the receiver expect?
• Solution: Are event schema, signature, timeout, retry, replay, and disable policy written down?
• Trade-off: At-least-once delivery is safer but creates duplicates, ordering concerns, and storage cost.
• Failure mode: Classify 4xx, 5xx, timeout, DNS failure, and invalid signature separately.
• Measurement: Track delivery success, latency, retry count, oldest pending event, DLQ, and receiver-disable rate.
• Security/cost: Apply secret rotation, replay windows, SSRF protection, payload minimization, and PII masking.

Event Contract

Each delivery should include stable id, type, version, occurredAt, producer, subject, and data fields:

{
  "id": "evt_01J9...",
  "type": "order.completed",
  "version": 1,
  "occurredAt": "2026-09-03T10:15:00Z",
  "producer": "order-service",
  "subject": "order/ord_123",
  "data": { "orderId": "ord_123", "total": 125.50 }
}

Keep schema changes additive and backward compatible. Use a new event type or version for breaking changes. Receivers should apply an allowlist for event type and version.

Signature Verification and Replay Protection

The sender signs the canonical body with HMAC and sends signature, timestamp, and key ID headers. The receiver should compare over the raw body in constant time; do not parse and re-serialize JSON before verification.

signature = HMAC-SHA256(secret, timestamp + "." + raw_body)

Apply a timestamp acceptance window, event-ID deduplication, and key rotation together. Never put secrets in URLs, logs, or error messages. A webhook endpoint must not initiate uncontrolled outbound calls for SSRF; validate source and tenant after signature verification as well.

Retry, Backoff, and Delivery State

A 2xx response is enough to accept delivery; do not treat the response body as the processed result. Use exponential backoff with jitter for timeouts and 5xx; use limited or no retries for 4xx. After the attempt or total-time budget is exhausted, move the event to a DLQ/review state.

Store states such as pending, delivered, retrying, failed, and disabled with the last failure reason. Manual replay may reuse the same event ID; generating a new delivery ID is not safe if the consumer is not idempotent.

Ordering, Fan-out, and Operations

Prefer ordering by subject or aggregate key over global ordering. Deliveries to one receiver should use bounded concurrency; one slow receiver must not block other tenants.

Track endpoint health, failure streak, retry queue age, and circuit state. When a receiver is temporarily disabled, retain events within a documented retention window and alert before retention expires so safe replay is possible.

Design webhooks together with background jobs and API security.