Operational Experience

Retries and Timeouts

Every webhook provider has to answer the same questions: how long will it wait for the listener to respond, what counts as a failure, how many times will it retry, and what happens when the listener stays down? Very few providers answer them the same way, and the answers determine how consumers must build their listeners.

What providers do today

ProviderResponse timeoutAutomatic retriesAfter repeated failures
GitHub10 secondsNone — redeliver manually or via the REST API—
GitLabGitLab.com limitNo automatic schedule documented — resend manually or via the APITemporarily disabled after 4 consecutive failures, permanently after 40
StripeNot published — "quickly return a 2xx"Up to 3 days with exponential backoff in live mode; 3 attempts over a few hours in sandboxes—
Shopify5 seconds8 retries over 4 hoursSubscriptions may be removed after multiple failures
Zoom3 secondsUp to 3 retries (5, 20, and 60 minutes) — only on 5xx and some connection errorsSubscription disabled after 6 consecutive failed revalidations
Microsoft Graph3 seconds (10 seconds on retries)Up to 4 hoursSlow endpoints are delayed; very slow endpoints have notifications dropped
Okta3 secondsAt most 1 retry — only on 5xx—
Linear5 seconds3 retries (1 minute, 1 hour, 6 hours)May be disabled; must be re-enabled manually
Asana10 secondsExponential backoff for up to 24 hoursWebhook deleted after 24 hours without a successful delivery
Mailgun—Over 8 hours (5m, 10m, 15m, 1h, 2h, 4h)—
PayPal—Up to 25 times over 3 days—
WorkOS—Up to 6 times over 3 days—
OpenAI"A few seconds"Up to 72 hours with exponential backoff—
Meta (Facebook)—Immediately, then with decreasing frequency over 36 hoursDropped after 36 hours

What counts as a failure

Most providers treat anything other than a 2xx response as a failed delivery, along with timeouts, connection errors, and TLS errors. The edge cases differ:

  • Redirects are failures. Stripe and OpenAI do not follow 3xx responses — the URL you register must be the final destination. This is also an important SSRF control for providers.
  • Some 4xx responses are not retried. Zoom and Okta only retry 5xx responses. Mailgun treats 406 Not Acceptable as "don't retry."
  • 410 Gone can unsubscribe you. Asana deletes a webhook that responds with 410, and the Standard Webhooks specification recommends that providers disable endpoints that return it.

Respond fast, process later

Short timeouts — 3 seconds at Zoom, Okta, and Microsoft Graph — leave no room for real work inside the request. Stripe, Shopify, and Microsoft Graph all recommend the same pattern: validate the signature, persist or enqueue the event, return a 2xx immediately, and process it asynchronously.

app.post('/webhook', express.raw({ type: 'application/json' }), async (req, res) => {
  if (!isValidSignature(req)) {
    return res.status(401).send('Unauthorized')
  }
  // Persist first, then acknowledge — don't do the work inside the request
  await queue.publish(req.body)
  res.status(202).send()
})

A queue also absorbs bursts. Stripe notes that subscription renewals at the start of the month can produce large spikes in deliveries.

Retries create duplicates and reorder events

A listener that finishes its work but responds after the timeout will receive the same event again. Retries also mean an older event can arrive after a newer one. Both are normal behavior for any provider with at-least-once delivery, so consumers should plan for:

Retried deliveries may also carry new signatures and timestamps — Stripe generates a new timestamp for every attempt — so replay prevention windows apply to each attempt, not to when the event first happened.

Recovering from long outages

When the retry window closes, the event is gone unless the provider gives you another way to get it:

  • Manual redelivery: GitHub, GitLab, and Stripe let you resend deliveries from their dashboards or APIs. Stripe allows resending from the Dashboard for 15 days and from the CLI for 30 days.
  • Event APIs: Stripe exposes events and their related objects through its API, so consumers can fetch what they missed after an outage.
  • Reconciliation: Shopify and Asana recommend a periodic job that fetches current state through the API, since webhooks alone can't guarantee you never miss a change.

Recommendations

For providers:

  • Document your timeout, retry schedule, what responses are retried, and when endpoints are disabled — the table above was assembled from a dozen different docs pages
  • Retry with exponential backoff and jitter over a window of at least a day; the Standard Webhooks specification recommends a multi-day schedule and a request timeout between 15 and 30 seconds
  • Notify the consumer (email, dashboard) before disabling an endpoint
  • Offer delivery logs and manual redelivery — see Testing and Local Development

For consumers:

  • Acknowledge quickly and process asynchronously
  • Make processing idempotent
  • Monitor for disabled endpoints and gaps in delivery, and reconcile with the provider's API after outages

Related: Heartbeat, Queueing, and Circuit Breaking

Previous
← Resiliency