Heartbeat, Queueing, and Circuit Breaking

Webhook listeners — like any other piece of software — will eventually become unstable or unavailable due to network or system issues. Some webhook providers — such as Bolt, Aftership, BitBucket, and Square — recognize availability as an inevitable problem and provide resiliency features to reduce the burden on webhook consumers. Examples of interesting features include:

Heartbeat checks: Webhook providers regularly reach out to webhook listeners to check their status. If a listener is unavailable, the provider takes action — such as queueing notifications or sending alerts to system admins — until the webhook listener gets fixed. This feature is helpful, especially in webhooks that don't send messages often. Asana, for example, sends a heartbeat with an empty events array every eight hours and deletes webhooks that go 24 hours without a successful response.
```
app.post('/webhook', (req, res) => {
  if (!req.rawBody) {
    // empty value = heartbeat check. Return 200 or 204 - No Content
    res.status(204).send()
  }else{
    ...
```
Queueing: webhook providers can keep track of historical information of webhook events sent and allow consumers to send messages again after failures or unavailability. Queues can be implemented on listener errors or in conjunction with heartbeat checks. See Retries and Timeouts for how long major providers keep retrying.
Circuit Breaking: Some webhook providers implement circuit breaking logic to throttle and alleviate requests to webhook listeners when webhook requests cross an error or response time threshold. GitLab temporarily disables a webhook after four consecutive failures — for one minute at first, growing to 24 hours — and permanently after 40. Microsoft Graph delays notifications for 10 minutes when more than 10% of responses take longer than 3 seconds.

These features help consumers keep their service resilient and recover from issues faster and without message loss. On the consumer side, resiliency means acknowledging quickly and processing from a queue, and handling the duplicates and out-of-order events that retries produce.