Architecting a Resilient Webhook Delivery Engine: At-Least-Once Guarantees, Jittered Backoff, and HMAC Signatures

Webhooks are the standard reactive protocol of the modern programmable web. When an invoice settles in Stripe, a branch merges in GitHub, or an order ships in Shopify, an HTTP POST payload fires across the public internet to notify external customer infrastructure. Yet building a webhook delivery system that remains dependable under high load is one of the most deceptively complex challenges in distributed systems engineering.

A naive implementation—issuing an HTTP call inside the primary request thread or firing unbuffered background tasks—quickly collapses in production. Customer endpoints crash, network sockets exhaust, slow client timeouts monopolize worker threads, and retry avalanches trigger cascading thundering herds that inadvertently DDoS downstream customer systems.

In this architectural masterclass, we will design and construct a resilient, high-throughput webhook delivery engine from first principles. We will examine the transactional outbox architecture, implement cryptographically secure HMAC-SHA256 signing with replay protection, model mathematical exponential backoff with Full Jitter, enforce per-host concurrency throttling, and build a production-grade dispatcher using .NET 9 and modern async channels.

1. The Four Fallacies of Naive Webhook Dispatch

Most webhook architectures begin with a straightforward design: an application event occurs (for example, OrderCreated), and the backend service loops through registered subscriber URLs, executing HttpClient.PostAsync(url, payload). In production, this approach encounters four critical failure modes:

1. The Slow Client Contention Trap: If a customer's receiving endpoint takes 15 seconds to process a payload, your thread pool or asynchronous socket limit becomes saturated. A burst of 1,000 events targeting slow endpoints will starve the primary application of HTTP connections and CPU execution contexts.
2. Dual-Write Inconsistency: If your database transaction commits but the HTTP call fails or crashes before dispatch, the event is permanently lost. Conversely, if you dispatch the webhook before committing the database transaction, customers receive notifications for state changes that might roll back.
3. The Thundering Herd Retry Avalanche: When a popular customer service experiences an outage (e.g., returning HTTP 503 or timing out), retrying events on a fixed schedule (such as every 30 seconds) synchronizes all queued failures into massive simultaneous traffic spikes the instant the customer service attempts to recover, knocking it offline again.
4. Security and Man-In-The-Middle Spoofing: Without cryptographic signatures, malicious actors can forge payloads directly to the customer's webhook endpoint, triggering unauthorized downstream actions such as provisioning inventory or fulfilling unpaid orders.

2. Transactional Outbox: Guaranteeing At-Least-Once Delivery

To eliminate dual-write inconsistency, event creation must be atomically bound to your business data modifications. We employ the Transactional Outbox Pattern. When a domain event occurs, the database transaction writes both the entity state and a new row into an OutboxEvents table within the same ACID transaction boundary:

-- Atomic Outbox Table Schema
CREATE TABLE WebhookOutbox (
    EventId UNIQUEIDENTIFIER PRIMARY KEY DEFAULT NEWID(),
    TenantId VARCHAR(64) NOT NULL,
    EventType VARCHAR(128) NOT NULL,
    Payload NVARCHAR(MAX) NOT NULL,
    CreatedAt DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME(),
    ProcessedAt DATETIME2 NULL,
    Status VARCHAR(24) NOT NULL DEFAULT 'PENDING'
);

CREATE INDEX IX_WebhookOutbox_Pending 
ON WebhookOutbox (Status, CreatedAt) 
WHERE ProcessedAt IS NULL;

A decoupled background worker polls the pending events or tails the database transaction log via Change Data Capture (CDC). In high-volume SQL Server environments, we utilize keyset pagination combined with READPAST and UPDLOCK hints to allow multiple parallel workers to claim batches without locking table pages or experiencing row contention:

-- Non-blocking batch claim for distributed workers
WITH NextBatch AS (
    SELECT TOP (50) EventId, Status, ProcessedAt
    FROM WebhookOutbox WITH (UPDLOCK, READPAST)
    WHERE Status = 'PENDING'
    ORDER BY CreatedAt ASC
)
UPDATE NextBatch
SET Status = 'PROCESSING', ProcessedAt = SYSUTCDATETIME()
OUTPUT inserted.EventId;

3. Cryptographic Payload Verification: HMAC-SHA256 & Replay Defense

Customers must be able to verify beyond doubt that an incoming webhook originated from your platform and was not altered in transit. We implement the industry-standard HMAC-SHA256 signing scheme, patterned after Stripe's security architecture.

The Webhook Signature Header Protocol

Each outgoing HTTP request includes an X-Signature header containing two values:

• t: A Unix epoch timestamp (seconds) representing the exact instant the payload was signed.
• v1: The hexadecimal HMAC-SHA256 signature calculated over the combined string $"{t}.{rawJsonPayload}" using the customer's shared endpoint secret.

POST /api/webhooks/listener HTTP/1.1
Host: customer.example.com
Content-Type: application/json; charset=utf-8
X-Webhook-Event-Id: evt_01JBCDEFGH890123456789ABCD
X-Signature: t=1730001245,v1=9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

{
  "id": "evt_01JBCDEFGH890123456789ABCD",
  "type": "order.fulfilled",
  "created_at": "2026-10-22T13:00:00Z",
  "data": {
    "order_id": "ord_884920",
    "total_usd": 149.50,
    "status": "completed"
  }
}

Defending Against Timing Attacks and Replay Exploits

Signature verification code must satisfy two strict criteria:

1. Timestamp Drift Window: Receiving endpoints reject any payload where |currentTime - t| > 300 seconds (5 minutes). This prevents attackers from capturing a legitimate payload from network logs and re-transmitting it hours or days later.
2. Constant-Time Comparison: Developers must never use standard string equality (==) to verify cryptographic hashes. Standard string equality short-circuits on the first mismatched byte, allowing attackers to measure microscopic CPU latency differences (timing attacks) to reverse-engineer valid signatures byte-by-byte. We utilize constant-time verification via CryptographicOperations.FixedTimeEquals.

public static class WebhookSecurity
{
    public static string ComputeHmacSha256(string secret, long timestamp, string rawPayload)
    {
        var encoding = Encoding.UTF8;
        var keyBytes = encoding.GetBytes(secret);
        var messageBytes = encoding.GetBytes($"{timestamp}.{rawPayload}");

        using var hmac = new HMACSHA256(keyBytes);
        var hashBytes = hmac.ComputeHash(messageBytes);
        return Convert.ToHexString(hashBytes).ToLowerInvariant();
    }

    public static bool VerifySignature(
        string rawPayload, 
        string signatureHeader, 
        string secret, 
        TimeSpan tolerance)
    {
        // Parse header: t=1730000000,v1=abc...
        var parts = signatureHeader.Split(',', StringSplitOptions.TrimEntries);
        string? timestampStr = null;
        string? signatureHex = null;

        foreach (var p in parts)
        {
            if (p.StartsWith("t=")) timestampStr = p[2..];
            else if (p.StartsWith("v1=")) signatureHex = p[3..];
        }

        if (string.IsNullOrEmpty(timestampStr) || string.IsNullOrEmpty(signatureHex))
            return false;

        if (!long.TryParse(timestampStr, out var timestamp))
            return false;

        // 1. Enforce replay tolerance window
        var sentTime = DateTimeOffset.FromUnixTimeSeconds(timestamp);
        var now = DateTimeOffset.UtcNow;
        if (now - sentTime > tolerance || sentTime - now > tolerance)
            return false;

        // 2. Recompute expected hash
        var expectedHex = ComputeHmacSha256(secret, timestamp, rawPayload);

        // 3. Constant-time byte-by-byte comparison
        var expectedBytes = Encoding.UTF8.GetBytes(expectedHex);
        var actualBytes = Encoding.UTF8.GetBytes(signatureHex);

        return CryptographicOperations.FixedTimeEquals(expectedBytes, actualBytes);
    }
}

4. Mathematical Modeling of Exponential Backoff with Full Jitter

When an endpoint returns HTTP 429 (Too Many Requests), HTTP 500, 502, 503, or times out after 10 seconds, the delivery engine must reschedule the payload. A naive backoff strategy calculates delay deterministically:

t_{\text{backoff}} = \min(t_{\max}, t_{\text{base}} \times 2^{\text{attempt}})

While this spreads attempts across time, it retains a catastrophic systemic vulnerability: synchronization. If an outage affects an infrastructure provider hosting multiple customer endpoints, thousands of failed requests execute their retry waves at precisely identical intervals (e.g. 5s, 10s, 20s, 40s). The resulting traffic spikes overwhelm the recovering web servers, causing repeated secondary failures.

The Full Jitter Algorithm

To break synchronization, we adopt AWS's recommended Full Jitter approach. Rather than sleeping for the calculated exponential ceiling, we select a uniformly distributed random value between 0 and the exponential maximum:

\text{Cap} = \min(t_{\max}, t_{\text{base}} \times 2^{\text{attempt}})

t_{\text{sleep}} = \text{Uniform}(0, \text{Cap})

This guarantees that even if 5,000 requests fail simultaneously at timestamp $, their retry attempts are smoothly dispersed across the entire window, converting dangerous burst spikes into a flat, predictable baseline that downstream systems can easily absorb.

public static class BackoffCalculator
{
    private static readonly TimeSpan BaseDelay = TimeSpan.FromSeconds(5);
    private static readonly TimeSpan MaxDelay = TimeSpan.FromHours(6);

    public static TimeSpan CalculateFullJitterDelay(int attempt)
    {
        // Exponential cap: 5s * 2^(attempt-1), capped at 6 hours
        var factor = Math.Pow(2, Math.Min(attempt - 1, 14));
        var rawMilliseconds = BaseDelay.TotalMilliseconds * factor;
        var ceilingMs = Math.Min(rawMilliseconds, MaxDelay.TotalMilliseconds);

        // Uniform random distribution from 0 to ceiling
        var jitteredMs = Random.Shared.NextDouble() * ceilingMs;
        
        // Ensure a minimum delay of at least 1 second
        return TimeSpan.FromMilliseconds(Math.Max(1000, jitteredMs));
    }
}

5. Per-Host Concurrency Throttling & Circuit Breaking

A resilient delivery platform must prevent a single customer's misconfigured server from degrading system-wide queue throughput. If customer endpoint https://api.alpha.com/webhook hangs for 30 seconds on every request, allowing 100 concurrent workers to target that domain will monopolize worker bandwidth.

We implement a dual-layer protective gate:

1. Per-Host Semaphore: Workers resolve the destination host name (e.g. api.alpha.com) and acquire a token from a distributed key-bucket (e.g., maximum 5 concurrent in-flight HTTP requests per customer domain). If the quota is full, the item is deferred back to the queue without counting as a delivery attempt.
2. Consecutive Failure Circuit Breaker: If an endpoint returns 5 consecutive network timeouts or 5xx server errors, the domain transitions into a PAUSED state for 15 minutes. During this period, outgoing webhooks are automatically staged directly to the database without consuming network I/O, protecting both your delivery cluster and the customer's recovering servers.

6. Production Implementation in .NET 9

Below is the complete core engine implementation utilizing modern .NET 9 asynchronous channels (System.Threading.Channels) and typed HTTP client pipelines:

public record WebhookDeliveryTask(
    Guid DeliveryId,
    Guid EventId,
    string EndpointUrl,
    string SharedSecret,
    string PayloadJson,
    int AttemptNumber);

public class WebhookDeliveryWorker : BackgroundService
{
    private readonly Channel<WebhookDeliveryTask> _channel;
    private readonly IHttpClientFactory _clientFactory;
    private readonly ILogger<WebhookDeliveryWorker> _logger;
    private readonly ConcurrentDictionary<string, SemaphoreSlim> _hostSemaphores = new();

    public WebhookDeliveryWorker(
        Channel<WebhookDeliveryTask> channel,
        IHttpClientFactory clientFactory,
        ILogger<WebhookDeliveryWorker> logger)
    {
        _channel = channel;
        _clientFactory = clientFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Webhook Delivery Engine online. Awaiting tasks...");

        // Scale reader tasks across available CPU cores
        var workerCount = Math.Max(2, Environment.ProcessorCount * 2);
        var workers = Enumerable.Range(0, workerCount)
            .Select(_ => Task.Run(() => ProcessQueueAsync(stoppingToken), stoppingToken));

        await Task.WhenAll(workers);
    }

    private async Task ProcessQueueAsync(CancellationToken ct)
    {
        var client = _clientFactory.CreateClient("WebhookClient");

        while (await _channel.Reader.WaitToReadAsync(ct))
        {
            while (_channel.Reader.TryRead(out var task))
            {
                var uri = new Uri(task.EndpointUrl);
                var semaphore = _hostSemaphores.GetOrAdd(uri.Host, _ => new SemaphoreSlim(5, 5));

                await semaphore.WaitAsync(ct);
                try
                {
                    await ExecuteDeliveryAttemptAsync(client, task, ct);
                }
                finally
                {
                    semaphore.Release();
                }
            }
        }
    }

    private async Task ExecuteDeliveryAttemptAsync(
        HttpClient client, 
        WebhookDeliveryTask task, 
        CancellationToken ct)
    {
        var timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
        var signature = WebhookSecurity.ComputeHmacSha256(task.SharedSecret, timestamp, task.PayloadJson);

        using var request = new HttpRequestMessage(HttpMethod.Post, task.EndpointUrl);
        request.Content = new StringContent(task.PayloadJson, Encoding.UTF8, "application/json");
        request.Headers.Add("X-Webhook-Event-Id", task.EventId.ToString());
        request.Headers.Add("X-Signature", $"t={timestamp},v1={signature}");
        request.Headers.Add("User-Agent", "Mangobaz-Webhook-Delivery-Engine/1.0");

        try
        {
            using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
            timeoutCts.CancelAfter(TimeSpan.FromSeconds(10)); // 10s strict client timeout

            var response = await client.SendAsync(request, timeoutCts.Token);

            if ((int)response.StatusCode >= 200 && (int)response.StatusCode < 300)
            {
                _logger.LogInformation("Delivery {Id} succeeded (HTTP {Status}) in attempt {Attempt}",
                    task.DeliveryId, (int)response.StatusCode, task.AttemptNumber);
                // Mark delivery as COMPLETED in database
                return;
            }

            _logger.LogWarning("Endpoint returned non-success HTTP {Status} for delivery {Id}",
                (int)response.StatusCode, task.DeliveryId);
            await HandleRetryAsync(task, $"Non-success status: {response.StatusCode}");
        }
        catch (Exception ex) when (ex is HttpRequestException or OperationCanceledException or SocketException)
        {
            _logger.LogWarning("Delivery attempt {Attempt} failed for {Id}: {Error}",
                task.AttemptNumber, task.DeliveryId, ex.Message);
            await HandleRetryAsync(task, ex.Message);
        }
    }

    private async Task HandleRetryAsync(WebhookDeliveryTask task, string reason)
    {
        const int MaxAttempts = 7;

        if (task.AttemptNumber >= MaxAttempts)
        {
            _logger.LogError("Terminal failure: Delivery {Id} exceeded {Max} attempts. Routing to DLQ. Reason: {Reason}",
                task.DeliveryId, MaxAttempts, reason);
            // Move task to Dead-Letter Queue (DLQ) in database & notify tenant
            return;
        }

        var delay = BackoffCalculator.CalculateFullJitterDelay(task.AttemptNumber + 1);
        _logger.LogInformation("Scheduling retry {NextAttempt} for delivery {Id} after {DelayMs}ms",
            task.AttemptNumber + 1, task.DeliveryId, (int)delay.TotalMilliseconds);

        // In production: Persist next execution timestamp to Outbox table
        // Background poller re-enqueues item when ScheduledAt 

7. Production Operational Checklist

Before launching a webhook delivery platform in high-traffic production, review this operational verification matrix:

8. Frequently Asked Questions

Why not use RabbitMQ or Kafka directly instead of a database Outbox?

Message brokers like Kafka and RabbitMQ are outstanding for high-throughput stream transport, but they cannot participate directly in an ACID database transaction alongside your relational tables (unless you introduce slow, fragile two-phase commit protocols). The Transactional Outbox pattern guarantees that if your SQL transaction commits, the event is guaranteed to exist. You can then stream from the Outbox table into Kafka or RabbitMQ safely.

What HTTP timeout should we enforce on customer webhooks?

We recommend a strict 10-second timeout. Customers should never perform heavy synchronous business processing inside their webhook listener. Best practice for receivers is to validate the HMAC signature, append the payload to an internal queue, and immediately return 200 OK within 100 milliseconds.

How should customers handle duplicate webhooks?

Because the network is unreliable, network partition drops during an HTTP 200 OK response can cause the sender to re-transmit an already-processed event. Customers must treat webhooks as idempotent: store the unique X-Webhook-Event-Id in a fast cache or unique database index. If an incoming event ID has already been recorded, the receiver should immediately return 200 OK without re-executing business logic.