Hooklane is an open-source, self-hosted service for sending webhooks, written in Rust. An application posts an event once, and Hooklane saves it, signs it and delivers it over HTTPS to every subscribed customer endpoint, with retries, a full delivery log and protection against slow or failing endpoints. The promise everything is built around: once the API answers 202, the event is never lost. Across every load and chaos test I ran, it lost zero events.
Measured on a single laptop (Apple M1 Pro, Docker limited to 6 CPUs) running the database, broker, workers, monitoring and load generator side by side, so these numbers are a floor, not a ceiling.
Most of my portfolio is product work: SaaS apps, Shopify apps and migrations. I wanted a piece that shows the other side of what I do: designing reliable, data-intensive backend infrastructure that scales horizontally, and proving it works under real load and failure testing.
Sending a webhook looks like one line of code: a POST to a URL the customer gave you. It stops being simple once real customers are involved. Their servers go down for an hour, answer slowly or get overloaded, and the sender still must never lose an event. Doing it properly touches most of the hard parts of distributed systems: durability, queues, retries, idempotency, backpressure, isolation between tenants, security and autoscaling. The idea came from my own product, Formsuite, which sends webhooks to its users. Hooklane itself is a standalone project and isn't used there.
Hooklane is one Rust binary that runs in four roles: an API, a relay, a scheduler and the delivery workers. Postgres is the source of truth for every event, delivery and attempt. Kafka is only the conveyor belt that hands work to the workers. Messages are keyed by endpoint across 64 partitions, so each customer endpoint is owned by exactly one worker at a time, and that one fact keeps ordering, rate limiting and circuit breaking simple.
Reliability comes from a few specific mechanisms. The API writes the event, its deliveries and its outbox rows in a single Postgres transaction and only answers 202 after the commit, so an event can never be saved but not queued. Delivery is at-least-once: Kafka offsets are committed only after the work is finished, and a scheduler job re-queues anything a crashed worker left behind. Before sending, a worker claims the delivery with a conditional update in Postgres, so a duplicated Kafka message, or two workers seeing the same delivery, still produce at most one request per attempt.
One customer's broken server can't slow down everyone else. Each endpoint gets a 15-second timeout, a token-bucket rate limit, a circuit breaker and its own lane inside the worker, and an endpoint that has failed continuously for 5 days is disabled automatically. Failed deliveries are retried 8 times over about 28 hours, with growing waits and random jitter, and Retry-After is honoured. A delivery that still fails ends in a visible dead state that can be inspected and replayed, not deleted.
Requests are signed to the Standard Webhooks spec (HMAC-SHA256), so customers can verify them with an off-the-shelf library, and secrets can be rotated without downtime. Because customers choose the URL the workers call, the SSRF protection checks the resolved IP address at connection time rather than when the endpoint is created, which closes the DNS-rebinding gap. API keys are hashed, endpoint secrets are encrypted and every query is scoped to its tenant.
Workers autoscale on Kafka consumer lag using KEDA on Kubernetes. CPU would be the wrong signal, because a worker waiting on slow endpoints uses almost no CPU while it falls behind. Everything is instrumented with OpenTelemetry: metrics go to Prometheus and a Grafana dashboard, and a single Jaeger trace follows an event from the API request through Kafka to the outbound HTTP call. The SLO is 99% of events to healthy endpoints delivered within 60 seconds.
Every load run is checked by a separate program that compares what Postgres says should have been delivered with what the fake customer endpoints actually received, id by id, so the lost and duplicate counts are exact. On a laptop, sustained throughput was about 1,785 events per second, with delivery latency of 0.15 s at p50 and 0.56 s at p99 under steady load. The largest run delivered 943,815 events with 0 lost and 0 duplicates.
The chaos tests were the part I cared about most. I killed workers with SIGKILL mid-run, restarted Postgres mid-run, and made 30% of endpoints return 5xx errors, time out or send 429s. No events were lost in any of them, and the broken endpoints didn't slow down the healthy ones, which still saw a p99 of 0.63 s. Killing workers did cause 17 duplicate sends. That is the expected trade-off of at-least-once delivery, and it's why the webhook id stays the same on every retry, so receivers can skip repeats. On a local Kubernetes cluster, a 3,000 events per second burst scaled the workers from 1 to 11 and back to 1, with all 77,634 events delivered.
Running it found problems that reading the code wouldn't have. The first full load run rejected 4,221 requests because the API's 20-connection database pool was capping throughput. Raising it to 64, and lifting a container file limit, took the run from 1,332 to 1,785 events per second with zero rejections. I chose not to turn off synchronous commit to go faster, because a 202 has to mean the event is safely on disk.
Every decision is documented in the repo along with the alternatives I rejected and why. Two examples: ordering is best effort rather than strict, because with strict ordering one failing event could block an endpoint for 28 hours, so receivers get a timestamp and a sequence number to spot late arrivals instead. And retry times live in a Postgres column rather than in Kafka retry topics, because Kafka can't say "deliver this in 30 minutes", while a column gives exact delays, honours Retry-After, survives crashes and shows up in the API.
This project is also an example of how I work: AI-first engineering, done with discipline. I designed the system and made every architectural decision, then wrote a full specification that states six guarantees and how each one must be enforced. A coding agent (Claude Code) built it in seven phases, and no phase could start until the previous one passed a gate of checks I could actually observe, from health checks returning 200 to load reports showing zero lost events.
The spec included rules to keep the results honest: no invented numbers, no weakening a test or a guarantee to make a gate pass, and anything that couldn't be run gets reported as not run. Verification doesn't trust the system under test either. Signatures are checked with the official Standard Webhooks libraries, integration tests run against a real Postgres and a real Kafka broker instead of mocks, and the crash tests really kill processes.
The obvious next step is measuring it on cloud hardware. The repo includes a Terraform setup for AWS (EKS and RDS) that has been validated but never applied, and every number so far comes from a laptop. Beyond that, the documented future work includes an optional strict-ordering mode, priority lanes, a customer-facing portal for endpoints and delivery logs, and table partitioning for long-term data retention.
A service whose whole job is not losing data benefits from a compiler that rejects data races, unhandled errors and null dereferences before the code ever runs. Rust is also a proven choice for this exact problem: Svix, a commercial webhook service, is built with it. Tokio provides the async runtime and axum the HTTP API.
Rarely, yes. If a worker crashes after sending a request but before recording the result, nobody knows whether it arrived, so it is sent again. Exactly-once delivery to a third party over a network isn't possible in general. The standard answer is at-least-once delivery plus deduplication by the receiver, which is why the webhook id never changes between retries.
If Kafka goes down, the API keeps accepting events because it only writes to Postgres. Outbox rows pile up and the relay publishes them when Kafka is back, so deliveries are delayed but nothing is lost. If Postgres goes down, the API answers 503 instead of crashing, so senders retry, and the other roles reconnect with backoff. In the restart test, zero accepted events were lost.
You can, and Hooklane ships that mode so the two can be compared. It's simpler and has a slightly lower median latency on one machine. The catch is that every poll is a write on the busiest table, and several workers can serve the same endpoint, so rate limits and circuit breakers are only per worker. In the demo, a failing endpoint got 10 attempts before parking in Postgres mode with two workers, and exactly 5 in Kafka mode.
No. It's a portfolio project, built to show how a system like this is designed and to measure it honestly. The architecture is real and every number comes from a test that was actually run, but only on a laptop and a local Kubernetes cluster. For a real product today, a managed service like Svix or Convoy is probably the right call.
© 2026 Brady Edgar. All rights reserved.