The Anatomy of a Lost Webhook

By Dana Wombat · July 14, 2026 · 6 min read

Every engineering team has the same ghost story: an order that never shipped, a user who never got provisioned, an invoice that never synced. When you dig in, the culprit is almost always the same — a webhook that fired once, failed quietly, and was never seen again.

Where webhooks actually go to die

Most lost webhooks are not dropped by the network. They die in one of three mundane places:

• The 30-second timeout. Your endpoint did the work but answered too slowly, so the sender gave up and marked it failed.
• The deploy window. For ninety seconds your service returned 502s, and the sender's retry policy quietly ran out.
• The duplicate scare. A retry arrived, your handler was not idempotent, so someone "fixed" it by dropping anything that looked familiar — including events that were not duplicates.

The fix is boring (that is the point)

Durable ingestion means acknowledging in milliseconds, persisting the payload before you process it, and replaying on your schedule instead of the sender's. Idempotency keys make retries safe; dead-letter queues make failures visible instead of silent.

Takeaways

If you cannot answer "where is that webhook right now?" in one query, you do not have an ingestion pipeline — you have a hope. Sit on your events. Patiently. Like a wombat.