There is a particular kind of phone call we get at Insiteful, and it always sounds the same. A brand rings up because their 3PL is missing orders. Not all of them. Not obviously. Just enough that somebody in the warehouse noticed a gap, and now the founder is asking how long this has been happening and nobody can answer. The integration was built two years ago. It worked. Nothing was changed. And yet orders are quietly falling on the floor.
Nine times out of ten the answer is webhooks. Specifically, a webhook subscription that Shopify deleted weeks ago after a handful of failures nobody was watching, and which has been silently not firing ever since. The endpoint is still there. The code is still correct. The subscription is gone. This article is the reliability playbook we apply to every integration we build, and it exists because webhook failure is the single most expensive silent bug in Australian ecommerce operations.
Why This Matters More Than It Used To
Australians spent $82.6 billion AUD online in 2025, up 14 percent year on year, with roughly 24 percent of all retail spend now going through a digital channel, according to the Australia Post eCommerce Report. That growth has a structural consequence that gets very little attention: the average Australian store now runs more integrations than it did three years ago, and each one is a place where an order can disappear.
A store doing $5M AUD a year through a native Shopify build might be pushing events into a 3PL, an ERP, an accounting ledger, a subscriptions platform, a review platform and a customer service desk. That is six webhook pipelines. If each one is 99.5 percent reliable, the compound probability that a given order lands cleanly in every system is about 97 percent. Three orders in every hundred have a problem somewhere. On 40,000 orders a year, that is 1,200 exceptions a year that somebody has to find and fix by hand, or worse, that nobody finds at all.
The fix is not a better app. The fix is treating the webhook layer as infrastructure with a defined failure behaviour, rather than as a piece of plumbing that either works or does not.
What Actually Happens When a Shopify Webhook Fails
Most guides on this topic are still wrong, and they are wrong in a way that will cost you money. Search “Shopify webhook retries” and you will find a large number of articles confidently telling you that Shopify retries for 48 hours. That was true once. It has not been true since 10 September 2024.
Per the Shopify developer changelog, webhooks are now retried a total of 8 times over 4 hours on an exponential backoff schedule. Shopify’s webhook verification documentation then adds the part that actually hurts: after 8 consecutive failures, the subscription is automatically deleted if it was configured using the Admin API. Warning emails go to the app’s emergency developer email address, which in a lot of agency-built integrations is an inbox nobody has opened since the build.
Read that sequence again, because the implications are severe. Your endpoint has a bad afternoon. Four hours later the retries are exhausted. The subscription is deleted. From that moment forward, events are not queued, not delayed and not retried. They simply never fire again, and your logs are silent because there is nothing to log. The system does not tell you it has stopped working. It just stops.
Two more constraints matter. Shopify enforces a one second connection timeout and a five second timeout for the entire request. And any response outside the 200 range counts as failure, including 3XX redirects. An endpoint that quietly 301s from the bare domain to the www subdomain is a failing endpoint, even though it looks fine in a browser.
The compressed retry window changes the engineering brief. Under a 48 hour curve, a human could plausibly notice an outage and fix it before delivery gave up. Under a 4 hour curve, on a Saturday, with an Australian team asleep during a North American traffic peak, no human is going to save you. The architecture has to survive on its own.
The Five Second Budget
The most common structural mistake we find is doing the work inside the request. A store’s order webhook handler receives the payload, then writes to the database, then calls the 3PL API, then calls the ERP, then returns 200. On a good day that takes 1.5 seconds. On a day when the 3PL is slow it takes six, and Shopify has already recorded a failure.
The rule is simple and non-negotiable: acknowledge fast, process later. The handler verifies the signature, writes the raw payload to durable storage, returns 200, and a separate worker does the actual integration work. The five second budget is then spent on two things you control, not on a third party API you do not.
A well-built handler looks like this, and it should be boring:
1. Read raw body ~1ms
2. Verify HMAC signature ~1ms
3. Check dedupe table for webhook ID ~5ms
4. INSERT payload into inbox table ~10ms
5. Return 200 OK ~20ms total
-- request ends here --
6. Worker picks up row, calls 3PL/ERP (seconds or minutes, does not matter)Everything past step 5 is your problem to retry, not Shopify’s. That is the entire point. You have converted an external dependency with a four hour patience limit into an internal queue you can retry for as long as you like.
Shopify’s documentation also notes that its delivery system uses HTTP Keep-Alive to reuse connections to the same host, so make sure Keep-Alive is enabled on your endpoint. On stores with heavy event volume this measurably reduces connection overhead during bursts, and bursts are exactly when you are closest to the timeout.
Verify Before You Trust
Every HTTPS webhook delivery carries a base64-encoded HMAC-SHA256 signature in the X-Shopify-Hmac-SHA256 header, generated from your app’s client secret and the raw request body. If you are not verifying it, your endpoint is an open door: anyone who learns the URL can post a fake order-paid event and trigger fulfilment on goods nobody bought.
The verification itself is short. Getting it wrong is easy, and there is one dominant cause.
const express = require('express');
const crypto = require('crypto');
const app = express();
// Raw body, NOT express.json(). This line is the whole ballgame.
app.use(express.raw({ type: '*/*' }));
app.post('/webhooks/orders-paid', (req, res) => {
const sent = req.headers['x-shopify-hmac-sha256'];
const digest = crypto
.createHmac('sha256', process.env.CLIENT_SECRET)
.update(req.body)
.digest('base64');
const ok = crypto.timingSafeEqual(
Buffer.from(digest, 'base64'),
Buffer.from(sent, 'base64')
);
if (!ok) return res.status(401).send('bad signature');
enqueue(req.body, req.headers);
return res.status(200).send('ok');
});The failure mode that catches most teams is body parsing. HMAC is computed over the exact bytes Shopify sent. If a JSON body parser runs first, it parses and re-serialises the payload, key order or whitespace shifts by a byte, and the signature no longer matches. The symptom is maddening: verification fails on every single delivery, the payload looks perfectly valid when you print it, and the code is correct. Middleware order is the bug. Put the webhook route and its raw parser ahead of any global JSON parser.
Use a timing-safe comparison, not a string equality check. And note a genuine operational trap from Shopify’s documentation: if you rotate your app’s client secret, it can take up to an hour for the HMAC digest to be generated using the new secret. If you rotate and immediately tighten verification to the new secret only, you will reject an hour of legitimate traffic, burn through the eight retries, and delete your own subscriptions. During a rotation window, accept either secret.
Deduplication: The Header Most Guides Get Backwards
Retries mean you will receive the same webhook more than once, so processing has to be idempotent. Fair enough. The interesting part is which header you deduplicate on, and this is where a lot of published advice is simply incorrect. We have seen multiple well-ranked guides instruct readers to use X-Shopify-Event-Id as the deduplication key. That is not what Shopify’s documentation says, and following it will cause data loss.
The distinction is precise. If you have more than one subscription for the same topic, you receive a separate delivery per subscription. Each delivery has a different X-Shopify-Webhook-Id but they share the same X-Shopify-Event-Id. So:
- X-Shopify-Webhook-Id identifies one delivery. Deduplicate on this.
- X-Shopify-Event-Id identifies the merchant action that caused the deliveries. Use it to correlate, trace and debug across subscriptions.
Deduplicate on the event ID and you will process the first subscription’s delivery and silently discard the second, which is a different endpoint doing a different job. Your 3PL gets the order and your ERP does not, and it looks exactly like the mystery we opened this article with.
The implementation we use is a unique constraint doing the work, rather than a read-then-write that races under concurrent retries:
CREATE TABLE webhook_inbox (
webhook_id TEXT PRIMARY KEY,
event_id TEXT NOT NULL,
topic TEXT NOT NULL,
shop_domain TEXT NOT NULL,
triggered_at TIMESTAMPTZ NOT NULL,
payload JSONB NOT NULL,
received_at TIMESTAMPTZ DEFAULT now(),
processed_at TIMESTAMPTZ
);
CREATE INDEX ON webhook_inbox (event_id);
CREATE INDEX ON webhook_inbox (processed_at) WHERE processed_at IS NULL;
-- Insert is idempotent by construction. A duplicate delivery
-- hits the primary key, does nothing, and still returns 200.
INSERT INTO webhook_inbox (webhook_id, event_id, topic, shop_domain, triggered_at, payload)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (webhook_id) DO NOTHING;Storing event_id alongside is what lets you answer “show me everything that happened as a result of this one order being paid” during an incident. That query has saved us hours on more than one reconciliation job.
Stale Payloads and Out of Order Delivery
The September 2024 changelog contains a detail that is easy to skim past and expensive to ignore: when webhooks are retried, they are delivered with the original payload from the time they were triggered.
Picture the sequence. An order is placed at 09:00. The orders/create delivery fails. At 09:04 the customer changes their shipping address and orders/updated fires and succeeds. At 09:20 the original 09:00 delivery finally succeeds on retry, carrying the old address. If your handler blindly writes whatever it receives, you have just overwritten good data with stale data, and you will ship to the wrong address.
Shopify’s guidance is to use the X-Shopify-Triggered-At header, or a timestamp from the payload, to determine whether a payload is stale. In practice we enforce this at the write, not in application logic, so it cannot be forgotten:
UPDATE orders
SET shipping_address = $2,
financial_status = $3,
source_triggered_at = $4
WHERE shopify_order_id = $1
AND source_triggered_at < $4; -- older event never winsOne line, and out of order delivery stops being a correctness problem. Shopify also notes you might receive webhooks up to a day late, so if a delay of that size would cause problems in your system, compare the payload timestamp against the current time and route anything suspicious to manual review rather than processing it as fresh.
App-Specific and Shop-Specific Subscriptions Fail Differently
This is the architectural decision with the largest blast radius, and it is usually made by accident.
Shop-specific subscriptions are created per store through the Admin API. These are the ones that get automatically deleted after eight consecutive failures. Recovering means your code has to detect the deletion and recreate the subscription, which means something in your system has to be actively checking.
App-specific subscriptions are declared in your app configuration and apply across every store that installs the app. Shopify’s own recovery documentation makes the practical difference explicit: after an extended outage you do not need to re-subscribe if your app uses app-specific webhook subscriptions. For shop-specific subscriptions you have to consult the code that created them and recreate what is missing.
Where the topic supports it, we declare subscriptions in app configuration. Where a store-by-store subscription is genuinely required, we treat the subscription list itself as monitored state. A scheduled job queries the live subscriptions, compares them against the expected set, recreates anything missing and raises an alert. That job takes an afternoon to build and it is the difference between a four hour gap and a four week gap.
query {
webhookSubscriptions(first: 50) {
edges {
node {
id
topic
endpoint { ... on WebhookHttpEndpoint { callbackUrl } }
}
}
}
}The Reconciliation Job That Makes The Whole Thing Safe
Every control above reduces the chance of loss. None of them eliminates it, and any architecture that assumes webhooks are sufficient on their own is wrong. Shopify’s own documentation recommends building a reconciliation job that periodically retrieves data you might have missed using the Shopify APIs. We treat this as mandatory on every integration we ship, not as a nice-to-have.
The pattern is a scheduled sweep that asks the Admin API for everything created or updated in a window, compares it against what your system holds, and repairs the difference:
query RecentOrders($since: DateTime!) {
orders(first: 250, query: "updated_at:>$since", sortKey: UPDATED_AT) {
edges {
node {
id
name
updatedAt
displayFulfillmentStatus
}
}
}
}Three design notes from running these in production. Overlap the window generously, so a job running hourly should look back three hours, because cheap duplicate checks are better than a boundary gap. Log every repair rather than fixing silently, because a reconciliation job that starts finding twenty orders a night is telling you something upstream has broken and that signal is worth more than the repair. And alert on the repair count, not just on job failure, since a reconciliation job that succeeds while quietly fixing hundreds of records is a fire alarm, not a green tick.
The Monitoring We Put On Every Integration
Shopify gives you real instrumentation and most merchants have never opened it. If your app was created in the Dev Dashboard or via Shopify CLI, the Monitoring page shows delivery counts, failure rates and 90th percentile response time by topic, and the Logs page shows individual deliveries with response codes.
Shopify’s documented threshold is useful and specific: a failed delivery rate above 0.5 percent is higher than average and worth investigating. We set alerts well below the point where subscriptions start getting deleted:
| Signal | Threshold | Why |
|---|---|---|
| Failed delivery rate | Above 0.5 percent | Shopify’s own “higher than average” mark |
| P90 response time | Above 2 seconds | Half the five second budget consumed; no headroom for a burst |
| Removed subscriptions | Any, ever | Silent total failure for that topic from this moment on |
| Unprocessed inbox rows | Older than 15 minutes | Delivery is fine, your worker is stuck |
| Reconciliation repairs | More than 5 per run | Something upstream is dropping events |
Two practical cautions. The dashboards cover the past 7 days, so an issue that started a fortnight ago is invisible there and you need your own retained logs to investigate it. And the logs are not real time, with data potentially delayed several minutes, so they are a diagnostic tool rather than an alerting source. Alerting belongs on your side of the wire.
The Failures We Get Called In To Fix
A pattern emerges after enough of these. Ranked by how much damage they do before anyone notices:
Deleted subscriptions nobody knew about. The most expensive by a wide margin, because the gap is measured in weeks. Usually traced back to a deploy or a hosting migration that caused a few hours of 5XX responses months earlier.
Work done inside the request. An integration that was fine at 200 orders a day starts intermittently timing out at 900. It presents as a mystery, since nothing was changed. Volume was the change.
A 301 in front of the endpoint. Someone adds a redirect rule, or moves to a new domain, or enables a canonical redirect. Every delivery now returns 3XX, which Shopify counts as failure. The endpoint returns a perfectly healthy 200 when tested in a browser, because the browser follows the redirect and Shopify does not.
Deduplicating on the wrong header. One system gets the data and another does not, permanently, and the pattern looks random until you inspect the headers.
No reconciliation job. Not a failure in itself. It is the reason the other four run for weeks instead of hours before someone notices.
How We Do It At Insiteful
We build and maintain native Shopify and Shopify Plus stores for Australian brands, and the integration layer is part of the build rather than something bolted on afterwards by a third party who never sees the storefront. That matters here more than in most areas, because webhook reliability sits precisely in the gap between “the store” and “the systems”, and gaps are where things get dropped.
Every integration we ship gets the same baseline. HMAC verification with raw body handling. An inbox table with a primary key on the delivery ID, so idempotency is a property of the schema and not of somebody remembering. Acknowledge in under 100 milliseconds with all real work moved to a worker. Timestamp guards on writes so a late retry cannot overwrite fresh data. App-specific subscriptions where the topic allows, plus a subscription audit job where it does not. A reconciliation sweep on a schedule. Alerting on failure rate, response time, subscription removal and repair volume.
None of that is exotic. It is roughly a day of work on top of a straightforward integration, and it converts a system that fails silently into one that either works or tells you loudly that it does not. On a store turning over $5M AUD, a single undetected week of dropped fulfilment events costs vastly more than that day.
We apply the same standard across the stack, whether we are wiring up an ERP, 3PL fulfilment, product information management, accounting or the marketing and support stack. Where the logic can live on the platform instead of on your infrastructure we prefer that too, using Shopify Flow for merchant-editable automation, because the most reliable webhook is the one you never needed to build.
Get Your Integrations Audited Properly
If orders are occasionally going missing, if your warehouse and your store disagree about what shipped, or if you genuinely do not know whether every webhook subscription you rely on is still alive right now, the answer is knowable and usually fixable in days rather than months. Most stores we audit have at least one deleted subscription they had no idea about.
See how our build process works and book a build assessment, and we will map every integration in your store, tell you which ones are silently failing, and show you what it takes to make them trustworthy.