Stopping email-blast HEAD storms from taking down a legacy news site

Industry: Media and events
Company size: 200-500 employees
Services: Site reliability, CDN and edge, WAF, incident response

The Challenge

Every time the client sent a marketing email, its own website went down for a few minutes. The site is a decade-old PHP application on ECS, sitting behind a CDN and a web application firewall, and it was months away from being replaced. It still had to survive until then.

The mechanism took some digging. Corporate mail clients run link-protection scanners that prefetch every URL in an email the moment it lands. A blast to a large list therefore produces thousands of HEAD requests to the same handful of links within seconds, spoofing a browser user agent and fanned out across hundreds of cloud IP addresses. The links in question were checkout pages marked no-store, so every prefetch reached the origin, and the application's HEAD handling sat after the cart-building code, so each one ran ten to thirty database queries and a session write. The PHP-FPM pool of 25 workers per task filled, the whole site stalled, and because the health check also ran through PHP, the load balancer started killing healthy tasks.

The first incident dropped about 14,000 prefetches on one checkout URL in seven minutes and stalled the site for seven minutes. Its signature was the confusing part: zero 5xx errors, every host healthy, dashboards green. The only trace was the load balancer's 460 code, a client giving up before the origin ever answered.

Our Approach

This was not one fix. It was five rounds over six weeks, each one measured against the next storm, and each one teaching us something the previous round had hidden.

• Round one: fix the code. Moving the HEAD short-circuit to the first line of the controller removed the cart build. Two days later a blast of 15,600 requests per minute still stalled the site for a minute, because the framework middleware, session and route-model binding, ran before the controller and still cost a worker per request.
• Round two: answer HEAD before PHP exists. An nginx rule answers HEAD on the checkout path with a 200 before the request touches PHP-FPM. A load test at twice the incident's volume, 92,000 requests, showed 100 percent short-circuited at about 2 milliseconds, zero reaching PHP, and CPU at 15 percent of the previously pegged baseline. We also split the health probes: liveness became a static nginx response so a saturated pod is never mistaken for a dead one, and readiness treats "busy" as healthy so load cannot eject the pods that are still serving.
• Round two, at the edge. Per-IP rate limits could never catch a scanner spread across 250 addresses a day. But real buyers never send HEAD, so a global, method-scoped WAF rule with zero false positives blocks checkout HEADs and returns a 200 that looks exactly like the nginx answer. The first day it blocked about 8,000 requests, 5,559 of them in a single five-minute window, with no origin impact. That same day we found the scanner also sends GETs, which have to reach PHP for real buyers, so a throttle keyed on user agent went in at 200 a minute with a 429 and a Retry-After, after a 30-day analysis of 3,684 distinct user agents showed the busiest human-plausible one peaked at 111.
• Round three: the scanner moves. Two weeks later an event email pointed the same scanner at the events pages: 13,649 requests a minute from one user agent across 1,229 addresses, and a three-minute outage. Origin Shield and query-string exclusions were already on and did not help. The real cause was synchronized cache expiry: a 30-minute TTL meant that at expiry a single blocking revalidation hit a saturated origin and every viewer behind it timed out. The fix was stale-while-revalidate on every CDN-cached page, so expiry serves stale and revalidates in the background. We also added the alarm that had been missing, on the 460 count itself, since every existing alarm watched signals that stay green during a storm.
• Round four: stop chasing paths. By the end of June the scanner had generalized to the footer links in every email, unsubscribe and subscription pages, at 461 and 1,225 times their baseline. The path-specific rules were catching under 2 percent of the damage. The durable discriminator turned out to be method plus path, not user agent, which the fleet rotates. One HEAD-by-path block rule replaced the whack-a-mole.
• Round five: the outage that broke the model. On July 1 the site stalled for 15 minutes at midnight under only 600 requests a minute. This was not volume. A broad crawl hit the expensive uncached pages, RSS feeds at 3.6 seconds, city landing pages at 6, search at 4, and enough of them in flight exhausted the pool. Autoscaling never fired because it watched request count, and the WAF was silent because the traffic was distributed. Two changes closed it. The feeds, which had 24 requests per URL, became edge-cached, cutting that origin load by more than 90 percent, and dead URLs got a cached 404 at the edge. And a timeout cascade, PHP-FPM at 20 seconds, nginx at 25, CDN at 30, means the origin reclaims a stuck worker before the edge gives up on it, instead of holding it for another minute.

The Results

• The next storm was four times bigger and did nothing. The morning after the final deploy, the same overnight scanner hit 2,281 requests a minute against the previous day's 580. p99 stayed at 1.13 seconds with seven dropped requests, against a 15-minute stall the day before.
• A real email blast at 423 requests a second at the edge from 15,834 addresses reached the origin at 87 a second, with a p99 of 2.2 seconds and zero 5xx errors. The origin never scaled and never needed to.
• When a storm does pin the pool, it now clears in about two minutes instead of fifteen, because the worker-reclaim cascade frees capacity while the storm is still running.
• The alarm that fires is the right one. The composite alarm on 460 count plus p99 paged zero times in its first week while each half fired alone briefly, which is the behavior you want.
• The last storm root-caused was internal: an admin search running 20-second full-table scans in a retry loop. The reclaim cascade contained it to five minutes with no page, and the query is now on the application team's list.

What we learned

• Green dashboards do not mean the site is up. Watch for clients giving up, not just servers failing.
• The cheapest place to answer a request is the one closest to the client. HEAD at nginx costs microseconds; at PHP it costs a worker.
• Rate limits keyed on what you can see are brittle. Method and path survive user-agent rotation. Per-IP limits cannot see a fleet, and they will eventually block your own office NAT. Ours did, for an afternoon.
• Cache expiry needs stale-while-revalidate or the first miss after expiry becomes a stampede.
• Saturation is about request cost, not request count. Autoscale on latency or busy workers, and make sure a stuck request is released by every layer, in order, before the edge times out.

Key Takeaway

Nobody would have rebuilt this application to fix the storms, and nobody had to. The defenses were a few nginx lines, a handful of firewall rules, cache headers, and three timeouts in the right order. What mattered was measuring every storm against the last change and being willing to say the previous fix was not enough.

Related guide: Why your website goes down when marketing sends an email

Does your site fall over when marketing hits send?

We'll pull your load balancer and CDN logs for the last blast, show you exactly which requests pinned the origin, and tell you which layer should be answering them.

Schedule a call