Service disruption across EU-West infrastructure, July 28 to 30, 2026
Between July 28 and July 30, an upstream compute provider experienced a progressive infrastructure failure that cascaded into Keirolabs EU-West servers. Affected API requests returned zero search results with an incorrect HTTP 200 status code. This document provides the complete root cause analysis, a day-by-day incident timeline, the six remediation measures deployed, and the service credit issued to all paid accounts.
50% service credit applied to all paid accounts for the next billing cycle
No action required. The credit covers every request across every endpoint for one full billing period. It has been applied automatically to all active paid subscriptions.
A single-provider dependency allowed a localized fiber failure to cascade across regions
On July 28, Keirolabs' primary compute provider experienced a fiber cut in one of three US-East availability zones. The failure was initially contained: retry logic at the load balancer masked most errors from end users, and the API error rate remained below 1% through the first 12 hours.
On July 29, the provider's capacity in US-East dropped by approximately 35% as the affected availability zone was fully drained. Without per-region circuit breakers, the overflow was routed to EU-West, which had no backpressure controls. Within two hours, EU-West queue depth exceeded 12x normal, P99 latency crossed 8 seconds, and the system began returning zero results for a significant fraction of requests.
A defect in the API error-handling path caused these zero-result timeout responses to be returned as HTTP 200 with an empty array, making the failure invisible to callers relying on status codes for error detection.
The incident reached SEV-1 at 06:55 UTC on July 29 when peak error rate hit 34%. Emergency compute on a secondary provider was activated at 08:10, reducing the error rate to 6% within 12 minutes. Full stability was restored by 09:00 UTC on July 30 after the upstream fiber was repaired and six remediation measures were deployed to prevent recurrence.
Request flow during the failure window
All three regional compute pools were backed by a single upstream provider. When that provider lost capacity in US-East, the failover path lacked backpressure, allowing unchecked overflow into EU-West.
Day-by-day chronological record
All timestamps in UTC. Total incident window: approximately 52 hours from first anomaly to confirmed full recovery.
Current system topology, effective August 1
Single-provider dependency eliminated. Each region is backed by two independent providers with circuit breakers, active health probes, and pre-warmed standby capacity.
Six changes deployed before this report was published
Each addresses a specific failure mode exposed by this incident. All six are live in production as of August 1.
Multi-provider compute distribution
Search workloads are now distributed across three independent infrastructure providers. No single provider handles more than 45% of total request volume. If any provider degrades below the latency SLO, traffic shifts to the next healthy pool within 15 seconds with zero manual intervention.
Per-region circuit breakers
Each region operates its own circuit breaker. When the error rate for a given provider in a given region exceeds 5% over a 30-second window, that path is opened and traffic is redirected. This prevents the cascading overflow that caused EU-West to absorb US-East failures.
Active latency-based health probes
Passive TCP health checks (30-second interval) have been replaced with active probes that execute real search queries against each provider every 5 seconds. The probe measures end-to-end query latency, not just socket liveness. Degradation detection is now approximately 6x faster.
Request queue backpressure
Each regional pool now enforces a queue depth ceiling. When the ceiling is reached, excess requests receive HTTP 429 with a Retry-After header rather than being silently queued and eventually timing out. This gives callers a clear signal to back off or route elsewhere.
Pre-warmed standby capacity
Every region maintains warm standby instances on a secondary provider, ready to accept production traffic in under 20 seconds. Previous cold-start provisioning required approximately 4 minutes, which was too slow to mitigate the cascading failure.
HTTP status code correction
The API no longer returns HTTP 200 when upstream compute times out and yields zero results. Timeout and empty-upstream conditions now surface as HTTP 503 with a Retry-After header. Genuine empty result sets (valid query, no matches) remain HTTP 200 with an explicit empty_reason field.
Three error-handling defects identified and corrected
The incident exposed latent defects in the API error path. All three have been corrected in production.
| Defect | Previous behavior | Corrected behavior |
|---|---|---|
| HTTP 200 on zero results from timeout | API returned 200 OK with an empty search_results array when upstream timed out. Callers could not distinguish a timeout from a legitimately empty result set. | Upstream timeouts now return 503 Service Unavailable with a Retry-After header. Genuine empty results remain 200 and include an empty_reason field. |
| Silent failover queue overflow | The failover path had no backpressure mechanism. When Provider A degraded, EU-West accepted every rerouted request into an unbounded queue, where they silently timed out. | Per-region queue depth is capped. Requests exceeding the cap receive 429 Too Many Requests with a retry hint, giving callers an actionable signal. |
| False-positive health checks | TCP-level health checks showed Provider A as healthy even when query latency exceeded 10x baseline. No automatic traffic rerouting was triggered because the socket was reachable. | Active probes issue real search queries every 5 seconds. If median latency exceeds 2x baseline for three consecutive probe cycles, the provider is marked degraded and traffic shifts automatically. |
We owe you better reliability. We shipped it.
The service credit is applied. For questions about this incident, the infrastructure changes, or your account, our engineering team is available.