A policy update containing unintended blank fields reached a code path in Service Control - the global layer that checks quota and policy on nearly every Google Cloud API call - that had a null-pointer defect and no feature-flag protection. Service Control crash-looped globally, returning 503s across dozens of products for roughly three hours; us-central1 took longest to recover due to a retry herd.
A race condition in the pipeline that propagates Google Cloud Load Balancer configuration caused a corrupt configuration to be pushed globally. Requests to sites fronted by GCLB began returning 404 errors even though the backends were healthy - taking down high-profile sites for roughly two hours until the configuration was rolled back.
GKE control-plane incidents are the classic split-brain outage: the Google-managed control plane - the Kubernetes API server, scheduler, and controller manager - becomes unreachable or unhealthy in a region, so kubectl, deployments, autoscaling, and self-healing all stall. Meanwhile the nodes and pods you already had running keep serving traffic, because the data plane runs independently of the control plane. You cannot deploy, scale, or recover from node failures until the control plane returns, but existing workloads generally stay up.
A migration to a new quota system left Google's central User ID Service with a near-zero storage quota. The identity service could not read the account data it needed, so requests that required authentication failed globally. Anything that had to log a user in - Gmail, YouTube, Docs, and the Google Cloud Console and APIs that authenticate through the same identity layer - returned errors for roughly 47 minutes of core impact.
A routine maintenance configuration change was applied to a far wider scope than intended, descheduling the network control-plane jobs in several US regions. The network fell back to preserving only high-priority traffic (like the control plane’s own signaling) while starving everything else, so Compute Engine, YouTube, Gmail, and Snapchat degraded severely for about four hours - worst in the US East.
Degradation in an internal blob-storage service that underpins Google Cloud Storage caused elevated error rates and latency for GCS operations, and for Google’s own products that store objects on the same layer - Gmail and Photos attachments among them - for roughly four hours.
A pair of overlapping events in Google's network configuration management system produced a race: as an unused IP block was being removed, the management system propagated a configuration in which the IP blocks for all of Google Compute Engine were withdrawn instead of just the one. A safety check that should have paused the bad config had a separate bug and let it through, so every GCE region lost external connectivity at once for roughly 18 minutes until the previous configuration was restored.
Four successive lightning strikes to the local utility grid feeding Google's europe-west1 facility in Belgium briefly cut power to a portion of the persistent-disk storage systems. Most I/O recovered quickly, but a small number of disks that were writing at the moment of the power loss saw errors, and an extremely small fraction of disk space - reported as around 0.000001 percent of the region - suffered permanent data loss because the writes never made it to stable storage.
7 min read
Recent incident log
Smaller incidents from the live feed (last 90 days). Major events graduate into full post-mortems above.
Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.
gcp