We are investigating elevated error rates with multiple products in us-east1

Technical impact

We are investigating elevated error rates with multiple products in us-east1

Timeline

1. MonitoringJul 22, 2025, 1:42 PM UTC

   ## \# Incident Report ## \#\# Summary On Friday, 18 July 2025 07:50 US/Pacific, several Google Cloud Platform (GCP) and Google Workspace (GWS) products experienced elevated latencies and error rates in the us-east1 region for a duration of up to 1 hour and 57 minutes. **GCP Impact Duration:** 18 July 2025 07:50 \- 09:47 US/Pacific : 1 hour 57 minutes **GWS Impact Duration:** 18 July 2025 07:50 \- 08:40 US/Pacific : 50 minutes We sincerely apologize for this incident, which does not reflect the level of quality and reliability we strive to offer. We are taking immediate steps to improve the pla

2. MonitoringJul 18, 2025, 10:08 PM UTC

   # Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) **GCP Impact start and

3. IdentifiedJul 18, 2025, 6:03 PM UTC

   The issue has been resolved for all affected products as of 2025-07-18 09:47 US/Pacific. From preliminary analysis, during a routine maintenance of our network in us-east1-b, we experienced elevated packet loss, causing service disruption in the zone. We will publish a full Incident Report with root cause once we have completed our internal investigations. We thank you for your patience while we worked on resolving the issue.

4. MonitoringJul 18, 2025, 5:32 PM UTC

   Our engineers have successfully recovered the network control plane in the affected us-east1 zones. We're seeing multiple services reporting full recovery, and product engineers continue to validate the remaining services. We'll provide another update with more details by 11:00 AM US/Pacific, July 18, 2025.

5. MonitoringJul 18, 2025, 4:58 PM UTC

   Our engineers have successfully recovered the network control plane in the affected us-east1 zones. We're seeing multiple services reporting full recovery, and product engineers are now validating the remaining services. We'll provide another update with more details by 10:30 AM US/Pacific, July 18, 2025.

6. MonitoringJul 18, 2025, 4:29 PM UTC

   Our engineers have confirmed that us-east1-b is partially affected. All other zones in us-east1 are currently operating normally. Our engineers have recovered the failed hardware and are currently recovering the network control plane in the affected zones. We'll provide another update by 10:00 AM US/Pacific, July 18, 2025.

7. InvestigatingJul 18, 2025, 3:54 PM UTC

   We're currently experiencing elevated latency and error rates for several Cloud services in the us-east1 region, beginning at 7:06 AM PDT today, July 18, 2025. Our initial investigation points to a hardware infrastructure failure as the likely cause. We apologize for any disruption this may be causing. We'll provide an update with more details by 9:15 AM PDT today.