Duration: 3 hours and 58 minutes

Affected Components: Event Aggregator (Ingress), Event Digest (Ingress), Notification (Egress), Seattle, WA, USA (us-west3), Route Propagation, Singapore (ap4), Hong Kong (ap2), Tokyo, Japan (ap1), Sydney, Australia (ap3), Osaka, Japan (ap5)

May 7, 10:19:00 GMT+0
Resolved - All outages have been restored. Please contact us as soon as possible if you deployed any custom workload with us in Osaka, otherwise, all managed services in Osaka will be seamlessly redirected to Tokyo..

May 7, 10:04:35 GMT+0
Identified - We identified a repeatedly tripped internal failsafe that has been adding bogus routes to Aria's global routing table. Expect to receive inconsistent routes in the meantime as we patch Aria on-the-fly. .

May 7, 09:45:47 GMT+0
Monitoring - We have brought all locations online except for Osaka which will be subjected to an extended maintenance period. We are also enabling a small portion of customers to interface with Caesar again to contain potential cascading failures..

May 7, 09:17:42 GMT+0
Monitoring - We rolled back a recent update to internal critical infrastructure in Hong Kong and Singapore and are proactively monitoring the result. Caesar Monitoring will be disabled globally in this moment and time..

May 7, 08:38:28 GMT+0
Monitoring - We implemented a fix for Tokyo and are incrementally routing in traffic again. .

May 7, 08:34:25 GMT+0
Identified - We have identified an additional outage in Singapore..

May 7, 08:30:47 GMT+0
Identified - We are commencing a cold restart of all services in Japan and removing routes destined for Tokyo in Hong Kong. .

May 7, 08:25:19 GMT+0
Investigating - We are currently investigating a simultaneous regional outage..

May 7, 08:21:18 GMT+0
Investigating - We are currently investigating this incident..

Duration: 5 hours

Affected Components: Volumetric Protection, Rate Limiter, Seattle, WA, USA (us-west3), Route Propagation, Fremont, CA, USA (us-west2), Singapore (ap4), Sydney, Australia (ap3)

May 3, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 3, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

May 3, 17:00:00 GMT+0
Identified - We are decommissioning all hosted Aegis instances in mainland China. Instead, potentially malicious traffic destined for mainland China will soon be routed to our APAC (Singapore, Australia) or US-West Hubs (Seattle, Fremont). We expect this migration to take no more than 5 hours, and during this period, we will occasionally pause all route propagation of Aria in the above-mentioned POPs. Please contact your customer success manager if your business requires uninterrupted route propagation..

Duration: 12 hours and 32 minutes

Affected Components: Certificate Authority (Internal), Event Digest (Ingress), Certificate Issuance

Mar 4, 02:34:50 GMT+0
Resolved - This incident has been resolved..

Mar 3, 14:49:54 GMT+0
Monitoring - We have deployed a revision to our CA and a public ACME client as a fallback in case further issues occur. Since exponential backoff was observed for many endpoints, this incident will be continually monitored until the revised build has successfully reissued all active certificates..

Mar 3, 14:39:17 GMT+0
Identified - We are continuing to work on a fix for this incident..

Mar 3, 14:07:13 GMT+0
Investigating - We are investigating a backlog of certificate signing requests that are failing internal challenges. This will impact all customer endpoints and Caesar sessions configured with 1-day expiration certificates. .

Mar 3, 14:02:41 GMT+0
Investigating - We are currently investigating this incident..

Duration: 29 minutes

Affected Components: Fremont, CA, USA (us-west2), Tokyo, Japan (ap1)

Mar 1, 16:54:07 GMT+0
Resolved - This incident has been resolved..

Mar 1, 16:24:45 GMT+0
Monitoring - We have implemented a fix for Inconsistent QoS Categorization. Customers who are affected will receive prorated bandwidth credit..