Maikiwi Global Network - Asia-Pacific Regional Outage – Incident details

All systems operational

Asia-Pacific Regional Outage

Resolved
Major outage
Started 3 months agoLasted about 4 hours

Affected

Maikiwi (Services)

Partial outage from 8:21 AM to 8:25 AM, Operational from 8:21 AM to 9:17 AM, Partial outage from 8:38 AM to 9:17 AM, Under maintenance from 9:17 AM to 9:45 AM, Partial outage from 9:45 AM to 10:04 AM, Degraded performance from 10:04 AM to 10:19 AM, Under maintenance from 10:04 AM to 10:19 AM, Operational from 10:19 AM to 12:19 PM

Aria

Operational from 8:21 AM to 10:04 AM, Degraded performance from 10:04 AM to 10:19 AM, Operational from 10:19 AM to 12:19 PM

Route Propagation

Operational from 8:21 AM to 10:04 AM, Degraded performance from 10:04 AM to 10:19 AM, Operational from 10:19 AM to 12:19 PM

Caesar Monitoring

Partial outage from 8:21 AM to 8:25 AM, Operational from 8:21 AM to 9:17 AM, Partial outage from 8:38 AM to 9:17 AM, Under maintenance from 9:17 AM to 9:45 AM, Partial outage from 9:45 AM to 10:04 AM, Under maintenance from 10:04 AM to 10:19 AM, Operational from 10:19 AM to 12:19 PM

Event Digest (Ingress)

Partial outage from 8:38 AM to 9:17 AM, Under maintenance from 9:17 AM to 9:45 AM, Partial outage from 9:45 AM to 10:04 AM, Under maintenance from 10:04 AM to 10:19 AM, Operational from 10:19 AM to 12:19 PM

Event Aggregator (Ingress)

Partial outage from 8:38 AM to 9:17 AM, Under maintenance from 9:17 AM to 9:45 AM, Partial outage from 9:45 AM to 10:04 AM, Under maintenance from 10:04 AM to 10:19 AM, Operational from 10:19 AM to 12:19 PM

Updates
  • Resolved
    Resolved

    All outages have been restored. Please contact us as soon as possible if you deployed any custom workload with us in Osaka, otherwise, all managed services in Osaka will be seamlessly redirected to Tokyo.

  • Identified
    Identified

    We identified a repeatedly tripped internal failsafe that has been adding bogus routes to Aria's global routing table. Expect to receive inconsistent routes in the meantime as we patch Aria on-the-fly.

  • Monitoring
    Update

    We have brought all locations online except for Osaka which will be subjected to an extended maintenance period. We are also enabling a small portion of customers to interface with Caesar again to contain potential cascading failures.

  • Monitoring
    Update

    We rolled back a recent update to internal critical infrastructure in Hong Kong and Singapore and are proactively monitoring the result. Caesar Monitoring will be disabled globally in this moment and time.

  • Monitoring
    Monitoring

    We implemented a fix for Tokyo and are incrementally routing in traffic again.

  • Identified
    Update

    We have identified an additional outage in Singapore.

  • Identified
    Identified

    We are commencing a cold restart of all services in Japan and removing routes destined for Tokyo in Hong Kong.

  • Investigating
    Update

    We are currently investigating a simultaneous regional outage.

  • Investigating
    Investigating
    We are currently investigating this incident.