Resolved

We've now resolved the incident. Thanks for your patience. Full RFO follows.

At 12:03 on the 5th of December one of Zettagrid's core routers experienced a cascading event. All software processes running within the routing engine of this router spontaneously started to crash and automatically restart. While this happened, core services failed over to the secondary core router as designed. As the router recovered, it started to regain control of services, before then crashing again

This resulted in a full outage for services going through both Perth core routers, as they both fought for control, also making it difficult for engineers to identify where and what the underlying cause of the outage was.

Once identified, the core router causing the issue was taken offline, with most services then failing over correctly to the other router quickly and back online around 1:15pm.

Some Metro Ethernet and NBN services continued to have issues until 3pm, due to issues caused by the original event.

Whilst the exact cause of the issue is yet to be identified, the faulty device has been brought back into operations after an upgrade to the latest recommended release from the vendor, restoring Zettagrid's Perth network to full capacity with expected fault tolerance.

Recovering

We've resolved the core issue, and customers should be back online.
If you are still experiencing issues please restart your onsite equiment or clear ARP tables for the gateway IP then if you are still not connected contact support - 1300 597 656 or https://support.zettagrid.com/

Identified

We've resolved the core issue, and most customers should be back online. Some customers may need to clear ARP tables for the gateway IP, or reboot onsite CPE if services do not restore. Some Perth ethernet customers are still offline and some Perth NBN customers are having performance issues - we have identified the issue and are working to resolve this. If other customers re still experiencing issues please restart your onsite equiment or clear ARP tables for the gateway IP then if still not working contact support - 1300 597 656 or https://support.zettagrid.com/

Recovering

We've resolved the core issue, and customers should be back online.
Some customers may need to clear ARP tables for the gateway IP, or reboot onsite CPE if services do not restore. If you are still experiencing issues please contact support - 1300 597 656 or https://support.zettagrid.com/

Updated

Engineers are continuing to investigate the issue and working to resolve connectivity. Further updates to be provided as soon as available.

Investigating

We've been alerted to network connectivity issues affecting the Perth zone. Investigation is currently underway. Updates will be provided as soon as possible.

Began at:

Affected components
  • AUS - Western Australia
    • Compute
    • Colocation
    • Internet Transit
    • Ethernet
    • Mobile Broadband
    • NBN
    • Simtex VoIP and Fax
    • Veeam Services
    • VMware Management & VMware Replication
    • Zerto Services