Resolved
After 4 days, 2 hours, and 26 minutes

The vendor has supplied a replacement switch. Engineers have attended the data centre and replaced the switch. All services have been tested and working as expected.

Recovering
After 1 hour and 24 minutes

We've fixed the core issue, and are waiting for things to recover. VMs should now be available. Technicans identified the issue as a partially failed switch which did not trigger a full fail-over to the second switch. The affected switch was removed from path and internet connectivity has been restored to the affected services. The failed switch will be taken offline, logs reviewed and root caused investigated.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 23 minutes

Technicians are attending the Perth data centre. We will provide an update at 8:15pm (WST).

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 7 minutes

This incident will also affect some Simtex calling as some of the VoIP servers are affected by this issue.

Investigating

Selected virtual servers are experiencing network connectivity issues on some hosts preventing access to the internet. The team are currently investigating and will update as soon as possible.

Began at:

Affected components
  • AUS - Perth
    • Virtual Data Centre (VDC)
    • Virtual Server (VPS)
    • ImageArchive & VDC Backup
    • Veeam Backup
    • Disaster Recovery Replication (VMware, Veeam, Zerto)
    • Simtex VoIP and Fax