Identified - Summary

In recent days, there has been significant media coverage concerning two published vulnerabilities concerning modern CPU architectures which are being referred to as Spectre and Meltdown.

These vulnerabilities break isolation between applications and operating system kernels (Meltdown) and between applications themselves (Spectre) which is essential in protecting sensitive information from unauthorised exposure.

While these vulnerabilities are the result of CPU design flaws, they must be mitigated through software.

What is Zettagrid doing?

Since becoming aware of the vulnerabilities, Zettagrid has been assessing the risk and exposure of Spectre and Meltdown to our infrastructure and working with vendors to ensure that appropriate mitigations are in place.

Remediation is ongoing and some services will require some scheduled maintenance to apply applicable patching and customers will be notified trough the Zettagrid Status Page.

What do customers need to do?

It’s important to understand that while patching is necessary at an infrastructure level, customers must patch their operating systems and in some cases, applications to ensure adequate protection.

Patches have, or soon will be released by major software vendors to help mitigate the vulnerability.

More Information

Up to date information on Spectre and Meltdown including links to statements and mitigation resources from major vendors is available from https://meltdownattack.com/.

Zettagrid continues to work with hardware and software vendors to help ensure mitigation and will post further updates as they become available.
Jan 8, 09:04 AWST

About This Site

This page is updated within any scheduled maintenance and unexpected disruptions to Zettagrid, ZettaNet and Simtex.

Global ? Operational
Operations Centre   ? Operational
Internet Transit   ? Operational
InterCapital Transit and MPLS   ? Operational
DNS   ? Operational
MyAccount Portal (Australia)   ? Operational
MyAccount Portal (Indonesia)   ? Operational
Australia - New South Wales ? Operational
Virtual Server (VPS)   ? Operational
Virtual Data Centre (VDC)   ? Operational
SecondSite   ? Operational
ImageArchive   ? Operational
Veeam Backup and Replication   ? Operational
VAST Object Storage   ? Operational
VoIP and Fax   ? Operational
Ethernet (Fibre/Midband)   ? Operational
Broadband   ? Operational
Data Centre Facilities   ? Operational
Australia - Queensland ? Operational
VoIP and Fax   ? Operational
Ethernet (Fibre/Midband)   ? Operational
Broadband   ? Operational
Data Centre Facilities   ? Operational
Australia - Victoria ? Operational
Virtual Server (VPS)   ? Operational
Virtual Data Centre (VDC)   ? Operational
SecondSite   ? Operational
ImageArchive   ? Operational
Veeam Backup and Replication   ? Operational
VAST Object Storage   ? Operational
VoIP and Fax   ? Operational
Ethernet (Fibre/Midband)   ? Operational
Broadband   ? Operational
Data Centre Facilities   ? Operational
Australia - Western Australia ? Operational
Virtual Server (VPS)   ? Operational
Virtual Data Centre (VDC)   ? Operational
SecondSite   ? Operational
ImageArchive   ? Operational
Veeam Backup and Replication   ? Operational
VAST Object Storage   ? Operational
VoIP and Fax   ? Operational
Ethernet (Fibre/Midband)   ? Operational
Broadband   ? Operational
Hosted Websites   ? Operational
Hosted Email   ? Operational
Data Centre Facilties   ? Operational
Indonesia - Jakarta ? Operational
Virtual Data Centre (VDC)   ? Operational
Virtual Server (VPS)   ? Operational
Veeam Backup and Replication   ? Operational
ImageArchive   ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Engineers are performing upgrades to the Perth Availability Zone. As a result, Management Access will be unavailable for no more than 4 hours during the maintenance window.

Customer virtual machines will not be affected by this work and are expected to remain fully operational.

The service impact is as follows:
- vCloud Management Access - unavailable
- Console Access - unavailable
- Provisioning of new services - expect delays
- Veeam Backup (image Archive) - no management & jobs paused
- Veeam Replication - no management & jobs paused
- Second Site replication will be paused during the window and will continue once completed

Note: All backup and replication jobs will be paused and will resume once the maintenance is complete.
Posted on Jan 19, 12:03 AWST
Scheduled - Engineers are performing upgrades to the Jakarta Availability Zone. As a result, Management Access will be unavailable for no more than 4 hours during the maintenance window.

Customer virtual machines will not be affected by this work and are expected to remain fully operational.

The service impact is as follows:
- vCloud Management Access - unavailable
- Console Access - unavailable
- Provisioning of new services - expect delays
- Veeam Backup (image Archive) - no management & jobs paused
- Veeam Replication - no management & jobs paused

Note: All backup and replication jobs will be paused and will resume once the maintenance is complete.
Posted on Jan 19, 16:25 AWST
Perth Data Centre VDC-PER09(DC2) will be undergoing Generator Transfer Test in order to conduct maintenance.

The data centre will run on generator power for up to three hours during the change window. The UPS will continue to maintain the protected power load at all times during the scheduled window.

Please note that this is a hazard notification only and there is no expected impact to services whilst this work is in progress.

The data centre has advised that during this works Data Centre engineers will be onsite monitoring all services.
Posted on Jan 19, 14:35 AWST
Sydney Data Centre VDC-SYD01 will be performing upgrades to UPS Systems between 30/1/18 and the 16/2/18 to further improve reliability and efficiency of the system.

Periodically, the UPS units will be placed into bypass as works occur. During these times, the site will remain protected by the onsite generators.

Please note that this is a hazard notification only and there is no expected impact to services whilst this work is in progress.

The data centre has advised that during these works Data Centre engineers will be onsite monitoring all services.
Posted on Jan 19, 16:48 AWST
Perth Data Centre VDC-PER09(DC2) will be undergoing routine preventative maintenance on UPS.

The UPS will continue to maintain the protected power load at all times during the scheduled window.

Please note that this is a hazard notification only and there is no expected impact to services whilst this work is in progress.

The data centre has advised that during this works Data Centre engineers will be onsite monitoring all services.
Posted on Jan 19, 16:58 AWST
System Metrics Month Week Day
Internet Transit Utilisation - Download ?
Fetching
Internet Transit Utilisation - Upload ?
Fetching
Broadband Backhaul Utilisation - Download ?
Fetching
Broadband Backhaul Utilisation - Upload ?
Fetching
Peering Transit Utilisation - Download ?
Fetching
Peering Transit Utilisation - Upload ?
Fetching
Past Incidents
Jan 19, 2018

No incidents reported today.

Jan 18, 2018
Completed - Zettagrid Engineers have identified an issue which will result in the rescheduling of these maintenance works to the coming week.
Updates will be posted once a definitive time is decided
Jan 18, 16:06 AWST
Scheduled - Engineers are performing upgrades to the Perth Availability Zone. As a result, Management Access will be unavailable for no more than 4 hours during the maintenance window.

Customer virtual machines will not be affected by this work and are expected to remain fully operational.

The service impact is as follows:
- vCloud Management Access - unavailable
- Console Access - unavailable
- Provisioning of new services - expect delays
- Veeam Backup (image Archive) - no management & jobs paused
- Veeam Replication - no management & jobs paused
- Second Site replication will be paused during the window and will continue once completed

Note: All backup and replication jobs will be paused and will resume once the maintenance is complete.
Jan 12, 15:43 AWST
Jan 17, 2018
Completed - The scheduled maintenance has been completed.
Jan 17, 19:00 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 17, 15:00 AWST
Scheduled - Engineers are performing essential upgrades to the Sydney Availability Zone. As a result, Management Access will be unavailable for no more than 4 hours during the maintenance window.

Customer virtual machines will not be affected by this work and are expected to remain fully operational.

The service impact is as follows:
- vCloud Management Access - unavailable
- Console Access - unavailable
- Provisioning of new services - expect delays
- Veeam Backup (image Archive) - no management & jobs paused
- Veeam Replication - no management & jobs paused
- Second Site replication will be paused during the window and will continue once completed

Note: All backup and replication jobs will be paused and will resume once the maintenance is complete.
Jan 12, 15:42 AWST
Jan 16, 2018
Completed - Maintenance has been completed
Jan 16, 17:18 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 16, 15:00 AWST
Scheduled - Engineers are performing essential upgrades to the Melbourne Availability Zone. As a result, Management Access will be unavailable for no more than 4 hours during the maintenance window.

Customer virtual machines will not be affected by this work and are expected to remain fully operational.

The service impact is as follows:
- vCloud Management Access - unavailable
- Console Access - unavailable
- Provisioning of new services - expect delays
- Veeam Backup (image Archive) - no management & jobs paused
- Veeam Replication - no management & jobs paused
- Second Site replication will be paused during the window and will continue once completed

Note: All backup and replication jobs will be paused and will resume once the maintenance is complete.
Jan 12, 15:41 AWST
Jan 15, 2018
Resolved - Mains power has been fully restored to the data centre.
Jan 15, 16:58 AWST
Identified - Perth data centre VDC-PER09 (a.k.a DC2) is presently operating on generator power due a Western Power mains fault in the vicinity.
All Zettagrid and Zettanet services remain fully operational.

The site is running as per design and Western Power restoration is anticipated by 4pm AWST (UTC+8)
Jan 15, 13:44 AWST
Jan 14, 2018

No incidents reported.

Jan 13, 2018

No incidents reported.

Jan 12, 2018
Completed - The scheduled maintenance has been completed.
Jan 12, 13:01 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 07:30 AWST
Scheduled - Additional maintenance is required to apply critical software patching and updates to ZettaNet website hosting platforms.

NOTE: If your website was disrupted during maintenance performed earlier this week, your website will NOT be affected in this scheduled maintenance window.

This maintenance is related to the Spectre and Meltdown Vulnerabilities notice posted earlier this week at http://status.zettagrid.com/incidents/92nmvxs9m4ll

At some stage during the scheduled window your ZettaNet hosted website will be unavailable for no more than 5 minutes, as servers and services are restarted.
The scheduled maintenance window:
Starts: 7:30am (AWST -UTC+8) on Friday 12th January 2018
Ends: 1:00pm (AWST -UTC+8) on Friday 12th January 2018

Upon conclusion, services are expected to automatically restore to normal operational status.

If your services do not fully restore or you have questions about this works, contact the Operations Centre on 1300 139 550 or support@zetta.net.au quoting change number CHG0033008.
Jan 11, 17:30 AWST
Completed - The scheduled maintenance has been completed.
Jan 12, 09:00 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 07:30 AWST
Scheduled - Engineers are performing critical security patch updates to our ( Zetta group) website and services.

During this scheduled maintenance window, the below listed websites and services will be unavailable for a period of no more than 5 minutes each.
- www.zetta.com.au
- www.zetta.net.au
- account.zettagrid.com
- payment.zettagrid.com
- www.simtex.com.au

This maintenance is related to the Spectre and Meltdown Vulnerabilities notice posted earlier this week at http://status.zettagrid.com/incidents/92nmvxs9m4ll

The scheduled maintenance window:
- Starts: 7:30 am (AWST -UTC+8) on Friday 12th January 2018
- Ends: 9:00 am (AWST -UTC+8) on Friday 12th January 2018

Upon conclusion of these works all services will be restored to normal operational status.
Jan 11, 18:09 AWST
Jan 11, 2018
Completed - This maintenance has been completed
Jan 11, 17:20 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 11, 07:30 AWST
Scheduled - Engineers are performing critical software patching and updates to ZettaNet Hosted Email and Webmail services.
This maintenance is related to the Spectre and Meltdown Vulnerabilities notice posted earlier this week http://status.zettagrid.com/incidents/92nmvxs9m4ll

As part of this works, all Email and Webmail services will be restarted and will be unavailable for no more than 15 minutes.
During this restart:
- All inbound email, sent to you, will be queued and delivered once service is restored
- Any attempt to send email (outbound) may fail and will need to be resent once service is restored.
- Your email client (Outlook, Apple Mail) may request that you re-enter your account details, if you receive this message we suggest you wait for 15 minutes then restart the application as your client should reconnect once service is restored.

This affects all emails services provided by ZettaNet including any customer domain hosted email.

If your services do not fully restore or you have questions about these works, contact the Operations Centre on 1300 139 550 or support@zetta.net.au quoting change number CHG0032997.
Jan 9, 16:13 AWST
Completed - The scheduled maintenance has been completed.
Jan 11, 13:00 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 10, 07:31 AWST
Scheduled - Engineers are performing critical software patching and updates to ZettaNet website hosting platforms.
This maintenance is related to the Spectre and Meltdown Vulnerabilities notice posted earlier this week at http://status.zettagrid.com/incidents/92nmvxs9m4ll

At some stage during the scheduled window your ZettaNet hosted website will be unavailable for no more than 5 minutes, as servers and services are restarted.

Upon conclusion, services are expected to automatically restore to normal operational status.

If your services do not fully restore or you have questions about this works, contact the Operations Centre on 1300 139 550 or support@zetta.net.au quoting change number CHG0032995.
Jan 9, 15:16 AWST
Resolved - The issue has been resolved and all VMs have now powered up on alternate hosts.
Jan 11, 12:56 AWST
Investigating - Engineers have been alerted to a host failure in our Sydney zone. The host has been removed from production and any affected VMs are rebooting on alternate hosts as per HA design.

If your machine does not respond in the next 10 - 15 minutes or you are having issues, please contact support on 1300 597 656.
Jan 11, 06:46 AWST
Completed - The scheduled maintenance has been completed.
Jan 11, 03:00 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 10, 22:00 AWST
Scheduled - Due to a planned maintenance being performed by our upstream provider, service(s) may experience a 10 minute break during the change window.

If the service(s) fail to restore after this interruption, please report the fault to our Operations Centre on 1300 139 550 or email to support@zetta.net.au

NOTE: This only affects "Telstra" Fibre/ Metro Ethernet service(s) in Melbourne
Jan 2, 11:29 AWST
Jan 10, 2018
Completed - The scheduled maintenance has been completed.
Jan 10, 16:00 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 10, 10:00 AWST
Scheduled - Zettagrid engineers will be upgrading all Veeam instances to version 9.5 update 3.

This maintenance affects the following services, all zones:
- VM Backup (a.k.a ImageArchive)
- Veeam Backup (a.k.a Cloud Connect)
- Veeam Replication (a.k.a Veeam Cloud Connect Replication)

Any jobs running at the start of the maintenance window will be paused and will resume upon completion of the works.
Any jobs scheduled to start, during the maintenance, window will be disabled and will resume according to the next scheduled job.

For Veeam Backup and Veeam Replication, customers should not attempt to upgrade to 9.5 update 3, until this maintenance works has been completed by Zettagrid.

If you did attempt to upgrade before Zettagrid, you will receive warning notices advising that your service provider is running outdated software with a further recommendation to postpone your upgrade; if you did then choose to proceed with your upgrade Zettagrid Veeam services will stop working.

If you have any questions about this works, contact the Operations Centre on 1300 597 656 or support@zettagrid.com quoting change reference CHG0032914.
Dec 13, 12:36 AWST
Jan 9, 2018

No incidents reported.

Jan 7, 2018

No incidents reported.

Jan 6, 2018

No incidents reported.

Jan 5, 2018
Resolved - This incident has been resolved.
Jan 5, 12:43 AWST
Monitoring - Zettagrid Engineers have Identified the issue all affected machines have rebooted on other hosts as part of HA.

If issues continue please contact Zettagrid support
Jan 5, 12:00 AWST
Investigating - Zettagrid Engineers have been alerted to a single host failure within the Perth availability zone.

Engineers are investigating and will update accordingly
Jan 5, 11:45 AWST