DAL05 Datacenter Network Issue
Incident Report for Pagoda Box

On February 3, 2016 at approximately 5:08 UTC, a backend private router in SoftLayer's DAL05 datacenter was disrupted due to a hardware failure. The disruption prevented all servers behind the router from communicating across the internal network, some of which were Pagoda Box servers.

SoftLayer engineers were immediately notified of the interruption and began to assess and mitigate the issue. At 06:01 UTC engineers reloaded the router's redundant management modules and restored network connectivity.

Most Pagoda Box applications affected by the interruption came back online immediately. However, some apps' recoveries were delayed as our internal Virtual Network Overlay Manager synced changes that occurred during the network outage. Once the network information was synced, service to all Pagoda Box apps was restored.

Thank you for your patience as the network issue was addressed. We know these types of interruptions have significant effects on our users' sites and businesses. A special thanks goes to the SoftLayer engineers who were able to mitigate the hardware failure and restore service to the affected servers.

Posted almost 2 years ago. Feb 03, 2016 - 16:43 UTC

Resolved
All affected services are now back online. There is some queuing in the transaction engine causing some transactions to progress slowly. However they are still progressing.
Posted almost 2 years ago. Feb 03, 2016 - 07:31 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted almost 2 years ago. Feb 03, 2016 - 06:19 UTC
Update
SoftLayer engineers have resolved the issue with the backend router and apps should now be back online. If your apps are still inaccessible, please submit a ticket.
Posted almost 2 years ago. Feb 03, 2016 - 06:17 UTC
Update
Update from SoftLayer: "As of Wednesday 03-Feb-2016 06:01 UTC In an attempt to stabilize bcr04.dal05 Network Engineers are reloading its redundant management modules. If successful, network connectivity to affected customer hosts will be restored shortly."
Posted almost 2 years ago. Feb 03, 2016 - 06:03 UTC
Update
Update from SoftLayer: "As of 03-Feb-2016 5:08 UTC, back-end private network connectivity is disrupted for customer hosts behind the back-end customer router (BCR) bcr04.dal05. Network Engineers and Datacenter Staff are investigating the issue and are working to restore service to affected hosts as quickly as possible."
Posted almost 2 years ago. Feb 03, 2016 - 05:45 UTC
Identified
Confirmed. There is an internal networking issue in SoftLayers DAL05 datacenter. A backend router stopped responding. SoftLayer engineers are aware of the issue and are currently addressing it.
Posted almost 2 years ago. Feb 03, 2016 - 05:39 UTC
Investigating
There appears to be a networking issue in the DAL05 datacenter. Engineers are assessing now.
Posted almost 2 years ago. Feb 03, 2016 - 05:32 UTC