On February 3, 2016 at approximately 5:08 UTC, a backend private router in SoftLayer's DAL05 datacenter was disrupted due to a hardware failure. The disruption prevented all servers behind the router from communicating across the internal network, some of which were Pagoda Box servers.
SoftLayer engineers were immediately notified of the interruption and began to assess and mitigate the issue. At 06:01 UTC engineers reloaded the router's redundant management modules and restored network connectivity.
Most Pagoda Box applications affected by the interruption came back online immediately. However, some apps' recoveries were delayed as our internal Virtual Network Overlay Manager synced changes that occurred during the network outage. Once the network information was synced, service to all Pagoda Box apps was restored.
Thank you for your patience as the network issue was addressed. We know these types of interruptions have significant effects on our users' sites and businesses. A special thanks goes to the SoftLayer engineers who were able to mitigate the hardware failure and restore service to the affected servers.