Slow Response Times in Virtual Network Layer
Incident Report for Pagoda Box

Pagoda Box lives on top of a custom-built provisioning system used to manage the underlying Pagoda Box infrastructure. A process running in this layer starved "Admin" processes of memory, causing the virtual network manager to "flap". This resulted in connections between users' apps' services to flap as well. Engineers killed the process consuming abnormal amounts of memory which restored all admin processes and normalized connections between apps' services.

All affected apps should now be back to normal. If your app is still unable to connect to services, please submit a ticket.

Posted almost 2 years ago. May 27, 2015 - 22:31 UTC

Resolved
Engineers identified a backend process that was consuming abnormal amounts of memory and starving the virtual network manager. They killed the process consuming the memory which normalized connections between apps' services.

All affected apps should now be back to normal. If your app is still unable to connect to services, please submit a ticket.
Posted almost 2 years ago. May 27, 2015 - 22:29 UTC
Update
The virtualized network has returned to normal but some apps may still be having issue connecting to services. Rebuilding affected services should resolve any connection problems.
Posted almost 2 years ago. May 27, 2015 - 21:48 UTC
Identified
Engineers have identified an issue with the virtualized network layer that manages network connections between servers and services.
Posted almost 2 years ago. May 27, 2015 - 21:36 UTC
Investigating
Engineers are investigating slow response times on a public hypervisor. Some user apps may be affected.
Posted almost 2 years ago. May 27, 2015 - 21:28 UTC