LA Data Center Routing Issue

Start: 
2011.05.31 1:10 PM ET
End: 
2011.05.31 1:40 PM ET

Some OnSIP customers experienced a loss of service on Tuesday 5/31/11 from approximately 1:10 pm until approximately 1:40pm. This loss of service included the inability to initiate an outbound call or to properly answer an inbound call. It was soon determined that the service interruption was do to a UUNet (Verizon) routing issue inside their network between LA and NY. Verizon networks were unable to reliably route traffic to large blocks of IP address, which contained OnSIP IP blocks.

In response to this issue, the OnSIP Engineering team routed our internal traffic around Verizon networks as OnSIP has real-time traffic that runs over the UUNet network between our LA and NYC data centers. In addition, we directed all users to the NY datacenter. We stopped using UUNet in LA, and the traffic was routed to the next best route. Service for non-Verizon customers began to return at this point - about 1:25 pm. Verizon customers may have continued to experience issues with both in-bound and outbound service until the the issue was resolved within the Verizon network by the 4:30pm time-frame. At all times and for all customers, the OnSIP service was up and inbound calls that went to either an Attendant Menu or voicemail were handled without issue or loss of call.