Network Disruption

Start: 
2011.05.27 10:45 AM ET
End: 
2011.05.27 12:30 PM ET

At 10:45 AM ET, we began tracking a network issue in our NYC data center that affected about 25% of customers.

At 11:15 AM, we diagnosed the problem within the NYC data center: One of the BGP peers that connects to one of our proxies was flapping, resulting in unreliable data transmission. This affected users were those registered to one of the New York proxies. At that time, we began moving all customers to the other NYC proxy via a new DNS entry. We reported the root cause and the efforts at 11:26 AM.

At 11:45 AM ET, services began restoring for customers. At 12:30 PM, the issue was confirmed to be resolved for all customers.