Detecting IP Packet Loss for UberNerds

I spent the good part of a couple of days last week helping track down packet loss at the edge of a PSTN carrier's IP network. They were dropping 1-5 packets out of every 10000 due to an incorrectly configured port on a gigabit ethernet switch. Now 0.05% packet loss doesn't seem like a lot, but at the end of the day it really depends on the phone you are using. If you are using a Polycom phone, you will likely never notice it. If you are using a Grandstream GXP-2000, the experience may be almost unbearable. It is my experience that the quality of IP phone software/hardware has by far the most significant effect on the end user's perception of call quality.

And then there is packet loss. And sometimes the two go hand in hand. Case in point is my hunt for missing packets last week. To review, the PSTN carrier in question was dropping 0.05% of the packets over the private interconnect my calls happened to be traveling. That's 5 out of 10000 packets. To put some context around that number, we need a bit more information. We were testing g711 at 20ms, so each packet contained 20ms of voice and thus we were sending 50 packets per second to carry one side of the voice conversation. That is, each packet represents 1/50 of a second of conversation and the carrier was losing up to 5 of these over 200 seconds. And those 5 lost packets were spread randomly across that 3+ minute period.

Now you might think someone would have a hard time hearing a 1/50 second gap in a 3 minute conversation. And you would be right. But if there is an IP network in the middle of the conversation, it turns out that drop detection depends greatly on the the piece of IP phone hardware/software one is talking on. In this case, the Polycom IP phone on my desk was useless as an empirical tool in helping me track down the 0.05% packet loss in question - I simply could not hear it. However, the Grandstream GXP-2000 (version we have in our test bed turned out to be an invaluable tool since it has the uncanny ability to turn the 1/50 second gap caused by a single lost packet into a multi-second garbled mess. So, if like me, you enjoy testing for dropped packets, I highly recommend adding the Grandstream to your toolbox - it is a great tool and worth far more than the retail price.