I moved houses over the weekend. Well, from an apartment to a house, anyway. Rather than transfer my existing service with Comcast over, I decided to treat myself and get 1 Gbps fiber from Century Link.
It cost me a couple hundred up front, and will probably be $109/month continuing. Not bad considering I was paying $80 for 120mbps via Comcast. Plus, having gigabit to the home is exciting.
There’s a catch, though. Connections to certain companies seem to be selectively slowed. We’ve noticed major problems connecting to both Steam, and Netflix.
So naturally, I spent an entire day trying to track down why.
Let’s do some speed tests!
First, fast.com from Netflix. This test is performed against the same infrasturcture that Netflix uses to stream videos.
And we get 10Mbps. Ten. Fun fact, 10Mbps is 1% of the speed I pay for through CenturyLink. I watched Tallulah last night on Netflix, and it was dead slow. I can definitely confirm from all the distortion and compression artifacts that 10Mbps is about what I was getting.
Now let’s head to speedtest.net.
502Mbps down and 268.72Mbps up. That’s about what I’d expect out of a residential 1Gbps link, so the speed itself isn’t surprising. What is surprising is the disparity between the Netflix test at 10Mbps and the Ookla test at 502Mbps. Why in the world is there an order of magnitude difference?
The next logical step is to see if there’s some network problem that exists between the router at my house and the servers at Netflix.
Since I already have Little Snitch handy, I’ll use that to grab the hostname of the fast.com server I’m connecting to.
ipv4_1-lagg0-c053.1.lax004.ix.nflxvideo.net. Neat. Let’s traceroute.
Nothing seems particularly out of line. No loss, no jitter; looks fine. So let’s ping all of the hosts mentioned in the trace to see if any of them feel overloaded. This won’t prove anything on its own, but it might give us a clue of where to go next.
Ping, traceroute, and mtr (the tool we’re about to break out) all use the Internet Control Message Protocol, or ICMP, messages to gather data. I could write a whole article on why dropped ICMPs aren’t a big deal, but here I’ll just say this: dropped pings are virtually meaningless on their own. The best they can do is point you towards your next troubleshooting step. Now that’s out of the way, let’s keep troubleshooting:
HERE is something to look at! dvr-brdr-02.inet.qwest.net looks angry. It’s consistently dropping quite a bit of icmp traffic. Let’s look at that guy a little closer. Am I always routed through that server, or is it just netflix traffic?
Looks like we’re not always routed through dvr-brdr-02, but a lot of the time we are. The most important trace is the one to speedtest.xmission.com.
See, we know that we can pull 502Mbps from speedtest.xmission.com as tested through speedtest.net. With that trace, we also know that we’re pulling that 502Mbps through our friend dvr-brdr-02. Since we care more about traffic through than to, we can rule out dvr-brdr-02 as a possible suspect. Though, that hop might be a tad overloaded.
So… What the hell? Netflix is still slow, and there doesn’t appear to be any real network issue between us. Well, let’s try a VPN.
Is our connection to Netflix still slow if it’s over a VPN? I subscribe to Goldenfrog’s VyprVPN, so let’s use that.
Wat. A four-fold speed increase. Not only that, but the speedtest.net results roughly match the fast.com results. This doesn’t conclusively prove anything, but it very strongly hints that CenturyLink is shaping based on destination. Bad bad bad.
Alright, let’s see what route we take to the vpn server.
There’s our old friend dvr-brdr-02 again. So we’re taking the same route out of Century Link’s network.
Now let’s see what route we take to the nflxvideo.net server when connected to VyprVPN and then compare it to our non-VPN’d trace.
It looks like either way we wind up on Cogent’s infrastructure with traffic destined to Netflix. If we were seeing destination based QoS/Throttling/Shaping from Cogent, we would probably see it over the VPN as well.
It’s not conclusive, but I doubt Cogent is throttling Netflix-destined traffic.
Well, the result is kind of muddy.
I don’t think Cogent or Netflix are responsible for the slow down. I think CenturyLink is responsible. It could be a misconfiguration, or some overloaded piece of gear, or some disgruntled engineer that hates Netflix, or it could be on purpose. I honestly don’t know, and I honestly can’t know because I don’t have access to CenturyLink’s network gear.
But why? What’s the motive? To sell more TV subscriptions?
Maybe… My guess is that they don’t have the capacity in either engineering or infrastructure to serve their clients at full speed.
It could also be the result of poor peering agreements. ¯\_(ツ)_/¯
In reality, there’s no way I can win a fight against an internet service provider. It doesn’t matter how many times I call technical support, or how many blog posts I write, or how ways I demonstrate the issue. CenturyLink will never admit fault, and they probably won’t fix the actual issue either.
So instead, I’m going to route traffic destined for Netflix over a persistent VPN link. It’s not optimal, and it’s really really shitty that I have to pay for an extra vpn service just to use the internet service I already pay for. Extremely shitty.
But… I guess that’s life.
I’m a bit worried about how many more of these I’m going to find. Am I going to have to route dropbox or Apple connections over the VPN too? At what point does it just make more sense to switch ISPs? I don’t know. We’ll see!
We’re just going to cancel and resubscribe to comcast at 250mbps. Having a real quarter gig is better than a whole fake gig. Google help us.
