Recently, 20th century software pioneer Jim Gettys has been doing a bang-up job raising awareness about performance problems with the internet known as “buffer-bloat.” The details are technical and complex but the gist of it is that networking equipment is often buffering way too much data, resulting in unnecessarily long latencies. High latencies (literally delays) result in unpleasant experiences when using a network as things take a long time. It’s important to recognize that even if your network’s bandwidth is extremely high, a long latency will make it feel very slow — the two measures of network speed are somewhat related, but mostly independent.
The simple way to counter buffer bloat is simply to reduce the size of the transmit buffer in each piece of your network gear. Most linux systems default to a transmit buffer of 1,000 packets, each of which can be 1.5 kilobytes, meaning that 1.5 megabytes of data can get queued up waiting for a chance to go across the network. Any application that is trying to move a lot of data through a clogged network will fill this buffer. That’s fine for the buffer-filling application, but any other application will suffer. So, for example, if you’re watching youtube and your roommate is trying to surf the net, your roommate’s web page requests will suffer very long latency, because their small web pages must get in this megabyte-long line along with your youtube video before they can be delivered. If your DSL line runs at say 10 mbps, then it’ll take 1.2 seconds for that 1.5 MB buffer to work fit through your pipe. Since it takes at least 2 round-trips to get a web page that means your roommate’s web page will take at least 2.4 seconds to show up, no matter how small it is!
Gettys quotes Kleinrock that the ideal size of a network buffer is (bandwidth) x (latency). Say your bandwidth is 10 mbps. Latency to any web page you’re likely to visit in the US should be less than 100ms, so let’s use that. This puts your ideal buffer size 125k. Buffer sizes are usually configured in terms of maximum number of packets. Typically the maximum packet size (MTU) is 1500 bytes, resulting in the ideal theoretical buffer size of 83 packets for a typical fastish home network line. Please redo these calculations yourself and experiment with how different numbers affect your system. (Be careful not to set your buffer size to zero as it could lock up the device’s network.) Remember that linux (which is likely what your wifi router is running) defaults to 1,000 packets!
[Update shortly after posting: a reader suggested I try setting my buffer to be much smaller still. So I went down to just 2 packets, and noticed that my ping times are much more reliable now when the network has more than one thing. His caveat which I will echo is that this will mess with your system if your router is trying to do any kind of traffic shaping, i.e. QoS. But otherwise protocols like TCP will keep everything running fine.]
If your home wifi access points are using DD-WRT as mine are, here’s how you set them to use a more sane buffer size:
1. Log in to your router’s admin web page.
2. Select the Administration tab and the Commands sub-tab
3. Type in the following commands into the box:
ifconfig eth0 txqueuelen 2
ifconfig eth1 txqueuelen 2
4. Click the “Save Startup” button at the bottom.
There — you’re done! For alternate techniques to configure your dd-wrt router for this kind of thing, see the wiki page on Startup Scripts.
I’m sharing this information because it took me a while to figure out. This problem is not well documented. I’m trying this out now on my house’s network now. In some controlled tests it seems like it might be somewhat better. But my tests have not been able to replicate the really horrible situations I’ve seen on our network which I suspect come from lots of simultaneous users. So it’ll be a while before we know for sure if this was a good change. To be clear, I don’t know if this advice is good or not. It could reduce your network’s maximum effective bandwidth, but hopefully it will do so by reducing the maximum latency, which is often a very good trade-off. This advice is consistent with the advice Gettys offers in terms of optimizing buffer sizes, and make sense to me. YMMV. If you try it out, please leave a comment on whether or not it helps you.