Balancing Connections Over Multiple Links

Tim Utschig <tim@tetro.net>

Contents

General Idea

Say you have access to multiple links to the Internet, such as several wireless networks in range. Wouldn't it be nice to combine all that bandwidth into one big fat pipe?

Unfortunately it's not so easy. You can't just trunk them together because they each have a different public IP address, gateway, etc.

What you can do however, thanks to some nifty Linux NetFilter extensions, is assign outgoing connections to different interfaces. This will allow protocols such as BitTorrent to utilize bandwidth from each of the links.

This document focuses on Linux iptables/NetFilter. You can achieve pretty much the same result with Linux Advanced Routing techniques. One small difference, as the link mentions, is that routes are cached, so connections to frequently used sites will always go over the same link. This may or may not be the behaviour you desire.

Prerequisites

You need a recent Linux kernel patched with support for the ROUTE target and either the "nth" or "random" match module. These patches are available in NetFilter's "patch-o-matic-ng" subversion module. I won't go into how to apply the patches, as more than sufficient documentation is included with them.

Testing I did was on Linux 2.6.14.2 patched with a copy of patch-o-matic-ng checked out with svn on 2005-11-18.

Setup

In the following examples, I use three interfaces: I use the connmark match/target to assign each connection to an interface, and make sure all the packets for the connection go over that one interface. Balancing the connections over the interfaces can be done with either "random" or "nth" match module. I will give you both examples, choose which ever one you prefer. The following commands are common to both methods.

Common commands:

# prevent incoming packets on masqueraded connections from being dropped 
# as "martians" due to the destination address being translated before the
# rp_filter check is performed
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/rausb0/rp_filter

# Load protocol-specific connection tracking modules so that new connections
# associated with existing connections have state "RELATED" and inherit the
# same connmark.
modprobe ip_conntrack_ftp

# masquerade outgoing connections on secondary interfaces
iptables -t nat -A POSTROUTING -o eth1   -s ! 172.16.0.0/16  -m state --state NEW,RELATED -j MASQUERADE
iptables -t nat -A POSTROUTING -o rausb0 -s ! 192.168.0.0/24 -m state --state NEW,RELATED -j MASQUERADE

# create a chain for processing new outgoing connetions
iptables -t mangle -N NEW_OUT_CONN

# Skip connections we want to always go out wired interface
iptables -t mangle -A NEW_OUT_CONN -d 192.168.1.0/24 -j RETURN
iptables -t mangle -A NEW_OUT_CONN -p tcp -m multiport --destination-ports 21,22,80,443,6667 -j RETURN
iptables -t mangle -A NEW_OUT_CONN -p udp --dport 53 -j RETURN

# have new outgoing connections pass through the above chain
iptables -t mangle -A OUTPUT -o eth0 -m state --state NEW -j NEW_OUT_CONN

# send packets out chosen interface
iptables -t mangle -A OUTPUT -m connmark --mark 2 -j ROUTE --gw 172.16.0.1 --continue
iptables -t mangle -A OUTPUT -m connmark --mark 3 -j ROUTE --gw 192.168.0.1 --continue

The "random" method:

# 34% of the time go out the default interface
iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 0
iptables -t mangle -A NEW_OUT_CONN -m random --average 34 -j RETURN

# 33% of the time go out eth1 (50% of the remaining probability)
iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 2
iptables -t mangle -A NEW_OUT_CONN -m random --average 50 -j RETURN

# else (hopefully 33% of the time) go out rausb0
iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 3

The "nth" method:

# 1st of every 3 connections goes out the default interface
iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 0
iptables -t mangle -A NEW_OUT_CONN -m nth --counter 1 --every 3 --packet 0 -j RETURN

# 2nd of every 3 connections goes out eth1
iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 2
iptables -t mangle -A NEW_OUT_CONN -m nth --counter 1 --every 3 --packet 1 -j RETURN

# 3rd of every 3 connections goes out rausb0
iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 3
iptables -t mangle -A NEW_OUT_CONN -m nth --counter 1 --every 3 --packet 2 -j RETURN

Handling when an interface goes down:

This script will make sure no packets get routed over a secondary interface that has gone down. Put it in your /etc/network/if-down.d/ (Debian), or equivalent, directory and chmod +x it.

#!/bin/sh

if [ "$IFACE" = "eth1" ]; then
  iptables -t mangle -D OUTPUT -m connmark --mark 2 -j ROUTE --gw 172.16.0.1 --continue 2>/dev/null
fi

if [ "$IFACE" = "rausb0" ]; then
  iptables -t mangle -D OUTPUT -m connmark --mark 3 -j ROUTE --gw 192.168.0.1 --continue 2>/dev/null
fi

exit 0

Handling when an interface comes back up:

This script will allow an interface to be used again when it comes back up. Put it in your /etc/network/if-up.d/ (Debian), or equivalent, directory and chmod +x it.

#!/bin/sh

if [ "$IFACE" = "eth1" ]; then
  iptables -t mangle -A OUTPUT -m connmark --mark 2 -j ROUTE --gw 172.16.0.1 --continue 2>/dev/null
fi

if [ "$IFACE" = "rausb0" ]; then
  iptables -t mangle -A OUTPUT -m connmark --mark 3 -j ROUTE --gw 192.168.0.1 --continue 2>/dev/null
fi

exit 0

Results

Screenshot of BitTornado
Not too shaby I think. Normally with my single DSL connection alone I get somewhere around 150 KB/s. Maybe if there's a smarter way to distribute connections this could be improved upon. Optimally my 3 test links combined would add up to 450 KB/s.

TODO

ChangeLog

Mon Jan 2 05:43:47 PST 2006
Michael Heimpold pointed out that --average 33 was wrong for the second -m random rule.

Fri Jan 13 09:37:12 PST 2006
Michael Heimpold figured out that RELATED connections (set as such by modules like ip_conntrack_ftp) inherit the same connmark. Changed the masquerading rules to also match RELATED packets. Now passive FTP works reliably.


Last Modified: Fri Jan 13 09:37:12 PST 2006