How to set up source nat routing (SNAT) in iptables to load balance outbound connections across multiple IP addresses

Photo by BobMical

In our earlier days, before IOFlood, some of us ran web based proxy services. These were popular at workplaces or schools for accessing websites that were blocked there, such as gmail and myspace (remember myspace?). One common problem that came up, was that with so many users each sharing one IP on one server, our traffic pattern would not look legitimate, and the most popular websites for the proxy might throttle or block our connections. So we needed a solution to load balance outbound connections across many IPs, and the software we were using did not really support that. Although this use case may not apply to you, the solution we came up with is relevant in a wider variety of circumstances, so I’d like to share that with you today.

What we ended up doing to solve this problem, is we used the source nat routing (SNAT) feature of IPTables. This allowed  the source address for outbound connections to come from a pool of IPs instead of all connections coming from the server’s primary IP address. IPTables allows a lot of flexibility here, so you can make this outbound pooling apply to only certain ports, certain destinations, etc. This can be useful for example, if you want all of your outbound email to be sent from one IP, without needing to reconfigure your mail client to make that happen. Or if you want connections to one particular website to come from one IP, but other connections to behave as normal, this is also something you can do with SNAT and IPTables.

SNAT can work either with a single IP as the source, or as a pool of IPs that it can rotate between. If you are using a pool, the pool of IPs must be contiguous. Also, the primary IP of the server MUST NOT be in the pool, or else the rest of the IPs in the pool will never be used. Also keep in mind that SNAT will use IP connection tracking, so make sure that your ip conntrack table is configured to an appropriate size! Exhausting your IP connection tracking table can cause poor network performance and dropped connections, as we explained in our previous article: nf_conntrack: table full, dropping packet — A solution for Centos Dedicated Servers.

Without further ado, I will share the IPtables rules we use on our web based proxies to load balance outbound connections using SNAT:

Using SNAT to load balance outbound connections across multiple IPs:

For our example, I will assume that you want to use SNAT for a pool of multiple IPs, and that you want “to SNAT” any connection with a destination port TCP 80, TCP 443, TCP 53, or UDP 53. This will allow dns queries, http, and https traffic to be SNAT’ed, with other traffic being unchanged. You can of course modify this to suit your own purposes.

For the following examples, I will assume your IPs range from 10.0.0.1 (the server primary IP) to 10.0.0.20. Please note that we start our pool from 10.0.0.2 in this case, because you cannot have the server primary IP as part of a pool. Of course, edit the following commands to match the IPs actually assigned to your server. Finally, we assume your ethernet interface is “eth0”. If it is not, change the following commands appropriately:

First, you should kill the IPTables connection tracking database so that IPTables changes will actually take effect:

iptables -t nat -F

Next, the following commands will redirect outbound requests to TCP port 80, TCP port 443, TCP port 53, and UDP port 53, to our IP pool:

iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 80 -j SNAT --to 10.0.0.2-10.0.0.20

iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 443 -j SNAT --to 10.0.0.2-10.0.0.20

iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 53 -j SNAT --to 10.0.0.2-10.0.0.20

iptables -t nat -A POSTROUTING -o eth0 -p udp --dport 53 -j SNAT --to 10.0.0.2-10.0.0.20

Next, test that the changes took effect by requesting a web page that shows your IP address, as such:

wget http://www.ipchicken.com -O out.html

Open the file in a text editor, and look for the portion that contains your IP address:

nano out.html

You should see the IP listed as one of the IPs from your pool. If you instead see the server primary IP address, try running the following commands again:

iptables -t nat -F

wget http://www.ipchicken.com -O out.html

nano out.html

If you get the results you’re looking for, now is a good time to save your IPTables commands so they will persist through a reboot:

iptables-save > /etc/sysconfig/iptables

That’s it! The above should allow all outbound connections from the server to destination ports 80, 443, and 53 (http, https, and dns) to go out via a rotating pool of IPs instead of the server primary IP address. The final command will also save the rules for later use, so they will persist through a reboot.

You can edit the above to also apply to other outbound connections on other ports, or you can make the rules more strict so you only use a pool when connecting to specific destination IPs. Finally, you can also specify a single IP instead of a range, if that suits your particular needs. There is a lot of flexibility here in the IPtables rules, so this kind of SNAT solution can be useful for a wide variety of problems.

If you have any questions about this information, or want to learn about IOFlood.com Phoenix Dedicated Servers, email us at sales [at] ioflood.com