nf_conntrack: table full, dropping packet — A solution for CentOS Dedicated Servers

A common problem you may experience is sluggish performance or disconnections from your Centos dedicated server, even though there is sufficient CPU, ram, disk i/o, etc. After some troubleshooting, you may come to believe you are being DDoS attacked, but you don’t see an unusual amount of traffic, and there’s no single IP or handful of IPs that are making an unusually large number of connections to your server. After looking over /var/log/messages, you’ll come to see a lot of messages like the following:

nf_conntrack: table full, dropping packet

This happens when your IPtables or CSF firewall is tracking too many connections. This can happen when you are being attacked, or is also very likely to happen on a busy server even if there is no malicious activity. Connections will be tracked if you have a firewall rule that does NAT or SNAT, or if you are tracking the number of connections per IP for rate limiting reasons. These scenarios are common either in linux router / firewalls, or in the case of firewall rules that are there for brute force protection / ddos protection.By default, Centos will set this maximum to 65,536 connections. This is enough for lightly loaded servers, but can easily be exhausted on heavily trafficked servers with a lot of firewall rules. On our heavy production servers, we’ve increased this limit to half a million, which has made a big improvement on the amount of workload those servers can handle.It is interesting to note, that the kind of servers most likely to have this problem, are ones where the user has set a lot of strict firewall rules to “help ward off attacks”. Unfortunately, the reality is that the firewall rules themselves are causing the downtime, not any attack! One way to solve the problem is to disable your firewall entirely, but before you go to that extreme, it is worth trying to increase the maximum connections here.In this article, I’ll give you instructions on how to increase the maximum allowed connections for the conntrack connection tracker in Centos. Centos 5 and Centos 6 store the relevant data in different places, so I’ll have instructions for each below. The instructions below assume you’ll be entering commands in an SSH shell / command prompt window:

Centos 5.x: Increasing maximum connection tracking for nf_conntrack

First of all, you may want to know what the maximum connection limit is already

cat /proc/sys/net/ipv4/ip_conntrack_max

This will output the current maximum number of connections that IPtables can track.If you want to see the current number of connections being tracked, you can run the following command:

cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count

You’ll be given a number of connections here. If this number is more than 20% of the maximum, it’s probably a good idea to increase the maximum.If you want to temporarily increase this to a half million, enter the following:

echo 524288 > /proc/sys/net/ipv4/ip_conntrack_max

And if you’d like the change to persist across reboots, you’ll need to edit the following file:

nano /etc/rc.d/rc.local

Copy / paste the following line to the end of the file, and then save your changes:

echo 524288 > /proc/sys/net/ipv4/ip_conntrack_max

That’s all there is to it. On heavily trafficked servers, it’s not unusual to see 100k – 200k connections being tracked even if there is no malicious activity. 500k should be a safe maximum, but if you really need to you could increase this further.

---

Centos 6.x: Increasing maximum connection tracking for nf_conntrack

On Centos 6, the general idea is the same as Centos 5, but the file locations are slightly different.To view the current maximum configured connections, run:

cat /proc/sys/net/netfilter/nf_conntrack_max

To see the current used connections, run:

cat /proc/sys/net/netfilter/nf_conntrack_count

To temporarily increase this to a half million, run:

echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max

To make this change persist after a reboot, you’ll need to edit the following file:

nano /etc/rc.d/rc.local

And copy and paste the following line to the end of the file, and then save your changes:

echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max

That’s it. You should be in good shape now. Just like in Centos 5, on heavily trafficked servers, it’s not unusual to see 100k – 200k connections being tracked even if there is no malicious activity. Therefore, 500k should be a safe maximum, but if you really need to you could increase this further.

---

The reason we have Centos instructions above is because we’re most familiar with Centos, using it for most of our internal systems. I understand that a lot of other people prefer Ubuntu or Debian. We don’t want to leave those folks out in the cold here, we just aren’t familiar with this fix for those OS’s. If you have any instructions on doing the same for Ubuntu, Debian, or other Linux distributions, please share them with us by emailing sales [at] ioflood.com. If you do send that along, we will be glad to post an update with that information, and also credit your contribution if you’d like.

Do you love servers?

If you love servers like we do, we’d love to work together! IOFLOOD.com offers dedicated servers to people like you, and as part of that service, if you have any problems with conntrack we would be happy to diagnose and resolve the issue for you, despite only offering unmanaged hosting. At IOFLOOD we feel that unmanaged doesn’t have to be unhelpful. To get started today, click here to view our dedicated servers, or email us at sales[at]ioflood.com to ask for a custom quote.