{"id":287,"date":"2015-02-19T11:14:12","date_gmt":"2015-02-19T18:14:12","guid":{"rendered":"http:\/\/blog.ioflood.com\/?p=287"},"modified":"2023-11-25T23:09:10","modified_gmt":"2023-11-26T06:09:10","slug":"nf_conntrack-table-full-dropping-packet-a-solution-for-centos-dedicated-servers","status":"publish","type":"post","link":"https:\/\/ioflood.com\/blog\/nf_conntrack-table-full-dropping-packet-a-solution-for-centos-dedicated-servers\/","title":{"rendered":"nf_conntrack: table full, dropping packet — A solution for CentOS Dedicated Servers"},"content":{"rendered":"
\"iptables

Photo by xmodulo<\/a> \"Creative<\/a><\/small><\/p><\/div>\r\n\r\nA common problem you may experience is sluggish performance or disconnections from your Centos dedicated server, even though there is sufficient CPU, ram, disk i\/o, etc. After some troubleshooting, you may come to believe you are being DDoS attacked, but you don’t see an unusual amount of traffic, and there’s no single IP or handful of IPs that are making an unusually large number of connections to your server. After looking over \/var\/log\/messages, you’ll come to see a lot of messages like the following:\r\n

nf_conntrack: table full, dropping packet<\/pre>\r\nThis happens when your IPtables or CSF firewall is tracking too many connections. This can happen when you are being attacked, or is also very likely to happen on a busy server even if there is no malicious activity. Connections will be tracked if you have a firewall rule that does NAT or SNAT, or if you are tracking the number of connections per IP for rate limiting reasons. These scenarios are common either in linux router \/ firewalls, or in the case of firewall rules that are there for brute force protection \/ ddos protection.\r\n\r\nBy default, Centos will set this maximum to\u00a065,536 connections. This is enough for lightly loaded servers, but can easily be exhausted on heavily trafficked servers with a lot of firewall rules. On our heavy production servers, we’ve increased this limit to half a million, which has made a big improvement on the amount of workload those servers can handle.\r\n\r\nIt is interesting to note, that the kind of servers most likely to have this problem, are ones where the user has set a lot of strict firewall rules to “help ward off attacks”. Unfortunately, the reality is that the firewall rules themselves are causing the downtime, not any attack! One way to solve the problem is to disable your firewall entirely, but before you go to that extreme, it is worth trying to increase the maximum connections here.\r\n\r\nIn this article, I’ll give you instructions on how to increase the maximum allowed connections for the conntrack connection tracker in Centos. Centos 5 and Centos 6 store the relevant data in different places, so I’ll have instructions for each below. The instructions below assume you’ll be entering commands in an SSH shell \/ command prompt window:\r\n
Centos 5.x: Increasing maximum connection tracking for nf_conntrack<\/h2>\r\nFirst of all, you may want to know what the maximum connection limit is already\r\n
cat \/proc\/sys\/net\/ipv4\/ip_conntrack_max<\/pre>\r\nThis will output the current maximum number of connections that IPtables can track.\r\n\r\nIf you want to see the current number of connections being tracked, you can run the following command:\r\n
cat \/proc\/sys\/net\/ipv4\/netfilter\/ip_conntrack_count<\/pre>\r\nYou’ll be given a number of connections here. If this number is more than 20% of the maximum, it’s probably a good idea to increase the maximum.\r\n\r\nIf you want to temporarily increase this to a half million, enter the following:\r\n
echo 524288 > \/proc\/sys\/net\/ipv4\/ip_conntrack_max<\/pre>\r\nAnd if you’d like the change to persist across reboots, you’ll need to edit the following file:\r\n
nano \/etc\/rc.d\/rc.local<\/pre>\r\nCopy \/ paste the following line to the end of the file, and then save your changes:\r\n
echo 524288 > \/proc\/sys\/net\/ipv4\/ip_conntrack_max<\/pre>\r\nThat’s all there is to it. On heavily trafficked servers, it’s not unusual to see 100k – 200k connections being tracked even if there is no malicious activity. 500k should be a safe maximum, but if you really need to you could increase this further.\r\n\r\n
\r\n\r\n
Centos 6.x: Increasing maximum connection tracking for nf_conntrack<\/h2>\r\nOn Centos 6, the general idea is the same as Centos 5, but the file locations are slightly different.\r\n\r\nTo view the current maximum configured connections, run:\r\n
cat \/proc\/sys\/net\/netfilter\/nf_conntrack_max<\/pre>\r\nTo see the current used connections, run:\r\n
cat \/proc\/sys\/net\/netfilter\/nf_conntrack_count<\/pre>\r\nTo temporarily increase this to a half million, run:\r\n
echo 524288 > \/proc\/sys\/net\/netfilter\/nf_conntrack_max<\/pre>\r\nTo make this change persist after a reboot, you’ll need to edit the following file:\r\n
nano \/etc\/rc.d\/rc.local<\/pre>\r\nAnd copy and paste the following line to the end of the file, and then save your changes:\r\n
echo 524288 > \/proc\/sys\/net\/netfilter\/nf_conntrack_max<\/pre>\r\nThat’s it. You should be in good shape now. Just like in Centos 5, on heavily trafficked servers, it’s not unusual to see 100k – 200k connections being tracked even if there is no malicious activity. Therefore, 500k should be a safe maximum, but if you really need to you could increase this further.\r\n\r\n
\r\n\r\nThe reason we have Centos instructions above is because we’re most familiar with Centos, using it for most\u00a0of our internal systems. I understand that a lot of other people prefer Ubuntu or Debian. We don’t want to leave those folks out in the cold here, we just aren’t familiar with this fix for those OS’s. If you have any instructions on doing the same for Ubuntu, Debian, or other Linux distributions, please share them with us<\/strong>\u00a0by emailing sales [at] ioflood.com.<\/strong>\u00a0If you do send that along, we will be glad to post an update with that information, and also credit your contribution if you’d like.\r\n\r\n<\/strong>\r\n
Do you love servers?<\/h2>\r\n <\/strong>\r\n\r\nIf you love servers like we do, we’d love to work together! IOFLOOD.com offers dedicated servers to people like you, and as part of that service, if you have any problems with conntrack we would be happy to diagnose and resolve the issue for you, despite only offering unmanaged hosting. At IOFLOOD we feel that unmanaged doesn’t have to be unhelpful. To get started today, click here to view our dedicated servers<\/a>, or email us at sales[at]ioflood.com to ask for a custom quote.\r\n\r\n\r\n<\/strong>","protected":false},"excerpt":{"rendered":"
A common problem you may experience is sluggish performance or disconnections from your Centos dedicated server, even though there is sufficient CPU, ram, disk i\/o, etc. After some troubleshooting, you may come to believe you are being DDoS attacked, but you don’t see an unusual amount of traffic, and there’s no single IP or handful […]<\/p>\n","protected":false},"author":1,"featured_media":394,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,12,35,3,11,28,115,9,10],"tags":[],"class_list":["post-287","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-centos","category-firewalls","category-iptables","category-linux","category-networking","category-performance-tuning","category-redhat-enterprise-linux-rhel","category-sysadmin","category-troubleshooting","cat-4-id","cat-12-id","cat-35-id","cat-3-id","cat-11-id","cat-28-id","cat-115-id","cat-9-id","cat-10-id","has_thumb"],"_links":{"self":[{"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/posts\/287","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/comments?post=287"}],"version-history":[{"count":11,"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/posts\/287\/revisions"}],"predecessor-version":[{"id":11231,"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/posts\/287\/revisions\/11231"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/media\/394"}],"wp:attachment":[{"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/media?parent=287"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/categories?post=287"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ioflood.com\/blog\/wp-json\/wp\/v2\/tags?post=287"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}