Troubleshooting high linecard CPU use / exhaustion on Brocade MLX and MLXe routers

Photo by Jemimus

Today we’re going to be talking about a poor design decision in the Brocade MLX and MLXe series routers that can lead to CPU exhaustion on your router line cards. Overall we’re very happy with the Brocade MLX. After all, every routing platform has its quirks, so one small problem that is easily solved is not a deal breaker. That said, the last thing you want to do is learn about them and have to fix them in a live environment, so testing and research is key. Luckily we were able to find this particular bug when we were testing this hardware, before putting it into production.

You may notice that traffic on your router which is destined to another subnet on your network, will max out at around 120 megabits / second no matter what you do. You may also notice that the affected router line card will peg the cpu at 100%, which you can see from the output of “show cpu lp”. The reason for this dates back several decades, as a feature to help prevent resource exhaustion on routers. Since that time, resource bottlenecks have shifted dramatically, and this feature now causes many more problems than it solves.

The particular culprit here is “icmp redirects”.

If you’re just looking to get your network back up and running properly, simply enter the following command from configuration mode on your Brocade device:

no ip icmp redirects

If you’d like to know more about what is happening here, read on:

ICMP redirects are control packets the router sends to inform end hosts that they shouldn’t be sending the router traffic. The router will do this if a packet is sent back out the same interface it was received on. This is intended to inform other routers to send traffic directly to destination hosts when possible instead of sending it to the router. After all, if the traffic is just sent back out the same interface it came in, the router didn’t need to see it, right? However, when the destination subnet or vlan is different, the notification is useless because it is not possible for the host to send the traffic directly without the aid of a router. In this case, the traffic coming in one port and leaving out the same port is normal and expected behavior. Unfortunately, the Brocade router doesn’t realize if the destination subnet or vlan was different than the source subnet or vlan, it will still send these notices anyway.

The best solution is simply to turn off these notifications, as generally the router has no trouble keeping up with sending the actual traffic you’re asking it to send, but does have difficulty keeping up with sending massive numbers of ICMP redirect packets. Again, this is because the decision to send ICMP redirects as a standard practice, was decided upon several decades ago, and the problem it was designed to solve largely does not exist any longer.

If you have any questions about this information, or want to learn about ioflood.com dedicated servers, email us at sales [at] ioflood.com