Photo by jackthegag

In case you’re still using the Brocade / Foundry FESX-448-PREM for doing routing on your network, I thought I would share some of the performance limitations we’ve seen on that platform. We’ve long since upgraded our network to use newer and more powerful Brocade MLX routers, but wanted to share our experience so others could benefit.

A while back, we were starting to see strange issues with performance and packet loss on the FESX448. As our network grew these became more severe. We started to see high CPU usage on the FESX (40% range steadily, 90% bursts frequently). The overall solution was to stop routing on the FESX and move it to a more powerful device.

The ARP table under stable operation on the FESX should be 1,000 – 2,000 ARP entries. Performance gets very bad at 3,000 ARP table entries, and the device is nearly unusable at 4,000 ARP table entries. The specified limit on the device is 64,000 but you will never reach this in practice. If you plan to use routing on the FESX-448, I suggest limiting this to networks with 1,000 ARP entries (roughly 2,000 IP addresses) as a maximum. If you need more than 1,000 ARP entries you will see better performance and reliability from a more powerful device.

The MAC table under stable operation on the FESX should also be 1,000 – 2,000 Mac address entries. Performance also gets very bad at 3,000 MAC addresses or larger. Much like with ARP, the specified limit on MAC addresses is much higher than what will work in real life. If you have more than 1,000 MAC addresses visible to your FESX, I recommend upgrading to a more powerful network device, or else you will likely see CPU exhaustion leading to other performance and reliability problems.

After doing some research, we found that the CPU on the FESX-448-PREM is a 300MHz Power PC processor. See below export of “show version”:

show version
 SW: Version 07.2.02eT3e3 Copyright (c) 1996-2010 Brocade Communications System s, Inc.
 Compiled on Oct 12 2011 at 15:24:57 labeled as SXR07202e
 (4046205 bytes) Primary SXR07202e.bin
 BootROM: Version 07.2.00T3e5 (FEv2)
 HW: Stackable FESX448-PREM (PROM-TYPE FESX448-L3U)
==========================================================================
 Serial #: FL36045829
 License: SX_V4_HW_ROUTER_SOFT_PACKAGE (LID: hnILFJKNHO)
 P-ASIC 0: type 00D1, rev D1 subrev 00
 P-ASIC 1: type 00D1, rev D1 subrev 00
 P-ASIC 2: type 00D1, rev D1 subrev 00
 P-ASIC 3: type 00D1, rev D1 subrev 00
==========================================================================
 300 MHz Power PC processor 8245 (version 129/1014) 66 MHz bus
 512 KB boot flash memory
 8192 KB code flash memory
 128 MB DRAM
The system uptime is 538 days 13 hours 25 minutes 34 seconds
The system : started=cold start

We only use these FESX-448-PREM for top-of-rack use now, layer2 only. They have performed well in this role. Due to their low price, it is a good option for this usage, but I do not recommend them for layer3 for any network of significant size. A Juniper EX4200 is better for a “small” router these days, as it has a more powerful CPU and is still available inexpensively on the secondary / used market. The Junuper EX4200 has a 1GHz PowerPC Processor, which is more than 3 times the speed of the FESX-448. For this reason, the Juniper-EX4200 should be ok for a small gateway / small router type of installation. For more serious BGP routing however you should consider something like a Brocade MLX / Cisco 6500 series / Juniper MX80 or similar.

If you have any questions about the above information, or want any information about ioflood.com dedicated servers, feel free to email us at sales [at] ioflood.com