Routed traffic slow after ip_forward=1

[aldebaran27]

Hi,

I am currently using CentOS v5.11, and I have run into a problem with my routed network throughput dropping to ~1/100 of what it is supposed to be after setting "ip_forward=1". I am not currently using iptables, since this is a private network and I have no need for a firewall.
Here is an example:
SystemA: eth0=10.10.4.1/24, eth1=10.10.3.1/24, ip_forward=1
SystemB: eth1=10.10.3.2/24, ip_forward=0
SystemC: eth1=10.10.2.1/24, eth1=10.10.3.3/24, ip_forward=1

In this configuration, SystemA is routing between 10.10.4.X and 10.10.3.X, and SystemB is routing between 10.10.3.X and 10.10.2.X.

SCP SystemA->SystemC(eth1, no routing): gigabit speeds
SCP SystemA->SystemC(eth0, routing): ~10M speeds
SCP SystemB->SystemC(eth1, no routing): gigabit speeds
SCP SystemB->SystemC(eth0, routing): gigabit speeds

Then, if I turn on "ip_forward=1" on SystemB, I get:
SCP SystemB->SystemC(eth1, no routing): gigabit speeds
SCP SystemB->SystemC(eth0, routing): ~10M speeds

Any help would be much appreciated!

[Super Jamie]

I drew this to understand your network a bit better:

Code: Select all

SystemA
eth0 = 10.10.4.1/24
eth1 = 10.10.3.1/24 --.
                      |
SystemB               |
eth1 = 10.10.3.2/24 --+
                      |
SystemC               |
eth0 = 10.10.2.1/24   |
eth1 = 10.10.3.3/24 --'

Software IP routing at Layer 3 is not as fast as software packet switching at Layer 2. If you want to route fast, use a proper router which does all this in hardware.

EL6 would likely be more efficient, EL7 would probably moreso, but not blindingly fast.

[TrevorH]

But 1/100th of the speed sounds like some sort of error not a result of software routing - unless you're trying to route 10GBe with a 486 processor.

[Super Jamie]

I assumed this "10M" means 10 Megabytes per second, is 100 Megabits, which is consistent enough with my experience using Linux boxes as routers.