Set persistent static routes for 4 interfaces on a server

Issues related to configuring your network
Post Reply
zackp
Posts: 4
Joined: 2012/08/13 01:26:52

Set persistent static routes for 4 interfaces on a server

Post by zackp » 2015/08/27 06:31:40

I am tasked to set up two test servers running CentOS 7.1 (the NetworkManager is disabled on both). Each one has four 10G ports. Server A resides on subnet A 192.168.15.0/24 and is connected to switch A. Server B resides on subnet B 192.168.16.0/24 and is connected to switch B. The two L3 switches A and B are connected to each other via a Link Aggregation Group (aka LAG). Switch A has VLAN0015 that has a 192.168.15.1 as gw for subnet A. Likewise, switch B has a VLAN0016 that has 192.168.16.1 as gw for subnet B. Each switch has a static route for routing traffic between the two VLANs.

The server network setup requirements are the following:
- All four interfaces on each server should be able to send/receive traffic independently (e.g. ping -I 192.168.15.100 -c 2 192.168.16.100 should see ICMP traffic between the pair only). No interface bonding is used - this is intentional.
- All four interfaces should know of the route to the subnet of the other server.

I have done:
- added to /etc/iproutes/rt_tables the following

Code: Select all

4       ens1f1table
3       ens1f0table
2       ens20f1table
1       ens20f0table

- introduced /etc/sysconfig/network-scripts/ route-* and rule-* for each of the four interfaces on each server (an example is given below)

For example, for server A's interface ens20f0, I have the following in /etc/sysconfig/network-scripts/route-ens20f0:

Code: Select all

192.168.15.0/24 dev ens20f0 src 192.168.15.100 table ens20f0table
default via 192.168.15.1 dev ens20f0 table ens20f0table
and in its /etc/sysconfig/network-scripts/rule-ens20f0:

Code: Select all

from 192.168.15.100/32 table ens20f0table
to 192.168.15.100/32 table ens20f0table
The setup "sort of works" but from time to time, from one server I couldn't ping any interface of the other server. After some tracerouting, I realized that some interfaces didn't have the right route for traffic. As a get around, I use the following

Code: Select all

/sbin/route add -net 192.168.16.0/24 gw 192.168.15.1 dev ...
to each of the ens20f0|ens20f1|ens1f0|ens1f1 to force the kernel routing table of e.g. server A to look like below:

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.11.3    0.0.0.0         UG        0 0          0 ens10f0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens20f0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens10f0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens20f1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens1f0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens1f1
192.168.11.0    0.0.0.0         255.255.255.0   U         0 0          0 ens10f0
192.168.15.0    0.0.0.0         255.255.255.0   U         0 0          0 ens20f0
192.168.15.0    0.0.0.0         255.255.255.0   U         0 0          0 ens20f1
192.168.15.0    0.0.0.0         255.255.255.0   U         0 0          0 ens1f0
192.168.15.0    0.0.0.0         255.255.255.0   U         0 0          0 ens1f1
192.168.16.0    192.168.15.1    255.255.255.0   UG        0 0          0 ens1f1
192.168.16.0    192.168.15.1    255.255.255.0   UG        0 0          0 ens1f0
192.168.16.0    192.168.15.1    255.255.255.0   UG        0 0          0 ens20f1
192.168.16.0    192.168.15.1    255.255.255.0   UG        0 0          0 ens20f0
Yes, I tried to use ip route add but it wouldn't add the desired route entry.

Obviously, the "get-around" setup is not persistent. I have tried to put in 192.168.16.0/24 via 192.168.15.1 dev ens20f0 into the route-ens20f0. After a /sbin/ifdown ens20f0 and then /sbin/ifup ens20f0, the desired route didn't show up in the kernel routing table. I am at a loss here what to do. Repeated reading of https://access.redhat.com/documentation ... lt_Gateway didn't help. I would be grateful to any hints as to what I have missed.

--Zack

Post Reply