Routing with multiple interfaces

Issues related to configuring your network
Post Reply
gmclean
Posts: 10
Joined: 2008/08/15 17:48:44
Contact:

Routing with multiple interfaces

Post by gmclean » 2008/08/15 18:08:38

Hi. I have two CentOS 4.6 servers, each with three network interfaces.
They do not behave the same, I want them to and I don't know why they don't.
One interface from each server is internal. The other two interfaces are both connected to the Internet.
This is done for redundancy and to balance or split the load.
The first server behaves the way I want. When I ping each interface from an external source, each interface replies out the interface that received the ping. The other server sends both replies out the default interface (not what I want).
The server that behaves correctly has static IP addresses. The misbahaving server has dynamic addresses.
On the misbehaving server I can make it work by adding a route to the external dude that is pinging me, but then the other interface sends the reply in the wrong direction.
Some config stuff to get the two interfaces to work:
added to the bottom of /etc/iproute2/rt_tables
2 gbp
200 pri
201 sec

created /etc/init.d/iproute
/sbin/ip rule add iif lo table gbp priority 500
/sbin/ip route add 75.47.88.0/24 dev eth1 src 75.47.88.163 table pri
/sbin/ip route add default via 75.47.88.162 table pri
/sbin/ip route add 69.23.64.0/22 dev eth2 src 69.23.64.235 table sec
/sbin/ip route add default via 76.45.96.1 table sec
/sbin/ip rule add from 75.47.88.163 table pri
/sbin/ip rule add from 69.23.64.235 table sec
/sbin/ip route flush cache
echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo "0" > /proc/sys/net/ipv4/conf/eth2/rp_filter

Don't ask me why, but I think this has something to do with the server having dynamic addresses and if you do a traceroute out that interface (Which is not the default route), the first hop is:
1 192.168.0.1 (192.168.0.1) 4.231 ms 1.477 ms 0.429 ms

somehow i think that 192.168 address is screwing stuff up, but I could be completely off base and am really hoping that I just have something configured differently that I don't know about.

any help you can give me on this is greatly appreciated.
Thank you. Greg...
oh, yes:
ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:60:97:05:19:22
inet addr:75.47.88.163 Bcast:75.47.88.255 Mask:255.255.255.0
inet6 addr: fe80::260:97ff:fe05:1922/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11118 errors:0 dropped:0 overruns:0 frame:0
TX packets:6576 errors:0 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:1000
RX bytes:2398681 (2.2 MiB) TX bytes:1583969 (1.5 MiB)
Interrupt:10 Base address:0xec80

route -n | grep eth1
75.47.88.162 75.47.88.162 255.255.255.255 UGH 0 0 0 eth1
75.47.88.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

traceroute -i eth1 199.89.xxx.xx
traceroute to 199.89.xxx.xx (199.89.xxx.xx), 30 hops max, 38 byte packets
1 192.168.0.1 (192.168.0.1) 0.505 ms 0.443 ms 0.451 ms
2 adsl-75-47-95-254.dsl.applwi.sbcglobal.net (75.47.95.254) 9.768 ms 8.255 ms 9.077 ms
3 dist1-vlan60.applwi.sbcglobal.net (67.38.56.242) 8.241 ms 8.436 ms 8.643 ms
4 bb2-g1-0.applwi.sbcglobal.net (67.38.56.100) 8.253 ms 9.468 ms 9.273 ms
5 151.164.189.80 (151.164.189.80) 13.834 ms 13.808 ms 13.397 ms
6 gar8.cgcil.ip.att.net (12.122.79.85) 14.028 ms 14.458 ms 14.440 ms
Icmp checksum is wrong
7 tbr1.cgcil.ip.att.net (12.122.99.6) 19.765 msIcmp checksum is wrong
19.918 msIcmp checksum is wrong
19.575 ms
8 12.122.99.37 (12.122.99.37) 18.454 ms 17.538 ms 17.897 ms
9 12.87.166.174 (12.87.166.174) 49.664 ms 35.652 ms 34.142 ms
10 206-40-100-138.netnet.net (206.40.100.138) 34.697 ms 35.262 ms 60.844 ms

Thank You!!

pjwelsh
Posts: 2589
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: Routing with multiple interfaces

Post by pjwelsh » 2008/08/15 21:51:15

You may have the "arp flux" problem. Google it for LOTS of pages or look at:
http://linux-ip.net/html/ether-arp.html

gmclean
Posts: 10
Joined: 2008/08/15 17:48:44
Contact:

Re: Routing with multiple interfaces

Post by gmclean » 2008/08/18 13:58:20

Thank you very much for the tip. I had never heard of the "arp flux" problem.
and while it was interesting reading, unfortunately, I do not beleive that this is my problem.
1. because after reading what you pointed me to, I tried it and it didnt work and
2. arp should not be involved in this case. The situation is that I have a server at my house. It has both a DSL connection and a cable connection. I am trying to ping the device from our corporate office. The two addresses for the server at my house are not on the local network and not something the corporate network would arp for. it just sends them down the line. The server at my house receives the pings, each on the correct interface. however, the cable connection is the default route on the server and its sends both ping replies back out the cable connection.

My house naturally is just a test server. We have a number of other locations set up in a very similar fashion (maybe two dsl lines or a T1 and a dsl, whatever). All other locations work correctly. They all have static addresses, my house is dynamic. All other locations send the ping reply back out the interface it came in on, as it should, not giving a rip (pun intended) about the default route on the server.

You can try it yourself... as of this writing on 8/18/08 my house has:
DSL Address: 75.47.88.163
DSL Modem Address: 75.47.88.162
Cable Address: 69.23.64.235
Cable Modem Address: 69.23.64.1

I have temporarially opened up ping to the world just so you can test. I will turn it back off in a couple of days. If you read this later on and would like ping opened back up, please email me.
You should see that both modem addresses will answer your ping. so will the cable address, but the dsl address will not becuase it sends the reply out the cable interface.

Any and all help is greatly appreciated.

User avatar
AlanBartlett
Forum Moderator
Posts: 9320
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk
Contact:

Re: Routing with multiple interfaces

Post by AlanBartlett » 2008/08/18 14:31:56

Results from an independent observer:

[code]
$ ping -c 5 75.47.88.163
PING 75.47.88.163 (75.47.88.163) 56(84) bytes of data.

--- 75.47.88.163 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3998ms

$ ping -c 5 75.47.88.162
PING 75.47.88.162 (75.47.88.162) 56(84) bytes of data.
64 bytes from 75.47.88.162: icmp_seq=1 ttl=238 time=145 ms
64 bytes from 75.47.88.162: icmp_seq=2 ttl=238 time=142 ms
64 bytes from 75.47.88.162: icmp_seq=3 ttl=238 time=143 ms
64 bytes from 75.47.88.162: icmp_seq=4 ttl=238 time=143 ms
64 bytes from 75.47.88.162: icmp_seq=5 ttl=238 time=141 ms

--- 75.47.88.162 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 141.503/143.214/145.146/1.185 ms
$ ping -c 5 69.23.64.235
PING 69.23.64.235 (69.23.64.235) 56(84) bytes of data.
64 bytes from 69.23.64.235: icmp_seq=1 ttl=44 time=149 ms
64 bytes from 69.23.64.235: icmp_seq=2 ttl=44 time=148 ms
64 bytes from 69.23.64.235: icmp_seq=3 ttl=44 time=158 ms
64 bytes from 69.23.64.235: icmp_seq=4 ttl=44 time=150 ms
64 bytes from 69.23.64.235: icmp_seq=5 ttl=44 time=150 ms

--- 69.23.64.235 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 148.137/151.527/158.311/3.520 ms
$ ping -c 5 69.23.64.1
PING 69.23.64.1 (69.23.64.1) 56(84) bytes of data.
64 bytes from 69.23.64.1: icmp_seq=1 ttl=236 time=142 ms
64 bytes from 69.23.64.1: icmp_seq=2 ttl=236 time=144 ms
64 bytes from 69.23.64.1: icmp_seq=3 ttl=236 time=142 ms
64 bytes from 69.23.64.1: icmp_seq=4 ttl=236 time=142 ms
64 bytes from 69.23.64.1: icmp_seq=5 ttl=236 time=140 ms

--- 69.23.64.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 140.745/142.458/144.190/1.116 ms
$
[/code]

gmclean
Posts: 10
Joined: 2008/08/15 17:48:44
Contact:

Re: Routing with multiple interfaces

Post by gmclean » 2008/08/19 16:34:16

OK. This is working now. Something is killing one of my route tables... Don't know what yet.
ip route show table pri
ip route show table sec
69.23.64.0/22 dev eth2 scope link src 69.23.64.235
default via 69.23.64.1 dev eth2

but the basic config is correct and working until the table gets trashed.
reload the table and works again for a bit.

User avatar
AlanBartlett
Forum Moderator
Posts: 9320
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk
Contact:

Re: Routing with multiple interfaces

Post by AlanBartlett » 2008/08/19 17:13:53

Now seeing:

[code]
$ ping -c 5 75.47.88.163
PING 75.47.88.163 (75.47.88.163) 56(84) bytes of data.

--- 75.47.88.163 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

$ ping -c 5 75.47.88.162
PING 75.47.88.162 (75.47.88.162) 56(84) bytes of data.
64 bytes from 75.47.88.162: icmp_seq=1 ttl=238 time=144 ms
64 bytes from 75.47.88.162: icmp_seq=2 ttl=238 time=144 ms
64 bytes from 75.47.88.162: icmp_seq=3 ttl=238 time=140 ms
64 bytes from 75.47.88.162: icmp_seq=4 ttl=238 time=140 ms
64 bytes from 75.47.88.162: icmp_seq=5 ttl=238 time=141 ms

--- 75.47.88.162 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 140.737/142.353/144.538/1.820 ms
$ ping -c 5 69.23.64.235
PING 69.23.64.235 (69.23.64.235) 56(84) bytes of data.

--- 69.23.64.235 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3998ms

$ ping -c 5 69.23.64.1
PING 69.23.64.1 (69.23.64.1) 56(84) bytes of data.
64 bytes from 69.23.64.1: icmp_seq=1 ttl=236 time=143 ms
64 bytes from 69.23.64.1: icmp_seq=2 ttl=236 time=148 ms
64 bytes from 69.23.64.1: icmp_seq=3 ttl=236 time=144 ms
64 bytes from 69.23.64.1: icmp_seq=4 ttl=236 time=144 ms
64 bytes from 69.23.64.1: icmp_seq=5 ttl=236 time=144 ms

--- 69.23.64.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 143.862/145.037/148.056/1.529 ms
$
[/code]

Post Reply