CentOS 7.5 Multiple NIC, Multiple Gateway/Routing issue

Issues related to configuring your network
cornekruger
Posts: 15
Joined: 2018/08/15 07:06:11

CentOS 7.5 Multiple NIC, Multiple Gateway/Routing issue

Post by cornekruger » 2018/08/15 09:50:41

Hi Fellow Underpaid SysAdmins

Hope you are all well.

I've got a Newly built Linux CentOS 3.10.0-862.9.1.el7.x86_64 system running in a vmware environment.

I've got a Draytek 3900 router doing natting from 3 different WAN`s over 3 Different Vlan to the same CentOS server (Urghhh)
The nattings works perfectly fine and was confirmed by a Draytek tech. And ive confirmed that the rules are working using Wireshark.

So whats happening is that the CentOS server is only replying using one nic, and i cannot access (For example) ssh from the other two NICs.
But if i disable the "Working" nic, then one of the other nics takes over the connection and the 2nd "NAT" works.

I suspect that the OS is accepting traffic from all NIC, but not replying back over that same nic. Traffic seems to be passing through a different nic on its return path.

I would like to get the server to reply back to the same Default gateway instead of a different gateway.
This is true for any port i try to test, so its not just ssh (Which listens on 0.0.0.0)

I have assigned 3 NIC`s to the server. Each nic is connected to a different Vlan (Vlan 110,111,112)
Vlan 110 = 192.168.110.0/24
Vlan 111 = 192.168.111.0/24
Vlan 112 = 192.168.112.0/24

My CentOS is configured with the following Settings
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME="ens192 192.168.111.240"
UUID=e6d31966-1a84-4674-a562-037895080936
DEVICE=ens192
ONBOOT=yes
MASK="255.255.255.0"
IPV6INIT=no
IPADDR=192.168.111.240
PREFIX=24
GATEWAY=192.168.111.254
DNS1=8.8.8.8
HWADDR=00:0C:29:7C:8B:AD
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=192.168.112.240
PREFIX=24
GATEWAY=192.168.112.254
DNS1=8.8.8.8
DEFROUTE=no
IPV4_FAILURE_FATAL=no
NAME=192.168.111.240
UUID=186c162e-1abc-3574-8be3-3a4a5409658f
ONBOOT=yes
AUTOCONNECT_PRIORITY=-999
HWADDR=00:0C:29:7C:8B:B7
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=192.168.110.240
PREFIX=24
GATEWAY=192.168.110.254
DNS1=8.8.8.8
DEFROUTE=no
IPV4_FAILURE_FATAL=no
NAME="Wired connection 2"
UUID=b234a56d-87b0-32fc-a209-49f994a8bbef
ONBOOT=yes
AUTOCONNECT_PRIORITY=-999
Here is my routing table
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.112.254 0.0.0.0 UG 103 0 0 ens224
0.0.0.0 192.168.111.254 0.0.0.0 UG 105 0 0 ens192
192.168.110.0 192.168.110.254 255.255.255.0 UG 0 0 0 ens256
192.168.110.0 0.0.0.0 255.255.255.0 U 104 0 0 ens256
192.168.111.0 0.0.0.0 255.255.255.0 U 105 0 0 ens192
192.168.112.0 192.168.112.254 255.255.255.0 UG 0 0 0 ens224
192.168.112.0 0.0.0.0 255.255.255.0 U 103 0 0 ens224
ip route list
default via 192.168.112.254 dev ens224 proto static metric 103
default via 192.168.111.254 dev ens192 proto static metric 105
192.168.110.0/24 via 192.168.110.254 dev ens256
192.168.110.0/24 dev ens256 proto kernel scope link src 192.168.110.240 metric 104
192.168.111.0/24 dev ens192 proto kernel scope link src 192.168.111.240 metric 105
192.168.112.0/24 via 192.168.112.254 dev ens224
192.168.112.0/24 dev ens224 proto kernel scope link src 192.168.112.240 metric 103
Please let me know if i have missed any information.

Hope you guys can help;. Have a great day!
Last edited by cornekruger on 2018/08/27 09:29:17, edited 1 time in total.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by jlehtone » 2018/08/17 08:18:02

I need a picture:

Code: Select all

      / lan1 \       / wan1
CentOS- lan2 -Draytek- wan2
      \ lan3 /       \ wan3
What NAT is in the picture? SNAT, DNAT?

Why are there 3 LANs between CentOS and Draytek?

cornekruger
Posts: 15
Joined: 2018/08/15 07:06:11

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by cornekruger » 2018/08/17 10:15:47

Hi Jlehtone

The 3 Vlans are to ensure high availability as 3 Different WAN connection, needs to be forwarded to the CentOS Server. Traffic from each WAN connection should pass back over that same WAN connection.

Well actually it is the other way around (Dnat)

Code: Select all

         / wan1 \       / lan1 \
Internet - wan2 -Draytek- lan2 -CentOS
         \ wan3 /       \ lan3 /
The server is private local and needs to be accessible from the public over 3 different IP`s.
ideal traffic Flow will be as follows

Code: Select all

        / wan1 \       / lan1 \       / wan1 \       / lan1 \         
Internet- wan2 -Draytek- lan2 -CentOS-  wan2 -Draytek- wan2 -Internet 
        \ wan3 /       \ lan3 /       \ wan3 /       \ lan3 /            
ASAIK ive working with Dnatting. Here is a screenshot of one of my NAT Rules
https://imgur.com/a/2KLRk1L
Image

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by jlehtone » 2018/08/17 14:09:27

In other words, the Draytek has three distinct public IPs, each in different (wan) subnet.

The Draytek forwards (some) incoming connections to the CentOS.
The Draytek is be able to route replies appropriately?

WAN is usually associated with INET (internet). A client on INET could reach all three public IPs, inet routers would forward them via appropriate WAN, and Draytek would receive them all. Draytek, when sending replies, is clever enough to use "originating WAN", even though replies of all connections would be destined to same client?

All that can be tested with just one subnet between the Draytek and the CentOS.


High Availability?
If that Draytek or CentOS fails, then all is down.
You would have HA on physical links though: three NIC-cable-NIC links.
One could have there one logical LAN and the three physical links bundled together with Team (or Bond).
That should still yield HA for the links.


However, if the Draytek can "multi-route", then why wouldn't the CentOS?

The 3-lan scenario probably requires "source routing" aka "policy-based routing".
You have now one routing table. The main.
You can create additional tables and populate them with almost similar routes (at least the default route).
You can add rules that packets originating from specific interface IP must use your custom tables.

Should be doable, I presume.

cornekruger
Posts: 15
Joined: 2018/08/15 07:06:11

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by cornekruger » 2018/08/17 16:53:44

Hi Jlehtone

Thank you for your reply.

yes the Draytek has routing policy rules, that works. We did test this separably by using one one network card at a time and tracking the packets. it works as expected. I also tested this with other machines and the Draytek policy routing works as expected, Just to be sure the Draytech tech also confirm that it is routing the packets fine.

In other words. the Draytek can route packets to the correct Centos NIC and IF (And a very big IF) the CentOS machine replies using the correct NIC. The traffic will pass using the intended routes back out through the Draytek to the WAN.

Regarding the High Availability. We are in the process of configuring configuring another Draytek device to take over incase (Draytek tech doing their thing)
The Centos server is on Vmware vSphere infrastructure. So backups and replication is in order. I Don't have to much worry about that really.
The 3-lan scenario probably requires "source routing" aka "policy-based routing".
You have now one routing table. The main.
You can create additional tables and populate them with almost similar routes (at least the default route).
You can add rules that packets originating from specific interface IP must use your custom tables.
Seems to be exactly what i`m looking for. Catch is. I seem to be to flat headed to actually get it working.
looking at my existing routing table, i`ve already tried to modify the gateway for each network, but failed.

Im doing some more research into source routing but il appreciate more advice, procedures, examples or guidance.
Thank you very much so far for you advice and effort!

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by hunter86_bg » 2018/08/20 04:44:45

Just a side question... Why is LACP not an option for this case? Why don't you let the Draytek do the hard work ?
Something like:

Code: Select all

         / wan1 \       
Internet - wan2 -Draytek- LACP -CentOS
         \ wan3 /       

cornekruger
Posts: 15
Joined: 2018/08/15 07:06:11

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by cornekruger » 2018/08/20 07:36:31

Morning Hunter

The 3 Vlans are on one physical cable, as the CentOS machine is a virtual machine. Its just got 3 nic assigned to it. Each NIC is tagged with a different Vlan. LACP is configured and Enabled on the Virtual Host as the host itself has multiple NICs. Which works normally for its intended purpose.
In essence all i need is that if a packet has been Dnatted from 192.168.110.254 -> 192.168.110.240. it should return using the same NIC. not the default NIC.
hunter86_bg wrote:
2018/08/20 04:44:45
Just a side question... Why is LACP not an option for this case? Why don't you let the Draytek do the hard work ?
Something like:

Code: Select all

         / wan1 \       
Internet - wan2 -Draytek- LACP -CentOS
         \ wan3 /       

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by hunter86_bg » 2018/08/20 07:46:45

Hm.. That changes the story.
Could you try to remove any default gateway temporarily (ip route del) and try again ?
Edit:
If you use different subnets. (looks so), you can assign default gateway for each interface and try this way.
Last edited by hunter86_bg on 2018/08/20 07:52:25, edited 1 time in total.

cornekruger
Posts: 15
Joined: 2018/08/15 07:06:11

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by cornekruger » 2018/08/20 07:51:54

hunter86_bg wrote:
2018/08/20 07:46:45
Hm.. That changes the story.
Could you try to remove any default gateway temporarily (ip route del) and try again ?
Im only getting the IP Route usage tip when copying and pasting the command. Should it suffice if i just comment out the DF Gateway entry in the NIX Config file (/etc/sysconfig/network-scripts/ifcfg/ens256)? and then restating the network? (systemctl restart network)?

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 3.10.0-862.9.1.el7.x86_64 Multiple NIC, Multiple Gateway/Routing issue

Post by hunter86_bg » 2018/08/20 09:14:48

You can try by commenting the gateway or adding gateways to all nics.

Post Reply