I'm Muneer from Switzerland. I just subscribed to this forum because I urgently need help from the CentOS geeks
We use CentOS boxes on VMware virtual-servers with nginx (plus) as reverse-proxies and load-balancers.
Till now everthing worked great. Unfortunately no longer after the last yum upgrade! Now all VMware NICs arn't accessable from the related servers anymore.
Just a simple overview. 3 VMs with CentOS and nignx with Windows 2016 Servers behind among 3 Network-Zones
DMZ ----> Reverse-Proxy / LB Zone 1 ---------> Frontend-Servers (Windows)
______________________________________ or ---------> LB Zone 2 ---------------> Backend-Servers (Windows)
______________________________________________________________ or ---------------> LB Zone 3 --------------> SQL Servers
and the windows Servers, from several different customers, are in separate vLans.
But the LBs (Zone 1 to 3) are used from all customers.
Here is what we have on the working env:
CentOS: 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
NICs (VMware):
Code: Select all
# ifconfig | grep -A1 ens
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.100.171 netmask 255.255.255.0 broadcast 10.0.100.255
--
ens193: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.12.13.16 netmask 255.255.255.0 broadcast 10.12.13.255
--
ens194: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.12.15 netmask 255.255.255.0 broadcast 10.10.12.255
--
ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.12.12.15 netmask 255.255.255.0 broadcast 10.12.12.255
--
ens224:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.12.12.17 netmask 255.255.255.0 broadcast 10.12.12.255
--
ens224:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.12.37 netmask 255.255.255.0 broadcast 10.10.12.255
--
ens225: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.11.13.16 netmask 255.255.255.0 broadcast 10.11.13.255
--
ens256: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.11.12.15 netmask 255.255.255.0 broadcast 10.11.12.255
--
ens256:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.11.12.17 netmask 255.255.255.0 broadcast 10.11.12.255
--
ens256:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.12.36 netmask 255.255.255.0 broadcast 10.10.12.255
--
ens257: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.13.16 netmask 255.255.255.0 broadcast 10.10.13.255
The reason why we have several is the routing to the vlans. In order to make it reachable we added a NIC for the specific vlan on the VM.
Besides that we used virtual interfaces (:0, :1) to separate traffic coming from different environments to avoid if-statements in the nginx conf.
Code: Select all
# ip route
default via 10.0.100.1 dev ens192 proto static metric 100
default via 10.12.13.1 dev ens193 proto static metric 101
default via 10.10.12.1 dev ens194 proto static metric 102
default via 10.12.12.1 dev ens224 proto static metric 103
default via 10.11.13.1 dev ens225 proto static metric 104
default via 10.11.12.1 dev ens256 proto static metric 105
default via 10.10.13.1 dev ens257 proto static metric 106
10.0.100.0/24 dev ens192 proto kernel scope link src 10.0.100.171 metric 100
10.10.12.0/24 dev ens194 proto kernel scope link src 10.10.12.15 metric 100
10.10.12.0/24 dev ens224 proto kernel scope link src 10.10.12.37 metric 101
10.10.12.0/24 dev ens256 proto kernel scope link src 10.10.12.36 metric 102
10.10.13.0/24 dev ens257 proto kernel scope link src 10.10.13.16 metric 100
10.11.12.0/24 dev ens256 proto kernel scope link src 10.11.12.15 metric 100
10.11.13.0/24 dev ens225 proto kernel scope link src 10.11.13.16 metric 100
10.12.12.0/24 dev ens224 proto kernel scope link src 10.12.12.15 metric 100
10.12.13.0/24 dev ens193 proto kernel scope link src 10.12.13.16 metric 100
# arp
Address HWtype HWaddress Flags Mask Iface
10.10.13.26 ether 00:50:56:bc:5b:1d C ens257
10.0.100.172 ether 00:50:56:bc:4d:7f C ens192
10.12.12.131 ether 00:50:56:bc:83:c0 C ens224
10.11.12.132 ether 00:50:56:bc:dd:d5 C ens256
10.11.12.131 ether 00:50:56:bc:8f:26 C ens256
10.12.12.111 ether 00:50:56:bc:4c:dd C ens224
10.11.13.15 ether 00:50:56:bc:de:55 C ens225
10.12.13.26 ether 00:50:56:bc:41:19 C ens193
10.11.13.26 ether 00:50:56:bc:6f:b6 C ens225
10.10.12.25 ether 00:50:56:bc:5b:b5 C ens194
10.10.12.47 ether 00:50:56:bc:5b:b5 C ens194
10.12.12.16 ether 00:50:56:bc:47:4f C ens224
10.12.12.27 ether 00:50:56:bc:7c:85 C ens224
10.11.12.25 ether 00:50:56:bc:72:e0 C ens256
10.11.12.251 ether 00:50:56:bc:3b:12 C ens256
10.12.13.5 ether 00:50:56:bc:b5:7c C ens193
10.11.13.5 ether 00:50:56:bc:de:55 C ens225
gateway ether cc:03:d9:02:b6:00 C ens192
10.10.12.46 ether 00:50:56:bc:5b:b5 C ens194
10.10.12.46 ether 00:50:56:bc:72:e0 C ens256
10.12.13.15 ether 00:50:56:bc:b5:7c C ens193
10.10.12.47 ether 00:50:56:bc:7c:85 C ens224
10.11.12.27 ether 00:50:56:bc:72:e0 C ens256
10.12.12.25 ether 00:50:56:bc:7c:85 C ens224
10.12.12.251 ether 00:50:56:bc:7a:f4 C ens224
# netstat -an | grep -w LISTEN
tcp 0 0 10.10.12.57:80 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.56:80 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.25:80 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.15:80 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.5:80 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.5:80 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.5:1433 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.7:1433 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.5:1433 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.5:1433 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2233 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.5:4443 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.5:4443 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.5:4443 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.5:4444 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.5:4445 0.0.0.0:* LISTEN
This weekend we upgraded it to
3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
and now these interfaces aren't reachable anymore from the windows servers in front of the LBs! No ping, no telnet.
BUT from one LB to the next it still works?!
For example. I can telnet from LB Zone 1 the IP 10.12.12.15, but not from one of the Win Servers in Zone 1, and therefore the whole Application fails.
- There is no blocking firewalld / iptables running, because we have pfsense instances in front of them. (No changes where done on them.)
Code: Select all
# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
The only changes were done on the OS was the yum upgrade.
I've tried once with all NICs, and their config, removed before upgrading, and then adding them back.
Still the same result.
On the not working env it looks like this:
Code: Select all
# ifconfig | grep -A1 ens
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.100.171 netmask 255.255.255.0 broadcast 10.0.100.255
--
ens193: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.12.13.16 netmask 255.255.255.0 broadcast 10.12.13.255
--
ens194: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.12.15 netmask 255.255.255.0 broadcast 10.10.12.255
--
ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.12.12.15 netmask 255.255.255.0 broadcast 10.12.12.255
--
ens224:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.12.12.17 netmask 255.255.255.0 broadcast 10.12.12.255
--
ens224:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.12.37 netmask 255.255.255.0 broadcast 10.10.12.255
--
ens225: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.11.13.16 netmask 255.255.255.0 broadcast 10.11.13.255
--
ens256: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.11.12.15 netmask 255.255.255.0 broadcast 10.11.12.255
--
ens256:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.11.12.17 netmask 255.255.255.0 broadcast 10.11.12.255
--
ens256:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.12.36 netmask 255.255.255.0 broadcast 10.10.12.255
--
ens257: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.13.16 netmask 255.255.255.0 broadcast 10.10.13.255
But some changes on the routing table. Different metric values and the two marked with XXX were missing before.
Code: Select all
# ip route
default via 10.0.100.1 dev ens192 proto static metric 100
default via 10.12.13.1 dev ens193 proto static metric 101
default via 10.10.12.1 dev ens194 proto static metric 102
default via 10.12.12.1 dev ens224 proto static metric 103
default via 10.11.13.1 dev ens225 proto static metric 104
default via 10.11.12.1 dev ens256 proto static metric 105
default via 10.10.13.1 dev ens257 proto static metric 106
10.0.100.0/24 dev ens192 proto kernel scope link src 10.0.100.171 metric 100
10.10.12.0/24 dev ens194 proto kernel scope link src 10.10.12.15 metric 102
10.10.12.0/24 dev ens224 proto kernel scope link src 10.10.12.37 metric 103
10.10.12.0/24 dev ens256 proto kernel scope link src 10.10.12.36 metric 105
10.10.13.0/24 dev ens257 proto kernel scope link src 10.10.13.16 metric 106
10.11.12.0/24 dev ens256 proto kernel scope link src 10.11.12.15 metric 105
10.11.12.0/24 dev ens256 proto kernel scope link src 10.11.12.17 metric 105 XXX
10.11.13.0/24 dev ens225 proto kernel scope link src 10.11.13.16 metric 104
10.12.12.0/24 dev ens224 proto kernel scope link src 10.12.12.15 metric 103
10.12.12.0/24 dev ens224 proto kernel scope link src 10.12.12.17 metric 103 XXX
10.12.13.0/24 dev ens193 proto kernel scope link src 10.12.13.16 metric 101
Besides that the arp cache looks pretty different:
Code: Select all
# arp
Address HWtype HWaddress Flags Mask Iface
10.11.12.27 ether 00:50:56:bc:72:e0 C ens256
10.0.100.172 ether 00:50:56:bc:4d:7f C ens192
gateway ether cc:03:d9:02:b6:00 C ens192
10.10.12.47 ether 00:50:56:bc:7c:85 C ens224
10.12.12.131 ether 00:50:56:bc:83:c0 C ens224
10.11.13.26 ether 00:50:56:bc:6f:b6 C ens225
10.12.13.26 ether 00:50:56:bc:41:19 C ens193
10.11.12.25 ether 00:50:56:bc:72:e0 C ens256
10.11.12.132 ether 00:50:56:bc:dd:d5 C ens256
10.12.12.27 ether 00:50:56:bc:7c:85 C ens224
10.12.12.25 ether 00:50:56:bc:7c:85 C ens224
10.10.12.25 ether 00:50:56:bc:5b:b5 C ens194
10.10.13.26 ether 00:50:56:bc:5b:1d C ens257
10.10.12.46 ether 00:50:56:bc:72:e0 C ens256
10.11.12.131 ether 00:50:56:bc:8f:26 C ens256
Code: Select all
# netstat -an | grep -w LISTEN
tcp 0 0 10.12.12.5:4444 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.5:4445 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.57:80 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.56:80 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.25:80 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.15:80 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.5:80 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.5:80 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.5:1433 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.7:1433 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.5:1433 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.5:1433 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2233 0.0.0.0:* LISTEN
tcp 0 0 10.10.12.5:4443 0.0.0.0:* LISTEN
tcp 0 0 10.11.12.5:4443 0.0.0.0:* LISTEN
tcp 0 0 10.12.12.5:4443 0.0.0.0:* LISTEN
tcp6 0 0 :::2233 :::* LISTEN
Does anybody have or had a similar issue?
Any hint would be highly appreciated!
Thanks in advance!
HotC