Upgrade from 7 to 7.4 breaks routing

Issues related to configuring your network
Post Reply
Puk
Posts: 4
Joined: 2017/11/28 02:24:15

Upgrade from 7 to 7.4 breaks routing

Post by Puk » 2017/11/28 02:35:52

So I recently got tasked with upgrading a clients sites infrastructure which is a mix of CentOS and Oracle Linux with version from 6 to 7.1. Don't ask me why but they haven't patched, like ever, and the developers have constantly pushed back as they are scared of what will break and have no idea how to re-install half the applications they use.

Anyway, patching the 6.x systems has gone fine, no issues at all. But when I patch any of the 7.1 systems to the latest 7.4 release, when they come back online the network stack is all messed up. They are all static IP in a single /24 subnet managed by NetworkManager. First notice of an issue is ping doesn't work inbound to the host. If you drop the nic and re-enable you'll get a single ping response but thats it. In /var/log/messages I see these which I am sure is the culprit.

Code: Select all

Nov 27 08:33:02 buildhost2 NetworkManager[952]: <error> [1511292782.5643] platform-linux: do-add-ip4-address[2: 10.10.10.14/24]: failure 17 (File exists)
Nov 27 08:33:02 buildhost2 NetworkManager[952]: <error> [1511292782.5646] platform-linux: do-add-ip4-route[2: 0.0.0.0/0 100]: failure 101 (Network is unreachable)
Nov 27 08:33:02 buildhost2 NetworkManager[952]: <warn>  [1511292782.5648] default-route: failed to add default route 0.0.0.0/0 via 10.10.10.254 dev 2 metric 100 mss 0 rt-src user with effective metric 100
And also with FirewallD I see

Code: Select all

Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: ICMP type 'beyond-scope' is not supported by the kernel for ipv6.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: beyond-scope: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: ICMP type 'failed-policy' is not supported by the kernel for ipv6.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: failed-policy: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: ICMP type 'reject-route' is not supported by the kernel for ipv6.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: reject-route: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' failed: iptables: No chain/target/match by that name.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Not for these servers some of them had like 1000+ outstanding yum updates FFS but I'm pretty sure its networkmanager that is causing it.

Traffic outbound from the servers is also not working so at the moment all I can really do is SSH in. If I do a route -n I get this is what I get AFTER patching

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.10.254   0.0.0.0         UG    100    0        0 ens32
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
And BEFORE patching

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.10.254   0.0.0.0         UG    100    0        0 ens32
10.10.10.0     0.0.0.0         255.255.255.0   U     100    0        0 ens32
Any thoughts? Spent a day and half on this and at a bit of a loss. I can see the route is different from before but when I try and add that 10.10.10.0 destination back in it fails.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Upgrade from 7 to 7.4 breaks routing

Post by TrevorH » 2017/11/28 09:51:20

Show the contents of /etc/sysconfig/network-scripts/ifcfg-* and /etc/sysconfig/network as well as any /etc/sysconfig/network-scripts/route* and rule* files.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Puk
Posts: 4
Joined: 2017/11/28 02:24:15

Re: Upgrade from 7 to 7.4 breaks routing

Post by Puk » 2017/11/28 19:14:31

TrevorH wrote:Show the contents of /etc/sysconfig/network-scripts/ifcfg-* and /etc/sysconfig/network as well as any /etc/sysconfig/network-scripts/route* and rule* files.
I may have spotted something actually. The VMs I am having the issue with were all 7.x machines that were built by our DevOps team who have all since left the company and surprise surprise they didn't document anything. When I upgrade these I get these issues, but when I upgrade any of the other 7.x VMs to the same patch level, these are all working fine. Those VMs were built from the image Ops built, whereas the DevOps image thats breaking was hand built each time by them, so its possible there is something there I am missing.

Anyway as requested.
ifcfg-ens32

Code: Select all

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=ens32
UUID=ee6f5f9b-5b87-4321-a178-3e92f58cf3d5
ONBOOT=yes
DNS1=10.10.0.1
DNS2=10.10.0.1
HWADDR=00:50:56:8D:3B:EE
IPADDR=10.10.10.123
PREFIX=24
GATEWAY=10.10.10.254
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
ifcfg-Profile_1

Code: Select all

TYPE=Ethernet
BOOTPROTO=none
DNS1=10.10.10.1
DNS2=10.10.10.2
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME="Profile 1"
UUID=8416ec5e-5119-479e-96c1-2090c0c6550a
ONBOOT=yes
IPADDR=10.10.10.14
PREFIX=24
GATEWAY=10.10.10.254
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
From that I can see that the ens32 still has the template IP address stored. If I go into network manager and toggle between ens32 and profile_1 I'll get a single ping response back.

If I compare those ifcfg files to what they were PRE UPGRADE they are identical and that is working fine.

There are no route-interface files but then again there aren't on the other working servers either.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Upgrade from 7 to 7.4 breaks routing

Post by TrevorH » 2017/11/28 19:58:55

I just noticed that your route table in the broken version uses virbr0 which means you have libvirt installed on your VM. Unless you're doing nested virtualization, you don't want libvirt inside the guest.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Puk
Posts: 4
Joined: 2017/11/28 02:24:15

Re: Upgrade from 7 to 7.4 breaks routing

Post by Puk » 2017/11/28 20:24:29

TrevorH wrote:I just noticed that your route table in the broken version uses virbr0 which means you have libvirt installed on your VM. Unless you're doing nested virtualization, you don't want libvirt inside the guest.
Ah good spot, I missed that one!

No we don't need nested virtualization but now you spotted that I checked and I've found that yes, we have some Devs running nested VMs and also some Docker installations on hosts we didn't know about. FML.

Puk
Posts: 4
Joined: 2017/11/28 02:24:15

Re: Upgrade from 7 to 7.4 breaks routing

Post by Puk » 2017/11/29 01:06:06

OK its definitly the missing route.

Before - Working

Code: Select all

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.10.254   0.0.0.0         UG    100    0        0 ens32
10.10.10.0     0.0.0.0         255.255.255.0   U     100    0        0 ens32
After - Broken

Code: Select all

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.10.254   0.0.0.0         UG    100    0        0 ens32
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
Note the missing one for the local broadcast domain. So anything form outside of that subnet IN<>OUT can talk fine, but anything in the subnet can't talk this host. If I run a ip route add 10.10.10.0/24 dev ens32 everything springs to life and work.

But then when I try and get it to persist with an entry in /etc/sysconfig/network-scripts/route-ens32 and/or route-Profile_1 is is lost on reboot anyway. So confusing why this is happening after a 7.4 upgrade.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Upgrade from 7 to 7.4 breaks routing

Post by jlehtone » 2017/11/30 11:14:43

Link-local routes should appear automatically, as part of connection. If addition of those routes fails for some reason, then explicit lines in route-* should fail for same reason.

IIRC, the CentOS 7.3 update did rebase NM noticeably.


Note 1: iproute2 shows info in bit different format than the old tools:

Code: Select all

# ip -4 ad
# ip ro
Note 2: NetworkManager can show what it knows about connections:

Code: Select all

# nmcli d s
# nmcli c s
Considering that you have two ifcfg-* files with different IP, on same subnet, most likely for same interface, ...
... sounds like "at least one too many, what is really going on"?

Note 3: there are two NM-packages that might help "old school"-like configs:

Code: Select all

# yum info NetworkManager-config-server NetworkManager-dispatcher-routing-rules

Post Reply