Upgrade from 7 to 7.4 breaks routing
Posted: 2017/11/28 02:35:52
So I recently got tasked with upgrading a clients sites infrastructure which is a mix of CentOS and Oracle Linux with version from 6 to 7.1. Don't ask me why but they haven't patched, like ever, and the developers have constantly pushed back as they are scared of what will break and have no idea how to re-install half the applications they use.
Anyway, patching the 6.x systems has gone fine, no issues at all. But when I patch any of the 7.1 systems to the latest 7.4 release, when they come back online the network stack is all messed up. They are all static IP in a single /24 subnet managed by NetworkManager. First notice of an issue is ping doesn't work inbound to the host. If you drop the nic and re-enable you'll get a single ping response but thats it. In /var/log/messages I see these which I am sure is the culprit.
And also with FirewallD I see
Not for these servers some of them had like 1000+ outstanding yum updates FFS but I'm pretty sure its networkmanager that is causing it.
Traffic outbound from the servers is also not working so at the moment all I can really do is SSH in. If I do a route -n I get this is what I get AFTER patching
And BEFORE patching
Any thoughts? Spent a day and half on this and at a bit of a loss. I can see the route is different from before but when I try and add that 10.10.10.0 destination back in it fails.
Anyway, patching the 6.x systems has gone fine, no issues at all. But when I patch any of the 7.1 systems to the latest 7.4 release, when they come back online the network stack is all messed up. They are all static IP in a single /24 subnet managed by NetworkManager. First notice of an issue is ping doesn't work inbound to the host. If you drop the nic and re-enable you'll get a single ping response but thats it. In /var/log/messages I see these which I am sure is the culprit.
Code: Select all
Nov 27 08:33:02 buildhost2 NetworkManager[952]: <error> [1511292782.5643] platform-linux: do-add-ip4-address[2: 10.10.10.14/24]: failure 17 (File exists)
Nov 27 08:33:02 buildhost2 NetworkManager[952]: <error> [1511292782.5646] platform-linux: do-add-ip4-route[2: 0.0.0.0/0 100]: failure 101 (Network is unreachable)
Nov 27 08:33:02 buildhost2 NetworkManager[952]: <warn> [1511292782.5648] default-route: failed to add default route 0.0.0.0/0 via 10.10.10.254 dev 2 metric 100 mss 0 rt-src user with effective metric 100
Code: Select all
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: ICMP type 'beyond-scope' is not supported by the kernel for ipv6.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: beyond-scope: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: ICMP type 'failed-policy' is not supported by the kernel for ipv6.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: failed-policy: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: ICMP type 'reject-route' is not supported by the kernel for ipv6.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: reject-route: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' failed: iptables: No chain/target/match by that name.
Nov 28 15:29:46 buildhost2 firewalld[20591]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Traffic outbound from the servers is also not working so at the moment all I can really do is SSH in. If I do a route -n I get this is what I get AFTER patching
Code: Select all
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.10.254 0.0.0.0 UG 100 0 0 ens32
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
Code: Select all
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.10.254 0.0.0.0 UG 100 0 0 ens32
10.10.10.0 0.0.0.0 255.255.255.0 U 100 0 0 ens32