Issue with auto-upgrade to Centos 6.10 from 6.9

General support questions
Post Reply
wildbi111
Posts: 4
Joined: 2018/07/05 23:14:53

Issue with auto-upgrade to Centos 6.10 from 6.9

Post by wildbi111 » 2018/07/05 23:23:10

Apparently an upgrade to Centos 6.10 from 6.9 early this morning broke my network connectivity for my google cloud instance. The network interfaces are up and have IP addresses. Unable to ping default gateway. Firewall rules (google and local) have not been changed/modified. This instance has been online for several years with no recent changes made. Any suggestions would be helpful and appreciated. Does anyone have an idea what may have happened.

Also, another issue is there are lots of "martian" errors being logged to /var/log/messages and to the console which is the only way I can connect to t he virtual instance.

Bill R.

pjsr2
Posts: 614
Joined: 2014/03/27 20:11:07

Re: Issue with auto-upgrade to Centos 6.10 from 6.9

Post by pjsr2 » 2018/07/06 08:32:50

"martian" errors are caused by misconfigurations in your network. They are a sign that somewhere a network interface uses a non-routable IP address (like 127.0.0.x or 192.168.1.255) or occur sometimes when a host uses multiple NICs with addresses in the same sub-net.

If you want to suppress these messages (but that does not really solve the problem) you can set the kernel parameters (with sysctl):
net.ipv4.conf.all.log_martians=0
net.ipv4.conf.default.log_martians=0

See for example https://www.cyberciti.biz/faq/linux-log ... addresses/

wildbi111
Posts: 4
Joined: 2018/07/05 23:14:53

Re: Issue with auto-upgrade to Centos 6.10 from 6.9

Post by wildbi111 » 2018/07/06 16:05:07

re: martian errors: I have no control of the network as it is google cloud-services. But I have verified that /proc/sys/net/ipv4/conf/{all/default}/log_martians are both set to 0 and that sysctl.conf has the appropriate martian entries but after a reboot both /proc entries are set back to 1 and the errors keep coming (below). The only way to stop them is downing the eth0 interface

Jul 6 10:58:47 aewa kernel: martian source 10.128.0.3 from 46.229.168.70, on dev eth0
Jul 6 10:58:47 aewa kernel: ll header: 42:01:0a:80:00:03:42:01:0a:80:00:01:08:00

I did do a tcpdump on eth0 while trying to remote ssh in and I saw the packets coming in from the ssh client but the server was not responding with any packets. Could this be behavior caused by selinux? I cannot figure what the issue is. All the networked services are starting up ok.

Additionally, I have stopped the shorewall firewall and iptables and verified that I am accepting packets.

All this is not easy with the darn martian errors scrolling on the console screen.

wildbi111
Posts: 4
Joined: 2018/07/05 23:14:53

Re: Issue with auto-upgrade to Centos 6.10 from 6.9

Post by wildbi111 » 2018/07/06 18:38:11

OK, I have a work-around for the martian errors cluttering up the console while the eth0 is enables/up -- I stopped rsyslogd.

running tcpdump on port 22 while attempting a remote ssh session shows the packets arriving but the server is not sending anything in reply. shorewall and iptables are both stopped.

iptables shows:
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
tcpdump (pub addr x'ed out) shows:
13:26:14.252031 IP x.x.x.x.36358 > 10.128.0.3.ssh: Flags [ S ], seq 3909836448, win 29200, options [mss 1460,sackOK,TS val 42333003 ecr 0,nop,wscale 7], length 0
13:26:15.288189 IP x.x.x.x.36358 > 10.128.0.3.ssh: Flags [ S ], seq 3909836448, win 29200, options [mss 1460,sackOK,TS val 42334041 ecr 0,nop,wscale 7], length 0
sestatus shows:
SELinux status: disabled
So the only think I can figure is it is something to do with the kernel. I'm looking at /boot but not sure how to force the VM to boot u p with a different kernel.

wildbi111
Posts: 4
Joined: 2018/07/05 23:14:53

Re: Issue with auto-upgrade to Centos 6.10 from 6.9

Post by wildbi111 » 2018/07/06 19:05:57

***SOLVED***

It's amazing sometimes how simple a solution can be! The issue turned out to be a missing default route. Once that was added the networking started to work. I know I looked at the route table a dozen times and never noticed the missing default route. My work around was to add the default route to the rc.local file

Bill

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Issue with auto-upgrade to Centos 6.10 from 6.9

Post by TrevorH » 2018/07/07 09:36:55

Right problem, sounds like the wrong solution though. You should have a GATEWAY= line in one of /etc/sysconfig/network or /etc/sysconfig/network-scripts/ifcfg-*. There should be one and one only or there will be a race condition and it'll be random as to which of the GATEWAY= lines wins.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

bryantw
Posts: 1
Joined: 2018/07/08 22:35:54

Re: Issue with auto-upgrade to Centos 6.10 from 6.9

Post by bryantw » 2018/07/08 22:37:41

I'm no help, just thought I'd echo that I'm having the same issue - Centos updated to 6.10 and reboot. Now I'm getting martian source errors on the console. I can ping the local internal IP address but nothing anywhere else.

I've turned off iptables and tested - still nothing - nothing else has changed with the exception of the update to CentOs 6.10.

Gateway is there. I can ping local address, but not gateway ip address.

Post Reply