No same subnet connections with NetworkManager

[mathog]

(Rats, this went into Centos 7. Can a moderator please move this to the Centos 6 Networking Support forum?)

Centos 6.9, two servers (Dell Poweredge T630 and a SuperMicro).

These machines would connect to other machines outside their subnet. However, they could not even ping a machine
on the same subnet or return a ping or accept a connection from one. After much head scratching NetworkManager was disabled and they were reverted to the old style eth0 device names and configuration files. Then they talked to other machines on the same subnet with no issues whatsoever. Turning iptables off didn't make any difference, before or after. Here is netstat -rn on one when it wouldn't route (it is in subnet 15):

Code: Select all

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
131.215.15.0    0.0.0.0         255.255.255.0   U         0 0          0 Auto_p4p1
131.215.15.0    0.0.0.0         255.255.255.0   U         0 0          0 Auto_em1
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
0.0.0.0         131.215.15.254  0.0.0.0         UG        0 0          0 Auto_em1 

and here when it would:

Code: Select all

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
131.215.15.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
0.0.0.0         131.215.15.254  0.0.0.0         UG        0 0          0 eth0

The ONLY change I made in the network-script file was to rename it and change two lines like this:

Code: Select all

diff HIDE_ifcfg-Auto_em1 ifcfg-eth0
13c13
< NAME="Auto em1"
---
> NAME="System eth0"
17c17
< DEVICE=Auto_em1
---
> DEVICE=eth0
19a20,21
> NM_CONTROLLED=no
> 

Then

Code: Select all

chkconfig NetworkManager off
service NetworkManager stop
service network restart

At which point normal subnet networking kicked in. I didn't configure these machines originally, maybe it has something to do with having both Auto_em1 and Auto_p4p1? Only one of the machines has the

Code: Select all

docker0

device, so this odd subnet exclusion happens whether or not docker is in play.

Just for completeness, here are the iptables settings:

Code: Select all

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:nfs 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:webcache 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:webcache 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

[mathog]

Forgot to mention, both are plugged into the same little netgear switch, and that plugs into the campus subnet. Not sure what model it is, 1000baseT 8 port, no wifi.

Also, before Friday they were working, both were communicating normally with other machines on their subnet. On Friday there was a subsecond power glitch. The little switch was power cycled since then, not sure about the campus switches. One machine was rebooted as well, which made no difference.