Packets from CentOS blocked by VPN

Issues related to configuring your network
Post Reply
adam_bise
Posts: 3
Joined: 2017/06/27 17:08:50

Packets from CentOS blocked by VPN

Post by adam_bise » 2017/06/27 17:34:37

I have the same issue, I have 2 newly installed 1611 servers on our internal network and hosts across our VPN to a satellite office cannot talk with them.

At our satellite office: (server C, Windows 2012, Site 2) In our main network: (server A and B, CentOS 1611, Site 1)

There is an IPsec vpn between site 1 (Cisco ASA 5512-x) and site 2 (Cisco 5506-x)

Server C can communicate with all other servers/hosts in site 1, except for Server A and B

Wireshark on server C shows packets destined for server A or B, tcpdump on server A and B sees those packets, and also sees packets sent back to Server C, but Server C doesn't see the return packets.

However, if server A or B establishes a connection with server C there is no issue.

The issue happens after some inactivity and persists until I repeatedly try to ping or connect, then it works fine for several minutes or until a period of inactivity.

On the CentOS servers I have tried all kinds of things, I set the firewall zone to trusted, verified traffic is allowed, tried shutting down firewalld. The odd part is that it is intermittent.

I have checked for another machine with a conflicting IP etc...

Server A and Server B can always connect to or ping server C, never an issue. When server C tried to talk to A or B the return packets don't make it thought the VPN or network.

What I haven't been able to figure out is why it is intermittent, and why are only the centOS 1611 servers affected.

We even have 1511 CentOS servers that are also unaffected.

I have tried installing a brand new CentOS 1611 VM and was able to replicate the issue. So I now have 3 CentOS 1611 machines with this issue.

I am at a complete loss, and any help is appreciated.

Adam
Last edited by adam_bise on 2017/06/27 22:33:57, edited 1 time in total.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Abruptly Lost Connections between two Centos 7 Servers

Post by TrevorH » 2017/06/27 18:20:59

I have split your post into its own topic since it could be significantly different to the topic you hijacked. Please read viewtopic.php?f=50&t=61715 for the original hijacked thread.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

adam_bise
Posts: 3
Joined: 2017/06/27 17:08:50

Re: Packets from CentOS blocked by VPN

Post by adam_bise » 2017/06/27 23:19:12

All I know to do is compare a working server with one of the non-working ones.

Any new centOS install has the issue, but this server does not.

Not sure what else to look for. We've looked at the VPN, there isn't any issue and everything else is working.

This is a server that doesn't have this problem, that was originally installed with 1511 media and updated. It is slightly different since it is running ntopng. There is a dummy interface and the nic is in promisc mode etc.


(Working Server)
# uname -a
Linux (edited hostname) 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:15:5d:01:97:59 brd ff:ff:ff:ff:ff:ff
3: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether ba:a4:f6:a8:23:5d brd ff:ff:ff:ff:ff:ff

# cat ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=eth0
UUID=f0701b9c-33aa-4a1c-a4db-0f2b64868217
DEVICE=eth0
ONBOOT=yes
IPADDR=192.168.x.x (edited)
PREFIX=21
GATEWAY=192.168.x.x (edited)
DNS1=192.168.x.x (edited)
DNS2=192.168.x.x (edited)
DOMAIN=edited
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no

# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: dhcpv6-client ssh
ports: 3000/tcp
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:

# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N FORWARD_IN_ZONES
-N FORWARD_IN_ZONES_SOURCE
-N FORWARD_OUT_ZONES
-N FORWARD_OUT_ZONES_SOURCE
-N FORWARD_direct
-N FWDI_public
-N FWDI_public_allow
-N FWDI_public_deny
-N FWDI_public_log
-N FWDO_public
-N FWDO_public_allow
-N FWDO_public_deny
-N FWDO_public_log
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-N IN_public
-N IN_public_allow
-N IN_public_deny
-N IN_public_log
-N OUTPUT_direct
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i eth0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o eth0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i eth0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 3000 -m conntrack --ctstate NEW -j ACCEPT

(Non-working server, fresh install)
# uname -a
Linux (edited hostname) 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 00:15:5d:01:97:76 brd ff:ff:ff:ff:ff:ff

# cat ifcfg-eth0
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="eth0"
UUID="37e608ab-5395-465a-bea5-72b5be6baf4b"
DEVICE="eth0"
ONBOOT="yes"
IPADDR="192.168.x.x" (edited)
PREFIX="21"
GATEWAY="192.168.x.x" (edited)
DNS1="192.168.x.x" (edited)
DNS2="192.168.x.x" (edited)
DOMAIN="edited"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_PRIVACY="no"
ZONE=trusted

# firewall-cmd --zone=trusted --list-all
trusted (active)
target: ACCEPT
icmp-block-inversion: no
interfaces: eth0
sources:
services:
ports:
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:

#iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N FORWARD_IN_ZONES
-N FORWARD_IN_ZONES_SOURCE
-N FORWARD_OUT_ZONES
-N FORWARD_OUT_ZONES_SOURCE
-N FORWARD_direct
-N FWDI_public
-N FWDI_public_allow
-N FWDI_public_deny
-N FWDI_public_log
-N FWDI_trusted
-N FWDI_trusted_allow
-N FWDI_trusted_deny
-N FWDI_trusted_log
-N FWDO_public
-N FWDO_public_allow
-N FWDO_public_deny
-N FWDO_public_log
-N FWDO_trusted
-N FWDO_trusted_allow
-N FWDO_trusted_deny
-N FWDO_trusted_log
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-N IN_public
-N IN_public_allow
-N IN_public_deny
-N IN_public_log
-N IN_trusted
-N IN_trusted_allow
-N IN_trusted_deny
-N IN_trusted_log
-N OUTPUT_direct
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i eth0 -j FWDI_trusted
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o eth0 -j FWDO_trusted
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDI_trusted -j FWDI_trusted_log
-A FWDI_trusted -j FWDI_trusted_deny
-A FWDI_trusted -j FWDI_trusted_allow
-A FWDI_trusted -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A FWDO_trusted -j FWDO_trusted_log
-A FWDO_trusted -j FWDO_trusted_deny
-A FWDO_trusted -j FWDO_trusted_allow
-A FWDO_trusted -j ACCEPT
-A INPUT_ZONES -i eth0 -j IN_trusted
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_trusted -j IN_trusted_log
-A IN_trusted -j IN_trusted_deny
-A IN_trusted -j IN_trusted_allow
-A IN_trusted -j ACCEPT

Again, it makes zero sense to me why the first server can communicate through the VPN with no issues and 3 other newly installed servers cannot.

I also don't understand why the problem is intermittent, seems to go away after pinging across the VPN from site 1 to site 2. Why the problem always comes back in a few minutes, why it is only on these CentOS boxes, and why the problem is only the return path in one direction of traffic, connection from site 2 to site 1 return packets back to site 2 are lost is the VPN

There are many other hosts and servers that talk back and forth over this VPN and the only ones that have issues are the newly installed CentOS machines.

Any ideas would be greatly appreciated. Running out of things to try!

adam_bise
Posts: 3
Joined: 2017/06/27 17:08:50

Re: Packets from CentOS blocked by VPN

Post by adam_bise » 2017/06/28 17:46:05

Installed a new debian VM, same issue there as well.

After installing was able to ping across VPN, after a few minutes reply packets are dropped.

Something wrong with our VPN that only affects certain Linux machines.. Frustrating!

One difference I can see is working Linux server is using pfifo_fast qdisc, but I am unclear how to change from mq to pfifo_fast or if that is even relevant.

Tried tc qdisc replace dev eth0 root pfifo_fast and saw the change in ip addr but barely understand qd's. Didn't seem to make a difference.

Post Reply