IPSEC VPN problem, tunnel established but no traffic possible

[z3us]

Does anybody know what to do now?
What logs or checks can I look into to discover the problem of no connectivity?

Please help, thanks!

[vtx]

You've come to the point where I would normally start debugging the libreswan scripts, in order to see what is going on...

Things that come to mind are for instance finding out if a tunnel is started or stopped, whether the _updown and _updown.netkey script in /usr/libexec/ipsec/ is called and with what parameters. Based upon your logging, I have the impression that _updown.netkey route-* is not called, but why....

What does "ipsec auto --route connectionname" do behind the scenes.

Also find out if the kernel parameters you set are indeed what they are supposed to like ip_forward = 1. Yes I noticed that you indeed set that one correctly (even though libreswan will do that for you), but IS it set correctly and does it remain that way... One of your sysctl params is containing a erroneous space by the way.

Note that rp_filter can play a big negative role in this, and can VERY silently cause big problems when you would be using policy routing (if you don't know here what I am talking about, you are not using it, and you're safe ).

Also plutodebug=none produces considerably less debugging output than plutodebug=all.

[z3us]

I've edited my /etc/sysctl.conf into:

Code: Select all

net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
kernel.sysrq = 0
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
kernel.core_uses_pid = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 4294967295
kernel.shmall = 268435456

I disabled rp_filter, see my ipsec verify results:

Code: Select all

Verifying installed system and configuration files

Version check and ipsec on-path                         [OK]
Libreswan 3.15 (netkey) on 2.6.32-642.6.1.el6.x86_64
Checking for IPsec support in kernel                    [OK]
 NETKEY: Testing XFRM related proc values
         ICMP default/send_redirects                    [OK]
         ICMP default/accept_redirects                  [OK]
         XFRM larval drop                               [OK]
Pluto ipsec.conf syntax                                 [OK]
Hardware random device                                  [N/A]
Checking rp_filter                                      [OK]
Checking that pluto is running                          [OK]
 Pluto listening for IKE on udp 500                     [OK]
 Pluto listening for IKE/NAT-T on udp 4500              [OK]
 Pluto ipsec.secret syntax                              [OK]
Checking 'ip' command                                   [OK]
Checking 'iptables' command                             [OK]
Checking 'prelink' command does not interfere with FIPS [PRESENT]
Checking for obsolete ipsec.conf options                [OK]
Opportunistic Encryption                                [DISABLED]

ipsec auto --route A-B
and
ipsec auto --route B-A
does nothing so far... I already rebooted both machines and retried...
I can try this while capturing packets?

[vtx]

Yes, but the fact is that ipsec auto --route con DOES something, even doesn't seem to do anything. The question is if does what it is supposed to do, but this is not so easy to explain, and even to implement.

The packet trace you showed earlier is that echo-request (ping) packets get eaten by the tunnel host A, even though they are supposed to go through the tunnel. You see the echo-request come in to the host, and there is supposed to go out an ESP packet containing that ping packet. It doesn't.

I have no idea why, but I notice there is an entry missing in your routing table. That entry should have been added by pluto as soon as the SA has been brought into the routed state, by calling the _updown.netkey script with one of the route-* options that scripts supports. On the other hand, that missing routing entry is redundant because it coincides with your default route. At least it seems so. You can verify by

Code: Select all

ip route get $remoteip

Another thing you can try is to add logging entries in all of the default five firewall mangle table chains for icmp and esp, to pinpoint it which point the packet disappears, but you might not see anything at all...

[z3us]

Yes, but the fact is that ipsec auto --route con DOES something, even doesn't seem to do anything. The question is if does what it is supposed to do, but this is not so easy to explain, and even to implement.

The packet trace you showed earlier is that echo-request (ping) packets get eaten by the tunnel host A, even though they are supposed to go through the tunnel. You see the echo-request come in to the host, and there is supposed to go out an ESP packet containing that ping packet. It doesn't.

I have no idea why, but I notice there is an entry missing in your routing table. That entry should have been added by pluto as soon as the SA has been brought into the routed state, by calling the _updown.netkey script with one of the route-* options that scripts supports. On the other hand, that missing routing entry is redundant because it coincides with your default route. At least it seems so. You can verify by

Code: Select all

ip route get $remoteip

Another thing you can try is to add logging entries in all of the default five firewall mangle table chains for icmp and esp, to pinpoint it which point the packet disappears, but you might not see anything at all...

I'm going to look into this deeper, thank you so far!

[z3us]

Found something that may be interested in pluto.log (after setting logmode to = all):

Code: Select all

Jan  6 22:04:10: | route owner of "B-A" erouted: self
Jan  6 22:04:10: | install_inbound_ipsec_sa() checking if we can route
Jan  6 22:04:10: | route owner of "B-A" erouted: self; eroute owner: self
Jan  6 22:04:10: | could_route called for B-A (kind=CK_PERMANENT)
Jan  6 22:04:10: |    routing is easy, or has resolvable near-conflict
Jan  6 22:04:10: | checking if this is a replacement state
Jan  6 22:04:10: |   st=0x7f3c0a2e79b0 ost=0x7f3c0a2e0420 st->serialno=#4 ost->serialno=#2
Jan  6 22:04:10: "B-A" #4: keeping refhim=4294901761 during rekey
Jan  6 22:04:10: | outgoing SA has refhim=4294901761
Jan  6 22:04:10: | looking for alg with transid: 12 keylen: 256 auth: 2

more lines of the same matter:

Code: Select all

Jan  6 22:04:10: | set up outgoing SA, ref=0/4294901761
Jan  6 22:04:10: | sr for #4: erouted
Jan  6 22:04:10: | route owner of "B-A" erouted: self; eroute owner: self
Jan  6 22:04:10: | route_and_eroute with c: B-A (next: none) ero:B-A esr:{(nil)} ro:B-A rosr:{(nil)} and state: #4
Jan  6 22:04:10: | eroute_connection replace eroute 10.106.0.0/24:0 --0-> 10.105.0.0/24:0 => tun.0@185.107.*.*>tun.0@185.107.*.* (raw_eroute)
Jan  6 22:04:10: | satype(9) is not used in netlink_raw_eroute.
Jan  6 22:04:10: | raw_eroute result=success
Jan  6 22:04:10: | route_and_eroute: firewall_notified: true
Jan  6 22:04:10: | route_and_eroute: instance "B-A", setting eroute_owner {spd=0x7f3c0a2dc108,sr=0x7f3c0a2dc108} to #4 (was #2) (newest_ipsec_sa=#2)

Does this matter for my routing issue?
I tried disabling the following line in ipsec.conf, but without any luck:

Code: Select all

#virtual_private=%v4:10.105.0.0/24,%v4:10.106.0.0/24

Also found these lines in pluto.log:

Code: Select all

Jan  6 22:04:11: | delete esp.5d7d55bf@185.107.*.*
Jan  6 22:04:11: | delete inbound eroute 10.105.0.0/24:0 --0-> 10.106.0.0/24:0 => unk255.10000@185.87.*.* (raw_eroute)
Jan  6 22:04:11: | raw_eroute result=success
Jan  6 22:04:11: | delete esp.5b47e213@185.87.*.*

do these lines mean my packets are deleted / getting lost?

more useful information from pluto.log:

Code: Select all

Jan  6 22:04:11: | shunt_eroute() called for connection 'B-A' to 'delete' for rt_kind 'unrouted'
Jan  6 22:04:11: | route owner of "B-A" unrouted: NULL
Jan  6 22:04:11: | command executing unroute-client
Jan  6 22:04:11: | executing unroute-client: PLUTO_VERB='unroute-client' PLUTO_VERSION='2.0' PLUTO_CONNECTION='B-A' PLUTO_INTERFACE='eth0' PLUTO_NEXT_HOP='185.107.*.*' PLUTO_ME='185.87.*.*' PLUTO_MY_ID='185.87.*.*' PLUTO_MY_CLIENT='10.106.0.0/24' PLUTO_MY_CLIENT_NET='10.106.0.0' PLUTO_MY_CLIENT_MASK='255.255.255.0' PLUTO_MY_PORT='0' PLUTO_MY_PROTOCOL='0' PLUTO_SA_REQID='16388' PLUTO_SA_TYPE='none' PLUTO_PEER='185.107.*.*' PLUTO_PEER_ID='185.107.*.*' PLUTO_PEER_CLIENT='10.105.0.0/24' PLUTO_PEER_CLIENT_NET='10.105.0.0' PLUTO_PEER_CLIENT_MASK='255.255.255.0' PLUTO_PEER_PORT='0' PLUTO_PEER_PROTOCOL='0' PLUTO_PEER_CA='' PLUTO_STACK='netkey' PLUTO_ADDTIME='0' PLUTO_CONN_POLICY='PSK+ENCRYPT+TUNNEL+PFS+UP+IKEV1_ALLOW+IKEV2_ALLOW+SAREF_TRACK+IKE_FRAG_ALLOW' PLUTO_CONN_ADDRFAMILY='ipv4' XAUTH_FAILED=0 PLUTO_IS_PEER_CISCO='0' PLUTO_PEER_DNS_INFO='' PLUTO_PEER_DOMAIN_INFO='' PLUTO_PEER_BANNER='' PLUTO_NM_CONFIGURED='0' ipsec _updown 2>&1

[vtx]

I missed this on you first post, but why do you have left* and right* inverted on both ends? Use exactly the same conn. pluto known if is left or right based upon its own IP address.

The only difference in this case should be that one side has auto=start, while the other only has auto=add. auto=start should also work, but for the sake of getting this going I'd recommend auto=add on one of the sides so that you have only one initiator.

[vtx]

For what it is worth this comes from a currently working setup (same on both ends), fairly similar to the one you want to set up:

Code: Select all

config setup
        klipsdebug=none
        plutodebug=none
        interfaces=%defaultroute
        uniqueids=no
        hidetos=no
        plutowait=no
        nat_traversal=yes
        protostack=netkey
        oe=off

Code: Select all

conn XdXsXv-Xo-XaXaXiX-Xe-XdXnXt
        leftid=213.xxx.yyy.zzz
        left=213.xxx.yyy.zzz
        leftsubnet=10.99.1.0/24
        leftsourceip=10.99.1.254
        leftnexthop=%defaultroute
        rightid=193.aaa.bbb.ccc
        right=193.aaa.bbb.ccc
        rightsubnet=172.28.99.0/24
        rightsourceip=172.28.99.254
        rightnexthop=%defaultroute
        authby=secret
        compress=no
        ikelifetime=60m
        keylife=6h
        keyingtries=%forever
        rekey=yes
        auto=start
        pfs=no
        ike=aes256-sha1-modp1024
        esp=aes256-sha1

Not everything in this config is really needed. Many options that are present simply specify the default value and could be omitted.

In any case this is working IPSec setup using netkey as protocol stack.

I notice now in this running config (for many many years) that this contains "uniqueids=no", this is probably not correct, and should be "uniqueids=yes". Don't know why this was changed, but in my case, it is a running production system, and since it is not broken I'm not going to attempt to fix it

[z3us]

Now I get this when capturing packets:
23:47:27.277615 IP 185.87.*.* > 185.107.*.*: ICMP 185.87.*.* udp port 500 unreachable, length 240

when restarting ipsec service....

Disabling firewall does not work

Also rebuild my config based on your settings, but still no luck in any results in the packet capture.
Only echo requests are shown, no replies or else

[vtx]

forceencaps=yes

damn, right, then ofcourse no esp visible in tcpdump since those were encapsulated in udp/4500.

who is blocking your udp/500?