Frontend doesn't forward nodes requets to internet

[mahmood]

I have faced an issue which wasn't present in the previous weeks.
The frontend node if able to ping 8.8.8.8 as shown below

Code: Select all

[root@rocks7 ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    425    0        0 enp2s0f0
10.1.1.0        0.0.0.0         255.255.255.0   U     425    0        0 enp3s0
172.20.54.0     0.0.0.0         255.255.255.0   U     425    0        0 enp2s0f0
rocks7.jupiterc rocks7.local    255.255.255.255 UGH   0      0        0 enp3s0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
224.0.0.0       0.0.0.0         255.255.255.0   U     0      0        0 enp3s0
255.255.255.255 0.0.0.0         255.255.255.255 UH    0      0        0 enp3s0
[root@rocks7 ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=112 time=93.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=112 time=93.0 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 93.028/93.473/93.919/0.540 ms

Now, while a node with IP 10.1.1.252 can ping 10.1.1.1 (the private interface of the frontend), it can not ping 8.8.8.8. Please see below

Code: Select all

[root@compute-0-2 ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         rocks7.local    0.0.0.0         UG    0      0        0 enp2s0f0
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 enp2s0f0
link-local      0.0.0.0         255.255.0.0     U     1003   0        0 enp2s0f0
rocks7          rocks7.local    255.255.255.255 UGH   0      0        0 enp2s0f0
224.0.0.0       0.0.0.0         255.255.255.0   U     0      0        0 enp2s0f0
255.255.255.255 0.0.0.0         255.255.255.255 UH    0      0        0 enp2s0f0
[root@compute-0-2 ~]# ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.134 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=64 time=0.158 ms
^C
--- 10.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.134/0.146/0.158/0.012 ms
[root@compute-0-2 ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

I don't know what is wrong that the node can ping the frontend but the frontend doesn't forward the request from its 10.1.1.1 interface to 172.20.54.1.

I also don't see anything suspicious in the log files. Any idea?

[jlehtone]

First a note: man route says:

This program is obsolete. For replacement check ip route.

That is not the problem though. It is just that, IMHO, the 'ip' is more informative:

Code: Select all

# ip route
default via 10.10.10.1 dev eth0 
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.69 

You can even test with it:

Code: Select all

# ip route get 8.8.8.8
8.8.8.8 via 10.10.10.1 dev eth0 src 10.10.10.69 
    cache

I honestly cannot read from the compute-0-2's output whether its route to 8.8.8.8 is via 10.10.10.1, but lets assume so.


What should happen on rocks7?

* Decision to route the packet out from rocks7 via correct interface.

Code: Select all

# ip route get 8.8.8.8 from 10.1.1.252 iif enp3s0

* Forwarding to be enabled

Code: Select all

# sysctl net.ipv4.ip_forward

* Netfilter to allow frowarding

Code: Select all

# iptables -S

* If outside subnet has no route to 10.1.1/24, then SNAT the outgoing packet

Code: Select all

# iptables -t nat -S

[mahmood]

Frontend:

Code: Select all

[root@rocks7 ~]# ip route
default via 172.20.54.1 dev enp2s0f0 proto static metric 425
10.1.1.0/24 dev enp3s0 proto kernel scope link src 10.1.1.1 metric 425
172.20.54.0/24 dev enp2s0f0 proto kernel scope link src 172.20.54.10 metric 425
172.20.54.10 via 10.1.1.1 dev enp3s0 scope link
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
224.0.0.0/24 dev enp3s0 scope link
255.255.255.255 dev enp3s0 scope link
[root@rocks7 mahmood]# ip route get 8.8.8.8
8.8.8.8 via 172.20.54.1 dev enp2s0f0 src 172.20.54.10
    cache

compute-0-2

Code: Select all

[root@compute-0-2 ~]# ip route
default via 10.1.1.1 dev enp2s0f0
10.1.1.0/24 dev enp2s0f0 proto kernel scope link src 10.1.1.252
169.254.0.0/16 dev enp2s0f0 scope link metric 1003
172.20.54.10 via 10.1.1.1 dev enp2s0f0
224.0.0.0/24 dev enp2s0f0 scope link
255.255.255.255 dev enp2s0f0 scope link
[root@compute-0-2 ~]# ip route get 8.8.8.8
8.8.8.8 via 10.1.1.1 dev enp2s0f0 src 10.1.1.252
    cache

So, we see that all things normal up to now. As you can see:
frontend: 10.1.1.1 (enp3s0)
compute-0-2: 10.1.1.252 (enp2s0f0)

However, The next command in your posts seems to be faulty here.

Code: Select all

[root@rocks7 ~]# ip route get 8.8.8.8 from 10.1.1.252 iif enp2s0f0
unreachable 8.8.8.8 from 10.1.1.252 dev lo
    cache iif *

[mahmood]

It seems that I have to write the frontend's interface that receive the packet from 10.1.1.252. However, it is still faulty.

Code: Select all

[root@rocks7 ~]# ip route get 8.8.8.8 from 10.1.1.252 iif enp3s0
unreachable 8.8.8.8 from 10.1.1.252 dev lo
    cache iif *

[mahmood]

OK. It seems that the forwarding was disabled

Code: Select all

[root@rocks7 ~]# sysctl -a | grep net.ipv4.ip_forward
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_use_pmtu = 0
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.enp2s0f0.stable_secret"
sysctl: reading key "net.ipv6.conf.enp3s0.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
sysctl: reading key "net.ipv6.conf.penp2s0f0.stable_secret"
sysctl: reading key "net.ipv6.conf.penp2s0f1.stable_secret"
sysctl: reading key "net.ipv6.conf.penp3s0.stable_secret"
sysctl: reading key "net.ipv6.conf.virbr0.stable_secret"
sysctl: reading key "net.ipv6.conf.virbr0-nic.stable_secret"

Now enabling that

Code: Select all

[root@rocks7 ~]# sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

And now compute-0-2 is able to ping 8.8.8.8

Code: Select all

[root@compute-0-2 ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=94.3 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 94.310/94.310/94.310/0.000 ms

And that seems to be solved. I remember that it was working but I don't know what happened since the restart plans on the frontend are in the order of months.

Thanks for your help.