NAT only partially working

[jabuzzard]

This is probably the weirdest problem I have come across with CentOS7.

Simple box has two NIC's one connects to the internal network, one is used to connect to a VDSL2 modem and a PPPoE link is brought up to get an internet connection.

I have edited sysctl.conf to enable IP forwarding, and after some shenanigans over getting interfaces into zones permanently I made my default zone external and edited the interface file in /etc/sysconfig/network-scripts to add line ZONE=internal for the internal ethernet interface.

On the face of it, it appears to be working. I am typing this post using the setup. However there is a range of things that simply don't work. Perhaps the most noticeable of which is BBC iPlayer, or in fact any of the UK based catch up services (so ITV player, 4oD/All4, Demand5 etc.) They neither work in a web browser or using the appropriate apps on an Android device, or on a Roku box.

But it is not just this, some websites simply don't work either, for example trying to go to http://fedoraforum.org/ and nothing happens. Browser sits there spinning for ever and then eventually times out giving ERR_CONNECTION_RESET in Chrome. Some other sites that don't work give ERR_TIMED_OUT instead.

It's not a problem with the internet connections because if I swap back in the crummy little router provided by my ISP everything works perfectly.

Everything on the internet suggests all I should need to do is put my external interface in the external zone, and enable IP forwarding in sysctl.conf and the job is a good one, and while it appears to be so at first sight it clearly is not.

I have done this previously using Shorewall on CentOS 5 and CentOS 6 for years, but with CentOS 7 and firewalld I seem to be snookered.

Anyone have any idea what I have missed and how do I get this working properly?

[TrevorH]

You know shorewall is in EPEL for el7 too?

[jabuzzard]

Though it is two years down the line I am going to post an update here because I now know what was going wrong back then, and the answer is very obscure and 99.99% of pages on how to do masquerading/NAT on CentOS7 using firewalld are missing a critical bit.

Basically in the end it turn out that the problem was the MTU of the PPPoE was SMALLER than the MTU of the internal ethernet connection. In particular it drops from 1500 to 1492. This manifests it self when some and only some https sites can't establish the secure connection and it times out. Sometimes it appears to be on ordinary http websites, but that is because they are immediately redirecting you to a https version. There is a suggestion that it might effect some ftp connections too, however I never saw that but I don't use ftp much these days.

Anyway the problem will always rear it's head if the MTU of the external interface is smaller than that of the internal interface. So you can also trigger it for example when you have jumbo frames on your internal interface and ordinary ethernet frames on your external interface.

Fix is to issue the following.

Code: Select all

firewall-cmd --direct --add-passthrough ipv4 -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Quite what the difference between CentOS5 with Shorewall is that you don't need to do anything like this at least by default I don't know.

[jlehtone]

Good to know.

Quite what the difference between CentOS5 with Shorewall is that you don't need to do anything like this at least by default I don't know.

I don't think that "firewall" is the issue here. Did CentOS5 with Shorewall use TCPMSS?

The MTU is part of network configuration. Either the connections are set up differently (network.service changes or NM) or kernel has changed how it handles packets and fragmentation (2.6.18 and 3.10.0 do differ).

[jabuzzard]

I don't know what CentOS 5 and Shorewall did because they have gone now. In fact getting rid of a CentOS 5 server a couple of weeks ago, where sticking a crummy router from the ISP in the middle was not an option was the reason I had actually work out what the hell was going on and solve the problem.

What I do know is that I didn't need to do any magic with TCPMSS to make it work with CentOS 5. Also I do know that the CentOS 7 is correctly getting an MTU of 1492 on the ppp0 interface. My gut feeling is that firewalld is clearly not as polished as Shorewall which was probably doing the right thing, and this is a pretty obscure thing because most of the time you masquerade stuff the MTU is the same on both interfaces.

Code: Select all

[jab@dizzy ~]$ ifconfig ppp0
ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1492
        inet 81.174.149.38  netmask 255.255.255.255  destination 195.166.130.254
        ppp  txqueuelen 3  (Point-to-Point Protocol)
        RX packets 29677804  bytes 22392580803 (20.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 35658983  bytes 40886719827 (38.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

There is a possibility that my "fix" is not 100% correct too. According to this ticket in the dd-wrt bug tracker the rule should be in the mangle table, but I am not familiar enough with firewalld to know if it is doing it right, However I no longer have problems accessing https websites so I am going with the notion it is sufficient for now.

http://svn.dd-wrt.com/ticket/3397

Although it is ancient the Linux Advanced Routing & Traffic Control HOWTO, was where I got my clue as to what was going wrong. I just happened to notice after several hours that the MTU of the pppoe link was lower and I had the idea that might be the problem, what if pmtu was failing, bit of Googling brought these links up

http://tldp.org/HOWTO/Adv-Routing-HOWTO ... overy.html
http://tldp.org/HOWTO/Adv-Routing-HOWTO ... u-mss.html

A quick adaption to firewalld syntax and it started working. Of course once you know what you are looking for you can find a couple of web pages that mention the same solution I came up with. However like I said 99.9% of pages describing NAT on CentOS 7 don't mention this and if you have a pppoe link you are masquerading then it's not going to work reliably unless you add the magic rule.