High memory usage on SSH sessions when having BGP

Issues related to configuring your network
assen
Posts: 80
Joined: 2013/02/22 12:48:05
Contact:

High memory usage on SSH sessions when having BGP

Post by assen » 2017/05/13 12:45:13

Hi,

Just stumbled onto a curious situation. Two boxes, CentOS 7 with all updates, both run stock Quagga (zebra + bgpd only). Each box has an external full view BGP and there is also an iBGP between the two boxes.

Everything works as expected with one weird exception: when I open an SSH session to any of the boxes (with an idle bash shell), the SSH session uses 700+ MB of RAM. This is actual RAM allocation (RSS) - and I get a "low memory" warning in the monitoring. As soon as I log out, the memory is freed and memory usage gets back to normal. All my other CentOS boxes behave normally (RSS of SSH is 5-10 MB per session), so there must be something related to the large number of routes (600K+) each problematic box has; still can't figure out how the number of routes may affect the memory that SSH requests upon login (it is SSH, not the bash shell, who uses the memory). I have this on both boxes and it is pretty consistent. Unfortunately, this is a prod environment and I can't stop services for further investigation.

Has anybody seen anything similar?

WWell,

Assen

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: High memory usage on SSH sessions when having BGP

Post by aks » 2017/05/15 16:16:17

Has anybody seen anything similar?
Nope.
it is SSH, not the bash shell, who uses the memory
If it *really* is sshd, you can have a "custom" sshd_config in the user you logged in as (in their $HOME) - but that's from memory (long degraded mind).

Also ~/.ssh/environment and ~/.ssh/rc can execute (if PermitUserEnvironment and PermitUserRC respectively are defined in sshd_config). There's also /etc/ssh/sshrc that can execute.

I'm just wondering if these additional "autoexec" type things are different on the problem machine vs the non problematic machines. Might point you in the right direction.

doctor123
Posts: 3
Joined: 2017/07/21 11:28:20

Re: High memory usage on SSH sessions when having BGP

Post by doctor123 » 2017/07/21 11:41:57

assen wrote: Has anybody seen anything similar?
Yes.

I had exactly the same issue! After a lot of trial and error I figured out it has to do with DNS resolving. If you either put the src ip of the host you are coming from in the hosts file on the server or completely disable DNS lookups server side in sshd_config:

UseDNS no

then memory use is back normal...

Currently I don't have time to follow up with a bug report, but my 2ct. is something like gethostbyname is failing in openssh daemon and goes over the full routing table allocating memory for each lookup until whole table is processed which is only released after session is closed (or something like that).

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: High memory usage on SSH sessions when having BGP

Post by TrevorH » 2017/07/21 14:52:44

I have a theory that would need testing to find out if it's true. Edit /etc/nsswitch.conf and locate the line that says

Code: Select all

hosts:      files dns myhostname
and put a # sign just before the myhostname parameter so it reads

hosts: files dns # myhostname

Now do whatever it is that needs to be done to make the problem happen and see if it still does.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

doctor123
Posts: 3
Joined: 2017/07/21 11:28:20

Re: High memory usage on SSH sessions when having BGP

Post by doctor123 » 2017/07/24 08:26:19

TrevorH wrote: and put a # sign just before the myhostname parameter so it reads

hosts: files dns # myhostname

Now do whatever it is that needs to be done to make the problem happen and see if it still does.
Yes, this indeed also fixes the problem. Thanks!

Care to explain what your theory about this is?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: High memory usage on SSH sessions when having BGP

Post by TrevorH » 2017/07/24 08:48:06

Raise a bug on bugzilla.redhat.com against systemd. This is some systemd thing to help resolve your own hostname and this is the second time I've seen a bug in it. The first one was a crash resolving an ipv6 hostname that was "too long".

I know of zero disadvantages of doing without this thing. If you want to get shot of it permanently, make sure you comment out the myhostname parameter but leave it on the same line as the rest of the hosts: line in nsswitch.conf as systemd updates via yum have a post-script that checks if it's present on the line and "helpfully" adds it back if it's not there. Luckily their stupidity doesn't stretch as far as finding out if there is a comment character in front of it so leaving it in there but commented on the same line means that it won't come back on yum updates.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

doctor123
Posts: 3
Joined: 2017/07/21 11:28:20

Re: High memory usage on SSH sessions when having BGP

Post by doctor123 » 2017/07/24 09:10:42

Nice...

Thanks again for figuring this out. When I get some time, I will file a bug report as you propose.

jikos
Posts: 2
Joined: 2018/02/01 12:22:10

Re: High memory usage on SSH sessions when having BGP

Post by jikos » 2018/02/01 12:24:32

Seems like I've encountered the exactly same issue (sshd is being OOM-killed on BGP router when there are too many parallel connection attempts from hosts without reverse IP). God bless systemd again.

Has the bugzilla already been created? If so, do you happen to have a link / bug#?

Otherwise I'd create it.

Thanks.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: High memory usage on SSH sessions when having BGP

Post by TrevorH » 2018/02/01 12:48:07

The bypass for this is pretty easy and I've been unable to detect any downside to it.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

jikos
Posts: 2
Joined: 2018/02/01 12:22:10

Re: High memory usage on SSH sessions when having BGP

Post by jikos » 2018/02/01 22:46:21

Sure, but I'd like to open bug against systemd with RH/Centos nevertheless. This has to be fixed in systemd; walking the whole routing table (and requiring O(N) memory to do so) in order to perform reverse lookup for one particular address doesn't make any sense, and has to be fixed.

Thanks.

Post Reply