Machine hangs with syslog-ng after a while

Issues related to applications and software problems
Post Reply
andreas123
Posts: 2
Joined: 2018/09/18 06:11:47

Machine hangs with syslog-ng after a while

Post by andreas123 » 2018/09/18 06:22:40

Hello,

I am facing a strange issue with syslog-ng v3.5.2 (also faced same issue with v3.9) running on centos 6.10.
After a few hours of running syslog-ng the machine becomes unresponsive, we cannot pass the ssh login screen, it accepts username but there's no password prompt. I cannot run any commands (crontab etc).
The only way to get into the machine, is to always keep an extra ssh session and login into that to kill syslog-ng. After killing syslog-ng machine becomes responsive again.
This machine accepts on average a couple of thousants packets per second from 300-400 hosts.

$ ss -s
Total: 330 (kernel 1024)
TCP: 1124 (estab 136, closed 958, orphaned 10, synrecv 0, timewait 948/0), ports 988

Transport Total IP IPv6
* 1024 - -
RAW 0 0 0
UDP 30 18 12
TCP 166 97 69
INET 196 115 81
FRAG 0 0 0

Anybody has faced a similar behaviour before?

Thanks

User avatar
TrevorH
Forum Moderator
Posts: 23884
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Machine hangs with syslog-ng after a while

Post by TrevorH » 2018/09/18 06:54:53

Did you look at `top` at the time? Or if you have sysstat installed then you could look at the output from `sar` to see some system stats. Sounds to me most likely you have something with a memory leak and it's using up all RAM and swapping like crazy at which point the machine will slow to a crawl...
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

andreas123
Posts: 2
Joined: 2018/09/18 06:11:47

Re: Machine hangs with syslog-ng after a while

Post by andreas123 » 2018/09/18 07:41:17

TrevorH wrote:
2018/09/18 06:54:53
Did you look at `top` at the time? Or if you have sysstat installed then you could look at the output from `sar` to see some system stats. Sounds to me most likely you have something with a memory leak and it's using up all RAM and swapping like crazy at which point the machine will slow to a crawl...
Thanks for replying, it seems that cpu and memory is quite low. running syslog-ng in verbose mode it shows this error at some point:

Code: Select all

iv_fd_epoll_timerfd_create: got error 24[Too many open files]
maybe this is related?

User avatar
TrevorH
Forum Moderator
Posts: 23884
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Machine hangs with syslog-ng after a while

Post by TrevorH » 2018/09/18 08:54:23

That could be the problem. You could increase the maximum that it's allowed to use from limits.conf perhaps but I would guess it's running as root anyway?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Post Reply