syscall recvmsg() seems to hang periodically

[sauteeed]

I was running a BIND9 DNS server on CentOS 6. I found that sometimes the number of queries per second of the server suddenly became 0, and then this lasts a few seconds, usually over 10 seconds. However, the number of incoming DNS query requests never drops below 500 per second.

I added some debug logs to BIND9 source code, rebuilt and ran again, then I noticed the thread would wait for a few seconds before recvmsg() returns, periodically. While it hangs, the Recv-Q becomes full because new packets keep coming. The version is BIND-9.9.6, but unfortunately when I switch to BIND- 9.9.9-P4, the problem is still there.

I then tried another DNS server written in Golang, I noticed that the problem never came. The query rate keeps stable as it should be.

I think the ReadFromUDP() function of Golang that server used just wraps the syscall recvfrom(), and the recvfrom() there wouldn't block, either. But ReadFromUDP() seems to keep trying recvfrom(), while BIND9 is using epoll. When data's successfully read, ReadFromUDP() will then return. I'm not sure whether there is actually any difference between those two cases.

I'm using CentOS 6.4, and the kernel version is 2.6.32-358-el6.x86_64.

Has anyone ever encountered such kind of problem?

[TrevorH]

6.4 is nearly 4 years old. You need to yum update to 6.8 and the latest kernel 2.6.32-642.6.1.el6 along with all the other 2000 packages that were updated in the last 4 years.

We only ship bind 9.8.2

[sauteeed]

Thank you, TrevorH.

I'm going to try the latest CentOS. By the way, is there any known bug corresponding the problem I posted?

[TrevorH]

No idea but rpm -q --changelog kernel-2.6.32-642.13.1.el6.x86_64 has the changes for 2.6.32-358 starting on line 23832 so that means there've been around 23,000 changes to the kernel since the one you're running.

[sauteeed]

Hi, TrevorH, after more investigations, I now believe that it's caused by disk IO, not recvmsg().

I made a mistake by trusting the log messages produced by BIND9, and it then led me to a wrong direction. In fact, it was the fflush() call that blocked all the worker threads. I put logs near the recvmsg() call, so it looked like there was something wrong with the syscall.

I then googled a lot about that, and noticed someone said there was a bug regarding JBD2 process that could possibly make what I encountered happen.

Could you please tell me whether there's indeed such a bug that could make the IO performance very low? It should help me a lot to make a conclusion.

Thank you.

[TrevorH]

No idea, you'll have to check bugs.centos.org and bugzilla.redhat.com for things matching your symptoms but before you attempt that, make sure you are running the latest kernel. CentOS 6.9's kernel is already in the CR repo which would take you to kernel-2.6.32-696.el6.x86_64 - there are over 25,000 lines in the rpm changelog between 2.6.32-358 and the latest so there are a _lot_ of bugs fixed between the two versions.

[sauteeed]

OK, thanks anyway.

[Whoever]

Are you sure that you don't have rate-limiting configured?