Apache httpd high CPU usage

[L6Echo]

Hi to all.
I've a CentOS 6.6 in a virtual envroiment (VMWare 5.1) with 4vCPU and 6GB of RAM and I use it as web server.
On this sever I've two tomcat perfectly running and an httpd service which is used as connector for tomcats and for run 10 virtual hosts.
Everything works good but, sometimes, httpd process take 100% of all vCPU (at random times, for periods from 5 minutes up to 50!).
During this period I checked how many httpd processes are triggered and I saw - more or less - 20 process: normally I have from 9 to 10 process.
During this period I have a normal number of connection, so I exclude also a [D]DOS kind of attack.
I tried also to google this problem but without success: most of the problems are generated by a wrong configuration of httpd.conf who can permit to create a lot of child but it's not my case.

Thanks to anyone who can help me out!

L6Echo

[aks]

Are you swapping?

[L6Echo]

Are you swapping?

No, no swap.

[aks]

Well then you'll have to gather more data.
RAM free, swap in/out, core utilisation (4 cores so up to 400%), disk I/O, check ballooning too.
Anything in the httpd error logs or in dmesg or messages?

[L6Echo]

First of all, thanks aks for your support.

RAM
RAM TOTAL: 5852 MB
RAM FREE: 3563 MB
SWAP: 31M
During the high CPU usage I have no significant changes in the consumption of ram.

CPU Details

Code: Select all

# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz		: 2000.000
model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz		: 2000.000
model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz		: 2000.000
model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz		: 2000.000

Disk I/O
Also the use of the disc does not change during the event.
I did an iostat and a vmstat in order to check both but nothing significant was found.

Apache error_log

Code: Select all

*** glibc detected *** /usr/sbin/httpd: double free or corruption (fasttop): 0x00007f5430e36040 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e66)[0x7f542d27ae66]
/etc/httpd/modules/libphp5.so(+0x1cb84b)[0x7f542421784b]
/usr/lib64/php/modules/soap.so(+0x16c6a)[0x7f541544dc6a]
/etc/httpd/modules/libphp5.so(zend_error+0x67c)[0x7f542426720c]
/lib64/libc.so.6(+0x326a0)[0x7f542d2376a0]
/lib64/libc.so.6(+0x81175)[0x7f542d286175]
/lib64/libc.so.6(__strdup+0x16)[0x7f542d285e86]
/etc/httpd/modules/libphp5.so(+0x1cb655)[0x7f5424217655]
/usr/lib64/php/modules/soap.so(+0x16c6a)[0x7f541544dc6a]
/etc/httpd/modules/libphp5.so(zend_error+0x67c)[0x7f542426720c]
/etc/httpd/modules/libphp5.so(+0x225376)[0x7f5424271376]
/etc/httpd/modules/libphp5.so(zend_parse_parameters+0x95)[0x7f5424271b15]
/etc/httpd/modules/libphp5.so(zif_fgets+0x4f)[0x7f54241c0caf]
/etc/httpd/modules/libphp5.so(+0x269698)[0x7f54242b5698]
/etc/httpd/modules/libphp5.so(execute+0x210)[0x7f542428c7e0]
/etc/httpd/modules/libphp5.so(zend_execute_scripts+0x15d)[0x7f54242669fd]
/etc/httpd/modules/libphp5.so(php_execute_script+0x1d8)[0x7f5424214a88]
/etc/httpd/modules/libphp5.so(+0x2a4055)[0x7f54242f0055]
/usr/sbin/httpd(ap_run_handler+0x70)[0x7f542ecf5e50]
/usr/sbin/httpd(ap_invoke_handler+0xce)[0x7f542ecf970e]
/usr/sbin/httpd(ap_process_request+0x1c0)[0x7f542ed04e00]
/usr/sbin/httpd(+0x37c48)[0x7f542ed01c48]
/usr/sbin/httpd(ap_run_process_connection+0x68)[0x7f542ecfd958]
/usr/sbin/httpd(+0x3fb97)[0x7f542ed09b97]
/usr/sbin/httpd(+0x3feaa)[0x7f542ed09eaa]
/usr/sbin/httpd(ap_mpm_run+0xc1c)[0x7f542ed0ab2c]
/usr/sbin/httpd(main+0xb90)[0x7f542ece1a30]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f542d223d5d]
/usr/sbin/httpd(+0x16999)[0x7f542ece0999]

======= Memory map: ========
7f540f30c000-7f540f311000 r-xp 00000000 fd:00 2752600                    /lib64/libnss_dns-2.12.so
7f540f311000-7f540f510000 ---p 00005000 fd:00 2752600                    /lib64/libnss_dns-2.12.so
7f540f510000-7f540f511000 r--p 00004000 fd:00 2752600                    /lib64/libnss_dns-2.12.so
7f540f511000-7f540f512000 rw-p 00005000 fd:00 2752600                    /lib64/libnss_dns-2.12.so
7f540f512000-7f540f518000 r-xp 00000000 fd:00 2752713                    /lib64/libnss_winbind.so.2
7f540f518000-7f540f717000 ---p 00006000 fd:00 2752713                    /lib64/libnss_winbind.so.2
7f540f717000-7f540f718000 rw-p 00005000 fd:00 2752713                    /lib64/libnss_winbind.so.2
7f540f718000-7f540f71e000 rw-p 00000000 00:00 0 
7f540f71e000-7f541371e000 rw-s 00000000 fd:00 1844966                    /tmp/apc.C0Korp (deleted)
7f541371e000-7f5413731000 r-xp 00000000 fd:00 2106164                    /usr/lib64/php/modules/zip.so
7f5413731000-7f5413930000 ---p 00013000 fd:00 2106164                    /usr/lib64/php/modules/zip.so
7f5413930000-7f5413932000 rw-p 00012000 fd:00 2106164                    /usr/lib64/php/modules/zip.so
7f5413932000-7f541396d000 r-xp 00000000 fd:00 1971331                    /usr/lib64/libxslt.so.1.1.26
...etc...

Now I updated glibc, will let you know.
Mauro

[L6Echo]

Even after updating the result does not change. Now I updated PHP.

[L6Echo]

Nothing happend. Same problem.
Someone has an idea?

[TrevorH]

When you say that you updated glibc - how did you do that? Also if you needed to update glibc then perhaps you have other pending updates that would go on if you ran yum update

[L6Echo]

Exactly, first I update with yum update glibc but, after the update, the behaviour didn't change.
Then I update all system (yum upgrade) but, nothing happend.

[aks]

First of all,

RAM FREE: 3563 MB

That's HUGE. I'd expect it to be lower (unless you've just rebooted or something like that).

SWAP: 31M

Used 31MB of how much?

*** glibc detected *** /usr/sbin/httpd: double free or corruption (fasttop): 0x00007f5430e36040 ***

That could well be part of the problem. Double free errors are (sometimes) C++ pointer problems. That's well above my "pay grade", sorry...

You seem (based on the graphic you posted) to have fairly (in a broad sense) rhythmic CPU spikes. Perhaps the exception is thrown and that causes the CPU spikes?

but, nothing happend.

What does that mean? Nothing was upgraded? As a hint, glibc is critical to the core system (C libraries tend to have that effect when dealing with C code). Make sure you've upgraded everything.