[FIXED] CPU usage VERY HIGH on a single core

[tommer]

I have an Asterisk 11 server (AsteriskNOW installed without FreePBX, using CentOS 6.6) running as a VM with 12 cores. The server has been updated with yum update to have the latest Asterisk and CentOS updates.

It is currently running around 1000 simultaneous SIP-channels that are evenly spread on 200 different ConfBridge conferences.

My problem is that using htop, top or mpstat i see a single CPU core being very loaded, but not the others, like if there was a single thread using a lot of CPU. But if i change htop or top to look at threads instead of processes, then there is no single thread using more than a few % CPU.
Another strange thing is that Load Average shows very low values.

See the screenshot of htop below (PID: 2270 is the complete Asterisk process, and the asterisk PID's below it are all the threads it consists of)


I am wondering:

• Is the info about one CPU core being very loaded correct? What will happen when it reaches 100%?

• If the info is correct, why does CentOS not move some of the load to other CPU's so the load is even, when you can clearly see that there is not a single thread using alot of CPU?

• Why is Load Average so low when you can clearly see that the Asterisk process uses 432% CPU? - Load Average should show something close to 4.0-4.5

To test if Load Average was working correctly i have tried to create a bash loop script to put 100% CPU load on one core, and then run the script multible times simultanously. If i run the script twice to put 100% load on 2 CPU cores then Load Average shows 2.0 correctly.

Can anybody explain this, or give me some insights?

[TrevorH]

Asterisk does not scale well with increasing connections. Once it hits a certain number the cpu usage goes sky high for only a small increase in the number of channels. That's an asterisk problem.

[tommer]

This problem does not occur suddenly when the server reaches a certain number of connections.
I can see the same with just very few connections, that one CPU is loaded more than others - just not as much as with 1000 SIP-channels ofcourse.

Also if this was an Asterisk problem, why don't I see an asterisk thread using alot of CPU?
I see alot of asterisk threads, all using very little CPU each. That suggests that Asterisk scales the load correctly on many different threads, but for some reason most of the threads are processed by the same CPU core.

I have an old CentOS 5.7 server also running Asterisk with a similar workload, where all Asterisk threads scales evenly on all CPU cores.

[TrevorH]

I'd suspect your network card uses an IRQ that's only accessible on cpu 0. Is irqbalance installed and running?

[tommer]

irqbalance is installed and running.
But it seems you are on to something about the IRQ settings.

If i look in /proc/interrupts for eth0 it shows that eth0 is using IRQ18 and has only had interrupts on CPU0. (See attachment)
If i then look in /proc/irq/18/smp_affinity_list it only shows 0 (CPU0 i suppose?) and smp_affinity shows 001

Shouldn't irqbalance balance this so that eth0 uses more than one CPU?

I can see some suggesting in forums to disable irqbalance and manually set the IRQ eth0 is using to all CPU's, but wouldn't that impact other processes like Asterisk not being load balanced between the CPU's?

I should also mention that the server is a VM running on VMware vSphere with a Intel E1000 virtual nic, without VMware Tools installed - would whether or not VMware Tools is installed or not, do any difference?

What would you recommend?

[tommer]

It seemed the problem was the E1000 vNIC i was using which did not support Side Scaling and therefor only used one CPU core.
After switching to VMXNET3 vNIC the load from the NIC is spread out on 8 cores.