High ksoftirqd cpu usage

General support questions
Post Reply
dan223
Posts: 64
Joined: 2006/11/13 18:12:14
Contact:

High ksoftirqd cpu usage

Post by dan223 » 2017/04/19 08:52:15

We have a high ksoftirqd cpu usage on one of our CentOS 6 which is running Xen. There are a number of VM's running on here but no more than other nodes we have of a similar spec.

Find below the output of proc/interrupts, i'm not really sure what is causing it or how to resolve it.
https://www.dropbox.com/s/fjx5678qqh5ky ... s.txt?dl=0

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: High ksoftirqd cpu usage

Post by aks » 2017/04/19 16:20:32

Yeah I didn't look at your output. The thing with softirq is that it does all that irq "tempering" stuff (like coalescing and so on). It manages IRQ per CPU (you'll have a softirq per CPU).

It gets involved heavily in I/O - specifically networking and (although I'm not 100% sure) disk I/O. Have a look at those things and see if they are substantially different to the other physical nodes running the same software, may indicate problems in those areas (rather than interrupts per se).

dan223
Posts: 64
Joined: 2006/11/13 18:12:14
Contact:

Re: High ksoftirqd cpu usage

Post by dan223 » 2017/04/19 17:03:23

I had a look at both disk and networking, the disk doesn't seem to be having any issues and the networking is a fair amount but there are more on other nodes from what I can see.

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: High ksoftirqd cpu usage

Post by aks » 2017/04/21 16:33:16

Not I/O then? In that case what else could generate such HUGE (you say) of interrupts. The next prime candidate is your virtualisation layer. Unlikely to be Xen, but what are the guests doing? Perhaps CPU/interrupt usage per guest?

dan223
Posts: 64
Joined: 2006/11/13 18:12:14
Contact:

Re: High ksoftirqd cpu usage

Post by dan223 » 2017/04/21 19:10:57

I don't know thats why I'm asking! I don't think it is I/O as read/writes appear to be fine, we have had other servers where read/writes are very slow but still hasn't caused this type of interrupts. I don't know whether it is huge as such but it shows in top as using 100~% of the cpu most of the time. The guests will be doing various things, they are used by customers, but xen should normally be isolated so it isn't something we have really come across before.

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: High ksoftirqd cpu usage

Post by Whoever » 2017/04/22 03:34:01

Wasn't there a bug related to this? Is your system fully up to date?

dan223
Posts: 64
Joined: 2006/11/13 18:12:14
Contact:

Re: High ksoftirqd cpu usage

Post by dan223 » 2017/04/22 08:33:11

Yes fully up to date

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: High ksoftirqd cpu usage

Post by aks » 2017/04/24 16:53:51

You've checked the hardware, firmware aqnd device drivers and it's all okay? Beyond that, not much to go on. It may be that "hard" interrupts are happening faster than the hardware can keep up and passing it onto softirq (although the interrupts can still overwhelm softirq, obviously). In top, there's the CPU status lines (on SMP kernels one per CPU), you'll see the hi, si and st counters. hi : time spent servicing hardware interrupts, si : time spent servicing software interrupts, st : time stolen from this vm by the hypervisor. If you have high values for hi and si then it's probably a hardware problem.

Given you seem to be on a "larger" SMP system (what 32 cores?) Perhaps the issue is with affinity - in the sense that servicing is "moving" between cores and never completing. There is a smp_affinity variable for each process, but that's not really scalable (nor easy).

Looking at the interrupts you posted, we see the lines with larger values across all cores are meagsas (storage), em1 (network) and rescheduling interrupts (although the posted output is not in a really good format and I'm often cross-eyed). Perhaps you could compare those values to your baseline of your system and/or the other systems (with the same hardware) that are working well?

I guess in your position I'd load that data into (something like a) spreadsheet, sort per CPU and get the top (say 5-10) across each CPU. Whatever comes out on top is likely to be the source of the problem.

Honew
Posts: 1
Joined: 2017/05/03 21:31:15

Re: High ksoftirqd cpu usage

Post by Honew » 2017/05/03 21:35:10

thank you all for your responses. I was getting the same problem of high cpu usage. did found some seemingly good responses that should be helpful. at least i hope so. thanks! i started to get headaches because of this problem...or maybe it is because i started to take hgh? gonna start searching for hgh reviews and see what might be the problem :D

Post Reply