[migration/N] high cpu consumption

General support questions
Post Reply
perucinci
Posts: 4
Joined: 2019/06/09 14:06:45

[migration/N] high cpu consumption

Post by perucinci » 2019/06/09 14:21:48

Dear Support,

We are running Centos and started seeing very frequently 'High CPU Load' especially process [ migrtion/n] eating up CPU anything
Version : CentOS release 6.9 (Final)
Kernal : 2.6.32-696.el6.x86_64
Load avg: 07:10:49 up 35 days, 15:48, 2 users, load average: 14.08, 13.26, 14.99
No of CPU : 40

Example,
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
107 root RT 0 0 0 0 S 44.6 0.0 1758:32 [migration/26]
131 root RT 0 0 0 0 S 39.0 0.0 1815:17 [migration/32]
19 root RT 0 0 0 0 S 37.7 0.0 2142:14 [migration/4]
91 root RT 0 0 0 0 S 36.7 0.0 1762:01 [migration/22]
27 root RT 0 0 0 0 R 34.4 0.0 2087:36 [migration/6]
115 root RT 0 0 0 0 S 31.1 0.0 1794:33 [migration/28]
51 root RT 0 0 0 0 S 27.5 0.0 1982:37 [migration/12]
23 root RT 0 0 0 0 S 20.3 0.0 638:17.39 [migration/5]
103 root RT 0 0 0 0 S 20.3 0.0 735:19.48 [migration/25]
99 root RT 0 0 0 0 S 16.1 0.0 1725:37 [migration/24]
59 root RT 0 0 0 0 S 8.8 0.0 2001:02 [migration/14]
147 root RT 0 0 0 0 S 3.6 0.0 1861:46 [migration/36]

Some time Load avg going upto 50+ though User process was not doing anything. We don't know if any scheduled kernel process doing this [ migration \] activity . If we looked at "Cpu(s)" seconds in the "top " then >98% of the CPU used by sys% only.

Is there known Bug in this Version and any Workaround ?


Thanks in advance,
Raj

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: [migration/N] high cpu consumption

Post by avij » 2019/06/09 14:39:08

That kernel of yours is more than two years old. Perhaps as a first step try yum update and reboot to see if the issue has been solved in the recent kernel updates.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: [migration/N] high cpu consumption

Post by TrevorH » 2019/06/09 15:53:52

No "perhaps" about it. Run yum update. There are many many more things fixed in the latest kernel series than just the possibility of fixing that.

You're also missing 505 fixes on https://access.redhat.com/errata/ for RHEL 6 that are dated post 21 March 2017, 63 of those are marked "Critical", 142 marked as "Important" and another 50 or so "Moderate" - the rest are "Low". So that's more than 200 security vulnerabilities that are classed as Important or Critical that you are missing from your system.

Update. And more often than every 2 years.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

perucinci
Posts: 4
Joined: 2019/06/09 14:06:45

Re: [migration/N] high cpu consumption

Post by perucinci » 2019/06/09 16:56:49

Thanks for the replies,

Does it mean that , we are hitting "known' Bug related to [ migration /n] process ?. If so could you please share the bug details.

Infact the server was build few months back and I think we ran 'yum update ' < 60 days before.

Anyhow will plan to update again and reboot the system as earliest .

Question : I came across the below update from some blog post,
" This appears to be a known bug. This openVZ forum post suggests to set the /proc/sys/kernel/sched_cpulimit_nr_balance to 0. "

Is it right workaround for my case ?

Note:
The file "/proc/sys/kernel/sched_cpulimit_nr_balance" not present in the system

Thanks,
Raj

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: [migration/N] high cpu consumption

Post by TrevorH » 2019/06/09 17:33:14

Openvz is not CentOS and uses a different kernel. Solutions for it may or may not be applicable but if the tuning know isn't there in CentOS then it's not applicable.
Infact the server was build few months back and I think we ran 'yum update ' < 60 days before.
How can you run `yum update` 60 days before you installed the system? It's patently obvious that you've never run yum update on this as it's still using the original CentOS 6.9 kernel and there were several updates to that in the lifetime of CentOS 6.9 and lots more during 6.10 so far.

And no, no-one knows if this is a known bug or not but the way that CentOS operates is that only the current version gets any support. If you're not running the current version then the first thing you will be told is: update.

If you have a requirement to stick on a single point release then you need to be using RHEL not CentOS and you also need to pay for an EUS RHEL subscription to allow you to stay fixed on one point release.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

perucinci
Posts: 4
Joined: 2019/06/09 14:06:45

Re: [migration/N] high cpu consumption

Post by perucinci » 2019/06/09 23:05:55

Thanks for your kind response and sorry for the confusion,

I mean to say, yum update might happen 60 days back of todays issue and not before server build :)

Will plan yum update along with down time and Keep you posted with the results.

Post Reply