kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

General support questions
Post Reply
Ebrahim
Posts: 4
Joined: 2019/04/26 07:25:32

kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by Ebrahim » 2019/04/26 07:44:57

Hi,

I have a dedicated server with with VMWare ESXi-6.5.0-20181103001-standard-customized installed which runs 8 small virtual machines. One of the VMs is used for hosting my wordpress site with Centos7 (centos-release-7-6.1810.2.el7.centos.x86_64) installed.

Everything was working okay, and suddenly I the server load started to jump (from ~1 to) as high as 50 and my website became unresponsive (too slow or at sometimes displayed Error 524 server down). And then after a minute or two, the load comes down back and the cycle repeats every 5-10 min...

At the times of high CPU load (or just before it starts to skyrocket), I very frequently get similar messages on my shell:

Code: Select all

Message from syslogd@web at Apr 26 08:45:20 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 25s! [ksoftirqd/4:29]

Message from syslogd@web at Apr 26 08:45:26 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:27134]

Message from syslogd@web at Apr 26 09:01:51 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [sw-collectd:7572]

Message from syslogd@web at Apr 26 09:03:38 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [lsphp:31825]

Message from syslogd@web at Apr 26 09:03:42 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [swapper/6:0]

Message from syslogd@web at Apr 26 09:03:43 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 24s! [swapper/5:0]

Message from syslogd@web at Apr 26 09:05:21 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 24s! [kworker/u16:1:22896]

Message from syslogd@web at Apr 26 09:05:50 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [ksoftirqd/2:19]

Message from syslogd@web at Apr 26 09:05:50 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [systemd-journal:3606]

Message from syslogd@web at Apr 26 09:05:50 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [ksoftirqd/2:19]

Message from syslogd@web at Apr 26 09:05:59 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ksoftirqd/2:19]

Message from syslogd@web at Apr 26 09:06:40 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [lsphp:32091]

Message from syslogd@web at Apr 26 09:06:40 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [lsphp:31812]

Message from syslogd@web at Apr 26 09:06:45 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [tuned:14968]

Message from syslogd@web at Apr 26 09:07:31 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [xfsaild/dm-0:3534]

Message from syslogd@web at Apr 26 09:07:45 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 25s! [lsphp:32094]

Message from syslogd@web at Apr 26 09:07:45 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [lsphp:32092]

Message from syslogd@web at Apr 26 09:13:53 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#7 stuck for 21s! [lsphp:32611]

Message from syslogd@web at Apr 26 09:13:53 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [swapper/2:0]

Message from syslogd@web at Apr 26 09:13:53 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [swapper/1:0]

Message from syslogd@web at Apr 26 09:13:55 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [htop:18808]

Message from syslogd@web at Apr 26 09:13:55 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 28s! [kworker/4:2:31820]

I am not an expert, and I searched through the internet.
I updated the kernel which had no effect.
I temporarily disabled csf which had no effect.
I read in the VMWare forum that this could be a result of memory overcommitment... But my dedicated server has 128GB of RAM and only 80GB is used (and it never goes up). The memory assigned to my web server is 32GB and all the other small servers won't reach 40GB in total...

I even saw another post with the same topic on this forum, but it also was not my problem, that is why I am sending a new post.

The problem starts to occur really out of the blue and after a few hours just goes away on its own for several hours later when it again shows up. It does not have any direct connection with the number of users on my site as far as I can see.


I desperately need to fix the issue since it is affecting my website hugely.
Thanks in advance

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by hunter86_bg » 2019/04/26 13:54:27

How many physical CPUs do you have on the host ?
How many logical cpus are assigned to the VMs ?

Edit: What about memory, disk usage on the host ?
You definately need to be sure if it is a result of overcommiting resources on the host or a kernel bug.

Another reason could be a very high load on your web server. What is the sar showing for your VM ?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by TrevorH » 2019/04/26 14:41:01

Check your cpu usage on the host. This looks like you've massively overcommitted the resources available to your server.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Ebrahim
Posts: 4
Joined: 2019/04/26 07:25:32

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by Ebrahim » 2019/04/26 15:13:33

Hi,
The dedicated is server is really powerful and the consumed host CPU NEVER reaches even to 40% and the consumed RAM is a fixed line 80%.
Free disk space on the host is 337 GB from 2TB.
There are 16 physical CPUs and 2 threads each:

Code: Select all

	16 CPUs x Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
---------------------
Logical processors	32
Processor type	Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Sockets	2
Cores per socket	8
Hyperthreading	Yes, enabled
Memory	127.89 GB
The Virtual Machine:

Code: Select all

CPU	8 vCPUs
Memory		32 GB
Hard disk 1	150 GB
Video card	4 MB
-----------------
Consumed host CPU		2.5 GHz
Consumed host memory	32.13 GB
Active guest memory		5.76 GB
Storage	
Provisioned			150 GB
Uncommitted			149.98 GB
Not-shared			320.97 GB
Used	320.97 GB
At the times when the problem occurs (which is usually once everyday which lasts for 2~3 hours), some the virtual machine cores would get stuck at 100% and would come down after 10 seconds or so...

The next time the problem occurs (which would be probably tomorrow morning) I would run sar on the vm and post it here, but as of now (when
everything is normal and the vm load average is 0.73 this is the output of sar:

Code: Select all

Linux 3.10.0-957.5.1.el7.x86_64 (web.dediserv01)        04/26/2019      _x86_64_        (8 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:02:01 AM     all      1.68      5.66      2.38      2.65      0.00     87.63
12:04:01 AM     all      0.12      6.17      2.15      0.01      0.00     91.55
12:06:01 AM     all      0.10      5.95      1.44      0.00      0.00     92.51
12:08:01 AM     all      0.11      6.52      1.79      0.03      0.00     91.54
12:10:01 AM     all      0.08      5.19      1.66      0.02      0.00     93.05
12:12:01 AM     all      0.09      5.78      1.73      0.06      0.00     92.34
12:14:01 AM     all      0.09      6.09      1.52      0.03      0.00     92.28
12:16:01 AM     all      0.09      5.65      1.44      0.00      0.00     92.82
12:18:01 AM     all      0.09      6.15      1.58      0.03      0.00     92.15
12:20:01 AM     all      0.09      5.59      1.47      0.00      0.00     92.85
12:22:01 AM     all      0.09      6.25      1.53      0.02      0.00     92.11
12:24:01 AM     all      0.08      6.51      1.64      0.00      0.00     91.77
12:26:01 AM     all      0.09      7.68      1.77      0.00      0.00     90.46
12:28:01 AM     all      0.08      5.42      1.78      0.02      0.00     92.70
12:30:01 AM     all      0.07      5.36      2.33      0.00      0.00     92.23
12:32:01 AM     all      0.09      5.87      1.78      0.04      0.00     92.22
12:34:01 AM     all      0.09      6.28      1.58      0.01      0.00     92.04
12:36:01 AM     all      0.09      7.07      2.40      0.00      0.00     90.44
12:38:01 AM     all      0.09      5.82      1.78      0.02      0.00     92.29
12:40:01 AM     all      0.08      6.13      1.54      0.02      0.00     92.23
12:42:01 AM     all      0.09      5.86      1.49      0.03      0.00     92.53
12:44:02 AM     all      0.08      6.18      1.88      0.00      0.00     91.85
12:46:01 AM     all      0.09      6.34      1.54      0.01      0.00     92.02
12:48:01 AM     all      0.10      5.52      1.49      0.05      0.00     92.84
12:50:01 AM     all      0.08      4.75      1.34      0.02      0.00     93.82
12:52:01 AM     all      0.08      4.47      1.57      0.03      0.00     93.85
12:54:01 AM     all      0.09      5.00      1.38      0.00      0.00     93.53
12:56:01 AM     all      0.10      6.14      1.54      0.01      0.00     92.21
12:58:01 AM     all      0.10      4.99      1.36      0.03      0.00     93.52
01:00:01 AM     all      0.10      4.54      1.57      0.01      0.00     93.77
01:02:01 AM     all      0.08      5.98      1.61      0.06      0.00     92.28
01:04:01 AM     all      0.08      6.70      2.04      0.02      0.00     91.16
01:06:01 AM     all      0.08      6.08      1.56      0.01      0.00     92.27
01:08:01 AM     all      0.09      6.87      1.75      0.03      0.00     91.26
01:10:01 AM     all      0.08      5.55      1.43      0.01      0.00     92.93
01:12:01 AM     all      0.08      6.14      1.56      0.03      0.00     92.18
01:14:01 AM     all      0.10      7.91      2.21      0.02      0.00     89.76
01:16:01 AM     all      0.09      7.11      1.80      0.00      0.00     91.00
01:18:01 AM     all      0.10      6.55      1.72      0.02      0.00     91.61
01:20:02 AM     all      0.07      4.81      1.28      0.00      0.00     93.84
01:22:01 AM     all      0.06      3.85      1.21      0.03      0.00     94.83
01:24:01 AM     all      0.07      4.19      1.16      0.00      0.00     94.58
01:26:01 AM     all      0.08      6.32      2.55      0.01      0.00     91.04
01:28:01 AM     all      0.08      4.63      1.42      0.05      0.00     93.81
01:30:01 AM     all      0.08      4.55      1.23      0.00      0.00     94.15
01:32:01 AM     all      0.09      5.51      1.43      0.03      0.00     92.95
01:34:01 AM     all      0.09      5.66      1.47      0.00      0.00     92.79
01:36:01 AM     all      0.07      5.15      1.39      0.00      0.00     93.39
01:38:01 AM     all      0.08      5.31      1.43      0.04      0.00     93.15
01:40:01 AM     all      0.09      6.55      1.54      0.01      0.00     91.81
01:42:01 AM     all      0.07      2.78      0.92      0.03      0.00     96.21
01:44:01 AM     all      0.07      3.61      1.40      0.00      0.00     94.92
01:46:01 AM     all      0.07      3.52      1.00      0.00      0.00     95.40

01:46:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
01:48:01 AM     all      0.07      3.91      1.16      0.02      0.00     94.83
01:50:01 AM     all      0.07      4.24      1.18      0.02      0.00     94.49
01:52:01 AM     all      0.09      5.10      1.38      0.03      0.00     93.40
01:54:01 AM     all      0.07      4.46      1.25      0.00      0.00     94.22
01:56:01 AM     all      0.09      4.28      1.24      0.01      0.00     94.38
01:58:02 AM     all      0.07      3.64      1.13      0.03      0.00     95.12
02:00:01 AM     all      0.10      4.77      1.45      0.01      0.00     93.68
02:02:01 AM     all      0.07      4.54      1.64      0.02      0.00     93.73
02:04:01 AM     all      0.08      5.17      1.51      0.01      0.00     93.24
02:06:01 AM     all      0.08      4.69      1.57      0.00      0.00     93.65
02:08:01 AM     all      0.08      4.15      1.69      0.03      0.00     94.05
02:10:01 AM     all      0.07      4.74      2.03      0.01      0.00     93.15
02:12:01 AM     all      0.07      4.45      1.29      0.02      0.00     94.17
02:14:01 AM     all      0.06      4.14      1.13      0.00      0.00     94.65
02:16:01 AM     all      0.07      4.09      1.15      0.01      0.00     94.68
02:18:01 AM     all      0.09      5.34      1.42      0.02      0.00     93.12
02:20:01 AM     all      0.08      4.79      1.61      0.01      0.00     93.53
02:22:01 AM     all      0.07      4.50      1.55      0.02      0.00     93.85
02:24:01 AM     all      0.06      2.96      0.95      0.00      0.00     96.03
02:26:01 AM     all      0.07      3.74      1.12      0.00      0.00     95.06
02:28:01 AM     all      0.07      3.51      1.44      0.02      0.00     94.97
02:30:01 AM     all      0.08      4.64      2.26      0.01      0.00     93.02
02:32:01 AM     all      0.06      2.87      1.25      0.44      0.00     95.38
02:34:01 AM     all      0.06      3.84      1.46      0.00      0.00     94.64
02:36:01 AM     all      0.06      2.88      0.91      0.01      0.00     96.14
02:38:01 AM     all      0.06      3.08      0.95      0.01      0.00     95.89
02:40:01 AM     all      0.06      3.32      1.02      0.00      0.00     95.59
02:42:01 AM     all      0.10      4.06      1.13      0.04      0.00     94.67
02:44:01 AM     all      0.11      4.24      1.24      0.06      0.00     94.35
02:46:01 AM     all      0.08      4.76      1.62      0.00      0.00     93.53
02:48:01 AM     all      0.07      4.01      1.47      0.02      0.00     94.43
02:50:01 AM     all      0.08      4.43      1.23      0.00      0.00     94.26
02:52:01 AM     all      0.07      3.63      1.09      0.02      0.00     95.19
02:54:01 AM     all      0.07      4.08      1.13      0.02      0.00     94.71
02:56:01 AM     all      0.10      4.60      1.56      0.01      0.00     93.72
02:58:01 AM     all      0.07      3.23      1.00      0.03      0.00     95.67
03:00:01 AM     all      0.10      3.84      1.14      0.00      0.00     94.92
03:02:01 AM     all      0.08     10.38      2.74      0.05      0.00     86.75
03:04:01 AM     all      0.06      5.48      1.92      0.00      0.00     92.54
03:06:01 AM     all      0.07      4.22      1.53      0.00      0.00     94.18
03:08:01 AM     all      0.07      3.30      1.02      0.06      0.00     95.56
03:10:01 AM     all      0.09      5.24      1.75      1.73      0.00     91.19
03:12:01 AM     all      0.18      5.40      2.26      1.16      0.00     90.99
03:14:01 AM     all      0.11      6.40      9.24      0.33      0.00     83.92
03:16:01 AM     all      9.23      4.01      1.36      0.11      0.00     85.29
03:18:01 AM     all      0.07      4.26      1.22      0.02      0.00     94.42
03:20:01 AM     all      0.07      3.30      1.01      0.00      0.00     95.62
03:22:01 AM     all      0.10      4.07      1.15      0.02      0.00     94.66
03:24:01 AM     all      0.07      4.18      1.19      0.00      0.00     94.56
03:26:01 AM     all      0.07      4.33      1.17      0.00      0.00     94.42
03:28:01 AM     all      0.06      2.90      0.90      0.02      0.00     96.12
03:30:01 AM     all      0.07      3.60      1.28      0.00      0.00     95.05
03:32:02 AM     all      0.07      4.35      1.20      0.02      0.00     94.36

03:32:02 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:34:01 AM     all      0.07      3.97      1.48      0.00      0.00     94.48
03:36:01 AM     all      0.07      3.70      1.10      0.00      0.00     95.14
03:38:01 AM     all      0.07      4.38      2.15      0.03      0.00     93.37
03:40:01 AM     all      0.06      3.45      1.56      0.01      0.00     94.93
03:42:01 AM     all      0.11      9.50      2.24      0.05      0.00     88.10
03:44:01 AM     all      0.08      3.79      1.25      0.01      0.00     94.87
03:46:01 AM     all      0.09      5.05      1.36      0.01      0.00     93.49
03:48:01 AM     all      0.06      3.08      1.37      0.02      0.00     95.46
03:50:01 AM     all      0.08      4.46      1.57      0.00      0.00     93.89
03:52:01 AM     all      0.07      4.25      1.21      0.15      0.00     94.32
03:54:01 AM     all      0.07      4.04      1.19      0.01      0.00     94.69
03:56:01 AM     all      0.09      3.79      1.84      0.02      0.00     94.25
03:58:01 AM     all      0.07      3.89      1.15      0.03      0.00     94.87
04:00:01 AM     all      0.10      4.41      1.57      0.01      0.00     93.91
04:02:01 AM     all      0.07      3.51      1.14      0.03      0.00     95.25
04:04:01 AM     all      0.06      3.75      1.09      0.02      0.00     95.08
04:06:01 AM     all      0.08      4.24      1.15      0.00      0.00     94.53
04:08:01 AM     all      0.07      3.72      1.11      0.02      0.00     95.08
04:10:01 AM     all      0.07      5.32      1.44      0.00      0.00     93.16
04:12:01 AM     all      0.09      5.93      2.14      0.03      0.00     91.82
04:14:01 AM     all      0.08      5.43      1.53      0.01      0.00     92.95
04:16:01 AM     all      0.07      4.07      1.09      0.01      0.00     94.76
04:18:01 AM     all      0.07      4.86      1.35      0.14      0.00     93.59
04:20:01 AM     all      0.06      3.12      0.97      0.00      0.00     95.85
04:22:01 AM     all      0.07      4.62      1.26      0.48      0.00     93.57
04:24:01 AM     all      0.07      4.26      1.18      0.57      0.00     93.92
04:26:01 AM     all      0.10      4.63      1.31      0.36      0.00     93.61
04:28:01 AM     all      0.08      4.79      1.43      0.56      0.00     93.15
04:30:01 AM     all      0.09      5.95      2.29      0.19      0.00     91.49
04:32:01 AM     all      0.06      3.51      1.39      0.01      0.00     95.03
04:34:01 AM     all      0.07      4.35      1.56      0.01      0.00     94.02
04:36:01 AM     all      0.07      4.41      1.77      0.08      0.00     93.67
04:38:01 AM     all      0.08      5.49      2.54      0.57      0.00     91.31
04:40:02 AM     all      0.08      6.23      2.94      0.47      0.00     90.28
04:42:01 AM     all      0.08      5.07      1.81      0.77      0.00     92.27
04:44:01 AM     all      0.10      7.13      1.77      2.00      0.00     89.01
04:46:01 AM     all      0.08      4.34      1.23      0.00      0.00     94.35
04:48:01 AM     all      0.08      4.88      1.43      0.03      0.00     93.57
04:50:01 AM     all      0.07      4.37      1.36      0.64      0.00     93.56
04:52:01 AM     all      0.08      3.91      1.37      1.47      0.00     93.17
04:54:01 AM     all      0.08      4.06      1.23      0.16      0.00     94.47
04:56:01 AM     all      0.09      3.11      1.01      0.01      0.00     95.78
04:58:01 AM     all      0.08      4.13      1.20      0.02      0.00     94.57
05:00:01 AM     all      0.12      3.55      1.09      0.08      0.00     95.16
05:02:02 AM     all      0.10      5.04      1.98      1.18      0.00     91.70
05:04:03 AM     all      0.14      9.53      7.54      0.04      0.00     82.76
05:06:12 AM     all      0.12      9.39      3.97      0.29      0.00     86.23
05:08:01 AM     all      0.12      7.89      3.25      0.13      0.00     88.61
05:10:01 AM     all      0.09      4.88      1.40      0.02      0.00     93.61
05:12:01 AM     all      0.07      3.85      1.25      0.48      0.00     94.35
05:14:01 AM     all      0.07      4.43      2.04      0.27      0.00     93.19
05:16:01 AM     all      0.07      5.06      3.22      0.08      0.00     91.57
05:18:01 AM     all      0.07      4.64      2.83      0.03      0.00     92.44

05:18:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:20:01 AM     all      0.08      6.03      2.11      0.06      0.00     91.72
05:22:06 AM     all      0.07      4.96      2.46      0.02      0.00     92.49
05:24:02 AM     all      0.09      8.72      2.21      0.22      0.00     88.76
05:26:01 AM     all      0.08      6.67      1.79      0.03      0.00     91.43
05:28:01 AM     all      0.10      5.95      1.55      0.04      0.00     92.36
05:30:02 AM     all      0.09      8.40      3.46      0.06      0.00     87.99
05:32:01 AM     all      0.13     10.42      2.65      1.01      0.00     85.79
05:34:01 AM     all      0.09      6.82      1.75      0.01      0.00     91.33
05:36:01 AM     all      0.08      5.66      1.45      0.07      0.00     92.74
05:38:03 AM     all      0.09      6.32      2.27      0.08      0.00     91.23
05:40:01 AM     all      0.08      6.13      2.15      0.05      0.00     91.60
05:42:01 AM     all      0.09      6.19      1.64      0.05      0.00     92.03
05:44:01 AM     all      0.08      4.75      1.30      0.03      0.00     93.84
05:46:01 AM     all      0.09      6.86      2.17      0.33      0.00     90.56
05:48:01 AM     all      0.09      6.39      1.68      0.03      0.00     91.81
05:50:01 AM     all      0.09      6.08      2.04      0.01      0.00     91.79
05:52:01 AM     all      0.11      9.83      2.84      0.49      0.00     86.74
05:54:01 AM     all      0.29     12.72      5.81      0.47      0.00     80.70
05:56:01 AM     all      0.14      7.73      2.70      0.43      0.00     89.00
05:58:01 AM     all      0.09      6.37      2.22      0.15      0.00     91.16
06:00:01 AM     all      0.11      6.17      1.86      0.74      0.00     91.11
06:02:01 AM     all      0.10      6.28      1.87      0.07      0.00     91.69
06:04:01 AM     all      0.08      4.81      1.34      0.00      0.00     93.76
06:06:01 AM     all      0.10      6.50      2.11      0.01      0.00     91.28
06:08:02 AM     all      0.10      7.17      2.62      1.43      0.00     88.69
06:10:03 AM     all      0.10      9.18      2.48      0.43      0.00     87.81
06:12:01 AM     all      0.09      5.58      2.00      0.03      0.00     92.30
06:14:01 AM     all      0.09      6.41      2.61      0.02      0.00     90.86
06:16:08 AM     all      0.10      7.21      2.65      0.03      0.00     90.02
06:18:01 AM     all      0.10      9.26      2.82      0.64      0.00     87.18
06:20:01 AM     all      0.09      6.22      1.61      0.03      0.00     92.06
06:22:05 AM     all      0.09      7.01      2.76      0.07      0.00     90.07
06:24:01 AM     all      0.10      8.13      2.51      1.12      0.00     88.14
06:26:02 AM     all      0.09      7.21      4.03      0.17      0.00     88.50
06:28:01 AM     all      0.10      7.60      2.76      0.05      0.00     89.48
06:30:01 AM     all      0.08      5.16      2.17      0.13      0.00     92.47
06:32:01 AM     all      0.10      8.09      3.34      0.96      0.00     87.51
06:34:01 AM     all      0.10      9.90      4.32      0.29      0.00     85.40
06:36:01 AM     all      0.10      7.70      3.87      0.25      0.00     88.09
06:38:01 AM     all      0.10      7.41      2.06      0.09      0.00     90.34
06:40:01 AM     all      0.10      8.52      2.84      0.37      0.00     88.18
06:42:01 AM     all      0.15      7.45      4.46      0.47      0.00     87.47
06:44:03 AM     all      0.19     10.30      4.42      0.84      0.00     84.25
06:46:03 AM     all      0.15     20.35      7.44      0.14      0.00     71.90
06:48:01 AM     all      0.10      8.32      2.39      0.25      0.00     88.94
06:50:01 AM     all      0.11      9.96      3.55      0.00      0.00     86.38
06:52:06 AM     all      0.10      8.59      3.38      0.36      0.00     87.56
06:54:01 AM     all      0.14     13.94      3.79      0.01      0.00     82.11
06:56:01 AM     all      0.12      7.66      1.99      0.03      0.00     90.21
06:58:01 AM     all      0.11     11.73      5.20      2.25      0.00     80.71
07:00:01 AM     all      0.12     10.18      6.68      0.65      0.00     82.37
07:02:01 AM     all      0.11      9.84      3.71      0.97      0.00     85.37
07:04:01 AM     all      0.11      9.78      3.69      0.03      0.00     86.40

07:04:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:06:01 AM     all      0.10      8.73      2.29      0.03      0.00     88.86
07:08:25 AM     all      0.10      8.65      6.53      5.51      0.00     79.21
07:10:01 AM     all      0.14     14.62      3.18      0.02      0.00     82.04
07:12:01 AM     all      0.11      8.74      2.09      0.05      0.00     89.00
07:14:01 AM     all      0.12      8.66      2.50      0.07      0.00     88.65
07:16:01 AM     all      0.09      4.60      4.17      0.22      0.00     90.92
07:18:01 AM     all      0.15     17.05      5.23      0.30      0.00     77.27
07:20:01 AM     all      0.10      8.75      2.14      0.01      0.00     89.00
07:22:16 AM     all      0.09      5.18      6.05      2.16      0.00     86.52
07:24:01 AM     all      0.18     22.31      6.32      0.34      0.00     70.85
07:26:01 AM     all      0.13     11.01      3.81      0.09      0.00     84.95
07:28:01 AM     all      0.12      9.74      2.45      0.08      0.00     87.61
07:30:01 AM     all      0.09      8.16      2.54      0.21      0.00     88.99
07:32:01 AM     all      0.10      7.92      2.72      0.10      0.00     89.16
07:34:01 AM     all      0.11      6.18      2.47      0.06      0.00     91.19
07:36:01 AM     all      0.11      8.47      2.77      2.48      0.00     86.16
07:38:01 AM     all      0.13     10.49      3.90      2.94      0.00     82.54
07:40:01 AM     all      0.12     10.34      2.89      0.02      0.00     86.63
07:42:02 AM     all      0.14     10.77      2.61      0.05      0.00     86.42
07:44:02 AM     all      0.12      9.41      4.06      0.19      0.00     86.22
07:46:01 AM     all      0.14     14.07      3.21      1.97      0.00     80.61
07:48:05 AM     all      0.15      3.01      6.06      3.91      0.00     86.87
07:50:02 AM     all      0.15     18.79      3.91      0.45      0.00     76.70
07:52:02 AM     all      0.11      7.04      2.18      0.44      0.00     90.23
07:54:01 AM     all      0.11      9.84      2.89      0.77      0.00     86.39
07:56:01 AM     all      0.14      8.43      3.24      0.06      0.00     88.13
07:58:13 AM     all      0.12     11.08      4.21      0.07      0.00     84.52
08:02:21 AM     all      0.78     18.35      7.53      0.76      0.00     72.59
08:01:55 AM     all      0.00      0.00      0.00      0.00      0.00      0.00
08:04:04 AM     all      3.27     27.62     18.45      2.71      0.00     47.96
08:06:01 AM     all      2.85     25.34      6.78      4.18      0.00     60.84
08:08:01 AM     all      4.33     12.41      5.53      1.89      0.00     75.84
08:10:02 AM     all      0.14     10.47      4.52      1.99      0.00     82.88
08:12:01 AM     all      0.15     15.99      3.50      0.07      0.00     80.30
08:14:01 AM     all      0.14     11.63      2.66      0.02      0.00     85.55
08:16:10 AM     all      0.14     11.71      2.99      0.97      0.00     84.19
08:18:02 AM     all      0.22     21.79      8.61      2.18      0.00     67.20
08:20:01 AM     all      0.13     12.03      3.98      0.01      0.00     83.85
08:22:01 AM     all      0.11      9.60      2.58      0.07      0.00     87.65
08:24:01 AM     all      0.13     12.25      2.79      0.32      0.00     84.51
08:26:02 AM     all      0.15     12.41      6.00      0.55      0.00     80.89
08:28:01 AM     all      0.17      7.93      3.47      5.28      0.00     83.15
08:30:01 AM     all      0.17     22.93      8.34      0.19      0.00     68.36
08:32:01 AM     all      0.12     11.88      3.74      0.06      0.00     84.19
08:34:01 AM     all      0.13     12.85      5.00      0.19      0.00     81.84
08:36:02 AM     all      0.20     24.03      6.99      0.12      0.00     68.66
08:38:01 AM     all      0.12     10.03      2.49      0.03      0.00     87.33
08:40:07 AM     all      0.13     11.83      5.03      0.00      0.00     83.01
08:42:03 AM     all      0.23     11.43      4.25      1.17      0.00     82.91
08:44:06 AM     all      0.15     10.58      6.06      1.11      0.00     82.09
08:46:01 AM     all      0.22     31.74     13.75      0.00      0.00     54.29
08:48:02 AM     all      0.14     16.23      4.99      0.03      0.00     78.60
08:50:03 AM     all      0.11     11.56      3.18      0.03      0.00     85.12

08:50:03 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:52:01 AM     all      0.13     13.51      4.33      1.50      0.00     80.53
08:54:01 AM     all      0.09      4.55      3.68      2.53      0.00     89.16
08:56:01 AM     all      0.25     21.28      6.59      0.20      0.00     71.68
08:58:01 AM     all      0.16     18.39      6.82      0.08      0.00     74.55
09:00:01 AM     all      0.12      8.40      2.53      0.03      0.00     88.93
09:02:05 AM     all      0.16     13.15      7.81      1.75      0.00     77.13
09:04:06 AM     all      0.15     12.20      8.63      8.08      0.00     70.93
09:06:05 AM     all      0.24     10.27     10.25     16.18      0.00     63.06
09:08:03 AM     all      0.59     24.56     42.71      1.33      0.00     30.82
09:10:01 AM     all      0.22     24.95      5.65      0.50      0.00     68.69
09:12:01 AM     all      0.13     11.06      3.26      0.07      0.00     85.49
09:14:06 AM     all      0.12      7.52      4.36     13.84      0.00     74.17
09:16:12 AM     all      0.14     12.89      3.91      2.09      0.00     80.96
09:18:01 AM     all      0.17     24.02      8.83      0.86      0.00     66.13
09:20:01 AM     all      0.13     10.73      3.05      0.00      0.00     86.10
09:22:01 AM     all      0.13     13.91      5.01      0.16      0.00     80.79
09:24:01 AM     all      0.12     11.78      5.18      0.02      0.00     82.90
09:26:01 AM     all      0.11      9.02      2.27      0.01      0.00     88.59
09:28:01 AM     all      0.12      8.91      2.27      0.03      0.00     88.67
09:30:01 AM     all      0.11     10.05      3.33      0.01      0.00     86.50
09:32:01 AM     all      0.12     10.21      2.90      0.06      0.00     86.71
09:34:01 AM     all      0.12     10.50      3.40      0.03      0.00     85.95
09:36:01 AM     all      0.10      9.58      3.63      0.00      0.00     86.68
09:38:01 AM     all      0.13     13.16      3.46      0.03      0.00     83.22
09:40:01 AM     all      0.13     11.89      3.76      0.01      0.00     84.21
09:42:01 AM     all      0.12     11.95      5.58      0.04      0.00     82.32
09:44:01 AM     all      0.11     11.29      5.22      0.00      0.00     83.37
09:46:01 AM     all      0.13     11.70      4.27      0.01      0.00     83.89
09:48:01 AM     all      0.13     12.33      3.98      0.06      0.00     83.50
09:50:01 AM     all      0.13     12.41      5.94      0.10      0.00     81.42
09:52:01 AM     all      0.11     11.55      3.07      0.05      0.00     85.23
09:54:01 AM     all      0.14     13.08      4.69      0.01      0.00     82.09
09:56:01 AM     all      0.15     12.50      4.36      0.01      0.00     82.98
09:58:01 AM     all      0.12     10.60      3.01      0.05      0.00     86.23
10:00:01 AM     all      0.14     12.40      4.96      0.01      0.00     82.49
10:02:01 AM     all      0.12     10.96      3.59      0.03      0.00     85.29
10:04:01 AM     all      0.13     11.99      3.33      0.00      0.00     84.55
10:06:01 AM     all      0.12     10.64      3.81      0.00      0.00     85.43
10:08:01 AM     all      0.11      9.38      4.62      0.06      0.00     85.83
10:10:01 AM     all      0.13     10.90      3.46      0.00      0.00     85.51
10:12:01 AM     all      0.13     10.86      3.11      0.02      0.00     85.89
10:14:01 AM     all      0.13     10.17      3.37      0.02      0.00     86.31
10:16:01 AM     all      0.11     10.36      3.82      0.01      0.00     85.71
10:18:01 AM     all      0.14     14.54      5.75      0.06      0.00     79.50
10:20:01 AM     all      0.13     12.35      3.71      0.00      0.00     83.81
10:22:01 AM     all      0.13     11.76      2.95      0.05      0.00     85.11
10:24:01 AM     all      0.12     11.37      3.93      0.01      0.00     84.58
10:26:01 AM     all      0.13     11.35      5.55      0.02      0.00     82.96
10:28:01 AM     all      0.11     10.92      5.83      0.03      0.00     83.11
10:30:01 AM     all      0.11     11.98      4.43      0.02      0.00     83.46
10:32:01 AM     all      0.13     13.10      4.97      0.04      0.00     81.75
10:34:01 AM     all      0.12     11.84      4.57      0.02      0.00     83.46
10:36:01 AM     all      0.14     12.11      4.03      0.00      0.00     83.71

10:36:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:38:01 AM     all      0.12     12.58      3.00      0.03      0.00     84.27
10:40:01 AM     all      0.10      8.36      4.37      0.00      0.00     87.17
10:42:01 AM     all      0.09      8.50      2.70      0.05      0.00     88.66
10:44:01 AM     all      0.10      8.44      2.71      0.01      0.00     88.74
10:46:01 AM     all      0.10     10.10      3.78      0.02      0.00     86.01
10:48:01 AM     all      0.12     11.49      2.86      0.04      0.00     85.50
10:50:01 AM     all      0.11     13.47      4.28      0.01      0.00     82.14
10:52:01 AM     all      0.13     13.76      4.91      0.04      0.00     81.16
10:54:02 AM     all      0.10      9.97      3.00      0.02      0.00     86.92
10:56:01 AM     all      0.15     13.17      3.70      0.00      0.00     82.98
10:58:01 AM     all      0.11     11.81      4.00      0.03      0.00     84.05
11:00:01 AM     all      0.13     10.60      2.60      0.00      0.00     86.67
11:02:01 AM     all      0.11     10.70      2.58      0.04      0.00     86.57
11:04:01 AM     all      0.10      8.86      2.79      0.00      0.00     88.25
11:06:01 AM     all      0.11     11.61      3.34      0.00      0.00     84.95
11:08:01 AM     all      0.12     10.78      3.89      0.02      0.00     85.19
11:10:01 AM     all      0.13      9.09      2.32      0.04      0.00     88.43
11:12:01 AM     all      0.12      9.66      3.01      0.06      0.00     87.15
11:14:01 AM     all      0.11      8.81      2.84      0.00      0.00     88.23
11:16:01 AM     all      0.11      9.42      3.59      0.01      0.00     86.87
11:18:01 AM     all      0.11      8.41      2.67      0.05      0.00     88.75
11:20:01 AM     all      0.11     11.44      5.97      0.00      0.00     82.48
11:22:01 AM     all      0.11     10.89      4.96      0.03      0.00     84.01
11:24:01 AM     all      0.12     11.10      4.48      0.03      0.00     84.28
11:26:01 AM     all      0.14     11.18      4.39      0.01      0.00     84.28
11:28:01 AM     all      0.10      9.18      3.04      0.03      0.00     87.66
11:30:01 AM     all      0.10      9.66      3.31      0.00      0.00     86.92
11:32:01 AM     all      0.13      9.16      3.80      0.04      0.00     86.88
11:34:01 AM     all      0.10      8.99      2.88      0.00      0.00     88.02
11:36:01 AM     all      0.10      8.78      2.29      0.00      0.00     88.83
11:38:01 AM     all      0.11     11.19      5.01      0.03      0.00     83.66
11:40:01 AM     all      0.12     13.47      4.79      0.01      0.00     81.61
11:42:01 AM     all      0.14      8.44      3.43      0.03      0.00     87.96
11:44:01 AM     all      0.10      8.08      3.69      0.01      0.00     88.12
11:46:01 AM     all      0.11     10.64      3.20      0.00      0.00     86.04
11:48:01 AM     all      0.13     12.59      3.69      0.04      0.00     83.55
11:50:01 AM     all      0.13     12.12      4.09      0.01      0.00     83.66
11:52:01 AM     all      0.11     10.06      2.61      0.05      0.00     87.18
11:54:01 AM     all      0.21     10.30      3.19      0.01      0.00     86.29
11:56:01 AM     all      0.14     10.39      3.22      0.01      0.00     86.25
11:58:01 AM     all      0.11     11.04      2.75      0.03      0.00     86.07
12:00:01 PM     all      0.14     10.98      3.37      0.01      0.00     85.50
12:02:01 PM     all      0.13     11.69      2.88      0.03      0.00     85.27
12:04:01 PM     all      0.12     11.84      3.82      0.01      0.00     84.21
12:06:01 PM     all      0.13     11.74      3.61      0.01      0.00     84.52
12:08:01 PM     all      0.11     11.49      3.12      0.05      0.00     85.23
12:10:01 PM     all      0.11     10.26      2.87      0.01      0.00     86.75
12:12:01 PM     all      0.13     11.76      3.47      0.03      0.00     84.61
12:14:01 PM     all      0.13     13.28      4.34      0.00      0.00     82.26
12:16:01 PM     all      0.12     11.88      3.00      0.01      0.00     84.99
12:18:01 PM     all      0.12     11.97      3.21      0.04      0.00     84.66
12:20:01 PM     all      0.11     11.11      2.71      0.01      0.00     86.05
12:22:01 PM     all      0.11     10.34      2.55      0.05      0.00     86.94

12:22:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:24:01 PM     all      0.13     13.25      3.71      0.01      0.00     82.89
12:26:01 PM     all      0.11      8.34      2.25      0.01      0.00     89.30
12:28:01 PM     all      0.11      8.81      2.28      0.04      0.00     88.76
12:30:01 PM     all      0.12     10.96      2.74      0.01      0.00     86.18
12:32:01 PM     all      0.11     10.62      2.62      0.03      0.00     86.62
12:34:01 PM     all      0.11     11.21      3.13      0.01      0.00     85.54
12:36:01 PM     all      0.12     11.33      3.85      0.00      0.00     84.70
12:38:01 PM     all      0.13     10.33      2.95      0.02      0.00     86.58
12:40:01 PM     all      0.14     11.49      2.76      0.01      0.00     85.61
12:42:01 PM     all      0.13     13.65      3.73      0.04      0.00     82.45
12:44:01 PM     all      0.11     10.79      2.76      0.02      0.00     86.33
12:46:02 PM     all      0.11     10.26      2.52      0.00      0.00     87.11
12:48:01 PM     all      0.11     10.56      3.34      0.02      0.00     85.97
12:50:01 PM     all      0.11     12.53      3.62      0.00      0.00     83.73
12:52:01 PM     all      0.12     10.96      2.75      0.04      0.00     86.12
12:54:01 PM     all      0.13     13.99      4.73      0.00      0.00     81.14
12:56:01 PM     all      0.14     12.01      2.97      0.01      0.00     84.88
12:58:01 PM     all      0.12     10.39      4.33      0.02      0.00     85.13
01:00:01 PM     all      0.18     13.32      5.62      0.01      0.00     80.88
01:02:01 PM     all      0.12     10.47      3.49      0.05      0.00     85.87
01:04:01 PM     all      0.11     11.50      4.05      0.00      0.00     84.34
01:06:01 PM     all      0.10     10.26      2.59      0.00      0.00     87.05
01:08:01 PM     all      0.11      9.68      2.46      0.04      0.00     87.72
01:10:01 PM     all      0.12     12.93      5.02      0.01      0.00     81.92
01:12:01 PM     all      0.12     11.09      4.00      0.03      0.00     84.76
01:14:01 PM     all      0.11     10.05      2.84      0.00      0.00     87.00
01:16:01 PM     all      0.13     10.16      3.71      0.00      0.00     86.00
01:18:01 PM     all      0.11     10.57      3.80      0.03      0.00     85.48
01:20:01 PM     all      0.12     12.87      4.39      0.01      0.00     82.62
01:22:01 PM     all      0.14     11.32      3.96      0.03      0.00     84.55
01:24:01 PM     all      0.10     10.01      3.85      0.01      0.00     86.02
01:26:01 PM     all      0.10      8.75      2.27      0.00      0.00     88.89
01:28:01 PM     all      0.11      8.89      2.36      0.03      0.00     88.62
01:30:01 PM     all      0.11     11.21      4.64      0.00      0.00     84.03
01:32:01 PM     all      0.12      9.42      3.66      0.04      0.00     86.76
01:34:01 PM     all      0.11     10.10      3.89      0.01      0.00     85.90
01:36:01 PM     all      0.11     12.18      4.20      0.04      0.00     83.46
01:38:01 PM     all      0.12     11.65      6.91      0.07      0.00     81.25
01:40:01 PM     all      0.12     11.44      4.80      0.01      0.00     83.63
01:42:01 PM     all      0.15     16.95      4.32      0.04      0.00     78.54
01:44:01 PM     all      0.13     12.35      3.90      0.01      0.00     83.61
01:46:01 PM     all      0.12     12.80      3.72      0.00      0.00     83.36
01:48:01 PM     all      0.12     13.16      3.80      0.05      0.00     82.86
01:50:01 PM     all      0.11     11.18      3.97      0.03      0.00     84.71
01:52:01 PM     all      0.13     14.06      4.16      0.02      0.00     81.62
01:54:01 PM     all      0.11      9.50      3.02      0.01      0.00     87.36
01:56:01 PM     all      0.14     11.30      3.47      0.01      0.00     85.08
01:58:01 PM     all      0.12     10.55      3.11      0.04      0.00     86.18
02:00:01 PM     all      0.15     14.68      4.13      0.00      0.00     81.04
02:02:02 PM     all      0.12     13.60      3.84      0.04      0.00     82.40
02:04:01 PM     all      0.12     11.42      2.96      0.00      0.00     85.49
02:06:01 PM     all      0.10     11.20      4.58      0.00      0.00     84.12
02:08:01 PM     all      0.12     10.50      2.76      0.03      0.00     86.60

02:08:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:10:01 PM     all      0.12     13.65      6.97      0.00      0.00     79.25
02:12:01 PM     all      0.11     11.80      6.10      0.07      0.00     81.91
02:14:01 PM     all      0.13     14.23      3.52      0.00      0.00     82.12
02:16:01 PM     all      0.12     12.31      2.99      0.00      0.00     84.57
02:18:01 PM     all      0.13     12.55      3.69      0.04      0.00     83.60
02:20:01 PM     all      0.11     12.81      3.71      0.01      0.00     83.35
02:22:01 PM     all      0.12     11.23      5.18      0.03      0.00     83.43
02:24:01 PM     all      0.13     14.70      6.82      0.02      0.00     78.34
02:26:01 PM     all      0.10      9.88      3.25      0.00      0.00     86.76
02:28:01 PM     all      0.11     10.98      3.41      0.11      0.00     85.39
02:30:01 PM     all      0.11     10.57      4.08      0.00      0.00     85.23
02:32:01 PM     all      0.13     11.69      3.23      0.05      0.00     84.90
02:34:01 PM     all      0.10     10.11      3.22      0.02      0.00     86.55
02:36:01 PM     all      0.11     10.05      2.43      0.00      0.00     87.41
02:38:01 PM     all      0.10      7.34      2.04      0.06      0.00     90.46
02:40:01 PM     all      0.11      9.71      2.52      0.01      0.00     87.64
02:42:01 PM     all      0.13     10.91      4.02      0.06      0.00     84.89
02:44:01 PM     all      0.12     11.32      3.53      0.01      0.00     85.01
02:46:01 PM     all      0.11     11.84      3.31      0.00      0.00     84.74
02:48:01 PM     all      0.11     10.44      3.19      0.07      0.00     86.19
02:50:01 PM     all      0.12     10.30      3.82      0.04      0.00     85.73
02:52:01 PM     all      0.12      9.25      3.03      0.10      0.00     87.49
02:54:01 PM     all      0.11      9.66      3.21      0.04      0.00     86.99
02:56:01 PM     all      0.13      9.00      2.50      0.01      0.00     88.35
02:58:01 PM     all      0.13     11.38      3.48      0.02      0.00     84.99
03:00:01 PM     all      0.16     11.78      4.11      0.03      0.00     83.92
03:02:01 PM     all      0.11     11.13      4.88      0.09      0.00     83.78
03:04:01 PM     all      0.10      9.75      4.28      0.00      0.00     85.87
03:06:01 PM     all      0.11      9.55      3.78      0.00      0.00     86.55
03:08:01 PM     all      0.11      9.61      3.19      0.03      0.00     87.06
03:10:01 PM     all      0.10      7.70      2.13      0.00      0.00     90.07
03:12:02 PM     all      0.12     11.26      3.53      0.04      0.00     85.06
03:14:01 PM     all      0.12     10.35      3.38      0.01      0.00     86.15
03:16:01 PM     all      0.12     11.07      3.46      0.01      0.00     85.35
03:18:01 PM     all      0.13     14.59      4.23      0.05      0.00     81.00
03:20:01 PM     all      0.14     12.85      4.92      0.03      0.00     82.06
03:22:01 PM     all      0.11     10.41      3.46      0.11      0.00     85.91
03:24:01 PM     all      0.10      9.33      2.56      0.02      0.00     88.00
03:26:01 PM     all      0.10     10.83      4.07      0.00      0.00     85.00
03:28:01 PM     all      0.13     14.21      3.55      0.03      0.00     82.09
03:30:01 PM     all      0.11     11.51      4.26      0.01      0.00     84.11
03:32:01 PM     all      0.13     11.97      3.06      0.03      0.00     84.81
03:34:01 PM     all      0.11      9.37      2.42      0.01      0.00     88.09
03:36:01 PM     all      0.11     11.30      2.83      0.00      0.00     85.76
03:38:01 PM     all      0.11      9.57      2.70      0.03      0.00     87.59
03:40:01 PM     all      0.13     11.12      4.05      0.00      0.00     84.71
03:42:01 PM     all      0.13     11.09      4.08      0.03      0.00     84.67
03:44:01 PM     all      0.12     12.72      3.76      0.01      0.00     83.39
03:46:01 PM     all      0.12     14.55      3.66      0.01      0.00     81.65
03:48:01 PM     all      0.11     12.25      3.18      0.22      0.00     84.24
03:50:01 PM     all      0.12     14.79      3.98      0.48      0.00     80.62
03:52:02 PM     all      0.12     14.91      4.27      0.12      0.00     80.58
03:54:01 PM     all      0.13     13.44      4.56      0.01      0.00     81.87

03:54:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:56:01 PM     all      0.12      9.89      3.23      0.01      0.00     86.75
03:58:01 PM     all      0.10      9.80      3.22      0.05      0.00     86.84
04:00:01 PM     all      0.12      9.43      2.52      0.00      0.00     87.93
04:02:01 PM     all      0.12     11.03      3.30      0.02      0.00     85.53
04:04:01 PM     all      0.13     14.58      5.37      0.03      0.00     79.90
04:06:01 PM     all      0.12     14.28      3.39      0.00      0.00     82.20
04:08:01 PM     all      0.13     13.86      6.12      0.03      0.00     79.87
04:10:01 PM     all      0.11     11.88      3.09      0.00      0.00     84.91
04:12:01 PM     all      0.12     11.34      3.25      0.03      0.00     85.26
04:14:01 PM     all      0.11     12.15      4.59      0.00      0.00     83.14
04:16:01 PM     all      0.09      7.98      3.60      0.01      0.00     88.32
04:18:01 PM     all      0.12      8.89      4.56      0.03      0.00     86.40
04:20:01 PM     all      0.11     10.48      3.95      0.00      0.00     85.46
04:22:01 PM     all      0.12     11.47      5.00      0.05      0.00     83.36
04:24:01 PM     all      0.11     12.35      3.82      0.02      0.00     83.70
04:26:01 PM     all      0.10     10.70      3.89      0.00      0.00     85.31
04:28:01 PM     all      0.12     13.20      5.34      0.05      0.00     81.29
04:30:01 PM     all      0.11     11.07      3.58      0.01      0.00     85.23
04:32:01 PM     all      0.11     10.87      3.23      0.07      0.00     85.72
04:34:01 PM     all      0.11      9.74      2.58      0.00      0.00     87.57
04:36:01 PM     all      0.12     13.42      4.46      0.00      0.00     82.00
04:38:01 PM     all      0.11     11.51      4.46      0.03      0.00     83.89
04:40:01 PM     all      0.11     12.01      5.44      0.00      0.00     82.43
04:42:01 PM     all      0.13     10.69      2.90      0.03      0.00     86.25
04:44:01 PM     all      0.11     10.92      3.50      0.03      0.00     85.44
04:46:01 PM     all      0.11     10.43      2.76      0.00      0.00     86.70
04:48:01 PM     all      0.11      9.98      2.61      0.04      0.00     87.26
04:50:01 PM     all      0.11     11.14      3.80      0.06      0.00     84.89
04:52:01 PM     all      0.13     11.80      3.81      0.03      0.00     84.23
04:54:01 PM     all      0.11     10.38      3.49      0.01      0.00     86.01
04:56:01 PM     all      0.13      9.43      3.23      0.02      0.00     87.20
04:58:01 PM     all      0.11      7.57      2.12      0.05      0.00     90.15
05:00:01 PM     all      0.14      9.83      3.34      0.00      0.00     86.69
05:02:01 PM     all      0.12     15.58      4.72      0.05      0.00     79.52
05:04:01 PM     all      0.12      9.87      2.65      0.01      0.00     87.35
05:06:01 PM     all      0.11     10.04      2.76      0.01      0.00     87.08
05:08:01 PM     all      0.13     11.48      6.12      0.05      0.00     82.22
Average:        all      0.14      8.95      2.99      0.17      0.00     87.75
Thanks.
Last edited by Ebrahim on 2019/04/26 15:58:55, edited 1 time in total.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by TrevorH » 2019/04/26 15:29:45

Personally I disable HT everywhere. Even when it works correctly it only gives you about a 5% improvement over a non-HT system but still enables a whole class of processor security vulnerabilities. It also manages to give a false impression of how much cpu is actually available as it counts an HT "core" as a real one giving the impression that you have twice as much cpu available as there really is.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Ebrahim
Posts: 4
Joined: 2019/04/26 07:25:32

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by Ebrahim » 2019/04/26 16:08:58

I tried to disable HT on the host by going to Advanced Settings and setting VMkernel.Boot.hyperthreading to false, but seemingly it does not take effect immediately.

I really do not want to restart the host at this point.

Is there any method by which I can find the actual cause of the problem?

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by hunter86_bg » 2019/04/27 08:42:19

As you know when this happens , try to run iostat and vmstat (in screen with | to tee -a) as every component usage should be checked.

What about a possible DDOS attack ? Have you checked the apache's logs ?

Anything in /var/log/messages , dmesg or journalctl ?

Ebrahim
Posts: 4
Joined: 2019/04/26 07:25:32

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by Ebrahim » 2019/04/28 04:53:13

Hi,
Thanks for the replies.
It's been 2 days that the problem has not repeated (After almost a week that it was there everyday). Now I wonder if it could have been really a DDOS attack. The website we host is behind cloudflare though...

How should I check whether the server is under DDOS attack if the issue occurred again?

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s

Post by hunter86_bg » 2019/04/28 05:04:34

Lots of entries in the /var/log/messages , or lots of incomplete connections checked via netstat or ss.
I'm not sure how cloudfare protects against DDOS as every bot will make a connection or 2 and will look like regular client.

Post Reply