CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Hi,
Following your advice I have created a tmpfs storage as a ramdisk, with sufficiend space, mounted at /var and redirected my application to use that storage, but I am still getting the soft lockup errors.
Any ideas?
Thank you in advance
Following your advice I have created a tmpfs storage as a ramdisk, with sufficiend space, mounted at /var and redirected my application to use that storage, but I am still getting the soft lockup errors.
Any ideas?
Thank you in advance
-
- Posts: 2019
- Joined: 2015/02/17 15:14:33
- Location: Bulgaria
- Contact:
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
What is the output of the sar command (previous posts)? Can you provide the output of top ?
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Hi,
Here is the output of 'sar'
and the output of 'top'
Thanks
Here is the output of 'sar'
Code: Select all
[root@localhost ~]# sar
Linux 3.10.0-327.36.2.el7.x86_64 (localhost.localdomain) 11/28/2016 _x86_64_ (8 CPU)
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 4.41 0.00 4.13 19.49 0.00 71.97
12:20:01 AM all 4.76 0.00 4.80 25.06 0.00 65.38
12:30:01 AM all 4.18 0.00 4.20 12.18 0.00 79.44
12:40:01 AM all 5.13 0.00 5.73 29.25 0.00 59.89
12:50:02 AM all 5.02 0.00 5.74 22.64 0.00 66.60
01:00:01 AM all 4.52 0.00 5.28 25.57 0.00 64.63
01:10:01 AM all 4.82 0.00 5.11 25.32 0.00 64.75
01:20:01 AM all 4.15 0.00 4.14 18.39 0.00 73.31
01:30:04 AM all 3.93 0.00 4.01 19.57 0.00 72.49
01:40:02 AM all 4.50 0.00 4.76 20.31 0.00 70.43
01:50:01 AM all 4.97 0.00 5.60 36.62 0.00 52.80
02:00:01 AM all 3.52 0.00 3.68 12.65 0.00 80.15
02:10:01 AM all 4.30 0.00 4.29 28.63 0.00 62.78
02:20:02 AM all 4.93 0.00 5.99 26.40 0.00 62.68
02:30:01 AM all 4.50 0.00 4.39 21.52 0.00 69.59
02:40:03 AM all 4.72 0.00 5.49 23.00 0.00 66.79
02:50:01 AM all 4.58 0.00 4.93 20.83 0.00 69.67
03:00:01 AM all 4.16 0.00 4.44 16.10 0.00 75.29
03:10:01 AM all 4.05 0.00 4.00 20.06 0.00 71.89
03:20:01 AM all 4.58 0.00 5.16 26.44 0.00 63.82
03:30:02 AM all 4.50 0.00 4.82 25.18 0.00 65.50
03:40:02 AM all 4.06 0.00 3.93 32.84 0.00 59.17
03:40:02 AM CPU %user %nice %system %iowait %steal %idle
03:50:01 AM all 3.55 0.00 3.61 13.33 0.00 79.51
04:00:10 AM all 4.53 0.00 5.27 23.44 0.00 66.76
04:10:01 AM all 4.66 0.00 4.86 27.88 0.00 62.60
04:20:01 AM all 4.43 0.00 4.91 17.62 0.00 73.04
04:30:02 AM all 4.46 0.00 4.82 20.88 0.00 69.85
04:40:03 AM all 5.16 0.00 5.83 24.01 0.00 64.99
04:50:09 AM all 4.62 0.00 5.02 23.45 0.00 66.91
05:00:01 AM all 3.39 0.00 3.05 18.12 0.00 75.44
05:10:01 AM all 4.28 0.00 4.25 25.93 0.00 65.54
05:20:01 AM all 4.24 0.00 4.11 19.23 0.00 72.41
05:30:03 AM all 3.86 0.00 4.24 21.40 0.00 70.50
05:40:05 AM all 3.44 0.00 3.30 23.19 0.00 70.06
05:50:01 AM all 3.95 0.00 3.37 21.47 0.00 71.22
06:00:03 AM all 4.67 0.00 5.14 21.25 0.00 68.94
06:10:01 AM all 4.57 0.00 4.73 25.41 0.00 65.29
06:20:03 AM all 4.65 0.00 4.77 21.53 0.00 69.05
06:30:06 AM all 4.46 0.00 4.96 20.19 0.00 70.39
06:40:02 AM all 5.12 0.00 5.82 26.47 0.00 62.59
06:50:03 AM all 3.76 0.00 3.67 20.75 0.00 71.82
07:00:02 AM all 4.77 0.00 5.57 21.55 0.00 68.12
07:10:01 AM all 4.94 0.00 5.03 22.61 0.00 67.41
07:20:01 AM all 4.47 0.00 4.85 25.05 0.00 65.63
07:20:01 AM CPU %user %nice %system %iowait %steal %idle
07:30:01 AM all 3.74 0.00 3.51 16.72 0.00 76.02
07:40:02 AM all 4.90 0.00 5.56 24.30 0.00 65.24
07:50:01 AM all 4.87 0.00 5.16 22.02 0.00 67.95
08:00:03 AM all 5.30 0.00 6.72 25.41 0.00 62.57
08:10:03 AM all 5.68 0.00 5.42 23.57 0.00 65.34
08:20:01 AM all 5.37 0.00 4.48 17.48 0.00 72.67
08:30:01 AM all 5.42 0.00 4.72 20.06 0.00 69.79
08:40:04 AM all 4.74 0.00 4.31 15.50 0.00 75.45
Average: all 4.47 0.00 4.67 22.06 0.00 68.81
08:45:15 AM LINUX RESTART
08:50:01 AM CPU %user %nice %system %iowait %steal %idle
09:00:02 AM all 3.83 0.00 3.02 8.88 0.00 84.27
09:10:01 AM all 4.25 0.00 3.47 10.83 0.00 81.46
09:20:02 AM all 4.71 0.00 4.03 14.67 0.00 76.59
09:30:01 AM all 4.85 0.00 4.45 19.01 0.00 71.68
09:40:01 AM all 5.12 0.00 4.81 21.01 0.00 69.07
09:50:01 AM all 5.83 0.00 5.54 19.78 0.00 68.85
10:00:02 AM all 4.78 0.00 4.64 24.67 0.00 65.91
10:10:02 AM all 5.34 0.00 5.47 42.44 0.00 46.75
10:20:01 AM all 4.46 0.00 5.07 54.95 0.00 35.51
10:30:01 AM all 5.46 0.00 5.94 26.65 0.00 61.95
10:40:01 AM all 4.53 0.00 4.66 24.60 0.00 66.21
10:50:01 AM all 5.62 0.00 5.52 25.20 0.00 63.66
11:00:01 AM all 4.10 0.00 3.33 21.85 0.00 70.71
11:10:02 AM all 5.72 0.00 4.88 24.13 0.00 65.28
11:20:01 AM all 7.75 0.00 6.29 23.77 0.00 62.19
11:30:01 AM all 6.46 0.00 5.37 19.03 0.00 69.14
11:40:02 AM all 5.28 0.00 5.11 25.98 0.00 63.62
11:50:02 AM all 5.00 0.00 5.97 21.54 0.00 67.48
12:00:02 PM all 4.54 0.00 4.53 20.34 0.00 70.59
12:10:01 PM all 4.58 0.00 4.75 23.46 0.00 67.20
12:20:01 PM all 3.83 0.00 3.21 21.21 0.00 71.75
12:30:01 PM all 5.36 0.00 5.35 23.72 0.00 65.57
12:30:01 PM CPU %user %nice %system %iowait %steal %idle
12:40:03 PM all 4.40 0.00 4.59 21.24 0.00 69.77
12:50:01 PM all 4.88 0.00 5.64 24.27 0.00 65.21
01:00:01 PM all 5.07 0.00 5.60 27.54 0.00 61.80
01:10:01 PM all 4.68 0.00 5.32 25.21 0.00 64.79
01:20:01 PM all 4.75 0.00 5.01 19.32 0.00 70.92
01:30:01 PM all 3.99 0.00 4.14 18.43 0.00 73.44
01:40:01 PM all 3.36 0.00 3.14 17.94 0.00 75.56
01:50:01 PM all 4.42 0.00 4.58 17.77 0.00 73.24
02:00:01 PM all 4.78 0.00 5.22 20.26 0.00 69.75
02:10:01 PM all 4.31 0.00 4.19 22.17 0.00 69.33
02:20:02 PM all 3.88 0.00 3.63 15.57 0.00 76.92
02:30:04 PM all 4.45 0.00 4.17 13.03 0.00 78.34
02:40:02 PM all 6.02 0.00 5.27 21.05 0.00 67.66
02:50:01 PM all 6.69 0.00 6.09 26.99 0.00 60.23
03:00:02 PM all 6.35 0.00 5.97 26.56 0.00 61.11
03:10:02 PM all 5.60 0.00 5.04 23.01 0.00 66.34
03:20:01 PM all 4.94 0.00 5.76 28.49 0.00 60.82
03:30:05 PM all 4.12 0.00 4.15 16.85 0.00 74.89
Average: all 4.89 0.00 4.74 22.18 0.00 68.18
Code: Select all
[root@localhost ~]# top
top - 15:39:29 up 6:54, 1 user, load average: 21.54, 17.72, 19.50
Tasks: 236 total, 13 running, 223 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.8 us, 3.8 sy, 0.0 ni, 68.4 id, 22.0 wa, 0.0 hi, 0.9 si, 0.0 st
KiB Mem : 16268088 total, 14379480 free, 813588 used, 1075020 buff/cache
KiB Swap: 8257532 total, 8257532 free, 0 used. 15154296 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
39 root 20 0 0 0 0 R 100.0 0.0 103:21.17 ksoftirqd/3
26821 apache 20 0 625492 25528 5648 R 97.4 0.2 0:36.37 httpd
26456 apache 20 0 627732 27784 5648 R 94.7 0.2 0:25.78 httpd
479 apache 20 0 627568 26144 4340 R 81.6 0.2 0:02.00 httpd
19491 apache 20 0 627796 27916 5712 R 81.6 0.2 0:14.07 httpd
29567 root 20 0 157680 2164 1504 R 44.7 0.0 0:00.29 top
469 root 20 0 0 0 0 S 18.4 0.0 4:44.20 xfsaild/dm-0
2377 ajaxterm 20 0 181252 8792 1392 S 2.6 0.1 1:30.68 python
1 root 20 0 43084 5572 2404 S 0.0 0.0 0:54.29 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.36 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:37.92 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:29.96 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2
-
- Posts: 2019
- Joined: 2015/02/17 15:14:33
- Location: Bulgaria
- Contact:
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
What worries me is the fact that you have 8 cores, no more than 4-5 used at 100% and still the load average is very high.
Can you provide the output from
The output can be a little bit longer.
Edit: Check this thread and if possible try his solution, it might help.
Link:Ubuntu over VmWare
Note: Replace step 6 with:
or
Edit: changed from step 5 to step 6
Can you provide the output from
Code: Select all
vmstat 2 10
Edit: Check this thread and if possible try his solution, it might help.
Link:Ubuntu over VmWare
Note: Replace step 6 with:
Code: Select all
grub2-mkconfig -o /etc/grub2.cfg
Code: Select all
grub2-mkconfig -o /etc/grub2-efi.cfg
Last edited by hunter86_bg on 2016/11/29 14:35:08, edited 1 time in total.
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Hi,
here is the output of vmstat
I will check out the thread you provided and will report back if the solution did the trick or not.
Thanks
here is the output of vmstat
Code: Select all
[root@localhost ~]# vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 4 0 13579408 948 1636440 0 0 3 227 49 4 5 5 69 22 0
4 5 0 13533668 948 1650132 0 0 30 1038 3793 7606 14 15 39 32 0
4 3 0 13580948 948 1643932 0 0 0 2816 980 1243 2 1 62 35 0
3 3 0 13611832 948 1641224 0 0 0 350 1174 1480 5 1 50 44 0
6 3 0 13612284 948 1640736 0 0 0 515 670 927 1 1 66 33 0
6 2 0 13610932 948 1640608 0 0 0 408 604 800 2 1 69 28 0
3 2 0 13615020 948 1635976 0 0 0 536 573 746 0 1 67 32 0
3 4 0 13612168 948 1640512 0 0 0 3476 563 754 2 1 62 35 0
6 3 0 13612020 948 1640532 0 0 0 330 471 583 1 1 61 38 0
5 2 0 13612500 948 1640556 0 0 0 416 613 870 2 1 65 33 0
Thanks
-
- Posts: 2019
- Joined: 2015/02/17 15:14:33
- Location: Bulgaria
- Contact:
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Note that kernel "kernel-3.10.0-514.el7.x86_64" is available.You might consider updating it.
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Hello again,
I tried the solution you suggested here
Thanks in advance.
I tried the solution you suggested here
But unfortunately it didn't work.hunter86_bg wrote:Edit: Check this thread and if possible try his solution, it might help.
Link:Ubuntu over VmWare
Note: Replace step 6 with:orCode: Select all
grub2-mkconfig -o /etc/grub2.cfg
Code: Select all
grub2-mkconfig -o /etc/grub2-efi.cfg
Thanks in advance.
-
- Posts: 2019
- Joined: 2015/02/17 15:14:33
- Location: Bulgaria
- Contact:
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Did you update the kernel also?
While I was checking oVirt , I saw that the priority of the resources used by the VM can be modified.Is this possible in your Virtualization environment.
If so, try to increase that priority.
While I was checking oVirt , I saw that the priority of the resources used by the VM can be modified.Is this possible in your Virtualization environment.
If so, try to increase that priority.
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
We're seeing similar issues on Centos 7. No problem on 6.8, but upgrading to 7 with the 3.10.0-514.6.1.el7.x86_64 kernel and nvidia driver 375.39 gives us seemingly identical behaviour, running directly on real hardware. I think this is a new kernel bug, or else something related to an nvidia driver bug. When this is going on, the CPU used by the migration task goes up to 100%.
Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
This problem usually manifests itself on disks. To check what you're waiting for you need to catch it waiting then use the various tools to discover what it's waiting on.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke