CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
Posted: 2016/11/16 09:12:58
Greetings,
I have CentOS running as a Virtual Machine on ESXi 5.5 for my company.
The VM has the following seup: CentOS 7 64-bit, 16GB memory, 8 vCores and 180GB storage, VM version 8.
Output of hostnamectl
Recently I started getting the following messages:
and the system is responding very slowly to all actions and navigation/browsing.
here is a cut from dmesg:
What info would you like me to provide to help resolve this issue?
Thank you in advance.
I have CentOS running as a Virtual Machine on ESXi 5.5 for my company.
The VM has the following seup: CentOS 7 64-bit, 16GB memory, 8 vCores and 180GB storage, VM version 8.
Output of hostnamectl
Code: Select all
Static hostname: localhost.localdomain
Icon name: computer-vm
Chassis: vm
Machine ID: 644ee04d4f2441618ed5648aa9be6090
Boot ID: 84496b3e999c4a67852760a61830b7f8
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-327.36.2.el7.x86_64
Architecture: x86-64
Code: Select all
[root@localhost ~]# grep 'soft lockup' /var/log/messages*
--- messages ommited ---
/var/log/messages-20161113:Nov 13 02:16:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:26:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:27:09 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:46:28 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 02:46:56 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 02:54:04 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 02:54:32 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:00:25 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:00:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:01:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:01:53 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:02:21 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:02:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:03:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:09:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:10:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:10:56 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:11:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:11:53 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:12:20 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:32:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:33:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:33:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:40:37 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:41:05 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:41:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:42:09 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:42:36 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:43:05 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:43:32 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
--messages ommited --
here is a cut from dmesg:
Code: Select all
[62120.118308] NMI backtrace for cpu 3
[62120.118309] CPU: 3 PID: 19 Comm: rcuos/1 Tainted: G L ------------ 3.10.0-327.36.2.el7.x86_64 #1
[62120.118309] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118310] task: ffff8804295e5080 ti: ffff88042960c000 task.ti: ffff88042960c000
[62120.118310] RIP: 0010:[<ffffffffa00663be>] [<ffffffffa00663be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
[62120.118310] RSP: 0018:ffff88043fcc3bc8 EFLAGS: 00000246
[62120.118311] RAX: ffffc90011b40000 RBX: ffff88042935c000 RCX: 0000000000000006
[62120.118311] RDX: ffff880425709600 RSI: ffff88042935c000 RDI: 000000000000000e
[62120.118312] RBP: ffff88043fcc3bd8 R08: 0000000000000002 R09: ffff88042935a0d8
[62120.118312] R10: ffff8804246a2bc0 R11: ffffea001091a800 R12: 0000000024702a42
[62120.118313] R13: ffff880424702a40 R14: ffff88042935c000 R15: ffff8804246a2bc0
[62120.118313] FS: 0000000000000000(0000) GS:ffff88043fcc0000(0000) knlGS:0000000000000000
[62120.118314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118314] CR2: 00007efdc21a896b CR3: 000000000194a000 CR4: 00000000000407e0
[62120.118314] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118315] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118315] Stack:
[62120.118316] ffff88042935cd30 000000000000003c ffff88043fcc3c80 ffffffffa0106729
[62120.118316] ffff88042935c008 040000000d44ff00 0000000000000006 ffff8804246a2bc0
[62120.118317] 0000000000000018 ffff88042935c188 0000006000000006 ffff880424702a70
[62120.118317] Call Trace:
[62120.118317] <IRQ> d [<ffffffffa0106729>] mptscsih_qcmd+0x249/0x820 [mptscsih]
[62120.118318] [<ffffffffa00812b0>] mptspi_qcmd+0x50/0xe0 [mptspi]
[62120.118318] [<ffffffff81417f4a>] scsi_dispatch_cmd+0xaa/0x230
[62120.118319] [<ffffffff81420ed1>] scsi_request_fn+0x501/0x770
[62120.118319] [<ffffffffa0000938>] ? dec_pending+0x188/0x2e0 [dm_mod]
[62120.118319] [<ffffffff812c75b3>] __blk_run_queue+0x33/0x40
[62120.118320] [<ffffffff812c7626>] blk_run_queue+0x26/0x40
[62120.118320] [<ffffffff8141f2f8>] scsi_run_queue+0x258/0x2f0
[62120.118321] [<ffffffff81421180>] scsi_next_command+0x20/0x40
[62120.118321] [<ffffffff814212f5>] scsi_end_request+0x155/0x1d0
[62120.118322] [<ffffffff814214d3>] scsi_io_completion+0x103/0x600
[62120.118322] [<ffffffff8134f700>] ? fbcon_add_cursor_timer+0xf0/0xf0
[62120.118323] [<ffffffff81416815>] scsi_finish_command+0xd5/0x130
[62120.118323] [<ffffffff814209aa>] scsi_softirq_done+0x12a/0x150
[62120.118323] [<ffffffff812d18e0>] blk_done_softirq+0x90/0xc0
[62120.118324] [<ffffffff81084baf>] __do_softirq+0xef/0x280
[62120.118324] [<ffffffff8164841c>] call_softirq+0x1c/0x30
[62120.118325] <EOI> d [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[62120.118325] [<ffffffff810844a4>] local_bh_enable+0x94/0xa0
[62120.118325] [<ffffffff81124412>] rcu_nocb_kthread+0x232/0x370
[62120.118326] [<ffffffff810a6b80>] ? wake_up_atomic_t+0x30/0x30
[62120.118326] [<ffffffff811241e0>] ? rcu_start_gp+0x40/0x40
[62120.118327] [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[62120.118327] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[62120.118328] [<ffffffff81646a58>] ret_from_fork+0x58/0x90
[62120.118328] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[62120.118329] Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 43 07
[62120.118329] NMI backtrace for cpu 4
[62120.118330] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G L ------------ 3.10.0-327.36.2.el7.x86_64 #1
[62120.118330] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118331] task: ffff880429625080 ti: ffff880429648000 task.ti: ffff880429648000
[62120.118331] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118332] RSP: 0018:ffff88042964be98 EFLAGS: 00000286
[62120.118332] RAX: 00000000ffffffed RBX: ffff880429648000 RCX: 0100000000000000
[62120.118333] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118333] RBP: ffff88042964be98 R08: 0000000000000000 R09: ffff88042723ba51
[62120.118333] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000004
[62120.118334] R13: ffff880429648000 R14: ffff880429648000 R15: 0000000000000000
[62120.118334] FS: 0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
[62120.118335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118335] CR2: 00007f0c3da70000 CR3: 0000000425ce0000 CR4: 00000000000407e0
[62120.118336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118336] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118336] Stack:
[62120.118337] ffff88042964beb8 ffffffff8101dbff ffff880429648000 ffffffff81a691a0
[62120.118337] ffff88042964bec8 ffffffff8101e506 ffff88042964bf20 ffffffff810d64a5
[62120.118338] ffff88042964bfd8 ffff880429648000 a6a41e832f87a923 28294442c384663c
[62120.118338] Call Trace:
[62120.118338] [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118339] [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118339] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118340] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118340] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[62120.118341] NMI backtrace for cpu 5
[62120.118341] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G L ------------ 3.10.0-327.36.2.el7.x86_64 #1
[62120.118342] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118342] task: ffff880429625c00 ti: ffff88042964c000 task.ti: ffff88042964c000
[62120.118343] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118343] RSP: 0018:ffff88042964fe98 EFLAGS: 00000286
[62120.118344] RAX: 00000000ffffffed RBX: ffff88042964c000 RCX: 0100000000000000
[62120.118344] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118345] RBP: ffff88042964fe98 R08: 0000000000000000 R09: 0000000000000001
[62120.118345] R10: 000000000000001b R11: 0000000000000018 R12: 0000000000000005
[62120.118346] R13: ffff88042964c000 R14: ffff88042964c000 R15: 0000000000000000
[62120.118346] FS: 0000000000000000(0000) GS:ffff88043fd40000(0000) knlGS:0000000000000000
[62120.118347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118347] CR2: 00007fff7dee1de8 CR3: 0000000426d3f000 CR4: 00000000000407e0
[62120.118348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118348] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118348] Stack:
[62120.118349] ffff88042964feb8 ffffffff8101dbff ffff88042964c000 ffffffff81a691a0
[62120.118349] ffff88042964fec8 ffffffff8101e506 ffff88042964ff20 ffffffff810d64a5
[62120.118350] ffff88042964ffd8 ffff88042964c000 4e5279df92a5e1dd 84c822f6fd3f86e7
[62120.118350] Call Trace:
[62120.118350] [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118351] [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118351] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118352] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118352] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[62120.118353] NMI backtrace for cpu 6
[62120.118353] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G L ------------ 3.10.0-327.36.2.el7.x86_64 #1
[62120.118354] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118354] task: ffff880429626780 ti: ffff880429650000 task.ti: ffff880429650000
[62120.118355] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118355] RSP: 0018:ffff880429653e98 EFLAGS: 00000286
[62120.118356] RAX: 00000000ffffffed RBX: ffff880429650000 RCX: 0100000000000000
[62120.118356] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118357] RBP: ffff880429653e98 R08: 0000000000000000 R09: 0000000000000000
[62120.118357] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000006
[62120.118357] R13: ffff880429650000 R14: ffff880429650000 R15: 0000000000000000
[62120.118358] FS: 0000000000000000(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
[62120.118358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118359] CR2: 00000000004487a0 CR3: 0000000426d3f000 CR4: 00000000000407e0
[62120.118359] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118360] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118360] Stack:
[62120.118360] ffff880429653eb8 ffffffff8101dbff ffff880429650000 ffffffff81a691a0
[62120.118361] ffff880429653ec8 ffffffff8101e506 ffff880429653f20 ffffffff810d64a5
[62120.118361] ffff880429653fd8 ffff880429650000 8a7a87c8cd22c7e2 b3c93cce69884ae5
[62120.118362] Call Trace:
[62120.118362] [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118362] [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118363] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118363] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118364] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[62120.118365] NMI backtrace for cpu 7
[62120.118365] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G L ------------ 3.10.0-327.36.2.el7.x86_64 #1
[62120.118366] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118366] task: ffff880429627300 ti: ffff880429654000 task.ti: ffff880429654000
[62120.118367] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118367] RSP: 0018:ffff880429657e98 EFLAGS: 00000286
[62120.118367] RAX: 00000000ffffffed RBX: ffff880429654000 RCX: 0100000000000000
[62120.118368] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118368] RBP: ffff880429657e98 R08: 0000000000000000 R09: 0000000000000001
[62120.118369] R10: 000000000000003b R11: 000000000000003b R12: 0000000000000007
[62120.118369] R13: ffff880429654000 R14: ffff880429654000 R15: 0000000000000000
[62120.118370] FS: 0000000000000000(0000) GS:ffff88043fdc0000(0000) knlGS:0000000000000000
[62120.118370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118371] CR2: 00007f991c2cb000 CR3: 00000000360d9000 CR4: 00000000000407e0
[62120.118371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118372] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118372] Stack:
[62120.118372] ffff880429657eb8 ffffffff8101dbff ffff880429654000 ffffffff81a691a0
[62120.118373] ffff880429657ec8 ffffffff8101e506 ffff880429657f20 ffffffff810d64a5
[62120.118373] ffff880429657fd8 ffff880429654000 5771fa10f83c71c9 0554c02bd6397f01
[62120.118374] Call Trace:
[62120.118374] [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118374] [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118375] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118375] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118376] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Thank you in advance.