CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

General support questions
digea
Posts: 9
Joined: 2016/11/16 08:43:51

CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/16 09:12:58

Greetings,

I have CentOS running as a Virtual Machine on ESXi 5.5 for my company.
The VM has the following seup: CentOS 7 64-bit, 16GB memory, 8 vCores and 180GB storage, VM version 8.

Output of hostnamectl

Code: Select all

   Static hostname: localhost.localdomain
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 644ee04d4f2441618ed5648aa9be6090
           Boot ID: 84496b3e999c4a67852760a61830b7f8
    Virtualization: vmware
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.36.2.el7.x86_64
      Architecture: x86-64
Recently I started getting the following messages:

Code: Select all

[root@localhost ~]# grep 'soft lockup' /var/log/messages*
--- messages ommited ---
/var/log/messages-20161113:Nov 13 02:16:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:26:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:27:09 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:46:28 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 02:46:56 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 02:54:04 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 02:54:32 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:00:25 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:00:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:01:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:01:53 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:02:21 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:02:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:03:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:09:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:10:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:10:56 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:11:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:11:53 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:12:20 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:32:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:33:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:33:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:40:37 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:41:05 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:41:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:42:09 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:42:36 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:43:05 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:43:32 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
--messages ommited --
and the system is responding very slowly to all actions and navigation/browsing.

here is a cut from dmesg:

Code: Select all

[62120.118308] NMI backtrace for cpu 3
[62120.118309] CPU: 3 PID: 19 Comm: rcuos/1 Tainted: G             L ------------   3.10.0-327.36.2.el7.x86_64 #1
[62120.118309] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118310] task: ffff8804295e5080 ti: ffff88042960c000 task.ti: ffff88042960c000
[62120.118310] RIP: 0010:[<ffffffffa00663be>]  [<ffffffffa00663be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
[62120.118310] RSP: 0018:ffff88043fcc3bc8  EFLAGS: 00000246
[62120.118311] RAX: ffffc90011b40000 RBX: ffff88042935c000 RCX: 0000000000000006
[62120.118311] RDX: ffff880425709600 RSI: ffff88042935c000 RDI: 000000000000000e
[62120.118312] RBP: ffff88043fcc3bd8 R08: 0000000000000002 R09: ffff88042935a0d8
[62120.118312] R10: ffff8804246a2bc0 R11: ffffea001091a800 R12: 0000000024702a42
[62120.118313] R13: ffff880424702a40 R14: ffff88042935c000 R15: ffff8804246a2bc0
[62120.118313] FS:  0000000000000000(0000) GS:ffff88043fcc0000(0000) knlGS:0000000000000000
[62120.118314] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118314] CR2: 00007efdc21a896b CR3: 000000000194a000 CR4: 00000000000407e0
[62120.118314] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118315] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118315] Stack:
[62120.118316]  ffff88042935cd30 000000000000003c ffff88043fcc3c80 ffffffffa0106729
[62120.118316]  ffff88042935c008 040000000d44ff00 0000000000000006 ffff8804246a2bc0
[62120.118317]  0000000000000018 ffff88042935c188 0000006000000006 ffff880424702a70
[62120.118317] Call Trace:
[62120.118317]  <IRQ> d [<ffffffffa0106729>] mptscsih_qcmd+0x249/0x820 [mptscsih]
[62120.118318]  [<ffffffffa00812b0>] mptspi_qcmd+0x50/0xe0 [mptspi]
[62120.118318]  [<ffffffff81417f4a>] scsi_dispatch_cmd+0xaa/0x230
[62120.118319]  [<ffffffff81420ed1>] scsi_request_fn+0x501/0x770
[62120.118319]  [<ffffffffa0000938>] ? dec_pending+0x188/0x2e0 [dm_mod]
[62120.118319]  [<ffffffff812c75b3>] __blk_run_queue+0x33/0x40
[62120.118320]  [<ffffffff812c7626>] blk_run_queue+0x26/0x40
[62120.118320]  [<ffffffff8141f2f8>] scsi_run_queue+0x258/0x2f0
[62120.118321]  [<ffffffff81421180>] scsi_next_command+0x20/0x40
[62120.118321]  [<ffffffff814212f5>] scsi_end_request+0x155/0x1d0
[62120.118322]  [<ffffffff814214d3>] scsi_io_completion+0x103/0x600
[62120.118322]  [<ffffffff8134f700>] ? fbcon_add_cursor_timer+0xf0/0xf0
[62120.118323]  [<ffffffff81416815>] scsi_finish_command+0xd5/0x130
[62120.118323]  [<ffffffff814209aa>] scsi_softirq_done+0x12a/0x150
[62120.118323]  [<ffffffff812d18e0>] blk_done_softirq+0x90/0xc0
[62120.118324]  [<ffffffff81084baf>] __do_softirq+0xef/0x280
[62120.118324]  [<ffffffff8164841c>] call_softirq+0x1c/0x30
[62120.118325]  <EOI> d [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[62120.118325]  [<ffffffff810844a4>] local_bh_enable+0x94/0xa0
[62120.118325]  [<ffffffff81124412>] rcu_nocb_kthread+0x232/0x370
[62120.118326]  [<ffffffff810a6b80>] ? wake_up_atomic_t+0x30/0x30
[62120.118326]  [<ffffffff811241e0>] ? rcu_start_gp+0x40/0x40
[62120.118327]  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[62120.118327]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[62120.118328]  [<ffffffff81646a58>] ret_from_fork+0x58/0x90
[62120.118328]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[62120.118329] Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 43 07
[62120.118329] NMI backtrace for cpu 4
[62120.118330] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G             L ------------   3.10.0-327.36.2.el7.x86_64 #1
[62120.118330] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118331] task: ffff880429625080 ti: ffff880429648000 task.ti: ffff880429648000
[62120.118331] RIP: 0010:[<ffffffff81058e96>]  [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118332] RSP: 0018:ffff88042964be98  EFLAGS: 00000286
[62120.118332] RAX: 00000000ffffffed RBX: ffff880429648000 RCX: 0100000000000000
[62120.118333] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118333] RBP: ffff88042964be98 R08: 0000000000000000 R09: ffff88042723ba51
[62120.118333] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000004
[62120.118334] R13: ffff880429648000 R14: ffff880429648000 R15: 0000000000000000
[62120.118334] FS:  0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
[62120.118335] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118335] CR2: 00007f0c3da70000 CR3: 0000000425ce0000 CR4: 00000000000407e0
[62120.118336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118336] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118336] Stack:
[62120.118337]  ffff88042964beb8 ffffffff8101dbff ffff880429648000 ffffffff81a691a0
[62120.118337]  ffff88042964bec8 ffffffff8101e506 ffff88042964bf20 ffffffff810d64a5
[62120.118338]  ffff88042964bfd8 ffff880429648000 a6a41e832f87a923 28294442c384663c
[62120.118338] Call Trace:
[62120.118338]  [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118339]  [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118339]  [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118340]  [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118340] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[62120.118341] NMI backtrace for cpu 5
[62120.118341] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G             L ------------   3.10.0-327.36.2.el7.x86_64 #1
[62120.118342] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118342] task: ffff880429625c00 ti: ffff88042964c000 task.ti: ffff88042964c000
[62120.118343] RIP: 0010:[<ffffffff81058e96>]  [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118343] RSP: 0018:ffff88042964fe98  EFLAGS: 00000286
[62120.118344] RAX: 00000000ffffffed RBX: ffff88042964c000 RCX: 0100000000000000
[62120.118344] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118345] RBP: ffff88042964fe98 R08: 0000000000000000 R09: 0000000000000001
[62120.118345] R10: 000000000000001b R11: 0000000000000018 R12: 0000000000000005
[62120.118346] R13: ffff88042964c000 R14: ffff88042964c000 R15: 0000000000000000
[62120.118346] FS:  0000000000000000(0000) GS:ffff88043fd40000(0000) knlGS:0000000000000000
[62120.118347] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118347] CR2: 00007fff7dee1de8 CR3: 0000000426d3f000 CR4: 00000000000407e0
[62120.118348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118348] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118348] Stack:
[62120.118349]  ffff88042964feb8 ffffffff8101dbff ffff88042964c000 ffffffff81a691a0
[62120.118349]  ffff88042964fec8 ffffffff8101e506 ffff88042964ff20 ffffffff810d64a5
[62120.118350]  ffff88042964ffd8 ffff88042964c000 4e5279df92a5e1dd 84c822f6fd3f86e7
[62120.118350] Call Trace:
[62120.118350]  [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118351]  [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118351]  [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118352]  [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118352] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[62120.118353] NMI backtrace for cpu 6
[62120.118353] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G             L ------------   3.10.0-327.36.2.el7.x86_64 #1
[62120.118354] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118354] task: ffff880429626780 ti: ffff880429650000 task.ti: ffff880429650000
[62120.118355] RIP: 0010:[<ffffffff81058e96>]  [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118355] RSP: 0018:ffff880429653e98  EFLAGS: 00000286
[62120.118356] RAX: 00000000ffffffed RBX: ffff880429650000 RCX: 0100000000000000
[62120.118356] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118357] RBP: ffff880429653e98 R08: 0000000000000000 R09: 0000000000000000
[62120.118357] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000006
[62120.118357] R13: ffff880429650000 R14: ffff880429650000 R15: 0000000000000000
[62120.118358] FS:  0000000000000000(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
[62120.118358] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118359] CR2: 00000000004487a0 CR3: 0000000426d3f000 CR4: 00000000000407e0
[62120.118359] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118360] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118360] Stack:
[62120.118360]  ffff880429653eb8 ffffffff8101dbff ffff880429650000 ffffffff81a691a0
[62120.118361]  ffff880429653ec8 ffffffff8101e506 ffff880429653f20 ffffffff810d64a5
[62120.118361]  ffff880429653fd8 ffff880429650000 8a7a87c8cd22c7e2 b3c93cce69884ae5
[62120.118362] Call Trace:
[62120.118362]  [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118362]  [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118363]  [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118363]  [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118364] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[62120.118365] NMI backtrace for cpu 7
[62120.118365] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G             L ------------   3.10.0-327.36.2.el7.x86_64 #1
[62120.118366] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
[62120.118366] task: ffff880429627300 ti: ffff880429654000 task.ti: ffff880429654000
[62120.118367] RIP: 0010:[<ffffffff81058e96>]  [<ffffffff81058e96>] native_safe_halt+0x6/0x10
[62120.118367] RSP: 0018:ffff880429657e98  EFLAGS: 00000286
[62120.118367] RAX: 00000000ffffffed RBX: ffff880429654000 RCX: 0100000000000000
[62120.118368] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[62120.118368] RBP: ffff880429657e98 R08: 0000000000000000 R09: 0000000000000001
[62120.118369] R10: 000000000000003b R11: 000000000000003b R12: 0000000000000007
[62120.118369] R13: ffff880429654000 R14: ffff880429654000 R15: 0000000000000000
[62120.118370] FS:  0000000000000000(0000) GS:ffff88043fdc0000(0000) knlGS:0000000000000000
[62120.118370] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[62120.118371] CR2: 00007f991c2cb000 CR3: 00000000360d9000 CR4: 00000000000407e0
[62120.118371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62120.118372] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[62120.118372] Stack:
[62120.118372]  ffff880429657eb8 ffffffff8101dbff ffff880429654000 ffffffff81a691a0
[62120.118373]  ffff880429657ec8 ffffffff8101e506 ffff880429657f20 ffffffff810d64a5
[62120.118373]  ffff880429657fd8 ffff880429654000 5771fa10f83c71c9 0554c02bd6397f01
[62120.118374] Call Trace:
[62120.118374]  [<ffffffff8101dbff>] default_idle+0x1f/0xc0
[62120.118374]  [<ffffffff8101e506>] arch_cpu_idle+0x26/0x30
[62120.118375]  [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[62120.118375]  [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[62120.118376] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
What info would you like me to provide to help resolve this issue?

Thank you in advance.
Last edited by digea on 2016/11/28 08:36:50, edited 1 time in total.
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 1864
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/16 11:19:37

Could you provide some cpu information from the VM ?

digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/16 13:13:30

Hi,

Here is the output of cat /proc/cpuinfo

Code: Select all

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 1
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 1
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 1
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
stepping        : 3
microcode       : 0x19
cpu MHz         : 3997.683
cache size      : 8192 KB
physical id     : 1
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dtherm xsaveopt
bogomips        : 7995.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
Thank you in advance.
Last edited by digea on 2016/11/28 08:37:08, edited 1 time in total.
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 1864
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/16 19:30:58

According to this KB article 1009996 one possible reason could be high levels of overcommitment (especially memory overcommitment) or other virtualization overheads.If you think that overcommitment is the reason , you can use the following:

Code: Select all

echo [time] >  /proc/sys/kernel/watchdog_thresh
where time cannot be more than 60.

digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/17 13:41:23

Hi,

The watchdog_thresh had it's default value of 10.
When I increase it to 30 using

Code: Select all

echo 30 > /proc/sys/kernel/watchdog_thresh

the messages stop appearing, but the system is still responding very slowly to all actions and navigation/browsing.
I fear that it doesn't solve the problem, but just hides the messages instead.

Any ideas?

Thank you in advance
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 1864
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/17 14:17:59

According to the VMware article - the main reason is probably overcommitting of the resources on the host. What is the load average on the host? Can you move the VM to another host ?

digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/17 15:22:21

Unfortunately I can't move the VM to another host at this time.
Here is the load average you requested (using 4 different commands):

Code: Select all

[root@localhost ~]# cat /proc/loadavg
13.36 14.86 17.19 8/326 10132
[root@localhost ~]# w
 17:17:16 up 1 day, 23:54,  1 user,  load average: 15.54, 15.26, 17.28
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    192.168.1.33     17:15    4.00s  0.48s  0.48s w
[root@localhost ~]# uptime
 17:17:24 up 1 day, 23:54,  1 user,  load average: 16.13, 15.39, 17.31
[root@localhost ~]# top
top - 17:18:09 up 1 day, 23:55,  1 user,  load average: 20.46, 16.62, 17.63
Tasks: 260 total,   7 running, 253 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.9 us,  0.3 sy,  0.0 ni, 13.0 id, 85.4 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem : 16268088 total, 10021028 free,   951016 used,  5296044 buff/cache
KiB Swap:  8257532 total,  8257532 free,        0 used. 14830240 avail Mem
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 1864
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/19 15:54:13

%Cpu(s): 0.9 us, 0.3 sy, 0.0 ni, 13.0 id, 85.4 wa, 0.0 hi, 0.5 si, 0.0 st
If I remember correctly "wa" should mean waiting for I/O and this should be further investigated.
What is the output of:

Code: Select all

iostat -x 2 5
Also

Code: Select all

iotop
could narrow the process that is writing so hard.If you don't have iotop (or not able to install it due to company policy) you can try this:

Code: Select all

for i in {1..10}; do ps -eo state,pid,cmd | grep "^D"; echo "------------" ; sleep 5; done
Reference: High IO wait troubleshooting

digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/24 12:22:25

Hi,

the output of iostat -x 2 5 is

Code: Select all

[root@localhost ~]# iostat -x 2 5
Linux 3.10.0-327.36.2.el7.x86_64 (localhost.localdomain)        11/24/2016      _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.46    0.00    4.30   18.97    0.00   73.27

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     8.50    1.15  146.82     5.98  1324.81    17.99     4.26   28.89   96.48   28.36   5.51  81.47
dm-0              0.00     0.00    1.14  155.23     5.86  1324.55    17.02    11.76   73.25   97.24   73.07   5.21  81.53
dm-1              0.00     0.00    0.00    0.00     0.00     0.00    16.69     0.00    5.34    5.34    0.00   5.02   0.00
dm-2              0.00     0.00    0.00    0.08     0.01     0.25     6.49     0.15 1809.58   22.44 1818.07 1343.16  10.91

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    1.09   31.05    0.00   67.86

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00    51.50    0.00  156.00     0.00  1534.50    19.67   144.15 1009.12    0.00 1009.12   6.41 100.00
dm-0              0.00     0.00    0.00  212.00     0.00  1354.50    12.78   208.82 1082.22    0.00 1082.22   4.72 100.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.58    0.00    1.24   41.74    0.00   56.43

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00    22.66    0.00  149.26     0.00  1077.83    14.44   152.67  767.63    0.00  767.63   6.70  99.95
dm-0              0.00     0.00    0.00  174.38     0.00  3176.85    36.44   174.14  795.85    0.00  795.85   5.73  99.95
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    1.23   53.53    0.00   45.24

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     2.49    0.00  151.74     0.00  2915.42    38.43   149.28 1128.01    0.00 1128.01   6.58  99.90
dm-0              0.00     0.00    0.00  152.24     0.00   618.91     8.13   153.42 1156.88    0.00 1156.88   6.56  99.90
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.32    0.00    0.79   22.67    0.00   76.22

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     2.50    0.00  134.50     0.00   650.50     9.67   144.02 1049.23    0.00 1049.23   7.45 100.20
dm-0              0.00     0.00    0.00  133.00     0.00   593.25     8.92   148.61 1093.15    0.00 1093.15   7.53 100.20
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
and

Code: Select all

[root@localhost ~]# for i in {1..10}; do ps -eo state,pid,cmd | grep "^D"; echo "------------" ; sleep 5; done
D   679 [xfsaild/dm-2]
D  3102 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
D 26164 cleanup -z -t unix -u
D 28473 [kworker/u16:2]
------------

Message from syslogd@localhost at Nov 24 14:21:38 ...
 kernel:BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
D  3102 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
D 26164 cleanup -z -t unix -u
D 28473 [kworker/u16:2]
------------
D 10582 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 11121 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 11146 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 11572 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 11676 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 11700 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 11766 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 11870 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12061 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12102 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12155 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 12335 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12398 [bc]
D 12401 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12412 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12416 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 3.5,3.5 -c 4.28,4.28 -l M
D 12425 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12431 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12442 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12445 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12476 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12592 [bc]
D 12593 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12594 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12597 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12602 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12603 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12605 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12609 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12613 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12630 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12631 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12633 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12636 [bc]
D 12646 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12648 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12653 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12656 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12664 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 1.00,1.00 -c 2.00,2.00 -l M
D 12674 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 12677 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12678 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12688 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12689 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12693 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12939 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12940 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12943 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 12944 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12948 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12953 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12961 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12965 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 12966 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12973 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12976 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12980 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12981 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12982 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12983 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12988 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12991 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 12993 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 12994 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12995 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 12996 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 12999 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 13002 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 13003 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 13008 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 13013 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 13019 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 13026 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 13043 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 13068 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 13080 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 13084 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 13093 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 13102 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 13108 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 13111 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 13113 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 13114 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 13119 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 13121 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 13122 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 13123 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14067 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14078 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14083 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14085 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14094 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14121 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14168 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14170 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14189 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14204 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14212 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14214 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14218 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14220 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14224 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14225 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14240 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14244 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 3.5,3.5 -c 4.28,4.28 -l M
D 14252 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14253 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14254 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14257 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14258 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14261 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14264 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14267 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14270 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14271 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14272 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14274 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14276 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 2.00,2.00 -c 2.60,2.60 -l G
D 14279 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14287 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14296 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14298 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14299 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14304 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14305 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14306 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14307 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14308 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14310 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14316 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14319 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14327 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14328 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14355 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14359 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14473 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14478 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14496 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14501 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14507 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14511 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14518 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14522 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14528 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14530 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14531 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 14534 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14537 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14615 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14635 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14651 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14655 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14664 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14669 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14739 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14754 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14757 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14758 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14775 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14780 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14781 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14790 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14798 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14799 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14805 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14819 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14823 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 3.5,3.5 -c 4.28,4.28 -l M
D 14832 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14887 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14892 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14896 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14899 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14903 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14907 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14909 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 14914 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14928 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 1.00,1.00 -c 2.00,2.00 -l M
D 14929 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14930 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14931 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 14933 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14934 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14936 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14942 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14946 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14947 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14949 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 14951 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14952 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14954 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14955 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14957 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 130.00,130.00 -l M
D 14958 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14960 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14964 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14965 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14967 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14968 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14972 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14975 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14976 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14981 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 14984 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14985 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14988 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 14989 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14991 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 14993 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 14998 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15007 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15013 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15014 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 15016 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15017 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15019 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15024 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15034 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15037 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15054 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15086 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15167 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15172 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15352 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15354 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15362 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15363 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15366 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15367 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15368 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15383 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15384 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15386 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15387 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15388 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15391 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15394 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15402 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15407 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 15408 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15410 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15411 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15417 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15423 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15427 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15428 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15452 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15458 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15477 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15489 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15491 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15516 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15519 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15525 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 15530 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15531 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15537 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 15540 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15546 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15548 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15551 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15553 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15555 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 80.00,80.00 -c 90.00,90.00 -l M
D 15556 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15565 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 15576 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15590 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15699 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15715 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15718 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 3.5,3.5 -c 4.28,4.28 -l M
D 15720 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15733 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15735 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15736 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15737 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15738 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15745 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 15747 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15748 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15753 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15758 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15759 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15762 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15770 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15781 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15782 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15786 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 15787 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15795 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15802 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15805 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15819 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15820 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15825 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15829 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15832 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15839 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 15840 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15849 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15899 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15913 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15915 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15924 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15926 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15927 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15937 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15939 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15940 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15941 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15942 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15943 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 15945 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 15946 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 10.00,10.00 -c 20.00,20.00 -l M
D 15947 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 15949 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15955 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 15958 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 15977 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 16000 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16001 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 16002 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 16005 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 16007 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16012 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16014 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16017 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 16018 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16019 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16020 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16023 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16026 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16040 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 16044 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 16058 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 100.00,100.00 -c 200.00,200.00 -l M
D 16089 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
D 16100 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16109 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16110 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 10.00,10.00 -c 20.00,20.00 -l M
D 16154 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16160 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16161 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16162 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16165 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16173 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16177 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D 16180 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 16183 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16189 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 16193 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l M
------------
------------
D 28473 [kworker/u16:2]
D 28788 [bc]
D 28794 [bc]
D 28881 [bc]
D 28882 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 28885 [bc]
D 28905 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 28908 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D 28938 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 28986 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
------------
------------
------------
D   463 [xfsaild/dm-0]
D  4708 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  4741 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  4769 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  4772 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D  4775 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  4777 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  4780 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  4797 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  4806 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  4854 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  4941 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  4970 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  4979 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  4988 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  4996 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  5017 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  5023 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D  5038 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  5040 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 50.00,50.00 -c 80.00,80.00 -l M
D  5042 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  5050 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 500.00,500.00 -c 800.00,800.00 -l M
D  5059 /bin/sh /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/10.xxx.yyy.zzz -w 5.00,5.00 -c 8.00,8.00 -l G
D 28473 [kworker/u16:2]
------------
D 28473 [kworker/u16:2]
------------
D   679 [xfsaild/dm-2]
D  3102 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
D 28473 [kworker/u16:2]
------------
Note: I have replaced all IPs with 10.xxx.yyy.zzz


Thank you in advance
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 1864
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/24 21:56:46

The output from iostat shows high disk utilization (~100% disk utilization) on sda and dm-0. As the amount of CPUs is 8 - I cannot decide if this is the guest itself or the host.In both cases this has to be resolved.

The second output shows that the check_rrdtraf is writing a lot on those disks and causes the issue.If those files are gathered ,so they can be processed - I could recommend you to create a tmpfs storage (probably mounted over the current folder) where you can store those files (I suppose that the total size is not too large) for processing and the processed files can be moved to a permanent storage location.As tmpfs is in the RAM - the processing speed will be increased several times, but a power failure could lead to unwanted data loss.

If this solution (tmpfs) is too risky - then you will have to increase the amount of disks (probably a raid, or iSCSI device from another server) where you can host the VM.

In this case - the issue is the high disk utilization (~100%) that is causing the soft lockup.

Post Reply