Kernel panic on CentOS 3.10.0-514.26.1.el7.x86_64

General support questions
Post Reply
popovmd.mdb
Posts: 8
Joined: 2017/05/02 09:04:31

Kernel panic on CentOS 3.10.0-514.26.1.el7.x86_64

Post by popovmd.mdb » 2018/01/18 09:08:01

Hello!
31.12.2017 04:52 Happened kernel panic with the server (Virtual Machine Vmware 5.5). Еhe server did not respond to the console, and did not execute the commands, after the reboot the server again worked correct.
We create a case in techsupport vmware, they not detected a problem with server from the point of view of the virtualization system.
In sosreport log in messages seen an errors:
INFO: task kworker/4:0:30059 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/4:0 D ffff88002bdcbdd8 0 30059 2 0x00000080
ffff88002bdcbbf0 0000000000000046 ffff8802d9f70000 ffff88002bdcbfd8
ffff88002bdcbfd8 ffff88002bdcbfd8 ffff8802d9f70000 ffff88002bdcbd58
ffff88002bdcbd60 7fffffffffffffff ffff8802d9f70000 ffff88002bdcbdd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff81322dee>] ? number.isra.2+0xbe/0x360
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/u35:3:31471 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u35:3 D ffff8803c0443dd8 0 31471 2 0x00000080
ffff8803c0443bf0 0000000000000046 ffff8802da7d6dd0 ffff8803c0443fd8
ffff8803c0443fd8 ffff8803c0443fd8 ffff8802da7d6dd0 ffff8803c0443d58
ffff8803c0443d60 7fffffffffffffff ffff8802da7d6dd0 ffff8803c0443dd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/7:1:1698 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/7:1 D ffff8804a3efbdd8 0 1698 2 0x00000080
ffff8804a3efbbf0 0000000000000046 ffff88034df91f60 ffff8804a3efbfd8
ffff8804a3efbfd8 ffff8804a3efbfd8 ffff88034df91f60 ffff8804a3efbd58
ffff8804a3efbd60 7fffffffffffffff ffff88034df91f60 ffff8804a3efbdd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff810d065c>] ? enqueue_entity+0x26c/0xb60
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/1:2:2135 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/1:2 D ffff8803170c7dd8 0 2135 2 0x00000080
ffff8803170c7bf0 0000000000000046 ffff8800ba975e20 ffff8803170c7fd8
ffff8803170c7fd8 ffff8803170c7fd8 ffff8800ba975e20 ffff8803170c7d58
ffff8803170c7d60 7fffffffffffffff ffff8800ba975e20 ffff8803170c7dd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/0:0:2356 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:0 D ffff88055b183dd8 0 2356 2 0x00000080
ffff88055b183bf0 0000000000000046 ffff88010fd99000 ffff88055b183fd8
ffff88055b183fd8 ffff88055b183fd8 ffff88010fd99000 ffff88055b183d58
ffff88055b183d60 7fffffffffffffff ffff88010fd99000 ffff88055b183dd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/u39:2:3399 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u39:2 D ffff88010fbd7dd8 0 3399 2 0x00000080
ffff88010fbd7bf0 0000000000000046 ffff8805fb633000 ffff88010fbd7fd8
ffff88010fbd7fd8 ffff88010fbd7fd8 ffff8805fb633000 ffff88010fbd7d58
ffff88010fbd7d60 7fffffffffffffff ffff8805fb633000 ffff88010fbd7dd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/13:4:4099 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/13:4 D ffff8804b7dd7dd8 0 4099 2 0x00000080
ffff8804b7dd7bf0 0000000000000046 ffff88065c0d5e20 ffff8804b7dd7fd8
ffff8804b7dd7fd8 ffff8804b7dd7fd8 ffff88065c0d5e20 ffff8804b7dd7d58
ffff8804b7dd7d60 7fffffffffffffff ffff88065c0d5e20 ffff8804b7dd7dd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff8104e0fc>] ? native_smp_send_reschedule+0x4c/0x70
[<ffffffff810c1128>] ? resched_task+0xb8/0xd0
[<ffffffff810c1f55>] ? check_preempt_curr+0x75/0xa0
[<ffffffff810c1f99>] ? ttwu_do_wakeup+0x19/0xd0
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
Created slice user-0.slice.
Starting user-0.slice.
Started Session 22364 of user root.
Starting Session 22364 of user root.
INFO: task kworker/12:2:25698 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/12:2 D ffff8804f2197dd8 0 25698 2 0x00000080
ffff8804f2197bf0 0000000000000046 ffff88065b541f60 ffff8804f2197fd8
ffff8804f2197fd8 ffff8804f2197fd8 ffff88065b541f60 ffff8804f2197d58
ffff8804f2197d60 7fffffffffffffff ffff88065b541f60 ffff8804f2197dd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff811dcb61>] ? __slab_free+0x81/0x2f0
[<ffffffff81322062>] ? put_dec+0x72/0x90
[<ffffffff81323053>] ? number.isra.2+0x323/0x360
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/u35:3:31471 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u35:3 D ffff8803c0443dd8 0 31471 2 0x00000080
ffff8803c0443bf0 0000000000000046 ffff8802da7d6dd0 ffff8803c0443fd8
ffff8803c0443fd8 ffff8803c0443fd8 ffff8802da7d6dd0 ffff8803c0443d58
ffff8803c0443d60 7fffffffffffffff ffff8802da7d6dd0 ffff8803c0443dd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
INFO: task kworker/7:1:1698 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/7:1 D ffff8804a3efbdd8 0 1698 2 0x00000080
ffff8804a3efbbf0 0000000000000046 ffff88034df91f60 ffff8804a3efbfd8
ffff8804a3efbfd8 ffff8804a3efbfd8 ffff88034df91f60 ffff8804a3efbd58
ffff8804a3efbd60 7fffffffffffffff ffff88034df91f60 ffff8804a3efbdd8
Call Trace:
[<ffffffff8168c8f9>] schedule+0x29/0x70
[<ffffffff8168a339>] schedule_timeout+0x239/0x2c0
[<ffffffff810d065c>] ? enqueue_entity+0x26c/0xb60
[<ffffffff810c0d8f>] ? __sched_setscheduler+0x83f/0x880
[<ffffffff8168ccd6>] wait_for_completion+0x116/0x170
[<ffffffff810c54e0>] ? wake_up_state+0x20/0x20
[<ffffffff810b08e8>] kthread_create_on_node+0xa8/0x140
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810a883a>] create_worker+0xea/0x250
[<ffffffff810a8b36>] manage_workers.isra.24+0xf6/0x2d0
[<ffffffff810a9542>] worker_thread+0x3d2/0x410
[<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697858>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
logind: Failed to start autovt@tty2.service: Connection timed out

please help in diagnosing the cause of this problem

popovmd.mdb
Posts: 8
Joined: 2017/05/02 09:04:31

Re: Kernel panic on CentOS 3.10.0-514.26.1.el7.x86_64

Post by popovmd.mdb » 2018/01/29 07:57:15

Guys, hello!
Please, tell me, can i access to developers base kernel CentOS 7.4? I think what my question refers specifically to the OS kernel.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Kernel panic on CentOS 3.10.0-514.26.1.el7.x86_64

Post by TrevorH » 2018/01/29 09:28:07

First step: yum update

That kernel is ancient and there have been at least two whole series of kernels since then. That one comes from 7.2 which is now more than 2 years old. Run `yum update` and get up to date and see if the problem recurs. Current kernel is 3.10.0-693.17.1.el7.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply