CentOS 7.5 crash at list_del corruption. next->prev should be ffff880c1fa38040

[vvelmuru]

Hi ,

we are using Centos 7.5 and Kernel version is 3.10.0-862 for server.
While doing the operations in Centos Server, Server was crashed at

[177121.901681] WARNING: CPU: 7 PID: 7499 at lib/list_debug.c:62 __list_del_entry+0xdd/0xe0
[177121.901683] list_del corruption. next->prev should be ffff880c1fa38040, but was 0220954700f8b83c

dmesg details:
-------------------
####################################################################################################################
[177119.642265] BUG: unable to handle kernel paging request at 000000060000bf27
[177119.727014] IP: [<ffffffff811dfce8>] do_sys_poll+0x228/0x570
[177119.796083] PGD 80000001560ac067 PUD 0
[177119.843397] Oops: 0000 [#1] SMP
[177120.358632] ------------[ cut here ]------------
[177120.358640] WARNING: CPU: 4 PID: 5149 at lib/list_debug.c:59 __list_del_entry+0xbd/0xe0
[177120.358641] list_del corruption. prev->next should be ffff880c15916000, but was e290c3d53c9f36a0
[177119.883336] Modules linked in: ipmi_si ipmi_devintf ipmi_msghandler ipt_REJECT nf_reject_ipv4 xt_multiport iptable_filter avs_cserver(PO) cdd_sys(O) avsutil(O) ixgbe mdio e1000e mptsas mptscsih mptbase ext3 mbcache jbd loop iTCO_wdt iTCO_vendor_support sg i2c_i801 lpc_ich mfd_core tpm_infineon ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi igb i2c_algo_bit i2c_core ptp crc32c_intel serio_raw pata_jmicron
[177120.358642] Modules linked in:
[177120.358643] ipmi_si
[177120.358643] ipmi_devintf
[177120.358644] ipmi_msghandler
[177120.358644] ipt_REJECT
[177120.358645] nf_reject_ipv4
[177120.358645] xt_multiport
[177120.358646] iptable_filter
[177120.358647] avs_cserver(PO)
[177120.358648] cdd_sys(O)
[177120.358648] avsutil(O)
[177120.358649] ixgbe
[177120.358649] mdio
[177120.358650] e1000e
[177120.358650] mptsas
[177120.358651] mptscsih
[177120.358651] mptbase
[177120.358652] ext3
[177120.358652] mbcache
[177120.358653] jbd
[177120.358653] loop
[177120.358654] iTCO_wdt
[177120.358654] iTCO_vendor_support
[177120.358655] sg
[177120.358655] i2c_i801
[177120.358655] lpc_ich
[177120.358656] mfd_core
[177120.358656] tpm_infineon
[177120.358657] ip_tables
[177120.358657] xfs
[177120.358658] libcrc32c
[177120.358658] sd_mod
[177120.358658] crc_t10dif
[177120.358659] crct10dif_generic
[177120.358659] crct10dif_common
[177120.358660] ata_generic
[177120.358660] pata_acpi
[177120.358661] igb
[177120.358661] i2c_algo_bit
[177120.358662] i2c_core
[177120.358662] ptp
[177120.358663] crc32c_intel
[177120.358663] serio_raw
[177120.358663] pata_jmicron
[177120.358664] ata_piix
[177120.358664] megaraid_sas
[177120.358665] libata
[177120.358665] mpt2sas
[177120.358666] raid_class
[177120.358666] scsi_transport_sas
[177120.358667] pps_core
[177120.358667]
[177120.358671] CPU: 4 PID: 5149 Comm: kworker/4:0 Kdump: loaded Tainted: P O ------------ 3.10.0-862.tv.1 #1
[177120.358671] Hardware name: Supermicro X8DAH/X8DAH, BIOS 2.0d 07/25/12
[177120.358676] Workqueue: events cache_reap
[177120.358676]
[177120.358677] Call Trace:
[177120.358685] [<ffffffff81599bb6>] dump_stack+0x19/0x1b
[177120.358689] [<ffffffff8106bdc8>] __warn+0xd8/0x100
[177120.358690] [<ffffffff8106be4f>] warn_slowpath_fmt+0x5f/0x80
[177120.358692] [<ffffffff81309b7d>] __list_del_entry+0xbd/0xe0
[177120.358694] [<ffffffff81309bad>] list_del+0xd/0x30
[177120.358695] [<ffffffff811a9e51>] free_block+0xe1/0x1b0
[177120.358697] [<ffffffff811aac6c>] drain_array+0x9c/0x100
[177120.358699] [<ffffffff811aaf21>] cache_reap+0x81/0x230
[177120.358704] [<ffffffff8108cb25>] process_one_work+0x175/0x410
[177120.358706] [<ffffffff8108d51f>] worker_thread+0x11f/0x3d0
[177120.358708] [<ffffffff8108d400>] ? manage_workers.isra.25+0x2b0/0x2b0
[177120.358711] [<ffffffff81093df1>] kthread+0xd1/0xe0
[177120.358713] [<ffffffff81093d20>] ? kthread_create_on_node+0x130/0x130
[177120.358716] [<ffffffff815aa4f7>] ret_from_fork_nospec_begin+0x21/0x21
[177120.358718] [<ffffffff81093d20>] ? kthread_create_on_node+0x130/0x130
[177120.358719] ---[ end trace c0e0ae4c54594a74 ]---
[177121.901677] ------------[ cut here ]------------
[177121.901681] WARNING: CPU: 7 PID: 7499 at lib/list_debug.c:62 __list_del_entry+0xdd/0xe0
[177121.901683] list_del corruption. next->prev should be ffff880c1fa38040, but was 0220954700f8b83c
[177121.901683] Modules linked in:
[177121.901684] ipmi_si
[177121.901684] ipmi_devintf
[177121.901685] ipmi_msghandler
[177121.901686] ipt_REJECT
[177121.901686] nf_reject_ipv4
[177121.901687] xt_multiport
[177121.901687] iptable_filter
[177121.901688] avs_cserver(PO)
[177121.901689] cdd_sys(O)
[177121.901689] avsutil(O)
[177121.901690] ixgbe
[177121.901690] mdio
[177121.901691] e1000e
[177121.901691] mptsas
[177121.901691] mptscsih
[177121.901692] mptbase
[177121.901692] ext3
[177121.901693] mbcache
[177121.901693] jbd
[177121.901694] loop
[177121.901694] iTCO_wdt
[177121.901695] iTCO_vendor_support
[177121.901695] sg
[177121.901695] i2c_i801
[177121.901696] lpc_ich
[177121.901696] mfd_core
[177121.901697] tpm_infineon
[177121.901697] ip_tables
[177121.901698] xfs
[177121.901698] libcrc32c
[177121.901699] sd_mod
[177121.901699] crc_t10dif
[177121.901699] crct10dif_generic
[177121.901700] crct10dif_common
[177121.901700] ata_generic
[177121.901701] pata_acpi
[177121.901701] igb
[177121.901701] i2c_algo_bit
[177121.901702] i2c_core
[177121.901702] ptp
[177121.901703] crc32c_intel
[177121.901703] serio_raw
[177121.901704] pata_jmicron
[177121.901704] ata_piix
[177121.901704] megaraid_sas
[177121.901705] libata
[177121.901705] mpt2sas
[177121.901706] raid_class
[177121.901706] scsi_transport_sas
[177121.901707] pps_core
[177121.901707]
[177121.901709] CPU: 7 PID: 7499 Comm: CalypsoPlayStre Kdump: loaded Tainted: P W O ------------ 3.10.0-862.tv.1 #1
[177121.901710] Hardware name: Supermicro X8DAH/X8DAH, BIOS 2.0d 07/25/12
[177121.901710] Call Trace:
[177121.901714] [<ffffffff81599bb6>] dump_stack+0x19/0x1b
[177121.901716] [<ffffffff8106bdc8>] __warn+0xd8/0x100
[177121.901718] [<ffffffff8106be4f>] warn_slowpath_fmt+0x5f/0x80
[177121.901723] [<ffffffff810cd2d0>] ? get_futex_key+0x260/0x2d0
[177121.901724] [<ffffffff81309b9d>] __list_del_entry+0xdd/0xe0
[177121.901726] [<ffffffff81309bad>] list_del+0xd/0x30
[177121.901729] [<ffffffff811ab85c>] cache_alloc_refill+0x18c/0x2d0
[177121.901732] [<ffffffff811ac14d>] kmem_cache_alloc+0xad/0x270
[177121.901735] [<ffffffff811cb18c>] get_empty_filp+0x5c/0x1c0
[177121.901737] [<ffffffff811cb30e>] alloc_file+0x1e/0xd0
[177121.901741] [<ffffffff814abe96>] sock_alloc_file+0xa6/0x140
[177121.901744] [<ffffffff814af1fb>] SyS_socket+0x6b/0xd0
[177121.901745] [<ffffffff815aa5d5>] ? system_call_after_swapgs+0xa2/0x146
[177121.901747] [<ffffffff815aa695>] system_call_fastpath+0x1c/0x21
[177121.901749] [<ffffffff815aa5e1>] ? system_call_after_swapgs+0xae/0x146
[177121.901750] ---[ end trace c0e0ae4c54594a75 ]---

#################################################################################################################

[TrevorH]

CentOS 7.5 is out of date and unsupported. Run yum update tto get your system up to date and on 7.6.

There was at least one similar bug fixed in the 7.6 kernel.