CentOS Linux release 7.3.1611 (Core)
Linux xxxxx 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
This is one patch behind the latest from April 12, but looking through the CVEs for the new kernel I didn't see anything remotely resembling the characteristics of this crash, so I doubt patching will help.
The system has 512 GB of physical memory and 100GB of swap, and 96 hyperthread cores.
Here's the backtrace:
Code: Select all
[3287176.157114] ------------[ cut here ]------------
[3287176.157156] kernel BUG at mm/huge_memory.c:1994!
[3287176.157184] invalid opcode: 0000 [#1] SMP
[3287176.157212] Modules linked in: btrfs zlib_deflate raid6_pq xor msdos ext4 mbcache jbd2 fuse 8021q garp mrp stp llc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache team_mode_roundrobin team iTCO_wdt iTCO_vendor_support vfat fat intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sb_edac edac_core ioatdma ipmi_devintf lpc_ich sg hpilo hpwdt dca ipmi_si ipmi_msghandler shpchp acpi_power_meter wmi pcc_cpufreq nfsd nfs_acl lockd binfmt_misc grace auth_rpcgss sunrpc ip_tables xfs sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm bnx2x i2c_core hpsa mdio ptp pps_core scsi_transport_sas
[3287176.157767] libcrc32c fjes dm_mirror dm_region_hash dm_log dm_mod
[3287176.157820] CPU: 14 PID: 510 Comm: kswapd1 Tainted: G W L ------------ 3.10.0-514.10.2.el7.x86_64 #1
[3287176.157874] Hardware name: HP ProLiant DL580 Gen8, BIOS P79 07/20/2015
[3287176.157912] task: ffff881ffdd80000 ti: ffff881ffdd88000 task.ti: ffff881ffdd88000
[3287176.157951] RIP: 0010:[<ffffffff811eb3b5>] [<ffffffff811eb3b5>] __split_huge_page+0x735/0x7e0
[3287176.158010] RSP: 0018:ffff881ffdd8b8f8 EFLAGS: 00010297
[3287176.158041] RAX: 0000000000000001 RBX: ffff885ffbe10050 RCX: 0000000000000000
[3287176.158081] RDX: 0000000000000000 RSI: ffff883fff48f838 RDI: ffff883fff48f838
[3287176.158119] RBP: ffff881ffdd8b988 R08: 0000000000000096 R09: 0000000006398657
[3287176.158158] R10: 0000000000000000 R11: ffff881ffdd8b5f6 R12: 0000000000000000
[3287176.158196] R13: ffff881ffdd8bba8 R14: ffffea00e4a98000 R15: ffffea00e4a98000
[3287176.158236] FS: 0000000000000000(0000) GS:ffff883fff480000(0000) knlGS:0000000000000000
[3287176.158279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3287176.158312] CR2: 00002b66228e0000 CR3: 00000000019ba000 CR4: 00000000001407e0
[3287176.158350] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3287176.158388] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3287176.158426] Stack:
[3287176.158442] 0000000000000292 000000fc81317bff 000000000000f1f8 ffff885ffbe10080
[3287176.158487] 000000000000000e 00000000d3dadc00 00000002b0ce1400 ffff881ffdd8b960
[3287176.158533] ffff881ffdd8bba8 00000040e4a98000 0000000000000000 0000000000000000
[3287176.158578] Call Trace:
[3287176.158603] [<ffffffff811eb4d6>] split_huge_page_to_list+0x76/0xf0
[3287176.160090] [<ffffffff811c2d57>] add_to_swap+0x57/0x80
[3287176.161527] [<ffffffff8119529b>] shrink_page_list+0x64b/0xb00
[3287176.162959] [<ffffffff81195dda>] shrink_inactive_list+0x1fa/0x630
[3287176.164415] [<ffffffff81196975>] shrink_lruvec+0x385/0x770
[3287176.165912] [<ffffffff811ef59f>] ? mem_cgroup_iter+0x16f/0x2d0
[3287176.167758] [<ffffffff811ef59f>] ? mem_cgroup_iter+0x16f/0x2d0
[3287176.169850] [<ffffffff81196dd6>] shrink_zone+0x76/0x1a0
[3287176.176131] [<ffffffff8119807c>] balance_pgdat+0x48c/0x5e0
[3287176.177565] [<ffffffff81198343>] kswapd+0x173/0x450
[3287176.179011] [<ffffffff810b17d0>] ? wake_up_atomic_t+0x30/0x30
[3287176.180455] [<ffffffff811981d0>] ? balance_pgdat+0x5e0/0x5e0
[3287176.181847] [<ffffffff810b06ff>] kthread+0xcf/0xe0
[3287176.183247] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[3287176.184611] [<ffffffff81696a58>] ret_from_fork+0x58/0x90
[3287176.185944] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[3287176.192088] Code: 80 48 0f 44 c3 e9 2b fc ff ff 0f 0b 0f 0b 0f 0b 0f 0b 0f 0b 0f 0b 0f 0b 0f 0b f3 90 49 8b 07 a9 00 00 00 01 75 f4 e9 61 fa ff ff <0f> 0b 41 8b 57 18 8b 75 9c 48 c7 c7 d8 e7 8d 81 31 c0 83 c2 01
[3287176.194902] RIP [<ffffffff811eb3b5>] __split_huge_page+0x735/0x7e0
[3287176.196279] RSP <ffff881ffdd8b8f8>
Code: Select all
Apr 26 17:58:37 eand-idsenglnx2 rsyslogd-2177: imjournal: begin to drop messages due to rate-limitingResearching this error message, I found a couple of other situations where split_huge_page() had problems, one as far back as RHEL 6.2 in RH Solution #62340.
Another instance circa 2014 is here: https://patchwork.kernel.org/patch/3963021/
...and...+ * There's chance that iteration over interval tree will race with
+ * insert to it. Let's try catch new entries by retrying.
Needless to say, both of the above are presumably fixed, but hopefully they give a bit of insight as to what the contours of this crash may be. I'm wondering if the combination of high memory pressure and high CPU load conspired to trip something up in the kernel's anonymous huge page tracking.+ * By the time __split_huge_page_refcount() called all PMDs should be
+ * marked pmd_trans_splitting() and new mappings of the page shouldn't
+ * be created or removed. If number of mappings is changed it's a BUG().
I have the core dump, so if you'd like me to fetch information out of it I'd be happy to. Thanks for any suggestions.