Guest kernel panic after upgrading host kernel

General support questions
Post Reply
rom
Posts: 5
Joined: 2018/06/24 09:14:38

Guest kernel panic after upgrading host kernel

Post by rom » 2018/06/24 09:19:42

Hey,
I'm running guest centos Linux version 3.10.0-123.8.1.el7.x86_6.
Once upgrading host kernel to 3.10.0-693.21.1.el7, start getting kernel panics when spawning vms in parallel:

[ 0.067000] Enabling x2apic
[ 0.067000] Enabled x2apic
[ 0.068000] Switched APIC routing to physical x2apic.
[ 0.069000] ------------[ cut here ]------------
[ 0.070000] WARNING: at arch/x86/kernel/apic/apic.c:1400 setup_local_APIC+0x1a4/0x285()
[ 0.071000] Modules linked in:
[ 0.072000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-123.8.1.el7.x86_64 #1
[ 0.073000] Hardware name: Fedora Project OpenStack Nova, BIOS 0.5.1 01/01/2011
[ 0.074000] 0000000000000000 00000000104261ea ffff88003da8dda8 ffffffff815e237b
[ 0.077000] ffff88003da8dde0 ffffffff8105dee1 0000000000000001 0000000000000000
[ 0.081000] 00000000000000f0 0000000000000000 00000000ffffffff ffff88003da8ddf0
[ 0.084000] Call Trace:
[ 0.085000] [<ffffffff815e237b>] dump_stack+0x19/0x1b
[ 0.086000] [<ffffffff8105dee1>] warn_slowpath_common+0x61/0x80
[ 0.087000] [<ffffffff8105e00a>] warn_slowpath_null+0x1a/0x20
[ 0.088000] [<ffffffff815d1355>] setup_local_APIC+0x1a4/0x285
[ 0.089000] [<ffffffff81a1961d>] native_smp_prepare_cpus+0x294/0x35b
[ 0.090000] [<ffffffff81a09096>] kernel_init_freeable+0xce/0x22a
[ 0.091000] [<ffffffff815c3c20>] ? rest_init+0x80/0x80
[ 0.092000] [<ffffffff815c3c2e>] kernel_init+0xe/0x180
[ 0.093000] [<ffffffff815f29ac>] ret_from_fork+0x7c/0xb0
[ 0.094000] [<ffffffff815c3c20>] ? rest_init+0x80/0x80
[ 0.095000] ---[ end trace 2516da72cd341f2b ]---
[ 0.097000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.098000] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01)
[ 0.102000] divide error: 0000 [#1] SMP
[ 0.102000] Modules linked in:
[ 0.102000] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W -------------- 3.10.0-123.8.1.el7.x86_64 #1
[ 0.102000] Hardware name: Fedora Project OpenStack Nova, BIOS 0.5.1 01/01/2011
[ 0.102000] task: ffff88003da90000 ti: ffff88003da8c000 task.ti: ffff88003da8c000
[ 0.102000] RIP: 0010:[<ffffffff810bbeb4>] [<ffffffff810bbeb4>] clockevents_config.part.2+0x24/0xa0
[ 0.102000] RSP: 0000:ffff88003da8ddc0 EFLAGS: 00010246
[ 0.102000] RAX: ffffffffffffffff RBX: ffff88003fc0dc80 RCX: 0000000000000000
[ 0.102000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffffffffff
[ 0.102000] RBP: ffff88003da8ddd8 R08: 0000000000000001 R09: 00000000000000b6
[ 0.102000] R10: 00000000000000b5 R11: 0000000000000003 R12: 000000000000b008
[ 0.102000] R13: 000000000000b010 R14: 0000000000000001 R15: 0000000000000000
[ 0.102000] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 0.102000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.102000] CR2: 00000000ffffffff CR3: 00000000018d0000 CR4: 00000000000406f0
[ 0.102000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.102000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 0.102000] Stack:
[ 0.102000] ffff88003fc0dc80 000000000000b008 000000000000b010 ffff88003da8ddf0
[ 0.102000] ffffffff810bbf50 ffff88003fc0dc80 ffff88003da8de00 ffffffff815d10a6
[ 0.102000] ffff88003da8de40 ffffffff81a1b43c 0000000000000000 0000000000000000
[ 0.102000] Call Trace:
[ 0.102000] [<ffffffff810bbf50>] clockevents_config_and_register+0x20/0x30
[ 0.102000] [<ffffffff815d10a6>] setup_APIC_timer+0xe6/0xf2
[ 0.102000] [<ffffffff81a1b43c>] setup_boot_APIC_clock+0x4d8/0x4e4
[ 0.102000] [<ffffffff81a196cc>] native_smp_prepare_cpus+0x343/0x35b
[ 0.102000] [<ffffffff81a09096>] kernel_init_freeable+0xce/0x22a
[ 0.102000] [<ffffffff815c3c20>] ? rest_init+0x80/0x80
[ 0.102000] [<ffffffff815c3c2e>] kernel_init+0xe/0x180
[ 0.102000] [<ffffffff815f29ac>] ret_from_fork+0x7c/0xb0
[ 0.102000] [<ffffffff815c3c20>] ? rest_init+0x80/0x80
[ 0.102000] Code: c3 66 0f 1f 44 00 00 66 66 66 66 90 55 31 d2 89 f1 89 f6 41 b8 01 00 00 00 48 89 e5 41 55 41 54 53 48 89 fb 48 8b 7f 70 48 89 f8 <48> f7 f6 48 85 c0 74 0b 48 3d 58 02 00 00 41 89 c0 77 51 4c 8d
[ 0.102000] RIP [<ffffffff810bbeb4>] clockevents_config.part.2+0x24/0xa0
[ 0.102000] RSP <ffff88003da8ddc0>
[ 0.103000] ---[ end trace 2516da72cd341f2c ]---
[ 0.104000] Kernel panic - not syncing: Fatal exception

Any idea?

Found something similar in the web but could not find any resolution:
https://bugs.centos.org/view.php?id=14366

Thanks,
Rom

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Guest kernel panic after upgrading host kernel

Post by TrevorH » 2018/06/24 11:27:10

Update your guest. That 3.10.0-123 kernel is from the original 7.0 release from 2014. It's 4 years old, riddled with security bugs and unsafe to use. No idea if there is a specific fix in newer versions but the latest is 3.10.0-862.3.3.el7 so just 700+ revisions newer. There are over 65000 lines in the kernel rpm changelog since that -123 kernel - that's 65000 fixes you do not have.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

rom
Posts: 5
Joined: 2018/06/24 09:14:38

Re: Guest kernel panic after upgrading host kernel

Post by rom » 2018/06/24 13:29:34

Hey,
Thanks TrevorH.
Those are the images that that we currently use (test vm machines at scale), and it works completely fine on 3.10.0-514.10.2.el7.
We use it only for boot and run memory stress, so dont really want to bother.

What I am more concerned of, whether there is actually a hidden bug in the kernel that was missed.
It's easily reproducible on my environment while trying to spawn ~10 guests in parallel.

Thanks,
Rom

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Guest kernel panic after upgrading host kernel

Post by TrevorH » 2018/06/24 15:12:04

Only the current CentOS version (7.5.1804 at the moment) is supported. You're not running that on either your host or your guest.

If you doubt if you need to update please go read https://access.redhat.com/errata/#/?q=& ... _version=7 and ask it to display only critical or important bugs that have been fixed. You are missing ALL of them.

You need to yum update on both host and guest to get current.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

rom
Posts: 5
Joined: 2018/06/24 09:14:38

Re: Guest kernel panic after upgrading host kernel

Post by rom » 2018/06/25 06:54:08

TrevorH,
I got similar failure with Linux version 3.10.0-693.5.2.el7.x86_64
[ 0.000000] Linux version 3.10.0-693.5.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP
Fri Oct 20 20:32:50 UTC 2017

I know it's not the latest but still... how can it be that any old guest vm cannot run anymore? Does not make sense to me.

Thanks,
Rom

rom
Posts: 5
Joined: 2018/06/24 09:14:38

Re: Guest kernel panic after upgrading host kernel

Post by rom » 2018/06/25 14:35:54

And just got it in centos 7.5 kernel:

Linux version 3.10.0-862.3.2.el7.x86_64

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Guest kernel panic after upgrading host kernel

Post by hunter86_bg » 2018/06/26 20:03:59

You may try to boot your VMs with

Code: Select all

acpi=off
, but then powering off will require to stop the VMs from the Hypervisor .

rom
Posts: 5
Joined: 2018/06/24 09:14:38

Re: Guest kernel panic after upgrading host kernel

Post by rom » 2018/06/28 08:48:57

Hey,
I upgraded everythin - both host os and guest os to centos 7.5 3.10.0-862.3.3.el7

White trying to spawn vms in parallel, get the same failure.

This is something similar that I saw on the web: https://bugs.centos.org/view.php?id=14366, but not reply.

Thanks a lot for your help,
Rom

Post Reply