Boot after upgrade stops at first CPU-core

General support questions
litux
Posts: 5
Joined: 2018/01/11 16:49:54

Boot after upgrade stops at first CPU-core

Postby litux » 2018/01/11 16:59:58

A couple of days ago I have updated several servers with CentOS 7, everything went fine except one server. After the yum update the machine rebooted but stopped in the bootup process with accessing the first CPU core. I think this is because of the firmware upgrade in the yum upgrade. When I reversed the yum update I could reboot again with the previous working kernel

This was the yum update output (ignore the LOCALE messages and nvidia-messages) For now I will not upgrade because the machine will be useless. How to solve this issue?

yum history info 60
Loaded plugins: fastestmirror
Transaction ID : 60
Begin time : Wed Jan 10 10:26:54 2018
Begin rpmdb : 1114:472e9a6cef785eb5f36f53f2ea38a45cd65d0736
End time : 10:32:25 2018 (5 minutes)
End rpmdb : 1115:40480137b5004ec957789551e2fffa8a4883f43f
User : root <root>
Return-Code : Success
Command Line : update
Transaction performed with:
Installed rpm-4.11.3-25.el7.x86_64 @base
Installed yum-3.4.3-154.el7.centos.noarch @base
Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
Packages Altered:
Updated dracut-033-502.el7.x86_64 @base
Update 033-502.el7_4.1.x86_64 @updates
Updated dracut-config-rescue-033-502.el7.x86_64 @base
Update 033-502.el7_4.1.x86_64 @updates
Updated dracut-network-033-502.el7.x86_64 @base
Update 033-502.el7_4.1.x86_64 @updates
Updated iwl100-firmware-39.31.5.1-56.el7.noarch @base
Update 39.31.5.1-57.el7.noarch @updates
Updated iwl1000-firmware-1:39.31.5.1-56.el7.noarch @base
Update 1:39.31.5.1-57.el7.noarch @updates
Updated iwl105-firmware-18.168.6.1-56.el7.noarch @base
Update 18.168.6.1-57.el7.noarch @updates
Updated iwl135-firmware-18.168.6.1-56.el7.noarch @base
Update 18.168.6.1-57.el7.noarch @updates
Updated iwl2000-firmware-18.168.6.1-56.el7.noarch @base
Update 18.168.6.1-57.el7.noarch @updates
Updated iwl2030-firmware-18.168.6.1-56.el7.noarch @base
Update 18.168.6.1-57.el7.noarch @updates
Updated iwl3160-firmware-22.0.7.0-56.el7.noarch @base
Update 22.0.7.0-57.el7.noarch @updates
Updated iwl3945-firmware-15.32.2.9-56.el7.noarch @base
Update 15.32.2.9-57.el7.noarch @updates
Updated iwl4965-firmware-228.61.2.24-56.el7.noarch @base
Update 228.61.2.24-57.el7.noarch @updates
Updated iwl5000-firmware-8.83.5.1_1-56.el7.noarch @base
Update 8.83.5.1_1-57.el7.noarch @updates
Updated iwl5150-firmware-8.24.2.2-56.el7.noarch @base
Update 8.24.2.2-57.el7.noarch @updates
Updated iwl6000-firmware-9.221.4.1-56.el7.noarch @base
Update 9.221.4.1-57.el7.noarch @updates
Updated iwl6000g2a-firmware-17.168.5.3-56.el7.noarch @base
Update 17.168.5.3-57.el7.noarch @updates
Updated iwl6000g2b-firmware-17.168.5.2-56.el7.noarch @base
Update 17.168.5.2-57.el7.noarch @updates
Updated iwl6050-firmware-41.28.5.1-56.el7.noarch @base
Update 41.28.5.1-57.el7.noarch @updates
Updated iwl7260-firmware-22.0.7.0-56.el7.noarch @base
Update 22.0.7.0-57.el7.noarch @updates
Updated iwl7265-firmware-22.0.7.0-56.el7.noarch @base
Update 22.0.7.0-57.el7.noarch @updates
Updated java-1.8.0-openjdk-1:1.8.0.151-1.b12.el7_4.x86_64 @updates
Update 1:1.8.0.151-5.b12.el7_4.x86_64 @updates
Updated java-1.8.0-openjdk-headless-1:1.8.0.151-1.b12.el7_4.x86_64 @updates
Update 1:1.8.0.151-5.b12.el7_4.x86_64 @updates
Erase kernel-3.10.0-514.el7.x86_64 @anaconda
Install kernel-3.10.0-693.11.6.el7.x86_64 @updates
Install kernel-devel-3.10.0-693.11.6.el7.x86_64 @updates
Updated kernel-headers-3.10.0-693.11.1.el7.x86_64 @updates
Update 3.10.0-693.11.6.el7.x86_64 @updates
Updated kernel-tools-3.10.0-693.11.1.el7.x86_64 @updates
Update 3.10.0-693.11.6.el7.x86_64 @updates
Updated kernel-tools-libs-3.10.0-693.11.1.el7.x86_64 @updates
Update 3.10.0-693.11.6.el7.x86_64 @updates
Updated linux-firmware-20170606-56.gitc990aae.el7.noarch @base
Update 20170606-57.gitc990aae.el7.noarch @updates
Updated microcode_ctl-2:2.1-22.el7.x86_64 @base
Update 2:2.1-22.2.el7.x86_64 @updates
Updated python-perf-3.10.0-693.11.1.el7.x86_64 @updates
Update 3.10.0-693.11.6.el7.x86_64 @updates
Scriptlet output:
1 warning: file /lib/modules/3.10.0-514.el7.x86_64/updates: remove failed: No such file or directory
2
3 Good news! Module version 387.26 for nvidia.ko
4 exactly matches what is already found in kernel 3.10.0-693.11.6.el7.x86_64.
5 DKMS will not replace this module.
6 You may override by specifying --force.
7 Error! Module version 7AB4E4BAAF10299B99D1D68 for nvidia-uvm.ko
8 is not newer than what is already found in kernel 3.10.0-693.11.6.el7.x86_64 (387.26).
9 You may override by specifying --force.
10
11 Good news! Module version 387.26 for nvidia-modeset.ko
12 exactly matches what is already found in kernel 3.10.0-693.11.6.el7.x86_64.
13 DKMS will not replace this module.
14 You may override by specifying --force.
15
16 Good news! Module version 387.26 for nvidia-drm.ko
17 exactly matches what is already found in kernel 3.10.0-693.11.6.el7.x86_64.
18 DKMS will not replace this module.
19 You may override by specifying --force.
20 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
21 depmod: ERROR: fstatat(7, nvidia-uvm.ko): No such file or directory
22 depmod: ERROR: fstatat(7, nvidia.ko): No such file or directory
23 depmod: ERROR: fstatat(7, nvidia-uvm.ko): No such file or directory
24 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
25 depmod: ERROR: fstatat(7, nvidia-drm.ko): No such file or directory
26 depmod: ERROR: fstatat(7, nvidia.ko): No such file or directory
27 depmod: ERROR: fstatat(7, nvidia-uvm.ko): No such file or directory
28 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
29 depmod: ERROR: fstatat(7, nvidia-drm.ko): No such file or directory
30 depmod: ERROR: fstatat(7, nvidia-uvm.ko): No such file or directory
31 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
32 depmod: ERROR: fstatat(7, nvidia-drm.ko): No such file or directory
33 depmod: ERROR: fstatat(7, nvidia-drm.ko): No such file or directory
34 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
35 depmod: ERROR: fstatat(7, nvidia.ko): No such file or directory
36 depmod: ERROR: fstatat(7, nvidia.ko): No such file or directory
37 depmod: ERROR: fstatat(7, nvidia-uvm.ko): No such file or directory
38 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
39 depmod: ERROR: fstatat(7, nvidia-drm.ko): No such file or directory
40 depmod: ERROR: fstatat(7, nvidia-uvm.ko): No such file or directory
41 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
42 depmod: ERROR: fstatat(7, nvidia-drm.ko): No such file or directory
43 depmod: ERROR: fstatat(7, nvidia.ko): No such file or directory
44 depmod: ERROR: fstatat(7, nvidia-uvm.ko): No such file or directory
45 depmod: ERROR: fstatat(7, nvidia-modeset.ko): No such file or directory
46 depmod: ERROR: fstatat(7, nvidia-drm.ko): No such file or directory
47 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
48 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
49 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
50 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
51 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
52 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
53 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
54 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
55 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
56 /sbin/dracut: line 649: warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory
history info


The specs of the server are:

dmidecode -t processor
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.

Handle 0x0069, DMI type 4, 48 bytes
Processor Information
Socket Designation: SOCKET 0
Type: Central Processor
Family: Xeon
Manufacturer: Intel
ID: F1 06 04 00 FF FB EB BF
Signature: Type 0, Family 6, Model 79, Stepping 1
Flags:
FPU (Floating-point unit on-chip)
VME (Virtual mode extension)
DE (Debugging extension)
PSE (Page size extension)
TSC (Time stamp counter)
MSR (Model specific registers)
PAE (Physical address extension)
MCE (Machine check exception)
CX8 (CMPXCHG8 instruction supported)
APIC (On-chip APIC hardware supported)
SEP (Fast system call)
MTRR (Memory type range registers)
PGE (Page global enable)
MCA (Machine check architecture)
CMOV (Conditional move instruction supported)
PAT (Page attribute table)
PSE-36 (36-bit page size extension)
CLFSH (CLFLUSH instruction supported)
DS (Debug store)
ACPI (ACPI supported)
MMX (MMX technology supported)
FXSR (FXSAVE and FXSTOR instructions supported)
SSE (Streaming SIMD extensions)
SSE2 (Streaming SIMD extensions 2)
SS (Self-snoop)
HTT (Multi-threading)
TM (Thermal monitor supported)
PBE (Pending break enabled)
Version: Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz
Voltage: 0.2 V
External Clock: 100 MHz
Max Speed: 4000 MHz
Current Speed: 3400 MHz
Status: Populated, Enabled
Upgrade: Socket LGA2011-3
L1 Cache Handle: 0x0066
L2 Cache Handle: 0x0067
L3 Cache Handle: 0x0068
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 6
Core Enabled: 6
Thread Count: 12
Characteristics:
64-bit capable
Multi-Core
Hardware Thread
Execute Protection
Enhanced Virtualization
Power/Performance Control


dmidecode -t baseboard
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: MSI
Product Name: X99A GAMING PRO CARBON (MS-7A20)
Version: 1.0
Serial Number: G816034527
Asset Tag: Default string
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Default string
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0

Handle 0x0023, DMI type 10, 6 bytes
On Board Device Information
Type: Video
Status: Enabled
Description: To Be Filled By O.E.M.

Handle 0x0055, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Onboard IGD
Type: Video
Status: Enabled
Type Instance: 1
Bus Address: 0000:00:02.0

Handle 0x0056, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Onboard LAN
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:00:19.0

Handle 0x0057, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Onboard 1394
Type: Other
Status: Enabled
Type Instance: 1
Bus Address: 0000:03:1c.2


User avatar
TrevorH
Forum Moderator
Posts: 22583
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Boot after upgrade stops at first CPU-core

Postby TrevorH » 2018/01/11 17:15:35

Can you try just updating the microcode_ctl package and rebooting and see if that does it?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

litux
Posts: 5
Joined: 2018/01/11 16:49:54

Re: Boot after upgrade stops at first CPU-core

Postby litux » 2018/01/12 10:00:10

Update the package

Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : 2:microcode_ctl-2.1-22.2.el7.x86_64 1/2
Cleanup : 2:microcode_ctl-2.1-22.el7.x86_64 2/2

Verifying : 2:microcode_ctl-2.1-22.2.el7.x86_64 1/2
Verifying : 2:microcode_ctl-2.1-22.el7.x86_64 2/2

Updated:
microcode_ctl.x86_64 2:2.1-22.2.el7

Complete!


And yes the machine stopped Last message
smpboot: Booting Node 0, Processors #1

User avatar
TrevorH
Forum Moderator
Posts: 22583
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Boot after upgrade stops at first CPU-core

Postby TrevorH » 2018/01/12 12:06:16

So the problem is not the new kernel, it's the new microcode_ctl package. Exclude that for now and see if the update completes. There is a check in the unit file for microcode_ctl that is looking for the exact processor model that you have so I am not sure why it's even running. Do you have an old microcode_ctl.service file left lying around in /etc/systemd/system perhaps?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

litux
Posts: 5
Joined: 2018/01/11 16:49:54

Re: Boot after upgrade stops at first CPU-core

Postby litux » 2018/01/12 13:35:16

I found this microcode service file, keep it or remove it?

cat /etc/systemd/system/basic.target.wants/microcode.service
[Unit]
Description=Load CPU microcode update
After=basic.target
ConditionVirtualization=false
ConditionPathExists=/sys/devices/system/cpu/microcode/reload

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/usr/bin/bash -c 'grep -l GenuineIntel /proc/cpuinfo | xargs grep -l -E "model[[:space:]]*: 79$" > /dev/null || echo 1 > /sys/devices/system/cpu/microcode/reload'
[Install]
WantedBy=basic.target


ls -ltrh /etc/systemd/system/basic.target.wants/
total 0
lrwxrwxrwx. 1 root root 41 Aug 27 19:27 firewalld.service -> /usr/lib/systemd/system/firewalld.service
lrwxrwxrwx. 1 root root 41 Aug 27 19:27 microcode.service -> /usr/lib/systemd/system/microcode.service

User avatar
TrevorH
Forum Moderator
Posts: 22583
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Boot after upgrade stops at first CPU-core

Postby TrevorH » 2018/01/12 16:46:50

I think that one is fine as it's a symlink to the correct one under /usr/lib/system/system. I discovered a bug in systemctl that means that if you ever run systemctl edit $service, it copies the existing unit file to /etc/systemd/system and then, even if you quit out of the systectl edit without making changes, it leaves it there. Systemd then uses that one in preference to the one in /usr/lib/systemd/system so any changes made by subsequent package updates are ignored as the old copy from /etc is used instead. Bug closed WONTFIX :-(

What could be worthwhile is taking that grep from that file and testing it to see if it matches on your processor. Obviously make sure it doesn't echo into /sys/devices/system/cpu/microcode/reload' or it'll hang again!
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

litux
Posts: 5
Joined: 2018/01/11 16:49:54

Re: Boot after upgrade stops at first CPU-core

Postby litux » 2018/01/12 17:19:45

The grep from the file

grep -l GenuineIntel /proc/cpuinfo | xargs grep -l -E "model[[:space:]]*: 79$"
/proc/cpuinfo

User avatar
TrevorH
Forum Moderator
Posts: 22583
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Boot after upgrade stops at first CPU-core

Postby TrevorH » 2018/01/12 18:17:17

Yes I know that but only you have your /proc/cpuinfo to test it with. It's checking for one specific processor model in that grep and that one looks like yours. The mystery is why it isn't matching and bypassing the reload.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

litux
Posts: 5
Joined: 2018/01/11 16:49:54

Re: Boot after upgrade stops at first CPU-core

Postby litux » 2018/01/23 11:06:07

After last update to microcode_ctl-2.1-22.5.el7_4.x86_64 this problem seems to be solved

User avatar
TrevorH
Forum Moderator
Posts: 22583
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Boot after upgrade stops at first CPU-core

Postby TrevorH » 2018/01/23 13:27:37

That's probably because the latest update to microcode_ctl is actually backing out the changes the previous one introduced.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke