Reverting microcode update

General support questions
Post Reply
ajg13
Posts: 5
Joined: 2014/07/24 21:07:28

Reverting microcode update

Post by ajg13 » 2018/01/13 19:18:32

Yesterday I ran a yum update on Centos 6.9 which resulted in an unbootable system.

The sequence hangs at

microcode: CPU0 sig=0x406f1, pf=0x1, revision=0xb000010
platform microcode: firmware: requesting intel-ucode/06-2f-01

so presumably it's related to microcode changes for the Intel bugs.

Unfortunately, the system won't boot into an older kernel for the same reason; though weirdly, in that case it gets through the first 5 CPU updates and hangs with the same error at CPU6.

What is the safest way to back out this update given that I can't boot the system (other than from a USB key)?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Reverting microcode update

Post by TrevorH » 2018/01/13 23:40:44

Boot from the USB key and rm the symlink in /etc/rc3.d on the installed system that points to ../init.d/microcode_ctl
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

ajg13
Posts: 5
Joined: 2014/07/24 21:07:28

Re: Reverting microcode update

Post by ajg13 » 2018/01/14 17:29:04

Thanks Trevor. Unfortunately neither the symlink nor the file it is supposed to link to exist on the system. They are also not present on our other unpatched Centos 6.9 machine (sorry I forgot to mention the Centos version before). Also not on a Centos 7 machine we have.

In addition, passing dis_ucode_ldr as a kernel parameter does not bypass the problem - it still hangs at that point. So I'm not sure where it is picking this up from.

This is not the first time the Xeon E5-2643 has had the problem:

https://bugs.debian.org/cgi-bin/bugrepo ... bug=862606

UPDATE: I erased the microcode loader from rescue mode by chrooting to the installed system and doing "rpm -e microcode_ctl.x86_64"

Not ideal but at least we are back up and running.

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: Reverting microcode update

Post by avij » 2018/01/14 18:31:35

I don't have such a physical CentOS 6 system readily available so that I could check, but I assume TrevorH is right. When you booted from USB, are you sure you checked the installed system's /etc/rc3.d and not the USB's /etc/rc3.d ?

There might indeed be a problem with the Intel-provided microcode. For example, Dell has this to say regarding microcode updates:
Dell wrote:*** Intel has communicated a potential issue with the microcode included in these BIOS updates for Intel Xeon Haswell and Broadwell processors listed below. This issue is currently under investigation by Intel and we will provide further updates as available. See Intel's statement for more details.

Intel® Xeon® Processor E3-1200 v4 Product Family
Intel® Xeon® Processor E5v4 Product Family
Intel® Xeon® Processor E7v3 Product Family
Intel® Xeon® Processor E5v3 Product Family
Intel® Xeon® Processor E3-1200 v3 Product Family
Expect updated microcodes at some point in the future..

ajg13
Posts: 5
Joined: 2014/07/24 21:07:28

Re: Reverting microcode update

Post by ajg13 » 2018/01/15 01:52:56

avij wrote:I don't have such a physical CentOS 6 system readily available so that I could check, but I assume TrevorH is right. When you booted from USB, are you sure you checked the installed system's /etc/rc3.d and not the USB's /etc/rc3.d ?
Yes definitely not present on our running Centos 6.9 and 7 systems:

> ls /etc/init.d/
NetworkManager* cpuspeed* iscsi* mrmonitor* oddjobd* rpcsvcgssd* vboxautostart-service*
abrt-ccpp* crond* iscsid* msm_profile* portreserve* rsyslog* vboxballoonctrl-service*
abrt-oops* cups* jexec* netcf-transaction* postfix* sandbox* vboxdrv*
abrtd* dnsmasq* kdump* netconsole* pppoe-server* saslauthd* vboxweb-service*
acpid* ebtables* killall* netfs* psacct* sendmail* vivaldiframeworkd*
atd* firstboot* ksm* network* quota_nld* single* vncserver*
auditd* functions ksmtuned* nfs* radvd* smartd* winbind*
autofs* haldaemon* libvirt-guests* nfs-rdma* rdisc* snmpd* wpa_supplicant*
avahi-daemon* halt* libvirtd* nfslock* rdma* snmptrapd* ypbind*
blk-availability* htcacheclean* lm_sensors* nscd* restorecond* sshd*
bluetooth* httpd* lvm2-lvmetad* nslcd* rngd* sssd*
certmonger* ip6tables* lvm2-monitor* ntpd* rpcbind* svnserve*
cgconfig* iptables* mdmonitor* ntpdate* rpcgssd* sysstat*
cgred* irqbalance* messagebus* numad* rpcidmapd* udev-post*


I could not work out where it was picking up the option to run microcode_ctl at boot, as nothing obviously points to it:

> locate microcode_ctl
/sbin/microcode_ctl
/usr/share/doc/microcode_ctl-1.17
/usr/share/doc/microcode_ctl-1.17/LICENSE.microcode_amd
/usr/share/man/man8/microcode_ctl.8.gz
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/changed_by
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/checksum_data
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/checksum_type
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/command_line
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/from_repo
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/from_repo_revision
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/from_repo_timestamp
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/reason
/var/lib/yum/yumdb/m/7f6950fc875869a3d832f107ddda68dcd894e7e7-microcode_ctl-1.17-25.2.el6_9-x86_64/releasever

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Reverting microcode update

Post by TrevorH » 2018/01/15 07:19:42

It seems to be done from /lib/udev/rules.d/89-microcode.rules
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

ajg13
Posts: 5
Joined: 2014/07/24 21:07:28

Re: Reverting microcode update

Post by ajg13 » 2018/01/15 16:56:25

TrevorH wrote:It seems to be done from /lib/udev/rules.d/89-microcode.rules
Aha ! That is useful to know for future reference, thanks.

ajg13
Posts: 5
Joined: 2014/07/24 21:07:28

Re: Reverting microcode update

Post by ajg13 » 2018/01/18 16:14:51

Problem confirmed by Redhat, and they have rolled back the patch:

http://www.theregister.co.uk/2018/01/18 ... date_woes/

Post Reply