Meltdown and Spectre

Support for security such as Firewalls and securing linux
User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Meltdown and Spectre

Post by toracat » 2018/01/17 16:54:56

@bshoe24

Try the following:

Copy the microcode.dat file you downloaded from Intel to /lib/firmware/
Run microcode_ctl -f microcode.dat using that file. You may see:

microcode_ctl: writing microcode (length: 1613824)
microcode_ctl: microcode successfuly written to /dev/cpu/microcode
CentOS Forum FAQ

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Meltdown and Spectre

Post by TrevorH » 2018/01/17 16:58:06

The intel-ucode directory exists on CentOS 7 but not on 6 as 6's microcode_ctl is older and only supports the microcode.dat file method.

Current advice from Redhat is not to update the microcode at all as it's resulted in unbootable systems. Indeed, here, since applying the intel-ucode file for my Haswell E3-1245 v3 Xeon, I get processor related Machine Check Exceptions if the ibrs_enabled flag is turned on. Those do not occur on the old microcode nor do they happen if ibrs_enabled=0. On my work systems, on the machines that have newer microcode available via a BIOS update my cpu usage has *doubled* with ibrs_enabled=1 and only returns to approximately normal when it's disabled again. Dell have subsequently withdrawn the BIOS update that has the newer microcode.

Implicit in all this is what no-one - neither Dell nor Redhat - are saying and that is that the only source for processor microcode updates is Intel. You get the same updated microcode whether you download it from Intel, update to a newer microcode_ctl containing the new m/c or install a BIOS update containing it. It all comes from the same place and if it's broken via one route then it's most likely to be broken by another that installs the same m/c.

You'll also find that there's a newer microcode_ctl package released today for both CentOS 6 and 7 that backs out the updates that Redhat shipped last week.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Meltdown and Spectre

Post by toracat » 2018/01/17 17:10:21

The exact wording from Red Hat about the microcode update is as follows:
This update supersedes microcode provided by Red Hat with the CVE-2017-5715 (“Spectre”) CPU branch injection vulnerability mitigation. (Historically, Red Hat has provided updated microcode, developed by our microprocessor partners, as a customer convenience.) Further testing has uncovered problems with the microcode provided along with the “Spectre” mitigation that could lead to system instabilities. As a result, Red Hat is providing an microcode update that reverts to the last known good microcode version dated before 03 January 2018. Red Hat strongly recommends that customers contact their hardware provider for the latest microcode updates.

IMPORTANT: Customers using Intel Skylake-, Broadwell-, and Haswell-based platforms must obtain and install updated microcode from their hardware vendor immediately. The "Spectre" mitigation requires both an updated kernel from Red Hat and updated microcode from your hardware vendor.
CentOS Forum FAQ

User avatar
bshoe24
Posts: 22
Joined: 2015/03/11 15:38:54

Re: Meltdown and Spectre

Post by bshoe24 » 2018/01/17 17:44:04

Yes I've done that a few times successfully and dmesg reports:

microcode: error!Bad data in microcode data file

This whole situation is one big clusterfrak!
toracat wrote:@bshoe24

Try the following:

Copy the microcode.dat file you downloaded from Intel to /lib/firmware/
Run microcode_ctl -f microcode.dat using that file. You may see:

microcode_ctl: writing microcode (length: 1613824)
microcode_ctl: microcode successfuly written to /dev/cpu/microcode

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Meltdown and Spectre

Post by toracat » 2018/01/17 18:12:32

@bshoe24

In your earlier post, you said you ran "microcode_ctl -u". I used the "-f" flag. Maybe worth another try.
CentOS Forum FAQ

User avatar
bshoe24
Posts: 22
Joined: 2015/03/11 15:38:54

Re: Meltdown and Spectre

Post by bshoe24 » 2018/01/18 00:26:17

I figured it out finally. My initial test system was E3-1230 V2 and that does not seem to have any updated microcode yet from Intel. My newer test systems have E3-1231 v3 and there does seem to be newer Intel microcode which mitigates Variant #2 (Spectre). Here are my notes for CentOS6 and CentOS7 which have different steps.

Feel free to correct any mistakes anyone sees :)

#############################################################
#CentOS6 E3-1231 v3

#get /sbin/microcode_ctl tool to dynamically load intel microcode on CentOS6
cd /root
yum install microcode_ctl
modprobe microcode

#install microcode to /lib/firmware/intel-ucode/
https://downloadcenter.intel.com/downlo ... -Data-File
transfer (scp) microcode-20180108.tgz to host

mv /home/username/microcode-20180108.tgz /root
tar -zxvf microcode-20180108.tgz
mkdir /lib/firmware/intel-ucode/
rm /lib/firmware/intel-ucode/* -f
cp -v /root/intel-ucode/* /lib/firmware/intel-ucode/

#dynamically reload microcode
microcode_ctl -u /root/microcode.dat

#verify loaded
dmesg

#test

https://access.redhat.com/labs/speculativeexecution/
./spectre_meltdown.sh

#debugfs mount for Redhat checker script
mount -t debugfs nodev /sys/kernel/debug

OR

#add to /etc/fstab for Redhat checker script
nodev /sys/kernel/debug debugfs defaults 0 0

mount -a

https://github.com/speed47/spectre-meltdown-checker
./spectre-meltdown-checker.sh

#make permanent and reboot to test post reboot
dracut -f -v
reboot

#after reboot check
dmesg|grep micro
./spectre_meltdown.sh
./spectre-meltdown-checker.sh
cat /sys/kernel/debug/x86/pti_enabled
cat /sys/kernel/debug/x86/ibpb_enabled
cat /sys/kernel/debug/x86/ibrs_enabled

#result
Variant #1 (Spectre): Mitigated
Variant #2 (Spectre): Mitigated
Variant #3 (Meltdown): Mitigated

#############################################################
#CentOS7 E3-1231 v3

#install microcode to /lib/firmware/intel-ucode/
https://downloadcenter.intel.com/downlo ... -Data-File
transfer (scp) microcode-20180108.tgz to host

cd /root
mv /home/username/microcode-20180108.tgz /root
tar -zxvf microcode-20180108.tgz
mkdir /lib/firmware/intel-ucode/
rm /lib/firmware/intel-ucode/* -f
cp -v /root/intel-ucode/* /lib/firmware/intel-ucode/

#dynamically reload microcode
echo 1 > /sys/devices/system/cpu/microcode/reload

#verify loaded
dmesg

#test

#debugfs mount for Redhat checker script already mounted so nothing needed

https://access.redhat.com/labs/speculativeexecution/
./spectre_meltdown.sh

https://github.com/speed47/spectre-meltdown-checker
./spectre-meltdown-checker.sh

#make permanent and reboot to test post reboot
dracut -f -v
reboot

#after reboot check
dmesg|grep micro
./spectre_meltdown.sh
./spectre-meltdown-checker.sh
cat /sys/kernel/debug/x86/pti_enabled
cat /sys/kernel/debug/x86/ibpb_enabled
cat /sys/kernel/debug/x86/ibrs_enabled

#result
Variant #1 (Spectre): Mitigated
Variant #2 (Spectre): Mitigated
Variant #3 (Meltdown): Mitigated

aceprabhu
Posts: 4
Joined: 2018/01/19 12:39:25

Re: Meltdown and Spectre

Post by aceprabhu » 2018/01/19 13:00:22

I patched my CentOS 6.9 system with the security patches for Spectre and Meltdown. Changelog results indicate successful patch update. However, the spectre-meltdown-checker.sh throws up negative results. Details are below. I went through the script and I could find the check based on which the script is saying 'NO'. But, I am not sure why this should be the case for CentOS 6 while I read many posts where script posts 'YES' just on doing yum update for KPTI and IBRS support?

Code: Select all

uname -a 
2.6.32-696.18.7.el6.i686

Code: Select all

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Checking count of LFENCE opcodes in kernel:  UNKNOWN 
> STATUS:  VULNERABLE  (only 61 opcodes found, should be >= 70, heuristic to be improved when official patches become available)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigation 1
*   Hardware (CPU microcode) support for mitigation:  NO 
*   Kernel support for IBRS:  NO 
*   IBRS enabled for Kernel space:  NO 
*   IBRS enabled for User space:  NO 
* Mitigation 2
*   Kernel compiled with retpoline option:  NO 
*   Kernel compiled with a retpoline-aware compiler:  NO 
> STATUS:  VULNERABLE  (IBRS hardware + kernel support OR kernel with retpoline are needed to mitigate the vulnerability)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Kernel supports Page Table Isolation (PTI):  NO 
* PTI enabled and active:  NO 
> STATUS:  VULNERABLE  (PTI is needed to mitigate the vulnerability)

Code: Select all

rpm -q --changelog kernel | egrep 'CVE-2017-5715|CVE-2017-5753|CVE-2017-5754'
- [x86] spec_ctrl: svm: spec_ctrl at vmexit needs per-cpu areas functional (Waiman Long) [1519797 1519796] {CVE-2017-5715}
- [x86] spec_ctrl: Eliminate redundnat FEATURE Not Present messages (Waiman Long) [1519797 1519796] {CVE-2017-5715}
- [x86] spec_ctrl: enable IBRS and stuff_RSB before calling NMI C code (Waiman Long) [1519797 1519796] {CVE-2017-5715}
- [x86] spec_ctrl: skip CAP_SYS_PTRACE check to skip audit (Waiman Long) [1519797 1519796] {CVE-2017-5715}
- [x86] spec_ctrl: disable ibrs while in intel_idle() (Waiman Long) [1519797 1519796] {CVE-2017-5715}
- [x86] spec_ctrl: skip IBRS/CR3 restore when paranoid exception returns to userland (Waiman Long) [1519797 1519796] {CVE-2017-5715}
....

User avatar
bshoe24
Posts: 22
Joined: 2015/03/11 15:38:54

Re: Meltdown and Spectre

Post by bshoe24 » 2018/01/19 17:22:08

@aceprabhu - I am running the same kernel with all CentOS 6.9 updates but I have loaded the Intel microcode. All three say mitigated for me.

My new issue is that now with my Intel E3-1231 v3 CPU test system I have gotten 2 machine check exceptions 24 hours after running with this January 8, 2018 release microcode. I went to Intel's website and they have a support chat that I spoke with and was able to pass along the information. They actually did open a ticket on my behalf so I could report the errors. This makes me worry about applying this microcode to production! Is it stable?? :(

[root@serverxyz ~]# cat /var/log/mcelog
Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 0
TIME 1516320203 Fri Jan 19 00:03:23 2018
MCG status:
MCi status:
Corrected error
Error enabled
MCA: Internal parity error
STATUS 90000040000f0005 MCGSTATUS 0
MCGCAP c09 APICID 0 SOCKETID 0
CPUID Vendor Intel Family 6 Model 60

Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 0
TIME 1516327203 Fri Jan 19 02:00:03 2018
MCG status:
MCi status:
Corrected error
Error enabled
MCA: Internal parity error
STATUS 90000040000f0005 MCGSTATUS 0
MCGCAP c09 APICID 0 SOCKETID 0
CPUID Vendor Intel Family 6 Model 60

User avatar
bshoe24
Posts: 22
Joined: 2015/03/11 15:38:54

Re: Meltdown and Spectre

Post by bshoe24 » 2018/01/19 21:43:31

Has anyone running Windows VPS been able to get the Microsoft checker powershell script to reflect hardware support for Spectre #2 CVE-2017-5715 yet.

The Windows patches can be manually downloaded if you haven't gotten it via Windows update.

#2008 R2
http://www.catalog.update.microsoft.com ... =KB4056894
#2012 R2
http://www.catalog.update.microsoft.com ... =KB4073576

#Microsoft powershell checker script
https://support.microsoft.com/en-gb/hel ... -execution
https://support.microsoft.com/en-in/hel ... powershell

Even though on my test CentOS7 KVM fully patched system With Intel microcode update it shows all 3 variants "mitigated" the Microsoft power shell script reports hardware support not present.

#CentOS7 / Redhat spectre_meltdown.sh script output:
Variant #1 (Spectre): Mitigated
Variant #2 (Spectre): Mitigated
Variant #3 (Meltdown): Mitigated

#Microsoft powershell script output:
Hardware support for branch target injection mitigation is present: False
Suggested actions
* Install BIOS/firmware update provided by your device OEM that enables hardware support for the branch target injection mitigation.

I am wondering if there is something else I need to enable on the KVM node?

I already tried:

echo 2 > /sys/kernel/debug/x86/ibrs_enabled #IBRS enabled for User space
Both QEMU and HOST-Model CPUs models #for real cpu passthrough vs QEMU

Neither resulted in detecting hardware support in this Microsoft test script.

and... of course its still generating MCE's with the microcode update. Not crashing yet though.
Jan 19 21:07:26 centos7kvmtest kernel: mce: [Hardware Error]: Machine check events logged

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Meltdown and Spectre

Post by TrevorH » 2018/01/19 22:22:59

Just about everyone has now withdrawn the microcode updates except Intel. Dell and Lenovo have both pulled their BIOS updates that shipped the newer microcode. Redhat have rereleased the microcode_ctl packages and removed the updated microcode from it.

My response from Intel to the similar bug I raised with them about MCEs on my e3-1245v3 was "I don't have any real information on when a fix will be available but will let you know as soon as I get information. My expectation is that we are still a few weeks away. Most of that is for testing the proposed fixes.".

Basically we're all just going have to live with the possibility of spectre being exploited for now. There is no newer microcode to resolve the issue and the one that supposedly does causes worse problems - like unbootable machines.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply