Suppressing spurious mcelog parity errors

Issues related to hardware problems
Post Reply
saba
Posts: 5
Joined: 2018/02/20 06:45:17

Suppressing spurious mcelog parity errors

Post by saba » 2018/02/20 07:06:55

On a Centos7 server running a couple of virtual machines (qemu/kvm/libvirt) my syslog is sometimes flooded with the following entries:

mcelog: Hardware event. This is not a software error.
mcelog: MCE 0
mcelog: CPU 1 BANK 0
mcelog: TIME 1519109153 Tue Feb 20 07:45:53 2018
mcelog: MCG status:
mcelog: MCi status:
mcelog: Corrected error
mcelog: Error enabled
mcelog: MCA: Internal parity error
mcelog: STATUS 90000040000f0005 MCGSTATUS 0
mcelog: MCGCAP c09 APICID 2 SOCKETID 0
mcelog: PPIN 90000040000f0005
mcelog: CPUID Vendor Intel Family 6 Model 60

Google led me to the following bug report
https://bugs.centos.org/view.php?id=14333
which has been closed because this obviously is a "hardware error".

But further research led me to this Intel erratum document http://www.intel.com/content/dam/ww...c ... update.pdf
There, item HSW131 suggests that this is not really a memory error, rather is it spuriously generated by processors of the Xeon E3-1200 product family and it "may be safely ignored".

So here comes my question: Is it possible to suppress those spurious errors but without filtering out other MCE errors which could indicate more serious issues?

Regards,
-Rainer

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Suppressing spurious mcelog parity errors

Post by TrevorH » 2018/02/20 12:04:46

Your link is broken but should go to https://www.intel.com/content/dam/www/p ... update.pdf

I suspect your problem is caused by the recent Intel microcode updates for Spectre which they currently do not recommend are used. Back out the microcode update and the problem will most likely go away.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

saba
Posts: 5
Joined: 2018/02/20 06:45:17

Re: Suppressing spurious mcelog parity errors

Post by saba » 2018/02/22 10:17:20

Thank you for your answer and fixing the link!
I'm currently running microcode version 0x1c but I'm not sure it this version is problematic.

Code: Select all

# cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
stepping        : 3
microcode       : 0x1c
cpu MHz         : 3357.367
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bogomips        : 6800.33
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:
[...]
However going back trough my logs I found that the errors appeared long before Spectre emerged.

I found also an Ubuntu bug report on the same problem:
https://bugs.launchpad.net/qemu/+bug/1307225
There, in comment 16, it is suggested that running virtual machines with both 32 bit and 64 bit OSes at the same time might trigger this issue.
This might also be the case on my server. If that is true, moving the 32bit VMs off the server should solve the problem - I'll let you know then.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Suppressing spurious mcelog parity errors

Post by TrevorH » 2018/02/22 13:09:40

I have an e3-1245-v3 and it shows microcode 0x22 and the broken update took it to 0x23 and caused those exact symptoms. Yours using 0x1c seems old given that it looks like a cpu in the same series - maybe you have an old processor and don't have microcode_ctl installed and enabled?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

saba
Posts: 5
Joined: 2018/02/20 06:45:17

Re: Suppressing spurious mcelog parity errors

Post by saba » 2018/02/23 10:38:43

Indeed, I was missing microcode_ctl! I installed it and now I also have 0x22.

So far I did not see any further MCE parity errors. Let's wait and see if it stays that way.

saba
Posts: 5
Joined: 2018/02/20 06:45:17

Re: Suppressing spurious mcelog parity errors

Post by saba » 2018/02/23 12:34:27

Unfortunately the microcode update did not help. Just got another parity error. The rate might be lower, though.

Code: Select all

mcelog: Hardware event. This is not a software error.
mcelog: MCE 0
mcelog: CPU 3 BANK 0
mcelog: TIME 1519388504 Fri Feb 23 13:21:44 2018
mcelog: MCG status:
mcelog: MCi status:
mcelog: Corrected error
mcelog: Error enabled
mcelog: MCA: Internal parity error
mcelog: STATUS 90000040000f0005 MCGSTATUS 0
mcelog: MCGCAP c09 APICID 6 SOCKETID 0
mcelog: CPUID Vendor Intel Family 6 Model 60

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Suppressing spurious mcelog parity errors

Post by TrevorH » 2018/02/23 15:45:17

Did you consider that this might be a real MCE?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

saba
Posts: 5
Joined: 2018/02/20 06:45:17

Re: Suppressing spurious mcelog parity errors

Post by saba » 2018/02/28 14:25:35

Actually not.
The status register "90000040000f0005" matches the description of erratum HSW131 "[...] the valid field (bit 63) set, the uncorrected error field (bit 61) not set, a Model Specific Error Code (bits [31:16]) of 0x000F, and an MCA Error Code (bits [15:0]) of 0x0005". So I _hope_ that "These corrected errors may be safely ignored", as the erratum says.

Post Reply