Kernel messages printed into console cannot be disabled

[adamr]

Hi all,

I have some CentOS 7.3 VMs on CentOS 7.3 KVM host and in VMs I get following kernel error messages:

Code: Select all

[ 2121.962794] Uhhuh. NMI received for unknown reason 20 on CPU 0.
[ 2121.964239] Do you have a strange power saving mode enabled?
[ 2121.965493] Dazed and confused, but trying to continue

I tried to solve the root cause but it looks like it's a common issue when you pass-through CPU and the issue is completely harmless so I’d like to ignore.

The problem is that it’s very annoying to get this printed into console every 30 seconds. I tried to disable printing these messages into console using:

Replace “quiet” by “silent” in /etc/default/grub
and then grub2-mkconfig -o <path to grub cfg>

and

I put

Code: Select all

kern.* /dev/null

Into /etc/rsyslog.conf

And

Code: Select all

dmesg -n 1

And

Code: Select all

dmesg --console-off

Nothing above had any effect. Messages are still printed which is very annoying. Could you please help me to disable it?

Thank you,
Adam

[toracat]

Try using the nmi_watchdog=0 boot time kernel parameter.

[adamr]

Thank you toracat, unfortunately that didn't work.

I've just received it again on server with nmi_watchdog=0 in Grub.

It's super annoying - for example there is no way to use vim and edit file (unless I edit it very fast or scroll content to refresh screen and get rid of these messages).

Code: Select all

Message from syslogd@myserver at May 30 15:44:58 ...
 kernel:Uhhuh. NMI received for unknown reason 20 on CPU 0.

Message from syslogd@myserver at May 30 15:44:58 ...
 kernel:Do you have a strange power saving mode enabled?

Message from syslogd@myserver at May 30 15:44:58 ...
 kernel:Dazed and confused, but trying to continue

[aks]

I don't think software can really mask NMIs (aren't are unmaskable by design?) I think it's usually a hardware event that generates them. You're passing through the CPU to the VM, perhaps that is the event that is generating it (or perhaps any other pass through event)? There is a bucket load of hits on google about this error 20 with KVM (the only tangle solution that someone posted (after looking at around 50 hits) was to change the pass through to "QEMU emulated CPU" in VM settings, which someone else said worked and then later on said didn't work, so go figure).

Or do you actually have a power problem (can the power supply deliver enough reliable power to satisfy the power requirements of all the hardware in the machine)?

As I haven't used QEMU in production for a while, a link to the args: https://wiki.gentoo.org/wiki/QEMU/Options#Processor (that emulation thing is what I think the is the key to what worked and then didn't in the above).

Probably won't help, but 2 cents worth ....

[adamr]

Thank you for response aks.

It isn't easy to test this issue since sometime it happens and sometime it doesn't and it's completely random - not related for example to higher load of KVM host.

I definitely don't have power issue since I observed this issue on multiple different KVM hosts and the current KVM is server with XEON E5 and server grade components. And I disable power saving in BIOS since I thought that power saving might cause this issue.

Btw. funny thing is that I can see the issue even on CentOS with non-official 4.9 kernel and I can see it on Fedora VM with the current updates too (RPM family). But Ubuntu doesn't do it (Debian family)...

I found something about emulating CPU too but that's exactly what I don't want to do since passing through CPU will give you the best performance. There are not too many reason for emulating if you have modern Linux and you don't need to live-migrate VM between hosts with different CPU.

[aks]

Btw. funny thing is that I can see the issue even on CentOS with non-official 4.9 kernel and I can see it on Fedora VM with the current updates too (RPM family). But Ubuntu doesn't do it (Debian family)...

Are they the same version of the actual software (rather than the packages for example)? If yes, it could be something with how the software is compiled (where Debian is "masking" the message as you say, it doesn't seem to break anything, while RH is not).

I guess if it's important, file a bug ticket with CentOS.

[adamr]

I found partial solution. I added this into /etc/rsyslog.conf

Code: Select all

:msg, contains, "Uhhuh. NMI received for unknown reason " ~
:msg, contains, "Do you have a strange power saving mode enabled?" ~
:msg, contains, "Dazed and confused, but trying to continue" ~

and instead of getting messages like this:

Code: Select all

Message from syslogd@myhostname at Jul  4 11:06:14 ...
 kernel:Uhhuh. NMI received for unknown reason 30 on CPU 0.

Message from syslogd@myhostname at Jul  4 11:06:14 ...
 kernel:Do you have a strange power saving mode enabled?

Message from syslogd@myhostname at Jul  4 11:06:14 ...
 kernel:Dazed and confused, but trying to continue

I get messages like this:

Code: Select all

[ 2062.425052] Uhhuh. NMI received for unknown reason 20 on CPU 0.
[ 2062.426191] Do you have a strange power saving mode enabled?
[ 2062.427201] Dazed and confused, but trying to continue

That means that this ^ is not printed by rsyslog. What does it? I have no idea. Btw. I see same format of message when I'm not logged in.

[TrevorH]

Those don't look like the sort of messages you should ignore...

[adamr]

That's what I do since beginning. But it's very annoying.

Imagine that you're in vi editor and this message shows every 30 seconds on your screen. Of course this won't be saved into text file that you edit but still it's very annoying.

[TrevorH]

Perhaps you should fix the problem not ignore the symptom?