yum and curl showing illegal instruction

General support questions
lixdeg
Posts: 7
Joined: 2016/05/28 08:56:51

yum and curl showing illegal instruction

Post by lixdeg » 2016/05/28 09:53:02

Since the last update yum is no longer working correctly.
Yum itself doesn't seem to be broken. But I currently cannot pinpoint the cause for crashing.

Code: Select all

yum update
Loaded plugins: fastestmirror
Setting up Update Process
Loading mirror speeds from cached hostfile
Illegal instruction

Code: Select all

yum --version
3.2.29
  Installed: rpm-4.8.0-55.el6.x86_64 at 2016-05-25 20:39
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2016-05-10 23:49
  Committed: Lubos Kardos <lkardos@redhat.com> at 2016-03-29

  Installed: yum-3.2.29-73.el6.centos.noarch at 2016-05-25 20:39
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2016-05-11 20:37
  Committed: Johnny Hughes <johnny@centos.org> at 2016-05-10

  Installed: yum-plugin-fastestmirror-1.1.30-37.el6.noarch at 2016-05-25 20:39
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2016-05-10 21:04
  Committed: Valentina Mukhamedzhanova <vmukhame@redhat.com> at 2016-02-10
curl isn't working too.

Code: Select all

rpm -q curl
curl-7.19.7-52.el6.x86_64

curl https://www.google.com
Illegal instruction
Can anybody confirm these problems while using CentOS 6. Perhaps with a virtualization layer?

Thanks in advance.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: yum and curl showing illegal instruction

Post by TrevorH » 2016/05/28 10:32:32

Are you on a virtual machine? What is the output form grep -i aes /proc/cpuinfo ?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

lixdeg
Posts: 7
Joined: 2016/05/28 08:56:51

Re: yum and curl showing illegal instruction

Post by lixdeg » 2016/05/28 12:04:15

TrevorH wrote:Are you on a virtual machine? What is the output form grep -i aes /proc/cpuinfo ?
Yes, XenPV. But I have now found some machines which are on the same kernels and didn't show the issue.
The logfiles show no yum install errors on the machines with the issue when updating to CentOS 6.8.
The update itself went fine everywhere.

Perhaps there is a difference in the hostnodes under XEN. Cannot think of any other reason at the moment.

Output:

Code: Select all

grep -i aes /proc/cpuinfo
flags           : fpu tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm tpr_shadow vnmi flexpriority ept vpid
flags           : fpu tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm tpr_shadow vnmi flexpriority ept vpid
flags           : fpu tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm tpr_shadow vnmi flexpriority ept vpid

Code: Select all

grep -o aes /proc/cpuinfo
aes
aes
aes
Looks OK to me.

Thanks for the idea. Have seen some post about this issue on other websites popping up.
E.g. https://linuxacademy.com/community/post ... n-centos-6
Seems to be a bit tricky to pinpoint. Won't expect many users with this issue.

lixdeg
Posts: 7
Joined: 2016/05/28 08:56:51

Re: yum and curl showing illegal instruction

Post by lixdeg » 2016/05/28 13:28:51

Looks like this bug in the tracker matches:
https://bugs.centos.org/view.php?id=10930&nbn=5

Your idea with the AES flag was my motivation.
I checked the different hosts. The issue occure when CPU flag AES is available.
On hosts without AES flag no issues occur.

Thanks. Wonder if this is a issue on Hypervisor or CentOS side.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: yum and curl showing illegal instruction

Post by TrevorH » 2016/05/28 19:25:17

I think the actual problem is that the cpuinfo says it has AES but when the system tries to use it, it discovers the support is not there - hence the illegal instruction crash. Have you for example booted this guest on a host machine that has aes-ni and then migrated to one that hasn't? Or overridden the processor type reported to the guest in some way?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

gryzli
Posts: 2
Joined: 2016/01/05 09:30:46
Contact:

Re: yum and curl showing illegal instruction

Post by gryzli » 2016/05/29 16:27:38

You could use temporary fix of this problem, by setting the environment variable: "NSS_DISABLE_HW_GCM=1"

So for example if you run:
NSS_DISABLE_HW_GCM=1 yum search some_package
OR
NSS_DISABLE_HW_GCM=1 curl httpS://some_https_site.com
they must work.

I think the problem is introduced after the last NSS update, and affects every library which uses NSS for HTTPS connections.

Curl, Python, PHP, Perl ...the web libraries of all these seem to be broken.

lixdeg
Posts: 7
Joined: 2016/05/28 08:56:51

Re: yum and curl showing illegal instruction

Post by lixdeg » 2016/05/29 19:30:10

TrevorH wrote:I think the actual problem is that the cpuinfo says it has AES but when the system tries to use it, it discovers the support is not there - hence the illegal instruction crash.
So you mean that the error is with Xen or a misconfiguration of Xen exposing the wrong CPU flags?
The discussions in the bug tracker are unclear to me.
TrevorH wrote: Have you for example booted this guest on a host machine that has aes-ni and then migrated to one that hasn't? Or overridden the processor type reported to the guest in some way?
I have tested serveral virtual Xen instances with my provider.
When AES was exposed by Xen via CPU flag the issue occurs. When not, then no issues occur.
I would expect other OS with the same error when Xen was exposing a AES flag when it isn't available, right? Or is CentOS the only one using AES CPU support? Wouldn't debian or ubuntu distributions have the same issues under Xen?

I will have to wait and see what RedHat is preparing here (no full access to nonsubscribers like me, but looks they already recognize an issue with 6.8. https://access.redhat.com/solutions/2313911

lixdeg
Posts: 7
Joined: 2016/05/28 08:56:51

Re: yum and curl showing illegal instruction

Post by lixdeg » 2016/05/30 09:07:49

gryzli wrote:You could use temporary fix of this problem, by setting the environment variable: "NSS_DISABLE_HW_GCM=1"
Thanks for for joining the thread.
I had sucess with NSS_DISABLE_HW_AES=1 .

I wonder if there is a way, to set NSS_DISABLE_HW_AES=1 global and not for the session.

Other solution was to downgrade nss packages.
Posted here https://bugs.centos.org/view.php?id=10930#c26705

Code: Select all

export NSS_DISABLE_HW_AES=1 
yum -y downgrade nss nss-util nss-tools nss-sysinit
yum -y install yum-plugin-versionlock
yum versionlock add! nss-3.21.0-8.el6.x86_64 nss-sysinit-3.21.0-8.el6.x86_64 nss-tools-3.21.0-8.el6.x86_64 nss-util-3.21.0-2.el6.x86_64

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: yum and curl showing illegal instruction

Post by TrevorH » 2016/05/30 14:45:45

The redhat link says "no solution yet"
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

lixdeg
Posts: 7
Joined: 2016/05/28 08:56:51

Re: yum and curl showing illegal instruction

Post by lixdeg » 2016/05/30 21:02:19

TrevorH wrote:The redhat link says "no solution yet"
Yes. This issue seems to be not that important to them.

I found a post from 17th of May, where someone already posted a workaround to this issue.
Not many CentOS under virtualization. Or not many hosts with AES cpu flags.

Post Reply