Virtual machines not loading after update -(libvirtd) P11 - undefined symbol

Issues related to applications and software problems
Post Reply
Bruce J
Posts: 3
Joined: 2019/01/07 15:34:56

Virtual machines not loading after update -(libvirtd) P11 - undefined symbol

Post by Bruce J » 2019/01/07 16:47:00

I have a problem with a centos 7 machine unable to start virtual machines after an update

main issue appears to be libvirtd unable to start because of the following
/usr/sbin/libvirtd: symbol lookup error: /lib64/libgnutls.so.28: undefined symbol: p11_kit_pin_file_callback

Am a bit lost with his - am used to it working - do not understand the dependencies and interaction of the packages so need some guidance here.

I am managing this machine remotely over ssh to a command prompt.
I thought this machine had fairly vanilla repositories - base, updates, extras + epel but has been around a while.
Was running virtual machines fine until update

1. I have tested with selinux in permissive and reverting to the previous kernel.

2. /etc/pki doesn't appear to have been changed through the update
In fact /etc changes mostly seem to be in selinux policies.

3. Yum check with epel enabled and disabled re(libvirtd) displays no issue

Will really appreciate anyone who knows about kvm and libgnutls or can point me right.


Offending bootl log portion
Note this occurs on kernel kernel-3.10.0-957.1.3.el7.x86_64
but still occurs if I revert to 3.10.0-862.3.2.el7.x86_6


Jan 7 15:13:29 z3120ba systemd: Starting Virtualization daemon...
Jan 7 15:13:29 z3120ba systemd: Starting Permit User Sessions...
Jan 7 15:13:29 z3120ba systemd: Starting Availability of block devices...
Jan 7 15:13:29 z3120ba systemd: Started Availability of block devices.
Jan 7 15:13:29 z3120ba systemd: Started RPC bind service.
Jan 7 15:13:29 z3120ba systemd: Started Permit User Sessions.
Jan 7 15:13:29 z3120ba systemd: Started Job spooling tools.
Jan 7 15:13:29 z3120ba systemd: Started Command Scheduler.
Jan 7 15:13:29 z3120ba dbus[4284]: [system] Activating via systemd: service name='org.freedesktop.ColorManager' unit='colord.service'
Jan 7 15:13:29 z3120ba systemd: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked.
Jan 7 15:13:29 z3120ba systemd: Cannot add dependency job for unit iscsid.socket, ignoring: Unit is masked.
Jan 7 15:13:29 z3120ba systemd: Starting Manage, Install and Generate Color Profiles...
Jan 7 15:13:29 z3120ba libvirtd: /usr/sbin/libvirtd: symbol lookup error: /lib64/3.10.0-862.3.2.el7.x86_6.so.28: undefined symbol: p11_kit_pin_file_callback
Jan 7 15:13:29 z3120ba systemd: libvirtd.service: main process exited, code=exited, status=127/n/a
Jan 7 15:13:29 z3120ba systemd: Failed to start Virtualization daemon.
Jan 7 15:13:29 z3120ba systemd: Unit libvirtd.service entered failed state.
Jan 7 15:13:29 z3120ba systemd: libvirtd.service failed.

Jan 7 15:13:29 z3120ba systemd: Starting Suspend/Resume Running libvirt Guests...
Jan 7 15:13:29 z3120ba systemd: libvirtd.service holdoff time over, scheduling restart.
Jan 7 15:13:29 z3120ba systemd: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked.
Jan 7 15:13:29 z3120ba systemd: Cannot add dependency job for unit iscsid.socket, ignoring: Unit is masked.
Jan 7 15:13:29 z3120ba systemd: Stopped Virtualization daemon.
Jan 7 15:13:29 z3120ba systemd: Starting Virtual Machine and Container Registration Service...




Kernel detail
Linux z3120ba 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
rpm -qa kernel\* | sort
kernel-3.10.0-693.11.6.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-957.1.3.el7.x86_64
kernel-tools-3.10.0-957.1.3.el7.x86_64
kernel-tools-libs-3.10.0-957.1.3.el7.x86_64

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Virtual machines not loading after update -(libvirtd) P11 - undefined symbol

Post by TrevorH » 2019/01/07 18:00:32

/usr/sbin/libvirtd: symbol lookup error: /lib64/libgnutls.so.28: undefined symbol: p11_kit_pin_file_callback
So your /lib64/libgnutls.so.28 is not up to date. Since that file belongs to the gnutls package, you also need to yum update gnutls and make sure you end up with the right version which is gnutls-3.3.29-8.el7.x86_64. The version of the file from that package does contain that symbol - you can check with strings /lib64/libgnutls.so.28 | grep -i p11_kit_pin_file_callback and on the updated version it has the string.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Bruce J
Posts: 3
Joined: 2019/01/07 15:34:56

Re: Virtual machines not loading after update -(libvirtd) P11 - undefined symbol

Post by Bruce J » 2019/01/07 18:36:16

Thanks Trevor you are correct

however I am at a loss as to how to fix using yum - -


*edit* or is it the callback that is failing on some unspecified specious hardware token ?
Although the existing library appears to have the string.

Code: Select all

 strings /lib64/libgnutls.so.28 | grep -i p11_kit_pin_file_callback
 p11_kit_pin_file_callback
 

Code: Select all

# yum install gnutls-3.3.29-8.el7
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.nsw.coloau.com.au
 * centosplus: centos.melbourneitmirror.net
 * epel: mirror.nsw.coloau.com.au
 * extras: centos.melbourneitmirror.net
 * updates: mirror.nsw.coloau.com.au
Package gnutls-3.3.29-8.el7.x86_64 already installed and latest version
Nothing to do

Code: Select all

find /usr -type f -ls|grep gnutls  2>> /dev/null
36753264   96 -rwxr-xr-x   1 root     root        95040 Oct 30 21:02 /usr/bin/gnutls-cli
36753265   72 -rwxr-xr-x   1 root     root        72440 Oct 30 21:02 /usr/bin/gnutls-cli-debug
36753266   72 -rwxr-xr-x   1 root     root        72104 Oct 30 21:02 /usr/bin/gnutls-serv
108839719  132 -rwxr-xr-x   1 root     root       131504 Nov  2 18:44 /usr/lib64/gio/modules/libgiognutls.so
67921546    4 -rw-r--r--   1 root     root           65 Oct 30 21:02 /usr/lib64/.libgnutls.so.28.43.3.hmac
68641567   36 -rwxr-xr-x   1 root     root        36768 Oct 30 21:02 /usr/lib64/libgnutls-dane.so.0.5.0
67921548 1272 -rwxr-xr-x   1 root     root      1300504 Oct 30 21:02 /usr/lib64/libgnutls.so.28.43.3
34486346    4 -rw-r--r--   1 root     root         1687 Sep  8  2017 /usr/share/doc/gnutls-3.3.29/AUTHORS
34542640   36 -rw-r--r--   1 root     root        35147 Jul 29  2014 /usr/share/doc/gnutls-3.3.29/COPYING
33555479   28 -rw-r--r--   1 root     root        26432 Mar 23  2015 /usr/share/doc/gnutls-3.3.29/COPYING.LESSER
34382900  332 -rw-r--r--   1 root     root       339398 Feb 16  2018 /usr/share/doc/gnutls-3.3.29/NEWS
33817404    8 -rw-r--r--   1 root     root         5987 Sep  8  2017 /usr/share/doc/gnutls-3.3.29/README
33817405   12 -rw-r--r--   1 root     root         9615 Sep  8  2017 /usr/share/doc/gnutls-3.3.29/THANKS
1759733    8 -rw-r--r--   1 root     root         5072 Sep  8  2017 /usr/share/doc/gnutls-utils-3.3.29/certtool.cfg
101268993   28 -rw-r--r--   1 root     root        27078 Oct 30 21:02 /usr/share/locale/cs/LC_MESSAGES/gnutls.mo
33817406   28 -rw-r--r--   1 root     root        25830 Oct 30 21:02 /usr/share/locale/de/LC_MESSAGES/gnutls.mo
108986097   28 -rw-r--r--   1 root     root        27544 Oct 30 21:02 /usr/share/locale/en@boldquot/LC_MESSAGES/gnutls.mo
33817407   28 -rw-r--r--   1 root     root        27532 Oct 30 21:02 /usr/share/locale/en@quot/LC_MESSAGES/gnutls.mo
33832570   28 -rw-r--r--   1 root     root        26247 Oct 30 21:02 /usr/share/locale/eo/LC_MESSAGES/gnutls.mo
101266630   28 -rw-r--r--   1 root     root        26874 Oct 30 21:02 /usr/share/locale/fi/LC_MESSAGES/gnutls.mo
33832608   20 -rw-r--r--   1 root     root        19343 Oct 30 21:02 /usr/share/locale/fr/LC_MESSAGES/gnutls.mo
100664461   28 -rw-r--r--   1 root     root        27752 Oct 30 21:02 /usr/share/locale/it/LC_MESSAGES/gnutls.mo
101132569   16 -rw-r--r--   1 root     root        14888 Oct 30 21:02 /usr/share/locale/ms/LC_MESSAGES/gnutls.mo
101268997   28 -rw-r--r--   1 root     root        27178 Oct 30 21:02 /usr/share/locale/nl/LC_MESSAGES/gnutls.mo
33882905   28 -rw-r--r--   1 root     root        27219 Oct 30 21:02 /usr/share/locale/pl/LC_MESSAGES/gnutls.mo
101269499   24 -rw-r--r--   1 root     root        23082 Oct 30 21:02 /usr/share/locale/sv/LC_MESSAGES/gnutls.mo
100889822   36 -rw-r--r--   1 root     root        34856 Oct 30 21:02 /usr/share/locale/uk/LC_MESSAGES/gnutls.mo
33883019   32 -rw-r--r--   1 root     root        29046 Oct 30 21:02 /usr/share/locale/vi/LC_MESSAGES/gnutls.mo
33883027   16 -rw-r--r--   1 root     root        14515 Oct 30 21:02 /usr/share/locale/zh_CN/LC_MESSAGES/gnutls.mo
1758111    4 -rw-r--r--   1 root     root         2234 Oct 30 21:02 /usr/share/man/man1/gnutls-cli-debug.1.gz
1759705    8 -rw-r--r--   1 root     root         4753 Oct 30 21:02 /usr/share/man/man1/gnutls-cli.1.gz
1759706    4 -rw-r--r--   1 root     root         3778 Oct 30 21:02 /usr/share/man/man1/gnutls-serv.1.gz


Bruce J
Posts: 3
Joined: 2019/01/07 15:34:56

Re: Virtual machines not loading after update -(libvirtd) P11 - undefined symbol

Post by Bruce J » 2019/01/08 04:10:36

Seem to have it working, but not sure what I have broken

First I found that the library in question seemed to call the p11_kit_pin_file_callback, not define it
Decades since I tried to do any of this stuff - left assembler at CP/M and have never had to do in Linux.

Code: Select all

libgnutls.so.28.43.3|grep callback
0000000000046560 T gnutls_sign_callback_get
0000000000046550 T gnutls_sign_callback_set
                 U p11_kit_pin_file_callback
                 U p11_kit_pin_register_callback
                 U p11_kit_pin_unregister_callback


readelf -s libgnutls.so.28.43.3|grep allback
    57: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND p11_kit_pin_file_callback
  1102: 0000000000046560    23 FUNC    GLOBAL DEFAULT   13 gnutls_sign_callback_get@@GNUTLS_1_4
  1188: 0000000000046550    15 FUNC    GLOBAL DEFAULT   13 gnutls_sign_callback_set@@GNUTLS_1_4


objdump -d  libgnutls.so.28.43.3| awk -F"\n" -v RS="\n\n" '$1 ~ /p11_kit_pin_file_callback/'
0000000000026bb0 <p11_kit_pin_register_callback@plt>:
   26bb0:       ff 25 a2 18 31 00       jmpq   *0x3118a2(%rip)        # 338458 <p11_kit_pin_register_callback>
   26bb6:       68 88 02 00 00          pushq  $0x288
   26bbb:       e9 60 d7 ff ff          jmpq   24320 <GNUTLS_1_4@@GNUTLS_1_4+0x24320>
So I looked at the p11-kit - this stuff is getting scary - right in the bowels of the system trust and I have no idea what I'm doing

Code: Select all

cd /usr/lib64
ls *p11* -l
lrwxrwxrwx. 1 root root      20 Jan  6 12:46 libp11-kit.so.0 -> libulockmgr.so.1.0.1
-rwxr-xr-x. 1 root root 1261848 Aug  4  2017 libp11-kit.so.0.3.0
lrwxrwxrwx. 1 root root      15 Jan  8 03:08 libp11.so.2 -> libp11.so.2.5.0
-rwxr-xr-x. 1 root root   65048 Aug  6 15:37 libp11.so.2.5.0
lrwxrwxrwx. 1 root root      19 Jan  6 12:46 p11-kit-proxy.so -> libp11-kit.so.0.3.0
lrwxrwxrwx. 1 root root      13 Jan  6  2016 p11-kit-trust.so -> libnssckbi.so
and I see the libp11-kit.so is linked to the ulockmgr ?? still no idea although I'm seeing references to hardware tokens in my readings
(Is the ulockmgr checking for a hardware token before granting access perhaps ?)
So I thought I might bypass this

Code: Select all

 rm libp11-kit.so.0

rm: remove symbolic link ‘libp11-kit.so.0’? y
[root@z3120ba lib64]# ln -s libp11-kit.so.0.3.0 libp11-kit.so.0
virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit
SUCCESS
Restart the machine and all going.

Code: Select all


virsh start CentOSDev1
Domain CentOSDev1 started



Maybe I've compromised the machine but this seems to have fixed it. Will tidy up and think about reinstall in the months to come.
As a bit of a check, I did confirm that the gpg keys in /etc/pki/rpm-gpg matched other systems I've installed from different sources.

If anyone can shed some light on this. please do.
Should I report this as a bug ? although I have no idea how the symlink got there, if it should be there or if I have broken anything.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Virtual machines not loading after update -(libvirtd) P11 - undefined symbol

Post by TrevorH » 2019/01/08 09:38:22

I have no idea how that symlink got there but it's totally wrong. Mine already points to libp11-kit.so.0.3.0 though I do have a /usr/lib64/libulockmgr.so.1.0.1 but that belongs to the package fuse-libs-2.9.2-11.el7.x86_64 so is nothing to do with p11-kit.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply