No longer able to switch users! Possible auditd.service issue

[jays]

Hi guys,

I'm posting here with the hope I can get some help on an issue that is happening in (possible) all our CentOS7 instances.
One of the procedures/policies in the company is to have the system update according to the latest packages release by CentOS.
After the last update made (clamav, java and kernel where updated) we noticed that, after rebooting some of the instances, we where no longer able to sudo into a different user, getting the following message:

Code: Select all

[jays@jays-tst ~]$ sudo su - jayb
[sudo] password for jays:
Last login: Mon Jul 30 12:02:27 EDT 2018 on pts/0
su: cannot open session: Cannot make/remove an entry for the specified session

I can run sudo but I can't sudo to any user.
By checking the journalctl:

Code: Select all

[jays@jays-tst ~]$ sudo journalctl -xe
Jul 31 06:35:26 jays-tst sudo[7620]: jays : TTY=pts/0 ; PWD=/home/jays ; USER=root ; COMMAND=/bin/su - jayb
Jul 31 06:35:26 jays-tst su[7624]: (to jayb) jays on pts/0
Jul 31 06:35:26 jays-tst su[7624]: pamttyaudit(su-l:session): error reading current audit status: Protocol not supported
Jul 31 06:35:26 jays-tst su[7624]: pam_unix(su-l:session): session opened for user jayb by jays(uid=0)
Jul 31 06:35:30 jays-tst sudo[7700]: jays : TTY=pts/0 ; PWD=/home/jays ; USER=root ; COMMAND=/bin/journalctl -xe

While trying to figure it out what was happening, we notice that some services started to fail on startup:

Code: Select all

[jays@jays-tst ~]$ systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
? auditd.service loaded failed failed Security Auditing Service
? NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online
? rc-local.service loaded failed failed /etc/rc.d/rc.local Compatibility

Some of our findings:

Code: Select all

[jays@jays-tst ~]$ sudo systemctl status auditd.service
? auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-07-30 11:42:23 EDT; 17min ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 729 ExecStart=/sbin/auditd (code=exited, status=1/FAILURE)

Jul 30 11:42:23 jays-tst systemd[1]: Starting Security Auditing Service…
Jul 30 11:42:23 jays-tst auditd[778]: Error - audit support not in kernel
Jul 30 11:42:23 jays-tst auditd[778]: Cannot open netlink audit socket
Jul 30 11:42:23 jays-tst auditd[778]: The audit daemon is exiting.
Jul 30 11:42:23 jays-tst systemd[1]: auditd.service: control process exited, code=exited status=1
Jul 30 11:42:23 jays-tst systemd[1]: Failed to start Security Auditing Service.
Jul 30 11:42:23 jays-tst systemd[1]: Unit auditd.service entered failed state.
Jul 30 11:42:23 jays-tst systemd[1]: auditd.service failed.

Code: Select all

[jays@jays-tst ~]$ sudo less /var/log/messages
Jul 30 11:19:22 jays-tst auditd[720]: Error - audit support not in kernel
Jul 30 11:19:22 jays-tst auditd[720]: Cannot open netlink audit socket
Jul 30 11:19:22 jays-tst auditd[720]: The audit daemon is exiting.
Jul 30 11:19:22 jays-tst auditd: Cannot daemonize (Success)
Jul 30 11:19:22 jays-tst auditd: The audit daemon is exiting.
Jul 30 11:19:22 jays-tst systemd: auditd.service: control process exited, code=exited status=1
Jul 30 11:19:22 jays-tst systemd: Failed to start Security Auditing Service.
Jul 30 11:19:22 jays-tst systemd: Unit auditd.service entered failed state.
Jul 30 11:19:22 jays-tst systemd: auditd.service failed

Code: Select all

[jays@jays-tst ~]$ sudo su - jayb
Last login: Mon Jul 30 12:04:43 EDT 2018 on pts/0
su: cannot open session: Cannot make/remove an entry for the specified session
[jays@jays-tst ~]$ sudo less /var/log/secure
Jul 30 12:02:27 jays-tst sudo: jays : TTY=pts/0 ; PWD=/home/jays ; USER=root ; COMMAND=/bin/su - jayb
Jul 30 12:02:27 jays-tst su: pamttyaudit(su-l:session): error reading current audit status: Protocol not supported
Jul 30 12:02:27 jays-tst su: pam_unix(su-l:session): session opened for user jayb by jays(uid=0)

We tried to rollback the updated packages, but, still, the issue persists.

Affected packages:

Code: Select all

[jays@jays-tst ~]$ sudo yum history info 88
Loaded plugins: fastestmirror
Transaction ID : 88
Begin time : Thu Jul 26 04:32:19 2018
Begin rpmdb : 596:d58235e358eedc190ea427e2b5a91ee27b52b023
End time : 04:35:34 2018 (195 seconds)
End rpmdb : 596:999ca89b0584dc699e31a6a2339b3fb930dc6de7
User : <jays>
Return-Code : Success
Command Line : update
Transaction performed with:
Installed rpm-4.11.3-32.el7.x8664 @base Installed yum-3.4.3-158.el7.centos.noarch @base Installed yum-metadata-parser-1.1.4-10.el7.x8664 @anaconda
Installed yum-plugin-fastestmirror-1.1.31-45.el7.noarch @base
Packages Altered:
Updated clamav-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated clamav-data-0.100.0-2.el7.noarch > @epel
Update 0.100.1-1.el7.noarch > @epel
Updated clamav-filesystem-0.100.0-2.el7.noarch > @epel
Update 0.100.1-1.el7.noarch > @epel
Updated clamav-lib-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated clamav-update-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated clamd-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated java-1.8.0-openjdk-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-accessibility-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-accessibility-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-debug-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-demo-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-demo-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-devel-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-devel-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-headless-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-headless-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-javadoc-1:1.8.0.171-8.b10.el75.noarch > @updates Update 1:1.8.0.181-3.b13.el75.noarch > @updates
Updated java-1.8.0-openjdk-javadoc-debug-1:1.8.0.171-8.b10.el75.noarch @updates Update 1:1.8.0.181-3.b13.el75.noarch @updates
Updated java-1.8.0-openjdk-javadoc-zip-1:1.8.0.171-8.b10.el75.noarch @updates Update 1:1.8.0.181-3.b13.el75.noarch @updates
Updated java-1.8.0-openjdk-javadoc-zip-debug-1:1.8.0.171-8.b10.el75.noarch @updates Update 1:1.8.0.181-3.b13.el75.noarch @updates
Updated java-1.8.0-openjdk-src-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-src-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Erase kernel-3.10.0-693.21.1.el7.x8664 > @updates Install kernel-3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-abi-whitelists-3.10.0-862.6.3.el7.noarch > @updates
Update 3.10.0-862.9.1.el7.noarch > @updates
Erase kernel-debug-3.10.0-693.21.1.el7.x8664 > @updates Install kernel-debug-3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-debug-devel-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Erase kernel-devel-3.10.0-693.21.1.el7.x8664 > @updates Install kernel-devel-3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-headers-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-tools-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-tools-libs-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-tools-libs-devel-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated python-perf-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
history info </jays>

We already try to run a filesystem check, but, as per my understanding, all seems ok:

Code: Select all

root@ttyS0:~# e2fsck /dev/sdb
e2fsck 1.42.13 (17-May-2015)
/dev/sdb: clean, 278372/3162112 files, 6167471/12632064 blocks
root@ttyS0:~#
root@ttyS0:~# e2fsck -f /dev/sdb
e2fsck 1.42.13 (17-May-2015)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb: 278372/3162112 files (0.2% non-contiguous), 6167471/12632064 blocks
root@ttyS0:~#

After rebooting, we still have the very same issue. We can't switch users as well, no auditing is being record.

By checking /etc/pam.d/su:

Code: Select all

[jays@jays-tst ~]$ cat /etc/pam.d/su
#%PAM-1.0
auth		sufficient	pam_rootok.so
# Uncomment the following line to implicitly trust users in the "wheel" group.
#auth		sufficient	pam_wheel.so trust use_uid
# Uncomment the following line to require a user to be in the "wheel" group.
#auth		required	pam_wheel.so use_uid
auth		substack	system-auth
auth		include		postlogin
account		sufficient	pam_succeed_if.so uid = 0 use_uid quiet
account		include		system-auth
password	include		system-auth
session		include		system-auth
session		include		postlogin
session		optional	pam_xauth.so

If I comment "session include system-auth" I will be able to switch users, but the issue remains.

Any help would be deeply appreciated!

Regards,
Jay

[avij]

That "Error - audit support not in kernel" sounds odd. Which kernel are you running? Please show the output of uname -a (with the hostname possibly edited out).

[jays]

Hi avij,

Please find below:

Code: Select all

$ uname -r
4.17.8-x86_64-linode110

We are using the default kernel from our provider, Linode.
It already come to my mind that it could be something related, however I don't have the expertise to troubleshoot it deeper.

Adding also:

Code: Select all

$ rpm -qa kernel
kernel-3.10.0-862.2.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64

I would deeply appreciate any help.

- Jay

[avij]

I think you should ask Linode about this. Perhaps they already know about the issue. The installed CentOS kernels don't matter, because they are not being used.

[jays]

That was my very first approach, but so far I wasn't able to make any progression.
I was having hope I could get some directions, since I'm running out of ideas.

- Jay

[avij]

I suppose you could try selecting an older kernel from Linode's control panel and rebooting to that older kernel for testing.

[jays]

Tried older kernels from Linode's control panel but with no luck.
The very same issues are still present

[avij]

I have a server at Linode as well, but it's running CentOS 6. I can "su - otheruser" just fine even with the newest 4.17.8-x86_64-linode110 kernel, but auditd is not running with this kernel. I get the same "audit support not in kernel" in the kernel log.

Prior to rebooting to the newest kernel I was using 4.15.13-x86_64-linode106 and auditd was running happily with this.

Perhaps something broke in between these kernel versions. Please file a support ticket at Linode and let them sort this out for you.

[jays]

Thank you very much @avij
You showed me the light

I did my tests with older Linode kernels but I didn't wen't that further
I tested with the next two prior versions, 4.17.2-x86_64-linode109 and 4.16.11-x86_64-linode108, which have the very same issue.
By changing to 4.15.13-x86_64-linode106 version, the problem indeed went away.

Its very likely that something could be indeed broke between these kernel version. I already update my support ticket at Linode with this findings.

EDIT:
Reproducing the case with a fresh Linode with a fresh/default CentOS7 installation and Linode latest kernel (4.17.8-x86_64-linode110) :

Code: Select all

li1040-151 login: root
Password:
Login incorrect

li1040-151 login: root
Password:
Last failed login: Tue Jul 31 14:56:41 UTC 2018 on ttyS0
There was 1 failed login attempt since the last successful login.
Last login: Tue Jul 31 14:21:41 on ttyS0
[root@li1040-151 ~]# auditctl -l
Error - audit support not in kernel
Cannot open netlink audit socket
[root@li1040-151 ~]#

Thank you once again for the prompt support and feedback.