CentOS 7.4: unable to mount kerberised NFS share

[Honest Abe]

Hi Guys,

I have three VMs -

10.10.100.1 - CentOS-Server1.example.exam [NFS server]
10.10.100.2 - CentOS-Client1.example.exam [NFS Client]
10.10.100.3 - Cent-Pro.example.exam [KDC]

Configs are here https://pastebin.com/P0h9e8d2 .I have one simple share (without kerberos) which mounts just fine.

I have checked firewall rules, nfs is allowed through and I didn't bother with rpc-bind and mountd since I want this to use NFSv4.

Code: Select all

[root@CentOS-Server1 ~]# firewall-cmd --permanent --list-services 
ssh dhcpv6-client iscsi-target dns http https samba nfs mountd

SELinux is enforced and context is set.

Code: Select all

[root@CentOS-Server1 ~]# ls -ldZ /nfs*
drwxr-xr-x. root      root unconfined_u:object_r:public_content_rw_t:s0 /nfs_k_share
drwxr-xr-x. nfsnobody root unconfined_u:object_r:public_content_rw_t:s0 /nfsshare

I have restarted nfs-server and nfs-client services on server & client machines respectively. (they are both CentOS 7.4)

Code: Select all

[root@CentOS-Client1 ~]# uname -a; cat /etc/redhat-release 
Linux CentOS-Client1.example.exam 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.4.1708 (Core) 

The unsecure mount is accessible from client, but I can't figure out the problem with the secure mount.

Code: Select all

[root@CentOS-Client1 ~]# mount -t nfs4  CentOS-Server1.example.exam:/nfsshare /mnt/nfs_unsecure/ -v
mount.nfs4: timeout set for Sun May 13 21:12:14 2018
mount.nfs4: trying text-based options 'vers=4.1,addr=10.10.100.1,clientaddr=10.10.100.2'
[root@CentOS-Client1 ~]# df -hPT | grep nfs
CentOS-Server1.example.exam:/nfsshare   nfs4       12G  4.2G  7.9G  35% /mnt/nfs_unsecure
[root@CentOS-Client1 ~]# mount -t nfs4  -o sec=krb5p,rw CentOS-Server1.example.exam:/nfs_k_share /mnt/nfs_secure/ -v
mount.nfs4: timeout set for Sun May 13 21:13:50 2018
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.1,addr=10.10.100.1,clientaddr=10.10.100.2'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.0,addr=10.10.100.1,clientaddr=10.10.100.2'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting CentOS-Server1.example.exam:/nfs_k_share

I have already searched the web, but couldn't get this working. I understand there are some changes from 7.0 to 7.1 and later. I would appreciate any help in this regard.

Let me know if any other data/config is required.

[hunter86_bg]

Can you try without any firewall on all servers? This will verify that nothing is missing in the firewall setup.

Can you also verify that all machines have the same time (ntp/chrony working) and that 'A' and 'PTR' records are available for all machines .A simple 'nslookup' on names and IPs will do the trick.

Also , what is the output of:

Code: Select all

systemctl status nfs-client.target

on the NFS client?

[Honest Abe]

Thanks hunter86_bg for your response.

No, stopping firewalld on both client and server machines do not work.

Client side -

Code: Select all

[root@CentOS-Client1 ~]# date; systemctl status -l nfs-client.target 
Mon May 14 10:42:24 IST 2018
● nfs-client.target - NFS client services
   Loaded: loaded (/usr/lib/systemd/system/nfs-client.target; enabled; vendor preset: disabled)
   Active: active since Mon 2018-05-14 10:38:47 IST; 3min 36s ago

May 14 10:38:47 CentOS-Client1.example.exam systemd[1]: Reached target NFS client services.
[root@CentOS-Client1 ~]# mount -t nfs -o sec=krb5p 10.10.100.1:/nfs_k_share /mnt/nfs_secure/ -v
mount.nfs: timeout set for Mon May 14 10:45:30 2018
mount.nfs: trying text-based options 'sec=krb5p,vers=4.1,addr=10.10.100.1,clientaddr=10.10.100.2'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5p,vers=4.0,addr=10.10.100.1,clientaddr=10.10.100.2'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5p,addr=10.10.100.1'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.10.100.1 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.10.100.1 prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 10.10.100.1:/nfs_k_share
[root@CentOS-Client1 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2018-05-14 10:42:08 IST; 3min 16s ago
     Docs: man:firewalld(1)
  Process: 745 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 745 (code=exited, status=0/SUCCESS)

May 14 10:38:46 CentOS-Client1.example.exam systemd[1]: Starting firewalld - dynamic firewall daemon...
May 14 10:38:51 CentOS-Client1.example.exam systemd[1]: Started firewalld - dynamic firewall daemon.
May 14 10:38:53 CentOS-Client1.example.exam firewalld[745]: WARNING: ICMP type 'beyond-scope' is not supported by the ...pv6.
May 14 10:38:53 CentOS-Client1.example.exam firewalld[745]: WARNING: beyond-scope: INVALID_ICMPTYPE: No supported ICMP...ime.
May 14 10:38:53 CentOS-Client1.example.exam firewalld[745]: WARNING: ICMP type 'failed-policy' is not supported by the...pv6.
May 14 10:38:53 CentOS-Client1.example.exam firewalld[745]: WARNING: failed-policy: INVALID_ICMPTYPE: No supported ICM...ime.
May 14 10:38:53 CentOS-Client1.example.exam firewalld[745]: WARNING: ICMP type 'reject-route' is not supported by the ...pv6.
May 14 10:38:53 CentOS-Client1.example.exam firewalld[745]: WARNING: reject-route: INVALID_ICMPTYPE: No supported ICMP...ime.
May 14 10:42:08 CentOS-Client1.example.exam systemd[1]: Stopping firewalld - dynamic firewall daemon...
May 14 10:42:08 CentOS-Client1.example.exam systemd[1]: Stopped firewalld - dynamic firewall daemon.
Hint: Some lines were ellipsized, use -l to show in full.

Server side -

Code: Select all

[root@CentOS-Server1 ~]# date;systemctl stop firewalld.service 
Mon May 14 10:41:49 IST 2018
[root@CentOS-Server1 ~]# ls -ldZ /nfs*
drwxr-xr-x. root      root unconfined_u:object_r:public_content_rw_t:s0 /nfs_k_share
drwxr-xr-x. nfsnobody root unconfined_u:object_r:public_content_rw_t:s0 /nfsshare
[root@CentOS-Server1 ~]# systemctl status -l nfs-server.service 
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Mon 2018-05-14 10:39:18 IST; 7min ago
  Process: 1387 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 1350 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=0/SUCCESS)
  Process: 1346 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 1387 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

May 14 10:39:17 CentOS-Server1.example.exam systemd[1]: Starting NFS server and services...
May 14 10:39:18 CentOS-Server1.example.exam systemd[1]: Started NFS server and services.
[root@CentOS-Server1 ~]# 

I also read few articles where it was mentioned that KDC might have some trouble with chrony, so I have set up NTP on my KDC.

so KDC has this -

Code: Select all

[root@Cent-Pro ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+fwdns2.vbctv.in 80.87.128.222    4 u   65   64  237   43.880   12.716   3.377
*139.59.43.68    193.6.176.19     3 u   30   64  373   12.945    8.666   2.667
[root@Cent-Pro ~]# ntpstat
unsynchronised
   polling server every 64 s

while the nfs server and nfs client use chrony, they refer to the KDC machine for time sync.

The client even users the nfs server as a peer.

Code: Select all

[root@CentOS-Client1 ~]# chronyc sources
210 Number of sources = 2
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
=? CentOS-Server1.example.e>     0   8     0     -     +0ns[   +0ns] +/-    0ns
^? Cent-Pro.example.exam         0   8     0     -     +0ns[   +0ns] +/-    0ns

Chrony on the NFS server.-

Code: Select all

[root@CentOS-Server1 ~]# chronyc sources
210 Number of sources = 5
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^? Cent-Pro.example.exam         0   8     0     -     +0ns[   +0ns] +/-    0ns
^+ ec2-13-126-37-14.ap-sout>     2   7    37    35  -3419us[-3419us] +/-   80ms
^* ec2-52-66-5-185.ap-south>     2   6   375   106  -5724us[-5610us] +/-   81ms
^+ ntp.slackware.in              3   6   357    44   -424us[ -424us] +/-  102ms
^+ fwdns2.vbctv.in               4   6   377    43  -5076us[-5076us] +/-  214ms

Also, I do not have a DNS,, but my hostfiles (/etc/hosts) are updated as mentioned in the first post. The servers can ping each other with FQDN/IP just fine

[hunter86_bg]

Next step that comes to my mind is to create a test user on the KDC ('ipa user-add testuser' if IPA is your identity management), set a password for that user and last -> log with that user on KDC, NFS server and NFS client and check if kerberos ticket has been granted.

[Honest Abe]

I have an user 'alice' set up at the KDC which authenticates against a password.. Linux user account 'alice' exists on the nfs server and nfs client , but there is no password set (shadow password.)

on NFS server -

Code: Select all

[ab@CentOS-Server1 ~]$ su - root
Password: 
Last login: Mon May 14 20:18:39 IST 2018 on pts/0
[root@CentOS-Server1 ~]# su - alice
Last login: Tue May 15 11:49:37 IST 2018 from centos-client1.example.exam on pts/1
[alice@CentOS-Server1 ~]$ kinit
Password for alice@EXAMPLE.EXAM: 
[alice@CentOS-Server1 ~]$ klist
Ticket cache: KEYRING:persistent:1006:krb_ccache_mLKqMyv
Default principal: alice@EXAMPLE.EXAM

Valid starting       Expires              Service principal
05/15/2018 11:53:16  05/16/2018 11:53:13  krbtgt/EXAMPLE.EXAM@EXAMPLE.EXAM
[alice@CentOS-Server1 ~]$ ssh pro
alice@pro's password: 
Last login: Tue May 15 11:48:04 2018 from 10.10.100.1

If this was working correctly, then I should not get the password prompt for alice@pro, right ? [pro is short name of KDC]

From NFS client, the behaviour is same -

Code: Select all

ab@CentOS-Client1 ~]$ su - root
Password: 
Last login: Mon May 14 20:18:54 IST 2018 on pts/0
[root@CentOS-Client1 ~]# su - alice
Last login: Tue May 15 11:48:46 IST 2018 on pts/0
[alice@CentOS-Client1 ~]$ kdestroy 
[alice@CentOS-Client1 ~]$ klist
klist: Credentials cache keyring 'persistent:1003:krb_ccache_CPjv47Q' not found
[alice@CentOS-Client1 ~]$ kinit; klist
Password for alice@EXAMPLE.EXAM: 
Ticket cache: KEYRING:persistent:1003:krb_ccache_CPjv47Q
Default principal: alice@EXAMPLE.EXAM

Valid starting       Expires              Service principal
05/15/2018 11:56:04  05/16/2018 11:56:02  krbtgt/EXAMPLE.EXAM@EXAMPLE.EXAM
[alice@CentOS-Client1 ~]$ ssh pro
The authenticity of host 'pro (10.10.100.3)' can't be established.
ECDSA key fingerprint is SHA256:SzzMho63yduLJc23AzDVzrCw07KU6PMvtgs1UG+FFNY.
ECDSA key fingerprint is MD5:b6:55:2a:71:82:1d:10:e5:15:63:62:c4:91:b2:a2:74.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'pro,10.10.100.3' (ECDSA) to the list of known hosts.
alice@pro's password: 
Last login: Tue May 15 11:53:30 2018 from 10.10.100.1
[alice@Cent-Pro ~]$ 

So, 'alice' has been granted ticket by KDC, but it still needs a password to authenticate against ssh. Is this expected ?
Also, are these setting wrt sshd_config okay ?
server -

Code: Select all

[root@CentOS-Server1 ~]# grep -i GSS /etc/ssh/sshd_config | grep -v ^#
GSSAPIAuthentication yes
GSSAPICleanupCredentials no

client -

Code: Select all

[root@CentOS-Client1 ~]# grep -i GSS /etc/ssh/sshd_config | grep -v ^#
GSSAPIAuthentication yes
GSSAPICleanupCredentials no

KDC-

Code: Select all

[root@Cent-Pro ~]# grep -i GSS /etc/ssh/sshd_config | grep -v ^#
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes

[hunter86_bg]

Usually you don't need to change sshd_config to enable kerberos login, but the /etc/krb5.keytab should contain a principal for the host similar to 'host/domain.com@REALM'.

Enable the verbosity for nfs by editing the /etc/sysconfig/nfs file, assign the “-vvv” string to the RPCIDMAPDARGS/RPCSVCGSSDARGS variables and restart the nfs-idmap/nfs-secure-server daemons.
Then check /var/log/messages or 'journalctl -f' during the mount.
Edit: Provide the contents of the kerberos keytab on nfs server/client via:

Code: Select all

 klist -k 

[Honest Abe]

Thanks for your continued help.
I'll post later today when I get to home.

[Honest Abe]

Okay, changes made in /etc/sysconfig/nfs -

Code: Select all

[root@CentOS-Server1 ~]# egrep -i "RPCIDMAPDARGS|RPCSVCGSSDARGS" /etc/sysconfig/nfs | grep -v ^#
RPCIDMAPDARGS="-vvv"

Restarted services on server as advised. then opened two terminals, one for monitoring 'tail -f /var/log/messages' and the other for 'journalctl -f'. and attempted to mount from the client.

On the client -

Code: Select all

[root@CentOS-Client1 ~]# date; mount.nfs4 -o sec=krb5p 10.10.100.1:/nfs_k_share /mnt/nfs_secure -v; date
Tue May 15 20:40:16 IST 2018
mount.nfs4: timeout set for Tue May 15 20:42:16 2018
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.1,addr=10.10.100.1,clientaddr=10.10.100.2'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.0,addr=10.10.100.1,clientaddr=10.10.100.2'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting 10.10.100.1:/nfs_k_share
Tue May 15 20:41:27 IST 2018

On the server, nothing during that timeframe -
End lines from tail -f /var/log/messages -

Code: Select all

May 15 20:40:01 CentOS-Server1 systemd: Created slice User Slice of root.
May 15 20:40:01 CentOS-Server1 systemd: Starting User Slice of root.
May 15 20:40:01 CentOS-Server1 systemd: Started Session 4 of user root.
May 15 20:40:01 CentOS-Server1 systemd: Starting Session 4 of user root.
May 15 20:40:01 CentOS-Server1 systemd: Removed slice User Slice of root.
May 15 20:40:01 CentOS-Server1 systemd: Stopping User Slice of root.

journalctl -f also does not have anything to show -

Code: Select all

May 15 20:40:01 CentOS-Server1.example.exam systemd[1]: Created slice User Slice of root.
May 15 20:40:01 CentOS-Server1.example.exam systemd[1]: Starting User Slice of root.
May 15 20:40:01 CentOS-Server1.example.exam systemd[1]: Started Session 4 of user root.
May 15 20:40:01 CentOS-Server1.example.exam systemd[1]: Starting Session 4 of user root.
May 15 20:40:01 CentOS-Server1.example.exam CROND[2220]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 15 20:40:01 CentOS-Server1.example.exam systemd[1]: Removed slice User Slice of root.
May 15 20:40:01 CentOS-Server1.example.exam systemd[1]: Stopping User Slice of root.
May 15 20:44:05 CentOS-Server1.example.exam systemd[1]: Starting Cleanup of Temporary Directories...
May 15 20:44:05 CentOS-Server1.example.exam systemd[1]: Started Cleanup of Temporary Directories.

Any other ideas ? or is it time to trash these two machines and start over ?
To give a bit of background, I am preparing for a cerification. I am not one to mug up commands, follow 'dumps' and score a cert, which is why I want to troubleshoot this, to get more in-depth knowledge.

[samblues000]

Hi, Did you get any resolution for the same.
I am also facing the same thing, when i try to mount the NFS Export from the NFS enabled kdc client, it says, access denied.

[hunter86_bg]

As usual start with checking:
1. Time is in sync
2. A & PTR records exists for both server and client
3. Kerberos keytab is present on both server & client
4. Kerberos keytab contain the NFS principal
5. Enable debug and restart the stack (don't forget the nfs-secure & nfs-secure server services)