CentOS6 with Kerberos/Winbind AD Auth - only root cannot ssh

[sfreireich]

I have a couple of servers that I set up to authenticate to Active Directory via the Kerberos/Winbind option. Everything works fine for domain user accounts (can ssh, su, samba, etc). However, the root account can no longer ssh using a password (can with SSH keys) but you can su to root without any problems. I used authconfig to configure /etc/krb5.conf file. I get the following in the secure log:
Mar 19 14:46:23 server_name sshd[13599]: Postponed keyboard-interactive for root from 127.0.0.1 port 44241 ssh2
Mar 19 14:46:25 server_name sshd[13599]: Connection closed by 127.0.0.1
Mar 19 14:46:25 server_name sshd[13599]: debug1: do_cleanup
Mar 19 14:46:25 server_name sshd[13596]: debug1: do_cleanup
Mar 19 14:46:25 server_name sshd[13596]: debug1: PAM: cleanup
Mar 19 14:47:12 server_name sshd[13588]: Received signal 15; terminating.
Mar 19 14:47:12 server_name sshd[13631]: Set /proc/self/oom_score_adj from 0 to -1000
Mar 19 14:47:12 server_name sshd[13631]: debug1: Bind to port 22 on 0.0.0.0.
Mar 19 14:47:12 server_name sshd[13631]: Server listening on 0.0.0.0 port 22.
Mar 19 14:47:12 server_name sshd[13631]: socket: Address family not supported by protocol
Mar 19 15:39:48 server_name sshd[13631]: debug1: Forked child 13766.
Mar 19 15:39:48 server_name sshd[13766]: Set /proc/self/oom_score_adj to 0
Mar 19 15:39:48 server_name sshd[13766]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Mar 19 15:39:48 server_name sshd[13766]: debug1: inetd sockets after dupping: 3, 3
Mar 19 15:39:48 server_name sshd[13766]: Connection from 127.0.0.1 port 44246
Mar 19 15:39:48 server_name sshd[13766]: debug1: Client protocol version 2.0; client software version OpenSSH_5.3
Mar 19 15:39:48 server_name sshd[13766]: debug1: match: OpenSSH_5.3 pat OpenSSH*
Mar 19 15:39:48 server_name sshd[13766]: debug1: Enabling compatibility mode for protocol 2.0
Mar 19 15:39:48 server_name sshd[13766]: debug1: Local version string SSH-2.0-OpenSSH_5.3
Mar 19 15:39:48 server_name sshd[13769]: debug1: permanently_set_uid: 74/74
Mar 19 15:39:48 server_name sshd[13769]: debug1: list_hostkey_types: ssh-rsa,ssh-dss
Mar 19 15:39:48 server_name sshd[13769]: debug1: SSH2_MSG_KEXINIT sent
Mar 19 15:39:48 server_name sshd[13769]: debug1: SSH2_MSG_KEXINIT received
Mar 19 15:39:48 server_name sshd[13769]: debug1: kex: client->server aes128-ctr hmac-md5 none
Mar 19 15:39:48 server_name sshd[13769]: debug1: kex: server->client aes128-ctr hmac-md5 none
Mar 19 15:39:48 server_name sshd[13769]: debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
Mar 19 15:39:48 server_name sshd[13769]: debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
Mar 19 15:39:48 server_name sshd[13769]: debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
Mar 19 15:39:48 server_name sshd[13769]: debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
Mar 19 15:39:48 server_name sshd[13769]: debug1: SSH2_MSG_NEWKEYS sent
Mar 19 15:39:48 server_name sshd[13769]: debug1: expecting SSH2_MSG_NEWKEYS
Mar 19 15:39:48 server_name sshd[13769]: debug1: SSH2_MSG_NEWKEYS received
Mar 19 15:39:48 server_name sshd[13769]: debug1: KEX done
Mar 19 15:39:48 server_name sshd[13769]: debug1: userauth-request for user root service ssh-connection method none
Mar 19 15:39:48 server_name sshd[13769]: debug1: attempt 0 failures 0
Mar 19 15:39:48 server_name sshd[13766]: debug1: PAM: initializing for "root"
Mar 19 15:39:48 server_name sshd[13769]: debug1: userauth-request for user root service ssh-connection method keyboard-interactive
Mar 19 15:39:48 server_name sshd[13769]: debug1: attempt 1 failures 0
Mar 19 15:39:48 server_name sshd[13769]: debug1: keyboard-interactive devs
Mar 19 15:39:48 server_name sshd[13769]: debug1: auth2_challenge: user=root devs=
Mar 19 15:39:48 server_name sshd[13769]: debug1: kbdint_alloc: devices 'pam'
Mar 19 15:39:48 server_name sshd[13766]: debug1: PAM: setting PAM_RHOST to "localhost"
Mar 19 15:39:48 server_name sshd[13766]: debug1: PAM: setting PAM_TTY to "ssh"
Mar 19 15:39:48 server_name sshd[13769]: debug1: auth2_challenge_start: trying authentication method 'pam'
Mar 19 15:39:48 server_name sshd[13769]: Postponed keyboard-interactive for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:39:55 server_name sshd[13770]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=root
Mar 19 15:39:57 server_name sshd[13766]: error: PAM: Authentication failure for root from localhost
Mar 19 15:39:57 server_name sshd[13766]: Failed keyboard-interactive/pam for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:39:57 server_name sshd[13769]: debug1: userauth-request for user root service ssh-connection method keyboard-interactive
Mar 19 15:39:57 server_name sshd[13769]: debug1: attempt 2 failures 1
Mar 19 15:39:57 server_name sshd[13769]: debug1: keyboard-interactive devs
Mar 19 15:39:57 server_name sshd[13769]: debug1: auth2_challenge: user=root devs=
Mar 19 15:39:57 server_name sshd[13769]: debug1: kbdint_alloc: devices 'pam'
Mar 19 15:39:57 server_name sshd[13769]: debug1: auth2_challenge_start: trying authentication method 'pam'
Mar 19 15:39:57 server_name sshd[13769]: Postponed keyboard-interactive for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:40:00 server_name sshd[13771]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=root
Mar 19 15:40:02 server_name sshd[13766]: error: PAM: Authentication failure for root from localhost
Mar 19 15:40:02 server_name sshd[13766]: Failed keyboard-interactive/pam for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:40:02 server_name sshd[13769]: debug1: userauth-request for user root service ssh-connection method keyboard-interactive
Mar 19 15:40:02 server_name sshd[13769]: debug1: attempt 3 failures 2
Mar 19 15:40:02 server_name sshd[13769]: debug1: keyboard-interactive devs
Mar 19 15:40:02 server_name sshd[13769]: debug1: auth2_challenge: user=root devs=
Mar 19 15:40:02 server_name sshd[13769]: debug1: kbdint_alloc: devices 'pam'
Mar 19 15:40:02 server_name sshd[13769]: debug1: auth2_challenge_start: trying authentication method 'pam'
Mar 19 15:40:02 server_name sshd[13769]: Postponed keyboard-interactive for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:40:04 server_name sshd[13776]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=root
Mar 19 15:40:07 server_name sshd[13766]: error: PAM: Authentication failure for root from localhost
Mar 19 15:40:07 server_name sshd[13766]: Failed keyboard-interactive/pam for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:40:09 server_name sshd[13769]: debug1: userauth-request for user root service ssh-connection method password
Mar 19 15:40:09 server_name sshd[13769]: debug1: attempt 4 failures 3
Mar 19 15:40:09 server_name sshd[13766]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=root
Mar 19 15:40:11 server_name sshd[13766]: debug1: PAM: password authentication failed for root: Authentication failure
Mar 19 15:40:11 server_name sshd[13766]: Failed password for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:40:14 server_name sshd[13769]: debug1: userauth-request for user root service ssh-connection method password
Mar 19 15:40:14 server_name sshd[13769]: debug1: attempt 5 failures 4
Mar 19 15:40:16 server_name sshd[13766]: debug1: PAM: password authentication failed for root: Authentication failure
Mar 19 15:40:16 server_name sshd[13766]: Failed password for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:40:19 server_name sshd[13769]: debug1: userauth-request for user root service ssh-connection method password
Mar 19 15:40:19 server_name sshd[13769]: debug1: attempt 6 failures 5
Mar 19 15:40:20 server_name sshd[13766]: debug1: PAM: password authentication failed for root: Authentication failure
Mar 19 15:40:20 server_name sshd[13766]: Failed password for root from 127.0.0.1 port 44246 ssh2
Mar 19 15:40:20 server_name sshd[13769]: Disconnecting: Too many authentication failures for root
Mar 19 15:40:20 server_name sshd[13769]: debug1: do_cleanup
Mar 19 15:40:20 server_name sshd[13766]: debug1: do_cleanup
Mar 19 15:40:20 server_name sshd[13766]: debug1: PAM: cleanup
Mar 19 15:40:20 server_name sshd[13766]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=root

I believe the issue is with PAM but tried several different options to remedy to no avail. Current PAM files are configured as:
# more sshd
#%PAM-1.0
auth required pam_sepermit.so
auth include password-auth
account required pam_nologin.so
account include password-auth
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session optional pam_keyinit.so force revoke
session include password-auth

# more system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth sufficient pam_krb5.so use_first_pass
auth sufficient pam_winbind.so use_first_pass
auth required pam_deny.so
account required pam_access.so
account required pam_unix.so broken_shadow
account [default=ignore success=1] pam_succeed_if.so uid < 16777216 quiet
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_krb5.so
account [default=bad success=ok user_unknown=ignore] pam_winbind.so
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3 type=
password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password sufficient pam_krb5.so use_authtok
password sufficient pam_winbind.so use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session optional pam_oddjob_mkhomedir.so umask=0077
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
session optional pam_krb5.so

more password-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth sufficient pam_krb5.so use_first_pass
auth sufficient pam_winbind.so use_first_pass
auth required pam_deny.so
account required pam_access.so
account required pam_unix.so broken_shadow
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_krb5.so
account [default=bad success=ok user_unknown=ignore] pam_winbind.so
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3 type=
password sufficient pam_unix.so md5 shadow nullok try_first_pass use_authtok
password sufficient pam_krb5.so use_authtok
password sufficient pam_winbind.so use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session optional pam_mkhomedir.so umask=0077
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
session optional pam_krb5.so

Any ideas here? The PAM files are in the state they were after authconfig. I do not nscd running and selinux is not a factor. Also, root is enabled in /etc/ssh/sshd_config file (root ssh worked before joining to the domain).

Thank you.