ssh fails to auth; immediate disconnect

Installing, Configuring, Troubleshooting server daemons such as Web and Mail
Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

ssh fails to auth; immediate disconnect

Post by Phylum » 2012/12/27 13:31:26

Although sshd is running, the moment a client it seems to fail to auth and immediately disconnects. It disconnects before one can enter in a user or password or before it even reads the key. It does not return an incorrect/unknown user or pass error - it just quits. All other services are working fine: apache, mysql, bind, mysql, imap, smtp etc.

Putty will return:
PuTTY Fatal Error Network error: Software caused connection abort

Standard Linux ssh will return:
Read from socket failed: Connection reset by peer

I've performed the following
- restarted ssh via whm
- bounced the server when restarting the service didn't fix the problem
- checked .ssh/authorized_keys to ensure they looked legit
- replaced sshd_config with a known working copy (another instance where sshd is responding normally)
- verified my pem/ppk files are ok
- tried via root and other users
- tried from multiple computers (windows, mac, Linux, Android )
- tried from computers outside of my home network (the office, aws, mobile network)
- tried from another vm within aws (another VM on the same subnet)
- didn't notice anything crazy in tcpdump while trying to connect via ssh
- checked iptables & flushed everything
- disabled lfd, & brute force protection
- verified aws security groups are properly setup
- tried adding 'ListenAddress 0.0.0.0' to sshd_config
- checked hosts.allow & hosts.deny
- user present in shadow


Its a virtual box not hosted on-site. I have no remote access (ssh, scp or telnet) to this machine. The only way I can 'get in' to retrieve logs or make configuration changes is to power it down, disconnect the volume, spin up another VM, attach & mount the volume. Once I grab what I need & make my changes changes, I reverse the process & power it back on. Again, after powering the box, it comes up fine & all services start (apache, mysql, imap, smtp, pop3, named, sshd etc) but ssh connection continue to fail.

This is a CentOS 5.8 vm - installed about 2 weeks ago. yum update'd before & after installing cpanel. Was working fine up until Dec 24th; I disconnected when the FiOS guy came to install the new service. Haven't been able to reconnect since.
The last couple of changes that I can think of was editing /etc/hosts but I don't see how that would break it.

More logs here
[url=http://pastebin.com/DG9ids8G]http://pastebin.com/DG9ids8G[/url]
[url=http://pastebin.com/VDjQqEDP]http://pastebin.com/VDjQqEDP[/url]
[url=http://pastebin.com/eL3mdTnW]http://pastebin.com/eL3mdTnW[/url]


Some output from ssh
[code]$ ssh -vvvo PreferredAuthentications=password user@10.0.0.22
OpenSSH_5.3p1, OpenSSL 1.0.0j-fips 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.0.0.22 [10.0.0.22] port 22.
debug1: Connection established.
debug1: identity file /home/ec2-user/.ssh/identity type -1
debug1: identity file /home/ec2-user/.ssh/id_rsa type -1
debug1: identity file /home/ec2-user/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 792 bytes for a total of 813
Connection closed by UNKNOWN

$ ssh -vvvo PreferredAuthentications=password user@10.0.0.22 -F /dev/null
OpenSSH_5.3p1, OpenSSL 1.0.0j-fips 10 May 2012
debug1: Reading configuration data /dev/null
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.0.0.22 [10.0.0.22] port 22.
debug1: Connection established.
debug1: identity file /home/ec2-user/.ssh/identity type -1
debug1: identity file /home/ec2-user/.ssh/id_rsa type -1
debug1: identity file /home/ec2-user/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 792 bytes for a total of 813
Connection closed by UNKNOWN

$ ssh -vvvo PreferredAuthentications=password -c aes256-ctr user@10.0.0.22
OpenSSH_5.3p1, OpenSSL 1.0.0j-fips 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.0.0.22 [10.0.0.22] port 22.
debug1: Connection established.
debug1: identity file /home/ec2-user/.ssh/identity type -1
debug1: identity file /home/ec2-user/.ssh/id_rsa type -1
debug1: identity file /home/ec2-user/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 496 bytes for a total of 517
Read from socket failed: Connection reset by peer

$ ssh -vvvo PreferredAuthentications=password -c aes256-ctr user@10.0.0.22 -F /dev/null
OpenSSH_5.3p1, OpenSSL 1.0.0j-fips 10 May 2012
debug1: Reading configuration data /dev/null
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.0.0.22 [10.0.0.22] port 22.
debug1: Connection established.
debug1: identity file /home/ec2-user/.ssh/identity type -1
debug1: identity file /home/ec2-user/.ssh/id_rsa type -1
debug1: identity file /home/ec2-user/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 496 bytes for a total of 517
Connection closed by UNKNOWN[/code]

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

ssh fails to auth; immediate disconnect

Post by toracat » 2012/12/27 17:29:47

Because cPanel makes extensive modifications to the system, we may not be able to provide proper assistance here. I saw you asking in the cPanel forums. If/when you get an answer that resolves the issue, please post back for future searchers.

Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

Re: ssh fails to auth; immediate disconnect

Post by Phylum » 2012/12/27 17:50:10

Will do - thanks for at least taking a look. Didn't want to put all my eggs in one basket in case they weren't willing/interested in helping troubleshoot this. I posted in as many places as made sense so I'll be sure to update them all.

Merry Christmas & Happy New Years all.

Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

Re: ssh fails to auth; immediate disconnect

Post by Phylum » 2012/12/28 19:07:12

Plan of action for o'dark thirty tomorrow morning since I can't take it down during the day:
- check .bashrc files
- uninstall/reinstall ssh: chroot then yum erase openssh-server && yum -y install openssh-server

I did manage to create a cron job earlier this morning to run a script & email me the results - finally had some time to review it: [url=http://pastebin.com/gz2AKvFJ]http://pastebin.com/gz2AKvFJ[/url]

Some things worth high lighting:
[code]## /var/log/messages
...
Dec 28 09:05:53 daniel sshd[18292]: Server listening on 0.0.0.0 port 22.
Dec 28 09:06:05 daniel kernel: [8101586.123735] sshd[18337]: segfault at 00005555c6a838a0 rip 00005555c6a838a0 rsp 00007fffe7021408 error 14
Dec 28 10:04:19 daniel kernel: [8105080.190032] sshd[26161]: segfault at 00005554fd2418a0 rip 00005554fd2418a0 rsp 00007fffb0861c48 error 14
Dec 28 10:05:51 daniel kernel: [8105171.946037] sshd[26176]: segfault at 000055557c5568a0 rip 000055557c5568a0 rsp 00007fff3154f1e8 error 14
Dec 28 11:56:24 daniel kernel: [8111805.096347] sshd[3013]: segfault at 00005554d596d8a0 rip 00005554d596d8a0 rsp 00007fffd8135dc8 error 14
Dec 28 11:57:50 daniel sshd[10747]: Did not receive identification string from 119.18.144.31
Dec 28 12:05:32 daniel kernel: [8112353.069199] sshd[11574]: segfault at 00005555cf2378a0 rip 00005555cf2378a0 rsp 00007fffde86c508 error 14
Dec 28 12:05:43 daniel kernel: [8112364.144714] sshd[12164]: segfault at 00005554dadf38a0 rip 00005554dadf38a0 rsp 00007fffd2cb0098 error 14
Dec 28 12:05:45 daniel kernel: [8112365.967406] sshd[12167]: segfault at 000055551db278a0 rip 000055551db278a0 rsp 00007fff8ff7e368 error 14
Dec 28 12:05:46 daniel kernel: [8112367.542307] sshd[12170]: segfault at 000055556a77b8a0 rip 000055556a77b8a0 rsp 00007fff43329708 error 14
#EOF

# I was connecting from another machine on the same subnet
## `sshd -dddDp 19999` results
Server listening on 0.0.0.0 port 19999.
debug3: fd 4 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 619
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 10.0.0.74 port 37821
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 619
debug2: parse_server_config: config /etc/ssh/sshd_config len 619
debug1: sshd version OpenSSH_4.3p2
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-dddDp'
debug1: rexec_argv[2]='19999'
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 19999 on 0.0.0.0.
Server listening on 0.0.0.0 port 19999.
debug3: fd 4 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 619
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 10.0.0.74 port 37823
#EOF

## /var/log/secure
Dec 28 07:19:27 daniel sshd[2096]: debug2: fd 3 setting O_NONBLOCK
Dec 28 07:19:27 daniel sshd[2096]: debug1: Bind to port 22 on 0.0.0.0.
Dec 28 07:19:27 daniel sshd[2096]: Server listening on 0.0.0.0 port 22.
Dec 28 08:14:50 daniel sshd[2096]: debug3: fd 4 is not O_NONBLOCK
Dec 28 08:14:50 daniel sshd[10962]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Dec 28 08:14:50 daniel sshd[2096]: debug1: Forked child 10962.
Dec 28 08:14:50 daniel sshd[2096]: debug3: send_rexec_state: entering fd = 7 config len 619
Dec 28 08:14:50 daniel sshd[2096]: debug3: ssh_msg_send: type 0
Dec 28 08:14:50 daniel sshd[2096]: debug3: send_rexec_state: done
Dec 28 08:14:50 daniel sshd[10962]: debug1: inetd sockets after dupping: 3, 3
Dec 28 08:14:50 daniel sshd[10962]: Connection from my.ip.add.ress port 21561
Dec 28 09:05:53 daniel sshd[2096]: Received signal 15; terminating.[/code]

Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

Re: ssh fails to auth; immediate disconnect

Post by Phylum » 2012/12/29 10:23:44

Ok so the situation is that sshd is segfaulting the moment a user connects. This is on a CentOS 5.6 x86_64 machine running OpenSSH 4.3p2-82.el5.

Its core dumping but I don't know what I should do next.

[code]# /usr/sbin/sshd -dddDp 19999
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 526
debug2: parse_server_config: config /etc/ssh/sshd_config len 526
debug1: sshd version OpenSSH_4.3p2
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-dddDp'
debug1: rexec_argv[2]='19999'
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 19999 on 0.0.0.0.
Server listening on 0.0.0.0 port 19999.
socket: Address family not supported by protocol
debug3: fd 4 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 526
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 41783
Segmentation fault (core dumped)

==> /var/log/messages <==
Dec 29 03:57:30 daniel kernel: [4386322.743762] sshd[11068]: segfault at 00005554de2e38a0 rip 00005554de2e38a0 rsp 00007fffcf7c1f08 error 14[/code]

Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

Re: ssh fails to auth; immediate disconnect

Post by Phylum » 2012/12/29 15:54:59

I'm not even going to pretend I know what I'm doing or talking about here. By doing a bit of research (a fancy way of saying I 'Googled it') I was able to get this far but its not looking very useful/promising.


core of /usr/sbin/sshd -dddDp 19999
[code]# gdb /usr/sbin/sshd core-sshd-11-0-0-1347-1356793916
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/sshd...(no debugging symbols found)...done.
BFD: Warning: /tmp/core/core-sshd-11-0-0-1347-1356793916 is truncated: expected core file size >= 806912, found: 802816.
[New Thread 1347]
Reading symbols from /lib64/libwrap.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libwrap.so.0
Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libaudit.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /usr/lib64/libfipscheck.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfipscheck.so.1
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `sshd: [accepted] '.
Program terminated with signal 11, Segmentation fault.
#0 0x0000555550c798a0 in ?? ()
(gdb) where
#0 0x0000555550c798a0 in ?? ()
#1 0x0000555555560b94 in main ()
(gdb) thread apply all bt

Thread 1 (Thread 0x2ada51905600 (LWP 1347)):
#0 0x0000555550c798a0 in ?? ()
#1 0x0000555555560b94 in main ()[/code]


core of /usr/sbin/sshd running as a daemon & segfaulting when someone tries to connect
[code]# gdb /usr/sbin/sshd core-sshd-11-0-0-1458-1356794096
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/sshd...(no debugging symbols found)...done.
BFD: Warning: /tmp/core/core-sshd-11-0-0-1458-1356794096 is truncated: expected core file size >= 806912, found: 802816.
[New Thread 1458]
Reading symbols from /lib64/libwrap.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libwrap.so.0
Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libaudit.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /usr/lib64/libfipscheck.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfipscheck.so.1
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `sshd: [accepted] '.
Program terminated with signal 11, Segmentation fault.
#0 0x000055552746d8a0 in ?? ()
(gdb) where
#0 0x000055552746d8a0 in ?? ()
#1 0x0000555555560b94 in main ()
(gdb) thread apply all bt

Thread 1 (Thread 0x2b13280f9600 (LWP 1458)):
#0 0x000055552746d8a0 in ?? ()
#1 0x0000555555560b94 in main ()[/code]

core of /usr/bin/ssh - trying to connect to a remote systsm
[code]# gdb /usr/bin/ssh core-ssh-11-0-0-2798-1356794504
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/ssh...(no debugging symbols found)...done.
[New Thread 2798]
Reading symbols from /usr/lib64/libfipscheck.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfipscheck.so.1
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `ssh 10.0.0.120'.
Program terminated with signal 11, Segmentation fault.
#0 0x00005554e3f13b80 in ?? ()
(gdb) where
#0 0x00005554e3f13b80 in ?? ()
#1 0x000055555557b67b in ?? ()
#2 0x0000555555565f50 in ?? ()
#3 0x0000555555566a36 in ?? ()
#4 0x000055555556781f in ?? ()
#5 0x000055555556b0f0 in ?? ()
#6 0x00005555555845cc in kexgex_client ()
#7 0x000055555558076d in kex_input_kexinit ()
#8 0x000055555557fabd in ?? ()
#9 0x000055555556b04e in ?? ()
#10 0x0000555555565be0 in ?? ()
#11 0x000055555555e1ca in main ()
(gdb) thread apply all bt

Thread 1 (Thread 0x2b0ee4df8e20 (LWP 2798)):
#0 0x00005554e3f13b80 in ?? ()
#1 0x000055555557b67b in ?? ()
#2 0x0000555555565f50 in ?? ()
#3 0x0000555555566a36 in ?? ()
#4 0x000055555556781f in ?? ()
#5 0x000055555556b0f0 in ?? ()
#6 0x00005555555845cc in kexgex_client ()
#7 0x000055555558076d in kex_input_kexinit ()
#8 0x000055555557fabd in ?? ()
#9 0x000055555556b04e in ?? ()
#10 0x0000555555565be0 in ?? ()
#11 0x000055555555e1ca in main ()[/code]

Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

Re: ssh fails to auth; immediate disconnect

Post by Phylum » 2012/12/29 18:10:34

After installing debug symbols...

### core of /usr/sbin/sshd -dddDp 19999
[quote]# gdb /usr/sbin/sshd /tmp/core/core-sshd-11-0-0-1347-1356793916
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/sshd...Reading symbols from /usr/lib/debug/usr/sbin/sshd.debug...done.
done.
BFD: Warning: /tmp/core/core-sshd-11-0-0-1347-1356793916 is truncated: expected core file size >= 806912, found: 802816.
[New Thread 1347]
Reading symbols from /lib64/libwrap.so.0...Reading symbols from /usr/lib/debug/lib64/libwrap.so.0.7.6.debug...done.
done.
Loaded symbols for /lib64/libwrap.so.0
Reading symbols from /lib64/libpam.so.0...Reading symbols from /usr/lib/debug/lib64/libpam.so.0.81.5.debug...done.
done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/lib64/libdl-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libselinux.so.1...
warning: the debug information found in "/usr/lib/debug//lib64/libselinux.so.1.debug" does not match "/lib64/libselinux.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/lib64/libselinux.so.1.debug" does not match "/lib64/libselinux.so.1" (CRC mismatch).

(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libaudit.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /usr/lib64/libfipscheck.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libfipscheck.so.1.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libfipscheck.so.1
Reading symbols from /lib64/libcrypto.so.6...Reading symbols from /usr/lib/debug/lib64/libcrypto.so.0.9.8e.debug...done.
done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libutil.so.1...Reading symbols from /usr/lib/debug/lib64/libutil-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libz.so.1...
warning: the debug information found in "/usr/lib/debug//lib64/libz.so.1.2.3.debug" does not match "/lib64/libz.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/lib64/libz.so.1.2.3.debug" does not match "/lib64/libz.so.1" (CRC mismatch).

(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/lib64/libnsl-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libcrypt.so.1...Reading symbols from /usr/lib/debug/lib64/libcrypt-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/lib64/libresolv-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libgssapi_krb5.so.2.2.debug...done.
done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5.so.3.3.debug...done.
done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libk5crypto.so.3.1.debug...done.
done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /usr/lib/debug/lib64/libcom_err.so.2.1.debug...done.
done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libnss3.so...Reading symbols from /usr/lib/debug/usr/lib64/libnss3.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/lib64/libc-2.5.so.debug...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.5.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libsepol.so.1...Reading symbols from /usr/lib/debug/lib64/libsepol.so.1.debug...done.
done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5support.so.0.1.debug...done.
done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/libnssutil3.so...Reading symbols from /usr/lib/debug/usr/lib64/libnssutil3.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...Reading symbols from /usr/lib/debug/usr/lib64/libplc4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...Reading symbols from /usr/lib/debug/usr/lib64/libplds4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libnspr4.so...Reading symbols from /usr/lib/debug/usr/lib64/libnspr4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/lib64/libpthread-2.5.so.debug...done.
[Thread debugging using libthread_db enabled]
done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `sshd: [accepted] '.
Program terminated with signal 11, Segmentation fault.
#0 0x0000555550c798a0 in ?? ()
(gdb) where
#0 0x0000555550c798a0 in ?? ()
#1 0x0000555555560b94 in sshd_exchange_identification (ac=, av=) at sshd.c:422
#2 main (ac=, av=) at sshd.c:1721
(gdb) thread apply all bt

Thread 1 (Thread 0x2ada51905600 (LWP 1347)):
#0 0x0000555550c798a0 in ?? ()
#1 0x0000555555560b94 in sshd_exchange_identification (ac=, av=) at sshd.c:422
#2 main (ac=, av=) at sshd.c:1721[/quote]

### core of /usr/sbin/sshd running normally
[quote]# gdb /usr/sbin/sshd /tmp/core/core-sshd-11-0-0-1458-1356794096
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/sshd...Reading symbols from /usr/lib/debug/usr/sbin/sshd.debug...done.
done.
BFD: Warning: /tmp/core/core-sshd-11-0-0-1458-1356794096 is truncated: expected core file size >= 806912, found: 802816.
[New Thread 1458]
Reading symbols from /lib64/libwrap.so.0...Reading symbols from /usr/lib/debug/lib64/libwrap.so.0.7.6.debug...done.
done.
Loaded symbols for /lib64/libwrap.so.0
Reading symbols from /lib64/libpam.so.0...Reading symbols from /usr/lib/debug/lib64/libpam.so.0.81.5.debug...done.
done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/lib64/libdl-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libselinux.so.1...
warning: the debug information found in "/usr/lib/debug//lib64/libselinux.so.1.debug" does not match "/lib64/libselinux.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/lib64/libselinux.so.1.debug" does not match "/lib64/libselinux.so.1" (CRC mismatch).

(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libaudit.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /usr/lib64/libfipscheck.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libfipscheck.so.1.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libfipscheck.so.1
Reading symbols from /lib64/libcrypto.so.6...Reading symbols from /usr/lib/debug/lib64/libcrypto.so.0.9.8e.debug...done.
done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libutil.so.1...Reading symbols from /usr/lib/debug/lib64/libutil-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libz.so.1...
warning: the debug information found in "/usr/lib/debug//lib64/libz.so.1.2.3.debug" does not match "/lib64/libz.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/lib64/libz.so.1.2.3.debug" does not match "/lib64/libz.so.1" (CRC mismatch).

(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/lib64/libnsl-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libcrypt.so.1...Reading symbols from /usr/lib/debug/lib64/libcrypt-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/lib64/libresolv-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libgssapi_krb5.so.2.2.debug...done.
done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5.so.3.3.debug...done.
done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libk5crypto.so.3.1.debug...done.
done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /usr/lib/debug/lib64/libcom_err.so.2.1.debug...done.
done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libnss3.so...Reading symbols from /usr/lib/debug/usr/lib64/libnss3.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/lib64/libc-2.5.so.debug...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.5.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libsepol.so.1...Reading symbols from /usr/lib/debug/lib64/libsepol.so.1.debug...done.
done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5support.so.0.1.debug...done.
done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/libnssutil3.so...Reading symbols from /usr/lib/debug/usr/lib64/libnssutil3.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...Reading symbols from /usr/lib/debug/usr/lib64/libplc4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...Reading symbols from /usr/lib/debug/usr/lib64/libplds4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libnspr4.so...Reading symbols from /usr/lib/debug/usr/lib64/libnspr4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/lib64/libpthread-2.5.so.debug...done.
[Thread debugging using libthread_db enabled]
done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `sshd: [accepted] '.
Program terminated with signal 11, Segmentation fault.
#0 0x000055552746d8a0 in ?? ()
(gdb) where
#0 0x000055552746d8a0 in ?? ()
#1 0x0000555555560b94 in sshd_exchange_identification (ac=, av=) at sshd.c:422
#2 main (ac=, av=) at sshd.c:1721
(gdb) thread apply all bt

Thread 1 (Thread 0x2b13280f9600 (LWP 1458)):
#0 0x000055552746d8a0 in ?? ()
#1 0x0000555555560b94 in sshd_exchange_identification (ac=, av=) at sshd.c:422
#2 main (ac=, av=) at sshd.c:1721[/code]

### core of /usr/bin/ssh - trying to connect to a remote systsm
[code]# gdb /usr/bin/ssh /tmp/core/core-ssh-11-0-0-2798-1356794504
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/bin/ssh...Reading symbols from /usr/lib/debug/usr/bin/ssh.debug...done.
done.
[New Thread 2798]
Reading symbols from /usr/lib64/libfipscheck.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libfipscheck.so.1.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libfipscheck.so.1
Reading symbols from /lib64/libcrypto.so.6...Reading symbols from /usr/lib/debug/lib64/libcrypto.so.0.9.8e.debug...done.
done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libutil.so.1...Reading symbols from /usr/lib/debug/lib64/libutil-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libz.so.1...
warning: the debug information found in "/usr/lib/debug//lib64/libz.so.1.2.3.debug" does not match "/lib64/libz.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/lib64/libz.so.1.2.3.debug" does not match "/lib64/libz.so.1" (CRC mismatch).

(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/lib64/libnsl-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libcrypt.so.1...Reading symbols from /usr/lib/debug/lib64/libcrypt-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/lib64/libresolv-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libgssapi_krb5.so.2.2.debug...done.
done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5.so.3.3.debug...done.
done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libk5crypto.so.3.1.debug...done.
done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /usr/lib/debug/lib64/libcom_err.so.2.1.debug...done.
done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libnss3.so...Reading symbols from /usr/lib/debug/usr/lib64/libnss3.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/lib64/libc-2.5.so.debug...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libplc4.so...Reading symbols from /usr/lib/debug/usr/lib64/libplc4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/lib64/libdl-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libkrb5support.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5support.so.0.1.debug...done.
done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/libnssutil3.so...Reading symbols from /usr/lib/debug/usr/lib64/libnssutil3.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplds4.so...Reading symbols from /usr/lib/debug/usr/lib64/libplds4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libnspr4.so...Reading symbols from /usr/lib/debug/usr/lib64/libnspr4.so.debug...done.
done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/lib64/libpthread-2.5.so.debug...done.
[Thread debugging using libthread_db enabled]
done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.5.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libselinux.so.1...
warning: the debug information found in "/usr/lib/debug//lib64/libselinux.so.1.debug" does not match "/lib64/libselinux.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/lib64/libselinux.so.1.debug" does not match "/lib64/libselinux.so.1" (CRC mismatch).

(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...Reading symbols from /usr/lib/debug/lib64/libsepol.so.1.debug...done.
done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.5.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `ssh 10.0.0.120'.
Program terminated with signal 11, Segmentation fault.
#0 0x00005554e3f13b80 in ?? ()
(gdb) where
#0 0x00005554e3f13b80 in ?? ()
#1 0x000055555557b67b in read_passphrase (
prompt=0x7fffc930aa20 "The authenticity of host '10.0.0.120 (10.0.0.120)' can't be established.\nRSA key fingerprint is 2c:b9:58:40:6e:6a:e5:fb:1b:6e:e6:10:c3:fb:32:67.\nAre you sure you want to continue connecting (yes/no)? ", flags=1) at readpass.c:139
#2 0x0000555555565f50 in confirm (
prompt=0x7fffc930aa20 "The authenticity of host '10.0.0.120 (10.0.0.120)' can't be established.\nRSA key fingerprint is 2c:b9:58:40:6e:6a:e5:fb:1b:6e:e6:10:c3:fb:32:67.\nAre you sure you want to continue connecting (yes/no)? ") at sshconnect.c:567
#3 0x0000555555566a36 in check_host_key (host=0x5555557b6e10 "10.0.0.120", hostaddr=, host_key=0x5555557bdc30, readonly=0,
user_hostfile=0x5555557b6cc0 "/root/.ssh/known_hosts", system_hostfile=0x5555557b6d70 "/etc/ssh/ssh_known_hosts") at sshconnect.c:772
#4 0x000055555556781f in verify_host_key (host=0x5555557b6e10 "10.0.0.120", hostaddr=0x5555557a22a0, host_key=0x5555557bdc30) at sshconnect.c:964
#5 0x000055555556b0f0 in verify_host_key_callback (hostkey=0x400) at sshconnect2.c:80
#6 0x00005555555845cc in kexgex_client (kex=0x5555557bb540) at kexgexc.c:120
#7 0x000055555558076d in kex_kexinit_finish (type=, seq=, ctxt=0x5555557bb540) at kex.c:241
#8 kex_input_kexinit (type=, seq=, ctxt=0x5555557bb540) at kex.c:210
#9 0x000055555557fabd in dispatch_run (mode=0, done=0x5555557bb5a8, ctxt=0x5555557bb540) at dispatch.c:93
#10 0x000055555556b04e in ssh_kex2 (host=, hostaddr=) at sshconnect2.c:142
#11 0x0000555555565be0 in ssh_login (sensitive=0x5555557a2320, orighost=, hostaddr=0x5555557a22a0, pw=,
timeout_ms=-1000) at sshconnect.c:1000
#12 0x000055555555e1ca in main (ac=, av=) at ssh.c:762
(gdb) thread apply all bt

Thread 1 (Thread 0x2b0ee4df8e20 (LWP 2798)):
#0 0x00005554e3f13b80 in ?? ()
#1 0x000055555557b67b in read_passphrase (
prompt=0x7fffc930aa20 "The authenticity of host '10.0.0.120 (10.0.0.120)' can't be established.\nRSA key fingerprint is 2c:b9:58:40:6e:6a:e5:fb:1b:6e:e6:10:c3:fb:32:67.\nAre you sure you want to continue connecting (yes/no)? ", flags=1) at readpass.c:139
#2 0x0000555555565f50 in confirm (
prompt=0x7fffc930aa20 "The authenticity of host '10.0.0.120 (10.0.0.120)' can't be established.\nRSA key fingerprint is 2c:b9:58:40:6e:6a:e5:fb:1b:6e:e6:10:c3:fb:32:67.\nAre you sure you want to continue connecting (yes/no)? ") at sshconnect.c:567
#3 0x0000555555566a36 in check_host_key (host=0x5555557b6e10 "10.0.0.120", hostaddr=, host_key=0x5555557bdc30, readonly=0,
user_hostfile=0x5555557b6cc0 "/root/.ssh/known_hosts", system_hostfile=0x5555557b6d70 "/etc/ssh/ssh_known_hosts") at sshconnect.c:772
#4 0x000055555556781f in verify_host_key (host=0x5555557b6e10 "10.0.0.120", hostaddr=0x5555557a22a0, host_key=0x5555557bdc30) at sshconnect.c:964
#5 0x000055555556b0f0 in verify_host_key_callback (hostkey=0x400) at sshconnect2.c:80
#6 0x00005555555845cc in kexgex_client (kex=0x5555557bb540) at kexgexc.c:120
#7 0x000055555558076d in kex_kexinit_finish (type=, seq=, ctxt=0x5555557bb540) at kex.c:241
#8 kex_input_kexinit (type=, seq=, ctxt=0x5555557bb540) at kex.c:210
#9 0x000055555557fabd in dispatch_run (mode=0, done=0x5555557bb5a8, ctxt=0x5555557bb540) at dispatch.c:93
#10 0x000055555556b04e in ssh_kex2 (host=, hostaddr=) at sshconnect2.c:142
#11 0x0000555555565be0 in ssh_login (sensitive=0x5555557a2320, orighost=, hostaddr=0x5555557a22a0, pw=,
timeout_ms=-1000) at sshconnect.c:1000
#12 0x000055555555e1ca in main (ac=, av=) at ssh.c:762[/quote]

Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

Re: ssh fails to auth; immediate disconnect

Post by Phylum » 2012/12/30 04:45:43

I performed an `strace -o /tmp/core/strace_sshd.txt /usr/sbin/sshd -dddDp 19999` but its too long to paste here apparently.
Check it out here: http://pastebin.com/KWZBSD4C


# ldd /usr/bin/ssh
[quote] libfipscheck.so.1 => /usr/lib64/libfipscheck.so.1 (0x00002acfa6eeb000)
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002acfa70ed000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002acfa743e000)
libz.so.1 => /lib64/libz.so.1 (0x00002acfa7642000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002acfa7856000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002acfa7a6e000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002acfa7ca7000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00002acfa7ebc000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002acfa80ea000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002acfa8380000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002acfa85a5000)
libnss3.so => /usr/lib64/libnss3.so (0x00002acfa87a7000)
libc.so.6 => /lib64/libc.so.6 (0x00002acfa8ad6000)
libplc4.so => /usr/lib64/libplc4.so (0x00002acfa8e2d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002acfa9031000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00002acfa9236000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002acfa943e000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00002acfa9649000)
libplds4.so => /usr/lib64/libplds4.so (0x00002acfa986f000)
libnspr4.so => /usr/lib64/libnspr4.so (0x00002acfa9a72000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acfa9cad000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00002acfa9ec9000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x00002acfaa0e1000)[/quote]
# ldd /usr/sbin/sshd
[quote] libwrap.so.0 => /lib64/libwrap.so.0 (0x00002b5238dea000)
libpam.so.0 => /lib64/libpam.so.0 (0x00002b5238ff3000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002b52391fe000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00002b5239403000)
libaudit.so.0 => /lib64/libaudit.so.0 (0x00002b523961b000)
libfipscheck.so.1 => /usr/lib64/libfipscheck.so.1 (0x00002b5239833000)
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002b5239a36000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002b5239d87000)
libz.so.1 => /lib64/libz.so.1 (0x00002b5239f8a000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002b523a19f000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b523a3b7000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002b523a5ef000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00002b523a805000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002b523aa33000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002b523acc8000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002b523aeee000)
libnss3.so => /usr/lib64/libnss3.so (0x00002b523b0f0000)
libc.so.6 => /lib64/libc.so.6 (0x00002b523b41e000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x00002b523b776000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00002b523b9bc000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002b523bbc5000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00002b523bdd0000)
libplc4.so => /usr/lib64/libplc4.so (0x00002b523bff5000)
libplds4.so => /usr/lib64/libplds4.so (0x00002b523c1fa000)
libnspr4.so => /usr/lib64/libnspr4.so (0x00002b523c3fd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b523c638000)[/quote]

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: ssh fails to auth; immediate disconnect

Post by TrevorH » 2012/12/30 20:32:32

Since pretty much the last thing it does is to open and read /etc/hosts.deny, do you have anything strange in that file?

Phylum
Posts: 10
Joined: 2012/12/16 16:59:13

Re: ssh fails to auth; immediate disconnect

Post by Phylum » 2013/01/02 19:24:09

Thanks for the reply & Happy New Year to you.

Both files are empty
[code]# cat /etc/hosts.*
#
# hosts.allow This file describes the names of the hosts which are
# allowed to use the local INET services, as decided
# by the '/usr/sbin/tcpd' server.
#

#
# hosts.deny This file describes the names of the hosts which are
# *not* allowed to use the local INET services, as decided
# by the '/usr/sbin/tcpd' server.
#[/code]

I renamed them & `touch`d them to recreate - still segfaults.

Short of doing a `yum reinstall *` and crossing my fingers, I don't know what to do.
I'm fully aware I should probably start over & run 6.x, but I fear I'll encounter this again, and if I do, then what? All that work for naught!

Locked