[SOLVED] SSH Hangs For Some Hosts

[albyva]

I'm having this problem where SSH from one host works flawlessly. SSH from another host hangs and never returns a password.
There are no firewall or routing issues, because the connection gets established and then just hangs. When I try to run a manual
test by telneting to port 22, I get the banner when I run it from the host that works, and I get nothing from the host that fails.

Does anybody have any idea why this problem is happening? Why is 1 host establishing a connection and getting a passwd prompt,
while another host is just being stalled and never getting beyond the connection established phase?

** Names, IP Addresses, etc have been removed for security reasons.

The host that doesn't work:
========================
[user@host1 ~]$ ssh -v -l user xx.xx.xx.xx
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/user/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/identity type -1
debug1: identity file /home/user/.ssh/identity-cert type -1
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1


^C
[user@host1 ~]$

[user@host1 ~]$ telnet xx.xx.xx.xx 22
Trying xx.xx.xx.xx...
Connected to xx.xx.xx.xx.
Escape character is '^]'.
^]q

telnet> q
Connection closed.



The Host that does work:
=======================
[user@host2 ~]$ ssh -v -l user xx.xx.xx.xx
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /home/user/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/identity type -1
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'xx.xx.xx.xx' is known and matches the RSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/user/.ssh/identity
debug1: Trying private key: /home/user/.ssh/id_rsa
debug1: Trying private key: /home/user/.ssh/id_dsa
debug1: Next authentication method: password
user@xx.xx.xx.xx's password:


[user@host2 ~]$ telnet xx.xx.xx.xx 22
Trying xx.xx.xx.xx...
Connected to xx.xx.xx.xx (xx.xx.xx.xx).
Escape character is '^]'.
SSH-2.0-OpenSSH_5.3
^]
telnet> q
Connection closed.

[TrevorH]

Your telnet experiment shows that the working host sends the expected "SSH-2.0-OpenSSH_5.3" banner but the other one does not. This indicates that it's either the remote openssh server that's at fault or some intervening firewall is being overzealous in scrubbing version numbers from banners - Cisco I'm looking at you!

[albyva]

Your telnet experiment shows that the working host sends the expected "SSH-2.0-OpenSSH_5.3" banner but the other one does not. This indicates that it's either the remote openssh server that's at fault or some intervening firewall is being overzealous in scrubbing version numbers from banners - Cisco I'm looking at you!

That is a good lead, because SSH from the host having issues doesn't have these problems SSHing to other servers. Just one in particular. Since the remote server accepts SSH
from another host, it doesn't "seem" like it's the openssh of the remote server. So maybe it is some intervening firewall along the path.

You don't suspect any openssh version issues, do you?

[albyva]

As I'm digging deeper, my issue might be a result of Asymmetric Routing.
I'll post more if that turns out to be the case. At least it'll be a good documented case if others encounter a similar issue.

[albyva]

Problem solved. Everything can be blamed on a Palo Alto Firewall.