[SOLVED] SSH out is fine! SSH in is NOT!

[bluethundr]

Hello! I have a little network at home of 11 machines. There is one physical CentOS 5.4 host and then 10 virtual hosts living on that machine hosted by Xen.

[code]
Linux lcent5-1 2.6.18-194.8.1.el5xen #1 SMP Thu Jul 1 20:25:56 EDT 2010 i686 i686 i386 GNU/Linux
[/code]
Recently I tried to install a memory module into the machine only to find that after I turned it on, the machine would not boot normally. It would take upwards of 45 mintues just to boot! As soon as it started to boot abnormally I removed the memory module from the machine only discovering that it would take this amount of time to boot. I tried several times and each time it took the better part of an hour.

Luckily I had another machine that was just like it! So I popped the hard drive out of the slow booting machine and then popped it into another one. Pretty soon my beloved CentOS box was booting as rapidly as it ever had and was seeming back to it's own self.

BUT!!!!

You could SSH out FROM the machine but no longer could you SSH **IN**!! Odd!

SSH out:

[code]
[bluethundr@lcent5-1:~]$:ssh bsd2
Last login: Tue Aug 24 00:13:23 2010 from 166.137.139.157
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.

FreeBSD 8.0-RELEASE (GENERIC) #0: Sat Nov 21 15:48:17 UTC 2009

#########################################################
# SUMMITNJHOME.COM #
# TITLE: FreeBSD 2 BOX #
# LOCATION: SUMMIT BASEMENT #
# #
#########################################################


[bluethundr@lbsd8-2:~]$:
[/code]
SSH back in?

[code]
[bluethundr@lbsd8-2:~]$:ssh lcent5-1
ssh: connect to host lcent5-1.summitnjhome.com port 22: Host is down
[/code]
So from another terminal window I have a look to see what could be going on!

Nothing seemed odd in messages:

[code]
[root@lcent5-1:~]$:tail -f /var/log/messages
Aug 23 18:00:08 lcent5-1 kernel: xenbr0: port 4(vif7.0) entering forwarding state
Aug 23 18:22:08 lcent5-1 dhclient: DHCPREQUEST on eth0 to 192.168.1.1 port 67
Aug 23 18:22:08 lcent5-1 dhclient: DHCPACK from 192.168.1.1
Aug 23 18:22:08 lcent5-1 dhclient: bound to 192.168.1.20 -- renewal in 42034 seconds.
Aug 23 20:09:08 lcent5-1 avahi-daemon[2903]: Invalid query packet.
Aug 23 20:39:33 lcent5-1 last message repeated 3 times
Aug 23 21:09:58 lcent5-1 last message repeated 3 times
Aug 23 21:40:23 lcent5-1 last message repeated 3 times
Aug 23 22:10:48 lcent5-1 last message repeated 3 times
Aug 23 22:41:13 lcent5-1 last message repeated 3 times
[/code]
or the secure logs:

[code]
[root@lcent5-1:~]$:tail -f /var/log/secure
Aug 22 20:20:10 lcent5-1 sshd[2687]: Server listening on :: port 22.
Aug 22 20:20:10 lcent5-1 sshd[2687]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 20:21:09 lcent5-1 gdm[4201]: pam_unix(gdm:session): session opened for user bluethundr by (uid=0)
Aug 22 20:21:54 lcent5-1 sudo: bluethundr : TTY=pts/2 ; PWD=/home/bluethundr ; USER=root ; COMMAND=/bin/bash
Aug 22 20:21:56 lcent5-1 userhelper[4517]: running '/usr/sbin/setup' with root privileges on behalf of 'root'
Aug 22 20:27:20 lcent5-1 sudo: bluethundr : TTY=pts/5 ; PWD=/home/bluethundr ; USER=root ; COMMAND=/usr/bin/setup
Aug 22 20:27:20 lcent5-1 userhelper[4739]: running '/usr/sbin/setup' with root privileges on behalf of 'root'
Aug 22 23:23:47 lcent5-1 sudo: bluethundr : TTY=pts/5 ; PWD=/home/bluethundr ; USER=root ; COMMAND=/bin/bash
Aug 22 23:51:53 lcent5-1 sshd[2687]: Received signal 15; terminating.
Aug 23 22:48:08 lcent5-1 sudo: bluethundr : TTY=pts/10 ; PWD=/home/bluethundr ; USER=root ; COMMAND=/bin/bash
[/code]
IPtables was OFF (for the time being, not permanently of course):

[code]
[root@lcent5-1:~]$:service iptables status
Firewall is stopped.
[/code]
And SELinux is disabled:

[code]
[root@lcent5-1:~]$:setenforce Permissive
setenforce: SELinux is disabled
[/code]
**AND** to make the story even more interesting... I can SSH in to virtual hosts that are hosted on the CentOS box that I cannot SSH into!

[code]
[bluethundr@lbsd8-2:~]$:ssh lcent5-1
ssh: connect to host lcent5-1.summitnjhome.com port 22: Host is down
[bluethundr@lbsd8-2:~]$:ssh virt1
Last login: Mon Aug 23 10:57:12 2010 from 192.168.1.44
#########################################################
# SUMMITNJHOME.COM #
# TITLE: VIRT1 BOX #
# LOCATION: SUMMIT BASEMENT #
# #
#########################################################


[bluethundr@virtCent01:~]$:
[/code]
[code]
[root@lbsd8-2:~]#:ssh virt2
root@virt2.summitnjhome.com's password:
Last login: Mon Aug 23 22:56:59 2010 from 192.168.1.44
FreeBSD 8.0-RELEASE (GENERIC) #0: Sat Nov 21 15:48:17 UTC 2009

#########################################################
# SUMMITNJHOME.COM #
# TITLE: Virt 2 BOX #
# LOCATION: SUMMIT BASEMENT #
# #
#########################################################



[root@virtCent02:~ ]#:
[/code]
Just to be clear, these above two machines are virtual Xen instances hosted on the machine that I am having trouble SSHing into at the moment!

[code]
[root@lcent5-1:~]$:xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 2259 2 r----- 2837.3
vm01 7 255 1 -b---- 57.1
vm02 2 255 1 -b---- 170.7
vm03 6 255 1 -b---- 165.2
vm04 5 255 1 -b---- 520.2
[/code]
Well I have to admit that at this stage I am stumped!! I would certainly appreciate any suggestions on how to proceed that anyone might have!!!

:-?

[pschaff]

Start with a google on [url=http://www.google.com/search?q=%22Address+already+in+use%22+sshd+ipv6+site%3Acentos.org&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a]"Address already in use" sshd ipv6 site:centos.org[/url]. What does [code]service sshd status[/code]show?

[bluethundr]

sshd is on, I am able to ssh out as mentioned:

[code]
[root@lcent5-1:~]$:service sshd status
openssh-daemon (pid 2799) is running...
[/code]

But I grepped "already in use" on secure log and found this:

[code]
[root@lcent5-1:~]$:grep -i "address already in use" /var/log/secure
Aug 22 11:09:15 lcent5-1 sshd[2519]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 11:19:36 lcent5-1 sshd[2517]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 11:28:06 lcent5-1 sshd[2516]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 12:19:59 lcent5-1 sshd[2519]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 12:56:46 lcent5-1 sshd[2201]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 13:07:50 lcent5-1 sshd[2520]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 13:37:11 lcent5-1 sshd[2544]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 13:47:03 lcent5-1 sshd[2563]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 20:09:00 lcent5-1 sshd[2709]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 22 20:20:10 lcent5-1 sshd[2687]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 24 21:31:33 lcent51 sshd[2799]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
[/code]


And saw that ssh was producing bind errors on port 22 claiming that the address is already. So I grepped netstat for 22 and this is what I saw:

[code]
[root@lcent5-1:~]$:netstat -tap | grep 22
tcp 0 0 lcent51.summitnjhome.c:2208 *:* LISTEN 2781/hpiod
tcp 0 0 *:time *:* LISTEN 2822/xinetd
tcp 0 0 *:rsync *:* LISTEN 2822/xinetd
tcp 0 0 192.168.122.1:domain *:* LISTEN 3884/dnsmasq
tcp 0 0 lcent51.summitnjhome.c:2207 *:* LISTEN 2786/python
tcp 0 0 192.168.1.20:53822 192.168.1.44:ssh TIME_WAIT -
[/code]

And just to be sure I made sure ipv6 was turned off:

[code]
[root@lcent5-1:~]$:service ip6tables status
Firewall is stopped.
[root@lcent5-1:~]$:chkconfig ip6tables off
[/code]

Does anyone know what may be taking up 22 and causing ssh in to timeout?

Thanks

[r_hartman]

The [i]Bind to port 22 on 0.0.0.0 failed: Address already in use.[/i] message you will always see when not binding sshd to a specific IP address. You can safely ignore that.

That ssh does not show up in your grepped netstat is because you do not specify the 'n' switch for numeric output.
You should either grep for 'ssh' or do a netstat -tapn.

Take it from there, but I suspect sshd is running fine, as you wouldn't be able to connect if it didn't. Your ssh command would just be sitting there, waiting to time out. So it is getting a response, but not the one it needs.

Keep in mind your host key probably changed when you swapped the box. You may want to run ssh -vvv to get some more details on what's going on.

EDIT: After some googling it seems you may be looking at ARP cache issues. One poster had the same issue, without an obvious cause, and it only disappeared after he reset his router. Sorry, don't know what else to offer.

[bluethundr]

It turned out that this box has two nic's. and networking was bound to the wrong interface that was glommed onto DHCP. I disabled the interface and bounced the network service and lo and behold all was normal at that point. Thanks for your help in this!

[AlanBartlett]

Thank you for reporting back with your success.

For posterity, this thread is marked [SOLVED].