Persistent tunnel. AutoSSH and Hamachi are occasionably unreliable

[mbhansen87]

I tried posting this on StackExchange but got no response so I'm hoping someone here is able to help.

I have a server running on site with clients and I've used LogMeIn (Hamachi) and AutoSSH (As a backup) for probably 2 years now.

It works great, however once in a while both methods fail. Either a port is taken all of the sudden or something "weird" happens. Just yesterday both methods failed and I cannot see any cause for why Hamachi failed. AutoSSH just shows this "autossh[1497]: ssh exited with error status 255; restarting ssh" which is I remember accurately that is a port issue. That's a huge problem as I have to reach out to the client to hard reset the server which will often take a day or more.

Most of the times only one of them will fail and I can get in and do a reboot and all is well.

I'm at ends here as to what to do. Any guide I can find will mention AutoSSH and I've tried every possible setting. It will work, but then 2-3 months will pass and it'll break.

Are anyone aware of a third alternative I can use for when both methods fail? I hate having to bother clients with this and its not good losing 1 - 2 days when its down as I use it everyday.

For OS one of them uses Raspbian and the other CentOS. Both have been unreliable from time to time.

Michael

[ahitch]

I have absolutely no experience with using anything LogMeIn (hamachi).....



As for the SSH issue, the best solution I've found is to run a bash script as a simple cron job as root to see if the daemon is running. If not, the routine issues a restart. I know it's not as eloquent as having another piece of software do the checking, but if you can wait 5 minutes (the timing of the cron job), it just works...

Another thing that I do on a regular basis is install a VNC server application. x11vnc seems pretty good, but I've got a couple out there running Tiger....

Instead of jumping through all the hoops of trying to get the stuff started as a service, and editing all the X11 / gdm / kdm / sddm / etc.. stuff, again, I create a simple cron job to check the service every 5 minutes or so. If it's not there, it gets a start.

In my case, I've always got another ssh daemon running on another machine that I can tunnel into and then do a vncview sesison to that problem machine. Our VMware stuff is set up for ssh on a non-standard port, using complex authentication, and ESXI provides a VERY nice tunnel via ssh, just like any other well-behaved Linux box.

I've even got a couple Netgear NAS boxes in a couple sites that allow me in when there's an emergency.... They provide forwarding as well, very nicely.

Now, I know that I'm probably not solving your particular issue, but it "SOUNDS" like you're having the same needs by company has had.... and continues to have.... so, maybe something I've mentioned will get you out of a bind.

The other alternative (and yup.... I've had to do it) is to make sure certain machines are plugged into certain managed UPS boxes, or a managed power box, and hard-bounce the thing.

Maybe this won't actually solve your problem, but if you can get some kind of remote connection to it, while the problem exists, you can probably track it down from there....

Good luck

[mbhansen87]

Thanks for the detailed response. Greatly appreciated. It does sound like you've had to deal with similar challenges for sure.

I actually had a script running before using AutoSSH but that would also fail eventually. That's why I went back to AutoSSH. Maybe my script wasn't good enough if you're saying yours "just works". I don't recall what the problem was as I killed that approach over a year ago. You wouldn't happen to have the script would you? I could use that as a third backup door.

I've even thought of just making a script that wgets a server of ours every 15 mins as a last resort. If it sees a reboot file (that I upload via FTP) it'll reboot. So this would be a last resort kinda approach. If I simply cannot get in then I'll upload the file, the script will see it and notify me then reboot and I'll delete the file. Its not pretty but that would also work as a final resort. Beats a hard reset that is for sure.

[hunter86_bg]

Does autossh has a systemd service file? If yes, you can modify it in such way that when the daemon dies - systemd will restart it.

[ahitch]

You wouldn't happen to have the script would you? I could use that as a third backup door.

Ultra-simple.... My solution isn't, by any means, eloquent, but it just seems to work. Here it is:

if (ps -ale|grep -1 x11vnc); then echo x11running >/var/log/myx11log ; else x11vnc -forever -bg -rfbauth /etc/x11vnc/x11vnc.pass -display :0; fi

Just name it something and put it in the root cron and run it every few minutes. It hardly makes a noticeable tick in the process overhead.

You can do the same thing with the ssh daemon ( ps -ale | grep -k sshd) to make sure it still runs.

Keep in mind, though, this works in Cent and other Red-Hat distros. Ubuntu doesn't like anything touching a desktop session until after login.... even when gdm is running! Under anything Red-Hat (including Fedora), this seems to be the easiest.... at least for me...


Also remember that sometimes (which I just ran into with a 3rd party-supplied Red Hat based server), IPTABLES (in this case, but any firewall daemon) can get in the way, as well as SELINUX configs...

I ain't smart.... just persistent....