Server hangs in the middle of the night

Issues related to hardware problems
User avatar
fawnridge
Posts: 16
Joined: 2016/04/01 12:04:10
Contact:

Server hangs in the middle of the night

Post by fawnridge » 2016/04/01 12:12:00

5.11 running on a custom built server. Supermicro motherboard, parallel hard drive, fixed IP address.

The server is exactly the same as another dozen I've sold over the years, so I know the hardware configuration works. The problem has been going on for a month. Sometimes it will work fine for several days, then in the middle of the night it will hang. Hang to the point where the ONLY thing the client can do is pull the power cord and reboot. Two days ago, I had them reload 5.11. This morning, the server is hung.

I think the problem may be with the static IP address and I'm going to have switch it off and go with DHCP today. Is it possible that CentOS is trying to contact the Internet in the middle of the night for updates? I was only able to ftp with this server during installation when I had the client switch temporarily to DHCP.

Thanks.

User avatar
TrevorH
Forum Moderator
Posts: 23632
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Server hangs in the middle of the night

Post by TrevorH » 2016/04/01 13:54:46

It's more likely that you have a hardware problem: perhaps a failing PSU or a fan has stopped working or something. Many things are kicked off in the middle of the night by cron so it may be that one of those is exercising the system in a way that makes it crash.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

User avatar
fawnridge
Posts: 16
Joined: 2016/04/01 12:04:10
Contact:

Re: Server hangs in the middle of the night

Post by fawnridge » 2016/04/01 17:37:35

Normally, I would agree with you and blame the hardware. But this server runs fine all day long with 20 users on it. They're printing, emailing, and working in a very large database with a tremendous amount of disk activity. This same server ran RedHat 9 until it was upgraded to CentOS 5.11 six or seven weeks ago. Ever since the upgrade, this problem has been occurring on an irregular basis: sometimes not for two or three days, sometimes every night for three nights in a row.

I've set their system to access their network with DHCP and added the current IP address, after a reboot, to /etc/hosts.

User avatar
TrevorH
Forum Moderator
Posts: 23632
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Server hangs in the middle of the night

Post by TrevorH » 2016/04/01 19:56:33

If it's network related (and I don't see any evidence either for or against that theory) then your system should still be alive if you access it via the local console. Check that.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

User avatar
fawnridge
Posts: 16
Joined: 2016/04/01 12:04:10
Contact:

Re: Server hangs in the middle of the night

Post by fawnridge » 2016/04/01 20:57:01

When it's hung, there's nothing you can do. The console is black. The keyboard has no effect. They have to pull the power cord to reboot.

User avatar
TrevorH
Forum Moderator
Posts: 23632
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Server hangs in the middle of the night

Post by TrevorH » 2016/04/01 21:07:41

Right, so it's pretty unlikely to be network then. Are the keyboard lights flashing? Does it ping? Have you tried setting up rsyslog to log to a remote machine so that you can see if there are any logs produced immediately prior to whatever-it-is?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

User avatar
fawnridge
Posts: 16
Joined: 2016/04/01 12:04:10
Contact:

Re: Server hangs in the middle of the night

Post by fawnridge » 2016/04/01 21:41:40

Don't know any of those answers. Yes, I can ping their router but I'm in South Florida and the client is in Portland, Oregon.

drk
Posts: 395
Joined: 2014/01/30 20:38:28

Re: Server hangs in the middle of the night

Post by drk » 2016/04/02 05:49:58

Middle of the night? Backup running? That will stress the system and a weak power supply/bad fan won't be able to keep up.

User avatar
fawnridge
Posts: 16
Joined: 2016/04/01 12:04:10
Contact:

Re: Server hangs in the middle of the night

Post by fawnridge » 2016/04/02 22:59:35

With the server getting its network access through DHCP, it ran all night.
re: backup - The only thing that gets backed up is a single partition on the hard drive that has the client's data and our programs. The entire backup - to a USB flash drive - takes less than 3 minutes.

User avatar
fawnridge
Posts: 16
Joined: 2016/04/01 12:04:10
Contact:

Re: Server hangs in the middle of the night

Post by fawnridge » 2016/04/05 20:31:34

Now 3 nights in a row. I don't want to tempt fate, but I'm fairly well convinced that the issue was DHCP.

Post Reply