Yesterday morning we had a power disruption, I rebooted the webserver and discovered that one of the drives failed in the process. The server did boot up thanks to RAID5, but when I powered it down again and rebooted we only get the error, "Kernel panic - not syncing" error message. Now I am at a loss! (But learning!)
BTW the webserver is running Centos 6.5 (in fact we have the exact same server as described in the tutorial "The Perfect Server").
I have been able to use the Centos 6.5 Live DVD to get access to the server and can see that all of the directories are still there. (Relief) But still at a loss as to what to do........
This is what I have done so far:
1) I replaced the failed hard drive with a new one but not sure what it is named in the raid i.e. /dev/sdb1 /dev/sdc1 /dev/sdd1 or /dev/sde1 (...I am running 5 disks on a software raid5). Therefore we have not copied any partitions over from another disk or gone any further.
2) Upon booting we escaped out of the boot screen and tried appending the kernel arguments by adding selinux=0 enforcing=0 but still get the "Kernel panic-not syncing" error on boot.
3) I have booted the system with the Centos 6.5 install disk and gone into the rescue Centos system option but just get a message that there are no Linux partitions even though the other disks are okay. Then it drops me out to a shell.
I managed to have a look at the boot log, in summary the only error that we can find is:
mdadm failed to start array /dev/md/array-name
input/output error
Logical Volume Management disabled at boot
Checking Filesystems OK
Mounting Filesystems OK
.......................everything else seems okay.
Anyone have any ideas as to how to get this server up again? Thanks in advance for taking the time to help....
How Do I Recover Server After Power and HD Failure?
-
- Posts: 3
- Joined: 2015/08/22 14:14:45
Re: How Do I Recover Server After Power and HD Failure?
Are you sure that you removed the failed drive and not one of the good drives?
-
- Posts: 3
- Joined: 2015/08/22 14:14:45
Re: How Do I Recover Server After Power and HD Failure?
Yes, I am sure that the failed drive was correctly replaced. But I think that I have also found my error. In locating the failed drive, I was a bit of an urban cowboy and sequentially unplugged each drive. In doing so, I think that caused each drive to drop out of the raid. This afternoon we are going to rebuild the array and hoping that it will all be fine since in reality there was still only one failed drive.
-
- Posts: 3
- Joined: 2015/08/22 14:14:45
Re: How Do I Recover Server After Power and HD Failure?
Good news is the raid is now rebuilding! Bad news is I can't even begin to explain everything that we had to do. (The "we" is a neighbor and myself.)
-
- Posts: 1
- Joined: 2015/08/22 22:06:32
Re: How Do I Recover Server After Power and HD Failure?
Hi not-defeated,not-defeated wrote:Yesterday morning we had a power disruption, I rebooted the webserver and discovered that one of the drives failed in the process. The server did boot up thanks to RAID5, but when I powered it down again and rebooted we only get the error, "Kernel panicnot syncing" error message. Now I am at a loss! (But learning!)
BTW the webserver is running Centos 6.5 (in fact we have the exact same server as described in the tutorial "The Perfect Server").
I have been able to use the Centos 6.5 Live DVD to get access to the server and can see that all of the directories are still there. (Relief) But still at a loss as to what to do........
This is what I have done so far:
1) I replaced the failed hard drive with a new one but not sure what it is named in the raid i.e. /dev/sdb1 /dev/sdc1 /dev/sdd1 or /dev/sde1 (...I am running 5 disks on a software raid5). Therefore we have not copied any partitions over from another disk or gone any further.
2) Upon booting we escaped out of the boot screen and tried appending the kernel arguments by adding selinux=0 enforcing=0 but still get the "Kernel panicnot syncing" error on boot.
3) After getting the "Kernel panicnot syncing" error on boot, I have booted the system with the Centos 6.5 install disk and gone into the rescue Centos system option but just get a message that there are no Linux partitions even though the other disks are okay. Then it drops me out to a shell.
I managed to have a look at the boot log, in summary the only error that we can find is:
mdadm failed to start array /dev/md/array-name
input/output error
Logical Volume Management disabled at boot
Checking Filesystems OK
Mounting Filesystems OK
.......................everything else seems okay.
Anyone have any ideas as to how to get this server up again? Thanks in advance for taking the time to help....
I have the same problem and I've searched on the web to find the solution. I noticed you have posted this question on several websites without receiving any answer solving your problem (ex: http://serverfault.com/questions/715997 ... hd-failure). Just here you claim you could get your server up.
Do you think I should remove the failed drive with a new one, or this didn't help you?
I don't want to post a new topic and hope you can help me on this page.
Thank you in advance