[SOLVED] EXT3-fs: unable to read superblock

cen201 · Post by **cen201** » 2010/09/27 23:39:03

Hi,

I am unable to boot my server and it is erroring with the message "Kernel panic - not syncing: Attempted to kill init!". Here are the details:

I have a brand new Dell PowerEdge T110 server with Quad-Core Xeon CPU and 2GB 1333MHz RAM. I have installed a hard disk (SATA, 7.2rpm) on this machine that I removed from another Dell PowerEdge T100 server with Dual-Core Xeon CPU that was running software RAID 1.

Now when I boot the new T110 server, it displays the following error messages:

Creating root device.
Mounting root filesystem.
EXT3-fs: unable to read superblock
mount: error mounting /dev/root on /sysroot as ext3: Invalid argument
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

I am running CentOs v5.5 x86-64. If I put the hard disk back into T100, it boots fine. I do not know why it is not working with T110 server. Please help!!!

pschaff · Post by **pschaff** » 2010/09/30 20:20:29

I expect it boots fine when in the array because the other member is being used for booting. Are you sure the RAID is functioning properly and is not degraded? What does [code]cat /proc/mdstat[/code]show on the T100?

Another possibility is that the hardware is different enough that the initrd needs to be updated. Can you boot the T110 from installation media in rescue mode and mount the installed system?

majun · Post by **majun** » 2010/09/30 21:23:28

Funny... I've just had to deal with the exact same issue. Took one disk out of a server, popped it into another server and had the same error message. In my case the md devices flip-flopped. On the old server md0 was swap and md1 was root, on the new one it was the other way around. Modifying the kernel line in grub and changing the root=/dev/md1 to root=/dev/md0 solved that problem temporarily and the server booted. It drives you mad when things like this happen for apparantly no apparent reason so I did what I always do: I started digging around. I found the culprit a few hours later: openSUSE 10.2. The server was about 300km away from me so I asked the local IT guy to pop in a linux CD to boot from in case all hell breaks loose - all he had was openSUSE 10.2, so here's what happened: CentOS uses Superblock v0.9 which does NOT carry information about the md enumeration. openSUSE 10.2 uses Superblock v1.0.x which DOES carry information about the md enumeration under Name, e.g. "linux:0" aka md0 or "linux:1" aka md1. Accordingly, openSUSE modified the superblock upon assembly on both partitions, adding "linux:0" to sda1 (which belonged to md1) and "linux:1" to sda2 (which belonged to md0). Once CentOS found that information it assembled the devices according to the new superblock enumeration and voila - total chaos ensued.

In the end I had to clone the server again and it's now back to normal. It drove me crazy though.

cen201 · Post by **cen201** » 2010/09/30 22:04:32

RAID was working fine on T100 and #cat /proc/mdstat was fine too. Because if you break any hard disk on T100 and reboot the server, it was booting fine.

Can you boot the T110 from installation media in rescue mode and mount the installed system?
The answer is Yes. I could even edit /etc/fstab and /boot/grub/grub.conf because I tried different options to make it work but no luck so far.

Another possibility is that the hardware is different enough that the initrd needs to be updated.
This is most likely the case. I have even installed the most recent kernel but it did not help.

Here is the output of /etc/fstab, grub.conf, and fdisk:
[code]
$cat /etc/fstab
/dev/md3 / ext3 defaults 1 1
/dev/md4 /var ext3 defaults 1 2
/dev/md2 /home ext3 defaults 1 2
/dev/md1 /usr ext3 defaults 1 2
/dev/md0 /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sdb5 swap swap defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0

$cal /boot/grub/grub.conf
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora (2.6.18-194.11.3.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.11.3.el5 ro root=/dev/md3
initrd /initrd-2.6.18-194.11.3.el5.img

#fdisk -l
Disk /dev/sda: 250.0 GB, 250000000000 bytes
255 heads, 63 sectors/track, 30394 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 127 1020096 fd Linux raid autodetect
/dev/sda2 128 12875 102398310 fd Linux raid autodetect
/dev/sda3 12876 15425 20482875 fd Linux raid autodetect
/dev/sda4 15426 19452 32346877+ 5 Extended
/dev/sda5 15426 15935 4096543+ 82 Linux swap /Solaris
/dev/sda6 15936 16445 4096543+ fd Linux raid autodetect
/dev/sda7 16446 16955 4096543+ fd Linux raid autodetect[/code][Moderator edit: Added [i]code[/i] tags to preserve formatting.]

pschaff · Post by **pschaff** » 2010/09/30 23:03:49

I'd try booting from installation media in rescue mode and running mkinitrd. Try a forum search for "rescue mkinitrd" for examples such as
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=23319&forum=44&post_id=91422#forumpost91422

cen201 · Post by **cen201** » 2010/10/08 22:02:49

I tried all the suggestions that were posted so far. But no luck. Finally, I decided to do a fresh install on Cent Os on T110 server.

AlanBartlett · Post by **AlanBartlett** » 2010/10/08 23:17:26

Thank you for reporting back.

For posterity, this thread is marked [SOLVED].

CentOS