[SOLVED] Problems with RAID 1 GRUB setup

[dejanst]

I recently upgraded from CentOS 5.7 to CentOS 6 (32 bit) and I have some issues with setting up GRUB on both HDDs.

The server is HP Proliant Microserver that is running 2 HDD's that have the same capacity. Both HDDs have the same partition layout, the partitions are properly combined in 8 RAID devices.

[code]

== BEGIN df -h ==
Filesystem Size Used Avail Use% Mounted on
/dev/md2 9.7G 349M 8.8G 4% /
tmpfs 440M 0 440M 0% /dev/shm
/dev/md0 291M 38M 239M 14% /boot
/dev/md7 73G 180M 69G 1% /komna
/dev/md6 49G 180M 46G 1% /telos
/dev/md5 2.0G 35M 1.8G 2% /tmp
/dev/md3 7.7G 883M 6.5G 12% /usr
/dev/md4 4.9G 215M 4.4G 5% /var

== BEGIN fdisk -l ==
Disk /dev/md0 doesn't contain a valid partition table
Disk /dev/md1 doesn't contain a valid partition table
Disk /dev/md2 doesn't contain a valid partition table
Disk /dev/md3 doesn't contain a valid partition table
Disk /dev/md4 doesn't contain a valid partition table
Disk /dev/md5 doesn't contain a valid partition table
Disk /dev/md6 doesn't contain a valid partition table

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000f15fe

Device Boot Start End Blocks Id System
/dev/sda1 1 1275 10240000 fd Linux raid autodetect
/dev/sda2 1275 1530 2048000 fd Linux raid autodetect
/dev/sda3 * 1530 1569 307200 fd Linux raid autodetect
/dev/sda4 1569 19458 143693824 5 Extended
/dev/sda5 1569 7943 51200000 fd Linux raid autodetect
/dev/sda6 7943 8963 8192000 fd Linux raid autodetect
/dev/sda7 8963 9600 5120000 fd Linux raid autodetect
/dev/sda8 9601 9856 2048000 fd Linux raid autodetect
/dev/sda9 9856 19458 77128704 fd Linux raid autodetect

Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003b78a

Device Boot Start End Blocks Id System
/dev/sdb1 1 1275 10240000 fd Linux raid autodetect
/dev/sdb2 1275 1530 2048000 fd Linux raid autodetect
/dev/sdb3 * 1530 1569 307200 fd Linux raid autodetect
/dev/sdb4 1569 19458 143693824 5 Extended
/dev/sdb5 1569 7943 51200000 fd Linux raid autodetect
/dev/sdb6 7943 8963 8192000 fd Linux raid autodetect
/dev/sdb7 8963 9600 5120000 fd Linux raid autodetect
/dev/sdb8 9601 9856 2048000 fd Linux raid autodetect
/dev/sdb9 9856 19458 77128704 fd Linux raid autodetect

Disk /dev/md0: 314 MB, 314560512 bytes
2 heads, 4 sectors/track, 76797 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md1: 2096 MB, 2096095232 bytes
2 heads, 4 sectors/track, 511742 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md2: 10.5 GB, 10484641792 bytes
2 heads, 4 sectors/track, 2559727 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md3: 8387 MB, 8387551232 bytes
2 heads, 4 sectors/track, 2047742 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md4: 5241 MB, 5241823232 bytes
2 heads, 4 sectors/track, 1279742 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md5: 2096 MB, 2096095232 bytes
2 heads, 4 sectors/track, 511742 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md6: 52.4 GB, 52427681792 bytes
2 heads, 4 sectors/track, 12799727 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md7: 79.0 GB, 78978674688 bytes
2 heads, 4 sectors/track, 19281903 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / Disk /dev/md7 doesn't contain a valid partition table
512 bytes
Disk identifier: 0x00000000

== END fdisk -l ==

== BEGIN blkid ==
/dev/sdb1: UUID="0b26d410-68bf-3813-7439-722b9944c54a" UUID_SUB="b4f29272-c47f-a723-0927-8c2d671b8a0d" LABEL="localhost.localdomain:2" TYPE="linux_raid_member"
/dev/sdb2: UUID="2c222a2b-0319-14d6-ff37-e9de8d67b86d" UUID_SUB="ea656011-3919-0b97-12ca-a5d931225e4b" LABEL="localhost.localdomain:1" TYPE="linux_raid_member"
/dev/sdb3: UUID="038f3aa1-3903-75fb-711f-ce7eb3f93b4a" UUID_SUB="6747df7a-cd6f-c6d0-f594-ddd5c0b60e8e" LABEL="localhost.localdomain:0" TYPE="linux_raid_member"
/dev/sdb5: UUID="c1366613-ace9-499a-698d-9f0d70ba42c0" UUID_SUB="f0ae8820-880e-b096-8971-16bd335b0599" LABEL="localhost.localdomain:6" TYPE="linux_raid_member"
/dev/sdb6: UUID="ab331e81-c714-593c-c407-3f3e22088f8b" UUID_SUB="fc3a348a-9243-9a10-580d-9669f771e9c2" LABEL="localhost.localdomain:3" TYPE="linux_raid_member"
/dev/sdb7: UUID="e6ef64c5-bc3b-dcf9-80dd-1dd292102140" UUID_SUB="be72b5f3-5210-6c6a-8078-4d38207423f5" LABEL="localhost.localdomain:4" TYPE="linux_raid_member"
/dev/sdb8: UUID="a5bf1ee9-904d-7510-9d41-34d86e5d2420" UUID_SUB="31f0aa42-4290-2dd7-9887-28621a205855" LABEL="localhost.localdomain:5" TYPE="linux_raid_member"
/dev/sdb9: UUID="309aafb0-e7fb-05b2-1b68-ef692ae37a2a" UUID_SUB="2ba852f3-156b-5633-1896-a6160f6cfea5" LABEL="localhost.localdomain:7" TYPE="linux_raid_member"
/dev/sda1: UUID="0b26d410-68bf-3813-7439-722b9944c54a" UUID_SUB="e7a528fe-d81b-1880-602f-78c1d0cb96dd" LABEL="localhost.localdomain:2" TYPE="linux_raid_member"
/dev/sda2: UUID="2c222a2b-0319-14d6-ff37-e9de8d67b86d" UUID_SUB="ae524eb7-2625-748c-acd9-2897d932fbeb" LABEL="localhost.localdomain:1" TYPE="linux_raid_member"
/dev/sda3: UUID="038f3aa1-3903-75fb-711f-ce7eb3f93b4a" UUID_SUB="4bfaeff8-6e21-2015-c3a2-9d216f8e46ed" LABEL="localhost.localdomain:0" TYPE="linux_raid_member"
/dev/sda5: UUID="c1366613-ace9-499a-698d-9f0d70ba42c0" UUID_SUB="7db77c08-0ccd-9ddf-a548-8ee9d963c16d" LABEL="localhost.localdomain:6" TYPE="linux_raid_member"
/dev/sda6: UUID="ab331e81-c714-593c-c407-3f3e22088f8b" UUID_SUB="29131f4a-9f8e-7769-86df-8613048bd747" LABEL="localhost.localdomain:3" TYPE="linux_raid_member"
/dev/sda7: UUID="e6ef64c5-bc3b-dcf9-80dd-1dd292102140" UUID_SUB="fd4d1009-468f-14db-9b15-7527a3d44849" LABEL="localhost.localdomain:4" TYPE="linux_raid_member"
/dev/sda8: UUID="a5bf1ee9-904d-7510-9d41-34d86e5d2420" UUID_SUB="445be3e3-bd74-b70d-d06d-3a35b76c8661" LABEL="localhost.localdomain:5" TYPE="linux_raid_member"
/dev/sda9: UUID="309aafb0-e7fb-05b2-1b68-ef692ae37a2a" UUID_SUB="85b5ac0c-5992-cc94-186a-8e03b5d9a17f" LABEL="localhost.localdomain:7" TYPE="linux_raid_member"
/dev/md0: UUID="72cf5870-d603-4d24-899a-976a4290b7f5" TYPE="ext2"
/dev/md1: UUID="e19e1ca8-ea57-44d0-99ca-c116b87b6b29" TYPE="swap"
/dev/md2: UUID="e7959d18-e82b-46a9-a8d4-0ebdf315869e" TYPE="ext4"
/dev/md3: UUID="ccbba718-54ee-437c-9d0d-3ab650c6b2ba" TYPE="ext4"
/dev/md4: UUID="3b2359e1-dc0a-491c-887e-91900e696d5a" TYPE="ext4"
/dev/md5: UUID="4c02597d-cb5b-4620-b55a-3d4c22e87e53" TYPE="ext4"
/dev/md6: UUID="43e63578-26fb-4019-bf46-cbbb05817d6e" TYPE="ext4"
/dev/md7: UUID="439ddd09-b01a-4144-b7f2-7f6ac4f186b6" TYPE="ext4"
== END blkid ==

== BEGIN cat /proc/mdstat ==
Personalities : [raid1]
md7 : active raid1 sda9[0] sdb9[1]
77127612 blocks super 1.1 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md6 : active raid1 sda5[0] sdb5[1]
51198908 blocks super 1.1 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md5 : active raid1 sda8[0] sdb8[1]
2046968 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md4 : active raid1 sda7[0] sdb7[1]
5118968 blocks super 1.1 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md3 : active raid1 sda6[0] sdb6[1]
8190968 blocks super 1.1 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md2 : active raid1 sda1[0] sdb1[1]
10238908 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md1 : active raid1 sda2[0] sdb2[1]
2046968 blocks super 1.1 [2/2] [UU]

md0 : active raid1 sda3[0] sdb3[1]
307188 blocks super 1.0 [2/2] [UU]

unused devices: <none>
== END cat /proc/mdstat ==

[/code]

I set up the RAID array during the installation procedure and everything is OK.

The second step for me was to make both members of the RAID1 /boot bootable. For this I followed this tutorial that works perfectly on CentOS 5.x:

http://wiki.centos.org/HowTos/SoftwareRAIDonCentOS5

But saddly this is not the case if I try to do the same thing on CentOS 6.

If I try the method where I set up the GRUB during boot (pres c when the GRUB splashchreen appers), I get the following error:

[code]
grub> root (hd0,0)
grub> setup (hd0)
Error 17: Cannot mount selected partition

grub> root (hd1,0)
grub> setup (hd1)
Error 17: Cannot mount selected partition
[/code]

If I try the method when I set up GRUB from a running machine, I get the following error:

[code]
grub> device (hd0) /dev/sda
device (hd0) /dev/sda
grub> device (hd1) /dev/sdb
device (hd1) /dev/sdb

grub> root (hd0,0)
root (hd0,0)
Filesystem type unknown, partition type 0xfd

grub> setup (hd0)
setup (hd0)
Error 1 : Cannot mount selected partition

grub> root (hd1,0)
root (hd1,0)
Filesystem type unknown, partition type 0xfd

grub> setup (hd1)
setup (hd1)
Error 1 : Cannot mount selected partition

grub> quit
quit
[/code]

I would appreciate a lot if someone takes the time to help me out with this problem. A big thanks in advance.

[dejanst]

If this is a bug that can not be fixed eazily - does anybody know of a different way to get the both members of the RAID1 /boot bootable. Because that is the main thing that is bothering me at the moment.

[WhatsHisName]

It would help to associate root with the correct partitions.

[quote]== BEGIN df -h ==
Filesystem Size Used Avail Use% Mounted on
...
[b]/dev/md0[/b] 291M 38M 239M 14% [b]/boot[/b]
...

== BEGIN cat /proc/mdstat ==
...
[b]md0[/b] : active raid1 [b]sda3[/b][0] [b]sdb3[/b][1]
307188 blocks super 1.0 [2/2] [UU]
...[/quote]
The assigned root should be [b](hd0,2)[/b] and [b](hd1,2)[/b], not (hd0,0) and (hd1,0).


You can also "find" the correct partitions. With separate /boot partitions:

[code]# grub

grub> find /grub/grub.conf
find /grub/grub.conf
(hd0,2)
(hd1,2)

grub> quit[/code]

[dejanst]

WhatsHisName - thank you very much, you were spot on.

It looks like the CentOS 6 installer uses a bit different md device layout than CentOS 5 and that caused a problem for a user that does not know much about GRUB and follows the online tutorials blindly ;)

Thank you very much for all the help. The issue is resolved.

[WhatsHisName]

You are welcome.

The howto should probably include the use of "grub> find ...", but even that will lead to some confusion. You really should do searches for both /boot/grub/grub.conf and /grub/grub.conf (no separete /boot vs. separate /boot partition, respectively) to cover all the bases.

[code]# grub

grub> find /boot/grub/grub.conf
find /boot/grub/grub.conf
(hd0,4)
(hd1,4)

grub> find /grub/grub.conf
find /grub/grub.conf
(hd0,5)
(hd0,6)
(hd0,7)
(hd1,5)
(hd1,6)
(hd1,7)

grub> quit[/code]
...for a multiboot system.

You can "find" almost anything present. I think most howtos "find" stage1 or stage2 instead of grub.conf, which is somewhat unique to rpm-based systems (i.e., Debian/Ubuntu have menu.lst (old) or grub.cfg (new)).

[AlanBartlett]

Thank you for reporting back.

For posterity (and on your behalf), this thread is marked [SOLVED].