Upgrading 5.7 to 3.x.x kernel will not boot with root on lvm
Posted: 2012/02/02 10:48:43
I am trying to update our standard build running Centos 5.4 to use a new 3.x.x series kernel. I need this kernel in order to support some new hardware PCIe devices. I have updated the system to use 5.7, but still get the following problem.
root is mounted on an LVM (output from getinfo.sh disk at the end of this post).
When I boot using any of the older kernels (vmlinuz-2.6.18-164.el5PAE Centos 5.4, vmlinuz-2.6.18-274.17.1.el5PAE Centos 5.7 or vmlinuz-2.6.28.tps.smp.9 which is my standard production kernel) th dm subsystem finds /dev/VolGroup00/LogVol01 and can boot the system fine.
When running a 3.x.x kernel (3.0.0-rc7, 3.1.2 and 3.3.3-stable) all fail to find any logical volumes and hence fail to find the root partition and then it fails with the standard panic message.
I have attached a (poor) screen shot of the kernel messages when trying to boot the 3.2.2 kernel, the key I think is the 'No Volume Groups Found'. Normally this reports /dev/VolGroup00/LogVol01' has been found.
I note that other posts on the web have reported this problem, but they seem to end up rebuilding their system without root on an LVM. With several hundred deployed systems this is not an option.
The 3.x.x kernels all have the correct dm- modules which are loaded correctly during the boot process.
I have also tried rebuilding mdadm tools form source.
Any suggestions of what to try next?
Information for disk problems.
[code]
== BEGIN uname -rmi ==
2.6.28.tps.smp.9 i686 i386
== END uname -rmi ==
== BEGIN rpm -qa \*-release\* ==
centos-release-notes-5.7-0
centos-release-5-7.el5.centos
== END rpm -qa \*-release\* ==
== BEGIN cat /etc/redhat-release ==
CentOS release 5.7 (Final)
== END cat /etc/redhat-release ==
== BEGIN getenforce ==
Disabled
== END getenforce ==
== BEGIN free -m ==
total used free shared buffers cached
Mem: 24364 670 23694 0 31 406
-/+ buffers/cache: 231 24132
Swap: 1023 0 1023
== END free -m ==
== BEGIN cat /etc/fstab ==
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
== END cat /etc/fstab ==
== BEGIN df -h ==
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
270G 36G 221G 14% /
/dev/cciss/c0d0p1 99M 41M 53M 44% /boot
tmpfs 12G 0 12G 0% /dev/shm
== END df -h ==
== BEGIN fdisk -l ==
Disk /dev/cciss/c0d0: 299.9 GB, 299966445568 bytes
255 heads, 63 sectors/track, 36468 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 13 104391 83 Linux
/dev/cciss/c0d0p2 14 36468 292824787+ 8e Linux LVM
Disk /dev/cciss/c0d1: 1200.1 GB, 1200186941440 bytes
255 heads, 63 sectors/track, 145914 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/cciss/c0d1p1 * 1 145914 1172054173+ 83 Linux
== END fdisk -l ==
root is mounted on an LVM (output from getinfo.sh disk at the end of this post).
When I boot using any of the older kernels (vmlinuz-2.6.18-164.el5PAE Centos 5.4, vmlinuz-2.6.18-274.17.1.el5PAE Centos 5.7 or vmlinuz-2.6.28.tps.smp.9 which is my standard production kernel) th dm subsystem finds /dev/VolGroup00/LogVol01 and can boot the system fine.
When running a 3.x.x kernel (3.0.0-rc7, 3.1.2 and 3.3.3-stable) all fail to find any logical volumes and hence fail to find the root partition and then it fails with the standard panic message.
I have attached a (poor) screen shot of the kernel messages when trying to boot the 3.2.2 kernel, the key I think is the 'No Volume Groups Found'. Normally this reports /dev/VolGroup00/LogVol01' has been found.
I note that other posts on the web have reported this problem, but they seem to end up rebuilding their system without root on an LVM. With several hundred deployed systems this is not an option.
The 3.x.x kernels all have the correct dm- modules which are loaded correctly during the boot process.
I have also tried rebuilding mdadm tools form source.
Any suggestions of what to try next?
Information for disk problems.
[code]
== BEGIN uname -rmi ==
2.6.28.tps.smp.9 i686 i386
== END uname -rmi ==
== BEGIN rpm -qa \*-release\* ==
centos-release-notes-5.7-0
centos-release-5-7.el5.centos
== END rpm -qa \*-release\* ==
== BEGIN cat /etc/redhat-release ==
CentOS release 5.7 (Final)
== END cat /etc/redhat-release ==
== BEGIN getenforce ==
Disabled
== END getenforce ==
== BEGIN free -m ==
total used free shared buffers cached
Mem: 24364 670 23694 0 31 406
-/+ buffers/cache: 231 24132
Swap: 1023 0 1023
== END free -m ==
== BEGIN cat /etc/fstab ==
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
== END cat /etc/fstab ==
== BEGIN df -h ==
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
270G 36G 221G 14% /
/dev/cciss/c0d0p1 99M 41M 53M 44% /boot
tmpfs 12G 0 12G 0% /dev/shm
== END df -h ==
== BEGIN fdisk -l ==
Disk /dev/cciss/c0d0: 299.9 GB, 299966445568 bytes
255 heads, 63 sectors/track, 36468 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 13 104391 83 Linux
/dev/cciss/c0d0p2 14 36468 292824787+ 8e Linux LVM
Disk /dev/cciss/c0d1: 1200.1 GB, 1200186941440 bytes
255 heads, 63 sectors/track, 145914 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/cciss/c0d1p1 * 1 145914 1172054173+ 83 Linux
== END fdisk -l ==