Dracut Shell after kernel upgrade

[thenameskam]

Story goes,

I have a set of servers i built then i have a set of servers that were here before me.

Same release level of Centos 7
Same bios
same Config for the most part.

When i updated the kernel on a server i built, i have no issue. it boots up fine.

When i update the kernel on a server that was built before mine, i get into the Dracut shell with the below errors. if i switch back to the old kernel, it boots up fine as well.

dracut-initqueue[565]: Warning: could not boot.
dracut-initqueue[565]: Warning: /dev/centos/root does not exist
dracut-initqueue[565]: Warning: /dev/mapper/centos-root does not exist

starting Dracut Emergency Shell...
Warning: /dev/centos/root does not exist
Warning: /dev/mapper/centos-root does not exist

generating "/run/initramfs/rdsosreport.txt"

Enetering emergency mode. Exit the shell to continue
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report.

dracut:/#

- ive tried rebuilding the initramfs for the new kernel, same result.
-tried running lvm vgscan and i get nothing returned besides the message stating it may take awhile to scan then it instantly moves down to the next line of "dracut:/#" while returning no results.
-ive compared the two lvm.conf files to see if there was any difference at all and i could not locate any.

anyone have any other ideas of what to try or look into?

something else to note, when i ran the dracut -f to rebuild the initramfs i got the below error.

Kernel version initramfs-3.10.0-862.11.6.el7.x86_64.img has no module directory /lib/modules/initramfs-3.10.0-862.11.6.el7.x86_64.img

but when i navigate to that directory i see files in there all named modules.xxx.

[TrevorH]

dracut wants just the kernel version number not the entire filename 'initramfs-3.10.0-862.11.6.el7.x86_64.img' so '3.10.0-862.11.6.el7.x86_64' instead.

You'd really want to run dracut -f /boot/initramfs-3.10.0-862.11.6.el7.x86_64.img 3.10.0-862.11.6.el7.x86_64

[thenameskam]

dracut wants just the kernel version number not the entire filename 'initramfs-3.10.0-862.11.6.el7.x86_64.img' so '3.10.0-862.11.6.el7.x86_64' instead.

You'd really want to run dracut -f /boot/initramfs-3.10.0-862.11.6.el7.x86_64.img 3.10.0-862.11.6.el7.x86_64

Good catch, when I removed the .img file i no longer received the lib/module error. But once i rebooted and selected the newer 11.6 Kernel i still received the same dracut shell with the same errors.

[TrevorH]

Post the output from ls -la /boot here.

[thenameskam]

Post the output from ls -la /boot here.

request is below.

dr-xr-xr-x. 5 root root 4096 Sep 17 17:51 .
dr-xr-xr-x. 19 root root 4096 Sep 17 17:26 ..
-rw-------. 1 root root 21508902 Sep 10 19:00 3.10.0-862.11.6.el7.x86_64.img
-rw-r--r--. 1 root root 126431 Aug 18 2016 config-3.10.0-327.28.3.el7.x86_64
-rw-r--r--. 1 root root 147859 Aug 14 22:02 config-3.10.0-862.11.6.el7.x86_64
-rw-r--r--. 1 root root 147823 May 21 23:50 config-3.10.0-862.3.2.el7.x86_64
drwxr-xr-x. 3 root root 16 Sep 14 2017 efi
drwxr-xr-x. 2 root root 42 Sep 26 2016 grub
drwx------. 5 root root 4096 Sep 17 17:51 grub2
-rw-r--r--. 1 root root 43577777 Sep 26 2016 initramfs-0-rescue-d915dc9e7bbd43a2a513c54652e73a24.img
-rw-------. 1 root root 21474520 Sep 14 07:37 initramfs-3.10.0-327.28.3.el7.x86_64.img
-rw-------. 1 root root 21982320 Sep 17 17:44 initramfs-3.10.0-862.11.6.el7.x86_64.img
-rw-------. 1 root root 21983227 Sep 17 17:51 initramfs-3.10.0-862.3.2.el7.x86_64.img
-rw-r--r--. 1 root root 611308 Sep 14 2017 initrd-plymouth.img
-rw-r--r--. 1 root root 252632 Aug 18 2016 symvers-3.10.0-327.28.3.el7.x86_64.gz
-rw-r--r--. 1 root root 305158 Aug 14 22:05 symvers-3.10.0-862.11.6.el7.x86_64.gz
-rw-r--r--. 1 root root 304943 May 21 23:52 symvers-3.10.0-862.3.2.el7.x86_64.gz
-rw-------. 1 root root 2964731 Aug 18 2016 System.map-3.10.0-327.28.3.el7.x86_64
-rw-------. 1 root root 3414344 Aug 14 22:02 System.map-3.10.0-862.11.6.el7.x86_64
-rw-------. 1 root root 3409912 May 21 23:50 System.map-3.10.0-862.3.2.el7.x86_64
-rwxr-xr-x. 1 root root 5156528 Sep 26 2016 vmlinuz-0-rescue-d915dc9e7bbd43a2a513c54652e73a24
-rwxr-xr-x. 1 root root 5159792 Aug 18 2016 vmlinuz-3.10.0-327.28.3.el7.x86_64
-rw-r--r--. 1 root root 171 Aug 18 2016 .vmlinuz-3.10.0-327.28.3.el7.x86_64.hmac
-rwxr-xr-x. 1 root root 6242208 Aug 14 22:02 vmlinuz-3.10.0-862.11.6.el7.x86_64
-rw-r--r--. 1 root root 171 Aug 14 22:02 .vmlinuz-3.10.0-862.11.6.el7.x86_64.hmac
-rwxr-xr-x. 1 root root 6228832 May 21 23:50 vmlinuz-3.10.0-862.3.2.el7.x86_64
-rw-r--r--. 1 root root 170 May 21 23:50 .vmlinuz-3.10.0-862.3.2.el7.x86_64.hmac

[thenameskam]

3.10.0-327.28.3.el7 is the only kernel version i can get it to boot up with and see the root partitions.

[TrevorH]

I would suggest using lsinitrd to make a list of the contents of both working and non-working initramfs files then somehow diff the names of the files present and see what is in the one that boots that is missing from the one that does not.

[thenameskam]

I would suggest using lsinitrd to make a list of the contents of both working and non-working initramfs files then somehow diff the names of the files present and see what is in the one that boots that is missing from the one that does not.

after poking around in excel for a bit, here is what i came up with after running the lsinitrd against both kernel versions. Under each Kernel version, is a list of files that the other kernel version does not have. Obviously 862 has more than 327 kernel.

3.10.0-327.28.3.el7
/weak-updates/megasr/megasr.ko
/weak-updates/megasr
/weak-updates
/extra/megasr/megasr.ko
/extra/megasr
/extra

3.10.0-862.11.6.el7
/kernel/net/core/devlink.ko
/kernel/net/core
/kernel/net
/kernel/drivers/video/fb_sys_fops.ko
/kernel/drivers/nvme/target/nvmet-fc.ko
/kernel/drivers/nvme/target/nvmet.ko
/kernel/drivers/nvme/target
/kernel/drivers/nvme/host/nvme-rdma.ko
/kernel/drivers/nvme/host/nvme-fc.ko
/kernel/drivers/nvme/host/nvme-fabrics.ko
/kernel/drivers/nvme/host/nvme-core.ko
/kernel/drivers/nvme/host
/kernel/drivers/nvme
/kernel/drivers/net/fjes/fjes.ko
/kernel/drivers/net/fjes
/kernel/drivers/net/ethernet/qlogic/qed/qed.ko
/kernel/drivers/net/ethernet/qlogic/qed
/kernel/drivers/net/ethernet/qlogic
/kernel/drivers/net/ethernet/mellanox/mlxsw/mlxsw_pci
/kernel/drivers/net/ethernet/mellanox/mlxsw/mlxsw_i2c
/kernel/drivers/net/ethernet/mellanox/mlxsw/mlxsw_cor
/kernel/drivers/net/ethernet/mellanox/mlxsw
/kernel/drivers/net/ethernet/mellanox/mlxfw/mlxfw.ko.
/kernel/drivers/net/ethernet/mellanox/mlxfw
/kernel/drivers/net/ethernet/mellanox
/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.
/kernel/drivers/net/ethernet/chelsio/libcxgb
/kernel/drivers/net/ethernet/chelsio
/kernel/drivers/iommu/amd_iommu_v2.ko
/kernel/drivers/iommu
/kernel/drivers/infiniband/core/rdma_cm.ko
/kernel/drivers/infiniband/core/iw_cm.ko
/kernel/drivers/infiniband/core/ib_core.ko
/kernel/drivers/infiniband/core/ib_cm.ko
/kernel/drivers/infiniband/core
/kernel/drivers/infiniband
/kernel/drivers/gpu/drm/amd/amdkfd/amdkfd.ko
/kernel/drivers/gpu/drm/amd/amdkfd
/kernel/drivers/gpu/drm/amd

[TrevorH]

That megasr.ko listed in the working kernel is most likely the driver for your RAID controller. No idea where that came from as I checked using yum provides '*/megasr.ko*' to see what package might provide it nad my list comes up empty. Do you have any output from rpm -qa kmod\* that might show an install kmod driver package that supplies it?

[thenameskam]

That megasr.ko listed in the working kernel is most likely the driver for your RAID controller. No idea where that came from as I checked using yum provides '*/megasr.ko*' to see what package might provide it nad my list comes up empty. Do you have any output from rpm -qa kmod\* that might show an install kmod driver package that supplies it?

as requested, the output is below.

rpm -qa kmod\*
kmod-20-21.el7.x86_64
kmod-megasr-2.2.3.rhtest60s10-1.el7.x86_64
kmod-libs-20-21.el7.x86_64

below is the output from the server i built that allows me to upgrade the kernel.

rpm -qa kmod\*
kmod-libs-20-15.el7_4.6.x86_64
kmod-20-15.el7_4.6.x86_64