Sysroot corruption following clean reboot

[rek3]

Facing a very strange issue with a CentOS 7.2 server where upon a normal, clean reboot the host is unable to mount /syslinux and boots to recovery mode only. Attempted to run a xfs_repair with -L on this but it just hangs; same story when attempting via bootable recovery media. Not sure how this happened and how I can recovery this system. The host's disk controller is reporting all disks and RAID 1 configuration are normal and (as far as I know) there were absolutely no changes or updates installed to the system prior to the reboot. Any ideas on how this happened or other methods on how to recover?

[TrevorH]

First, CentOS 7.2 is over 3 years old and should not be uised. The current version is 7.5 and 7.6 is being built as I type. Run yum update to get up to date ASAP.

Next, /syslinux? Are you sure? CentOS 7 uses grub2 not syslinux. Are you sure you don't mean selinux?

[rek3]

Technically CentOS 7 Core

Code: Select all

:/etc# cat os-release
NAME="dracut"
VERSION="7 (Core) dracut-033-359.el7"
ID=dracut
VERSION_ID=033-359.el7
PRETTY_NAME="CentOS Linux 7 (Core) dracut-033-359.el7 (Initramfs)"
ANSI_COLOR="0;34"

And yes I mean sysroot. Gets hung at "A start job is running for /sysroot" at startup and eventually enters recovery mode.

[TrevorH]

However you try to say it, you're still way out of date. I don't know how you managed to get the dracut version into the VERSION= string in os-release but the current dracut is dracut-033-535.el7_5.1.x86_64 and your strings say 033-359.

Anyhow, what you're seeing with a start job running for /sysroot most likely means your initramfs doesn't contain the correct driver for the hard disk controller that is being used for your root partition. Does one of the other kernels on the grub menu boot instead? If it does - most likely the rescue one will - then run `dracut -f --kver $version` where $version is the latest kernel that you have installed - you can see what it wants by looking at the directory names under /lib/modules/. Don't run dracut without getting the --kver right or you might overwrite the working initramfs used by the rescue kernel.

[rek3]

There is only the one kernel version shown in the boot menu (the normal OS and the recovery mode) both of which kick off the sysroot start job...

sysroot.JPG (32.28 KiB) Viewed 1676 times

Also 'dracut' is not available to run in the recovery mode so no good there. I did finally manage to grab the rdsosreport and I'm combing through this but there doesn't seem to be anything too useful here. (See attached.) Any other feedback or suggestions would be much appreciated.