Grub error after update from 7.5 to 7.6

General support questions
Post Reply
username
Posts: 76
Joined: 2013/03/04 13:17:23

Grub error after update from 7.5 to 7.6

Post by username » 2018/12/11 10:34:01

Hi,

Yesterday, I've updated a server running CentOS 7.5 to Centos 7.6. I saw there were two .rpmsave related to Grub and SELinux. After the update, the server wasn't rebooting anymore. I thought it could be a Grub error. So I tried a netboot using the custom kernel from my server provider. The server started correctly. I looked at the logs (/var/log/messages, /var/log/dsmeg, /var/log/boot) but I couldn't find anything related to the failed boot intent. On the other hand, there was an error in grubby log :

Code: Select all

4908: Mon Dec 10 13:46:11 2018: command line: --grub2 -c /boot/grub2/grub.cfg --add-kernel=/boot/vmlinuz-3.10.0-957.1.3.el7.x86_64 --copy-default --title CentOS Linux (3.10.0-957.1.3.el7.x86_64) 7 (Core) --args=root=/dev/md2  --remove-kernel=TI$
DBG: Image entry failed: uuid missing: rootdev (null), dev 23884405-b432-42a1-a47a-f34fe0a13cc6
DBG: menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-862.14.4.el7.x86_64-advanced-23884405-b432-42a1-a47a-f34fe0a13cc6' {
DBG:    load_video
DBG:    set gfxpayload=keep
DBG:    insmod gzio
DBG:    insmod part_gpt
DBG:    insmod part_gpt
DBG:    insmod diskfilter
DBG:    insmod mdraid09
DBG:    insmod ext2
DBG:    set root='mduuid/444484e5032bfce9a4d2adc226fd5302'
DBG:    if [ x$feature_platform_search_hint = xy ]; then
DBG:      search --no-floppy --fs-uuid --set=root --hint='mduuid/444484e5032bfce9a4d2adc226fd5302'  23884405-b432-42a1-a47a-f34fe0a13cc6
DBG:    else
DBG:      search --no-floppy --fs-uuid --set=root 23884405-b432-42a1-a47a-f34fe0a13cc6
DBG:    fi
DBG:    linuxefi /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=/dev/md2 ro crashkernel=auto rhgb quiet rd.md.uuid=444484e5:032bfce9:a4d2adc2:26fd5302 rootdelay=10 rootdelay=10 noquiet nosplash net.ifnames=0 biosdevname=0
DBG:    initrdefi /boot/initramfs-3.10.0-862.14.4.el7.x86_64.img
DBG: }
I tried to reinstall grub2, grubby and the new kernel. Then, I did a grub-mkconfig. There was no error during the process. Then, I rebooted again but the server failed to start.

I had another identical server to update (same hardware, same configuration). I excluded grub2* from the update and rebooted this server without any issue. I compared the 2 grub.conf from these two servers and checked in the /boot folder if there was not an initramfs missing for the new kernel. But I couldn't see anything weird.

What log files could be helpful to try solving this issue ?

EDIT : I also tried rebooting after restoring the grub.cfg.rpmsave and it didn't work. Then, I restored again the new grub.cfg but this time editing out the menu entry corresponding to the new kernel. It didn't work either. Finally, I disabled SELinux and rebooted again but without success.

EDIT 2 : I got an email from the support team that says : "The server stays frozen on startup with this message "grub rescue >"

Thanks
Last edited by username on 2018/12/13 16:08:15, edited 2 times in total.

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: Grub error after update from 7.5 to 7.6

Post by username » 2018/12/13 16:06:17

OK, I guess everything can be summerized by : "Is there a way to repair Grub on a dedicated server ?"

I don't really like the custom netboot kernel I'm using right know. Usually, they create some problems. For instance, this one doesn't support SELinux.

zorlac
Posts: 3
Joined: 2005/02/14 19:32:33
Contact:

Re: Grub error after update from 7.5 to 7.6

Post by zorlac » 2018/12/20 02:44:42

Are you using a SoYouStart/OVH server, by chance?

This happened to me to two servers and I got the exact same response from their support. The first one I gave up and just rebuilt the entire server and recovered from backups. The second one I'm determined to fix properly, since it's not a production server.

Running "yum update" and a "reboot" should NOT brick the server.

Oh, and look what else I just found:
https://forums.cpanel.net/threads/servi ... -6.642989/

I did both suggestions in rescue mode under chroot:
  • disabled selinux (it was actually already disabled)
  • grub2-mkconfig -o /boot/grub2/grub.cfg
  • grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
Make SURE you have mounted both /boot and then /boot/efi in rescue mode, as well as /proc, /sys & /dev. Check /etc/fstab for the correct /dev/ devices. For me it was /dev/sda2 and /dev/sda1, respectively

This worked for me, but the question remains - WTF does a yum update and reboot brick the server? I'm still not sure which grub.cfg it wants to use. It is in efi boot mode, but grub.cfg doesn't exist in the EFI folder on identical servers, which someone else in that cpanel thread pointed out as well. It used to reboot fine without it.

zorlac
Posts: 3
Joined: 2005/02/14 19:32:33
Contact:

Re: Grub error after update from 7.5 to 7.6

Post by zorlac » 2018/12/20 03:53:28

On another identical system, running "yum update" followed by immediately rebuilding both grub.cfg files and then rebooting fixed the issue.

desertcat
Posts: 843
Joined: 2014/08/07 02:17:29
Location: Tucson, AZ

Re: Grub error after update from 7.5 to 7.6

Post by desertcat » 2018/12/20 07:04:43

username wrote:
2018/12/13 16:06:17
OK, I guess everything can be summerized by : "Is there a way to repair Grub on a dedicated server ?"

I don't really like the custom netboot kernel I'm using right know. Usually, they create some problems. For instance, this one doesn't support SELinux.
Before saying, "You're hosed", let me weigh in. 7.6 is not as buggy as 7.5 was, but there are still bugs in it, especially IF you run KDE, have a PS/2 KVM switch, and have a PS/2 mouse. Oddly if you run GNOME you don't have this problem. Go figure. I developed a workaround so just today rolled over my workstation today. After having blown up my machines up 4 times, I knew what to expect when I rolled the machine over today for good. This workaround would be a PITA if I had to reboot the machine every day, but since the only time it gets rebooted is when a new kernel comes out or I have to shut down the machine for some other reason.

What I think *may* be wrong is that your boot file in BIOS is screwed up. This has been the case since CentOS 7 came out. Example: For reasons unknown *my* particular mobo etc.combo does not like UEFI, the default however seems to be UEFI (probably because the manufacturer figures EVERYONE is going to install Windoze). All devices however are presented as both UEFI and Non-UEFI (Legacy BIOS). In my case my BIOS is setup to run as a Legacy BIOS, NOT UEFI. Sometimes however the BIOS gets whacked and it defaults back to UEFI... and my machine freezes. IF you are running /boot/grub you should be using the Legacy BIOS, not UEFI BIOS. ...or vice versa. Go into BIOS and make sure that the booting media matches what the machine expects to see, otherwise the machine will hang. You should also pick one but not both /boot format/partition types -- either /boot partition, or /boot/efi partition, as the machine might get confused as which is the right one to refer to.

If when you go into BIOS and check the BIOS type and make sure that it is (say) Legacy BIOS and you have a /boot partition and the machine is still hanging, then you probably are hosed and it is simply easier to do a complete re-install... though *that* is a GIANT PITA than doing a simple roll over. (Trust me I am still re-configuring my test bed, but it is at least back up and running).

Alternately IF you backed up the machine simply roll the machine back to 7.5 until you can debug the problem. My guess is when you rolled over the machine your BIOS might have been re-set to the default UEFI, and the solution might be as simple going into BIOS and setting it back to Legacy BIOS and checking other options to make sure they are correct then saving and exiting BIOS, and rebooting the machine.

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: Grub error after update from 7.5 to 7.6

Post by username » 2018/12/22 15:47:07

zorlac wrote:
2018/12/20 02:44:42
Are you using a SoYouStart/OVH server, by chance?
Yep ! It's an OVH server (SP-16)...
zorlac wrote:
2018/12/20 02:44:42
Running "yum update" and a "reboot" should NOT brick the server.
I have been using CentOS as a server since 5 or 6 years and it's a first time I see that... I was quite confident about running simple updates like this. But now... not so much. And these servers are quite newly configured (like 6 months old) with nothing exotic running (just a LEMP stack). I've added exclude=grub* to my yum.conf. I don't want to update these packages anymore.
zorlac wrote:
2018/12/20 02:44:42
  • grub2-mkconfig -o /boot/grub2/grub.cfg
  • grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
OK, I only generated the first one because it seemed /boot/efi/EFI/ was not used. The server may be looking for the cfg there. I will give it another shot after Christmas holidays.

Thanks

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: Grub error after update from 7.5 to 7.6

Post by username » 2018/12/22 15:51:29

desertcat wrote:
2018/12/20 07:04:43
In my case my BIOS is setup to run as a Legacy BIOS, NOT UEFI. Sometimes however the BIOS gets whacked and it defaults back to UEFI... and my machine freezes.
Yeah, that could be it. It seemed /boot/efi/EFI/ was not used. But the server might be looking for the cfg there because of some kind of bug or BIOS misconfiguration. I will run another grub2-mkconfig to this path and give it another try after Christmas.

Thanks

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: Grub error after update from 7.5 to 7.6

Post by username » 2019/01/02 11:18:57

Code: Select all

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
That was it !

/boot/grub2 was used before the update. But for some unknown reason, after the reboot, the server was looking for a grub.cfg in /boot/efi/EFI/centos/. So I guess it's now booting in EFI. Why ? I don't know. My other server (same hardware, same server provider, same update) uses /boot/grub2/grub.cfg and there is no grub.cfg in /boot/efi/EFI/centos.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Grub error after update from 7.5 to 7.6

Post by TrevorH » 2019/01/02 12:15:04

Someone switched it to use UEFI in the BIOS? Or the battery ran out and it reset it all to defaults and did it that way?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: Grub error after update from 7.5 to 7.6

Post by username » 2019/01/03 10:24:40

TrevorH wrote:
2019/01/02 12:15:04
Someone switched it to use UEFI in the BIOS? Or the battery ran out and it reset it all to defaults and did it that way?
I don't really know what happened. It seems that both servers are using UEFI. If I run dmesg | grep efi, I get the same output on bother servers :

Code: Select all

[    0.000000] efi: EFI v2.50 by EDK II BIOS ID:S1200SP.86B.03.01.0026.092720170729
[    0.000000] efi:  SMBIOS=0x78633000  ACPI=0x79ffe000  ACPI 2.0=0x79ffe014 
[    0.000000] efi: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB) 
But on one server, ls /boot/efi/EFI/centos returns only these files :
fonts grubenv grubx64.efi

It was the same on the second server, but this one wasn't booting anymore before I did a grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg that solved the issue. I don't really get it either...

Post Reply