CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

General support questions
Oliver Jones
Posts: 4
Joined: 2018/03/08 09:34:01

CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by Oliver Jones » 2018/03/08 10:01:56

CentOS will boot in UEFI mode, detect CD/DVD-based install media and install normally on VirtualBox. It will even function normally, until the virtual machine is powered off. However, after a subsequent power-on, VirtualBox will not boot CentOS - and will jump straight into the UEFI Interactive Shell.

Steps to reproduce:

1) Create a new Linux VM in VirtualBox 5.2.8 (Red Hat Linux 64-bit, 4GB of memory, accept 8GB default disk size and type).
2) Go to Settings -> System -> Extended Features, and tick the box "Enable EFI (special OSes only)". Select the boot media for the CD/DVD device.
3) Start the virtual machine, and install a minimal install of CentOS, with all the defaults (including default disk layout.)
4) When the install is finished, click the "Reboot" button, and observe that the system will boot in UEFI mode.
5) Shut the system down (either select ACPI Shutdown, or power-off the machine manually from the command line.)
6) Start the virtual machine back up again. Note that it will not boot - you will get the UEFI Interactive Shell, instead of GRUB.

Note, under identical circumstances, this does not happen with RHEL 7.4. (Reproduce the above steps with RHEL 7.4 media instead of CentOS media, and step 6 will result in the machine booting successfully into RHEL 7.4.) CentOS would appear, at the very least, to be binary-incompatible with RHEL, when it comes to UEFI boot code: RHEL 7.4 boots consistently under UEFI mode in VirtualBox; CentOS 7.4 does not.

I've observed this issue in older versions of VirtualBox also (including the 5.1 branch), so this isn't something the VirtualBox team has broken lately. I need to test our products under UEFI boot conditions as well as legacy, so please do not ask me to use legacy boot as a workaround.

owl102
Posts: 413
Joined: 2014/06/10 19:13:41

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by owl102 » 2018/03/08 11:03:02

(TL;DR) It's the fault of the incomplete UEFI implementation in VirtualBox, and CentOS can't do anything about.
Oliver Jones wrote:CentOS will boot in UEFI mode, detect CD/DVD-based install media and install normally on VirtualBox. It will even function normally, until the virtual machine is powered off. However, after a subsequent power-on, VirtualBox will not boot CentOS - and will jump straight into the UEFI Interactive Shell.
This is a known issue. VirtualBox UEFI is not offering a permanent virtualized NVRAM, therefore the UEFI boot loader entry created by CentOS during installation gets lost as soon as you power off the virtual machine. To compensate for this problem, VirtualBox UEFI is able to boot some (known) guest operating systems anyway. That's why the (U)EFI support in VirtualBox is marked as "special OSos only".

I do not have a list of these "special OSos" at hand, but CentOS is not one of those. (But RHEL seems to be on this list. Yes, CentOS is binary compatible to RHEL, but the path of the UEFI boot module inside the ESP (EFI System Partition) is different, and while VirtualBox is able to find and start the RHEL boot loader on its own, i.e. without having a UEFI boot entry in NVRAM, it's not able to do so with CentOS.)

See also:
https://askubuntu.com/questions/428789/ ... untu-12-04
https://bbs.archlinux.org/viewtopic.php?id=158003
http://askubuntu.com/a/593302
... and many more.

(Following the third link you'll get a workaround to compensate for this issue, creating an UEFI start script called startup.nsh within the virtual machine.)
I need to test our products under UEFI boot conditions
IMHO the incomplete (U)EFI of VirtualBox is not a good precondition for testing products under UEFI boot conditions.

BTW: The VirtualBox User Manual clearly says: "Note that the VirtualBox EFI support is experimental and will be enhanced as EFI matures and becomes more widespread."
German speaking forum for Fedora and CentOS: https://www.fedoraforum.de/

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by TrevorH » 2018/03/08 11:18:19

Yes, recreated and it looks like a bug somewhere.

You can fix it if you boot the install DVD in rescue mode (off the troubleshooting menu) and copy /boot/efi/EFI/centos/grubx64.efi to /boot/efi/EFI/BOOT. So what I did was boot in rescue mode, tell it to drp to a shell - not mount anything then run

Code: Select all

vgchange -ay
mount /dev/centos/root /mnt/sysimage
mount /dev/sda2 /mnt/sysimage/boot
mount /dev/sda1 /mnt/sysimage/boot/efi
mount --bind /proc /mnt/sysimage/proc
mount --bind /sys /mnt/sysimage/sys
mount --bind /dev /mnt/sysimage/dev
chroot /mnt/sysimage
cp -p /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/BOOT/
exit
umount -R /mnt/sysimage
shutdown -r now
Please raise a bug for this on bugs.centos.org - if it doesn't do it on the same version of RHEL then it's a CentOS bug and needs to be reported on our bug tracker.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

owl102
Posts: 413
Joined: 2014/06/10 19:13:41

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by owl102 » 2018/03/08 11:41:27

TrevorH wrote:if it doesn't do it on the same version of RHEL then it's a CentOS bug and needs to be reported on our bug tracker.
The only thing which comes into my mind CentOS could do about this is using the same path to the boot loader within the ESP, but I guess this is not an option because of the "redhat" in the Path (which was replaced with "centos" by the CentOS team).
Last edited by owl102 on 2018/03/08 17:46:57, edited 1 time in total.
German speaking forum for Fedora and CentOS: https://www.fedoraforum.de/

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by avij » 2018/03/08 12:09:59

Adding some diagnostics to owl102's excellent response:

When I boot a VirtualBox CentOS machine after installation and run efibootmgr -v I see "Boot0003* CentOS HD(1,GPT,37c..,)/File(\EFI\centos\shimx64.efi)" but after shutdown that entry is gone. I can start the machine by typing fs0:\EFI\centos\shimx64.efi into the shell, and I can run efibootmgr -v from there to see the boot entries (which does not have CentOS anymore at this stage).

Also, an easier way to accomplish what TrevorH wrote above: cp fs0:\EFI\centos\grubx64.efi fs0:\EFI\BOOT in the UEFI shell will copy grubx64.efi to the other directory.

Separate from the above, I can add a boot menu entry with efibootmgr -c -l '\EFI\centos\shimx64.efi' -L CentOS that works after a reboot, but vanishes after a shutdown.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by TrevorH » 2018/03/08 12:21:52

The "it's a virtualbox bug" makes sense except that it works if you use the 1611 media.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Oliver Jones
Posts: 4
Joined: 2018/03/08 09:34:01

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by Oliver Jones » 2018/03/08 13:09:09

owl102 wrote:This is a known issue. VirtualBox UEFI is not offering a permanent virtualized NVRAM, therefore the UEFI boot loader entry created by CentOS during installation gets lost as soon as you power off the virtual machine. To compensate for this problem, VirtualBox UEFI is able to boot some (known) guest operating systems anyway. That's why the (U)EFI support in VirtualBox is marked as "special OSos only".
Thanks for the explanation. It's quite helpful to know this.
owl102 wrote:I do not have a list of these "special OSos" at hand, but CentOS is not one of those. (But RHEL seems to be on this list. Yes, CentOS is binary compatible to RHEL, but the path of the UEFI boot module inside the ESP (EFI System Partition) is different, and while VirtualBox is able to find and start the RHEL boot loader on its own, i.e. without having a UEFI boot entry in NVRAM, it's not able to do so with CentOS.)
I will go and ask the VirtualBox team to add CentOS to the list. To be honest, given how popular CentOS is, I find it hard to believe they didn't do it already: I did note that if you type "CentOS" as the guest name for a new VM, VirtualBox will pick OS type "Linux" and "Red Hat (64-bit)".
To be honest, I'm talking about CentOS - not Ubuntu, Arch Linux, et cetera. One eats an elephant one bite at a time - and if the issue can be solved satisfactorily for CentOS, I'm sure other Linuxes will follow.

Incidentally, I happen to be a big fan of FreeBSD, and FreeBSD 11.1 has absolutely no issue booting in UEFI mode in VirtualBox 5.2.8. FreeBSD is hardly as popular or well-supported as Linux (let alone CentOS), which is one reason I'm very surprised at this issue.
owl102 wrote:(Following the third link you'll get a workaround to compensate for this issue, creating an UEFI start script called startup.nsh within the virtual machine.)
To be honest, that's a horrible hack. There's no easy way to automate deployment of that (say, using scripts or Ansible). But thanks for the info.
owl102 wrote:IMHO the incomplete (U)EFI of VirtualBox is not a good precondition for testing products under UEFI boot conditions.
Unfortunately, VirtualBox ticks a lot of other boxes (such as the ability to run on an extremely diverse set of host operating systems), which is why I'm using it. If I had access to considerably more funds than I currently do, I'd call it a day and have everyone use VMware.
owl102 wrote:BTW: The VirtualBox User Manual clearly says: "Note that the VirtualBox EFI support is experimental and will be enhanced as EFI matures and becomes more widespread."
I take exception to this for two reasons:

a) That manual entry is probably a decade old, without any updates since it was written. It implies EFI isn't widespread, which is hilarious.

b) EFI has been around in mainstream kit since x86-64 became a thing (2006), and I challenge you to find any computer equipment made in the last decade that doesn't come with EFI - or, since Haswell, with EFI not enabled by default. Every Windows version, since Vista x64, supports EFI. It's Legacy BIOS, rather than enjoying mainstream status, that's increasingly seen as deprecated. (See also: Secure Boot.)
TrevorH wrote:Yes, recreated and it looks like a bug somewhere.

You can fix it if you boot the install DVD in rescue mode (off the troubleshooting menu) and copy /boot/efi/EFI/centos/grubx64.efi to /boot/efi/EFI/BOOT. So what I did was boot in rescue mode, tell it to drp to a shell - not mount anything then run

Code: Select all

vgchange -ay
mount /dev/centos/root /mnt/sysimage
mount /dev/sda2 /mnt/sysimage/boot
mount /dev/sda1 /mnt/sysimage/boot/efi
mount --bind /proc /mnt/sysimage/proc
mount --bind /sys /mnt/sysimage/sys
mount --bind /dev /mnt/sysimage/dev
chroot /mnt/sysimage
cp -p /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/BOOT/
exit
umount -R /mnt/sysimage
shutdown -r now
Please raise a bug for this on bugs.centos.org - if it doesn't do it on the same version of RHEL then it's a CentOS bug and needs to be reported on our bug tracker.
Thanks very much for this! To be honest, booting from rescue media also isn't my thing (can't be automated), but I did test out two variants of this approach, both of which worked well:

1) When installing, wait for the "Reboot" button to appear, then press Ctrl-Alt-F2 to get a shell (Ctrl-Alt-F6 to get back), then enter:

Code: Select all

cp -p /mnt/sysimage/boot/efi/EFI/centos/grubx64.efi /mnt/sysimage/boot/efi/EFI/BOOT/
2) On first boot, sudo to (or login as) root, then enter this line, copied verbatim from your post:

Code: Select all

cp -p /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/BOOT/
I will be using variant 2 for automated deployment, or variant 1 for manual installs.

As suggested, I have raised this as a bug on bugs.centos.org (and included a reference to this thread), here:
https://bugs.centos.org/view.php?id=14565
avij wrote:Adding some diagnostics to owl102's excellent response:

When I boot a VirtualBox CentOS machine after installation and run efibootmgr -v I see "Boot0003* CentOS HD(1,GPT,37c..,)/File(\EFI\centos\shimx64.efi)" but after shutdown that entry is gone. I can start the machine by typing fs0:\EFI\centos\shimx64.efi into the shell, and I can run efibootmgr -v from there to see the boot entries (which does not have CentOS anymore at this stage).

Also, an easier way to accomplish what TrevorH wrote above: cp fs0:\EFI\centos\grubx64.efi fs0:\EFI\BOOT in the UEFI shell will copy grubx64.efi to the other directory.
Thanks for this, also - especially the last part, which is a useful way to recover without using rescue media.
TrevorH wrote:The "it's a virtualbox bug" makes sense except that it works if you use the 1611 media.
It would be interesting to note why 1611 worked, yet 1708 does not. What changed?

owl102
Posts: 413
Joined: 2014/06/10 19:13:41

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by owl102 » 2018/03/08 15:05:51

Oliver Jones wrote:To be honest, I'm talking about CentOS - not Ubuntu, Arch Linux, et cetera.
It doesn't matter, the problem is the same: The NVRAM of VirtualBox is not persistent and therefore all EFI boot entries added during installation of CentOS, Debian, Ubuntu, Arch Linux et cetera get lost after a shutdown. (BTW: Debian isn't a "special OS" either, same problem with Debian 9)
a) That manual entry is probably a decade old, without any updates since it was written.
And the EFI support of VirtualBox is probably a decade old, without any updates since it was introduced. Therefore the current status of VirtualBox EFI support still matches the User Manual.

Furthermore the EFI option in VirtualBox clearly says "special OSes only". So as long as CentOS is no "special OS" this is not a bug, VirtualBox does not claim is should work, quite the opposite.
It implies EFI isn't widespread, which is hilarious.
Within VMs EFI isn't widespread. Only on real hardware.
German speaking forum for Fedora and CentOS: https://www.fedoraforum.de/

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by TrevorH » 2018/03/08 16:08:47

It would be interesting to note why 1611 worked, yet 1708 does not. What changed?
In 1611, the contents of /boot/efi/EFI/BOOT includes a file called fallback.efi and it's this that makes it work. If you yum update 1611 to 1708 then it's immediately broken as the grub2 update deletes fallback.efi and replaces it with an fbx64.efi. If you copy fbx64.efi to fallback.efi then it works again.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

tru_tru
Posts: 35
Joined: 2007/10/05 21:12:09
Location: France

Re: CentOS 7.4 [1708] - boot problems in UEFI mode on VirtualBox 5.x

Post by tru_tru » 2018/03/08 19:07:34

For the record the summary was posted at: https://forums.virtualbox.org/viewtopic.php?f=3&t=87077
and there is an open bug entry at https://www.virtualbox.org/ticket/15835

Thanks everyone :)

Post Reply