CentOS 7.5 PXE kernel/initrd mismatch

[jonfullmer]

I've been using the pxeboot kernel & initrd since 7.3. The current versions are failing to start the installation. Here's the output upon death:

anaconda 21.48.22.93-1 for CentOS Linux 7 started.
* installation log files are stored in /tmp during the installation
* shell is available on TTY2
15:50:00 Running pre-installation scripts
15:50:01 Not asking for VNC because of an automated install
15:50:01 Not asking for VNC because text mode was explicitly asked for in kickstart
Traceback (most recent call last):
File "/sbin/anaconda", line 1206, in <module>
setupDisplay(anaconda, opts, addon_paths)
File "/sbin/anaconda", line 715, in setupDisplay
anaconda.initInterface(addons)
File "/usr/lib64/python2.7/site-packages/pyanaconda/anaconda.py", line 220, in initInterface
self._intf = TextUserInterface(self.storage, self.payload,
File "/usr/lib64/python2.7/site-packages/pyanaconda/anaconda.py", line 168, in storage
self._storage.setDefaultFSType(self.instClass.defaultFS)
File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 1761, in setDefaultFSType
self._check_valid_fstype(newtype)
File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 1736, in _check_valid_fstype
raise ValueError("new value %s is not valid as a default fs type" % fmt)
ValueError: new value non-existent xfs filesystem is not valid as a default fs type
Exception AttributeError: "'NoneType' object has no attribute 'udev_unref'" in <bound method Context.__del__ of <pyudev.core.Context object at 0x7f8656fe0650>> ignored

Pane is dead

In the past when I'd come across this kind of error, it was usually because the kernel and initrd were older (or didn't match) the OS repository being installed (i.e., CentOS 7.3 pxeboot images attempting to install CentOS 7.5). But I'm using the most current pxeboot images.

It seems odd to me that the kernel and initrd.img files have different dates (they're usually the same). The vmlinuz is 20-Apr-2018, but the initrd.img is 03-May-2018.

And probably the most convicting, if you go to shell (after getting the above output), the /lib/modules directory (the intird.img) shows:

3.10.0-514.el7.x86_64

But a "uname -a" (the vmlinuz) reveals: 3.10.0-862.el7.x86_64

Kernel mismatch.

How we go about repairing these images? Or is there something that I might ignorantly be missing, in order to get these to work?

Any help is appreciated. Thanks!

[TrevorH]

They work for me. I have the following

Code: Select all

# sha256sum /srv/tftp/images/centos/x86_64/7.5/*
66242e0eb2611be498a3109ff2e79992a40ec4fcb4ee48b94ddf1869c930210a  /srv/tftp/images/centos/x86_64/7.5/initrd.img
9cb4e73e90c241c82f542e174077bd99381970dd97644751228ad77ea9f78973  /srv/tftp/images/centos/x86_64/7.5/vmlinuz

If you have mention of 3.10.0-514 in there then you have a file from 7.3 not from 7.5. Booting via PXE in rescue mode and looking in /lib/modules shows me only a 3.10.0-862.el7.x86_64 directory there, no -514 one.

[jonfullmer]

Wow, now I really don't know. We have the same files:

Code: Select all

# sha256sum *
66242e0eb2611be498a3109ff2e79992a40ec4fcb4ee48b94ddf1869c930210a  initrd.img
9cb4e73e90c241c82f542e174077bd99381970dd97644751228ad77ea9f78973  vmlinuz

[TrevorH]

Are you sure you're pulling the initrd.img file you think you are?

[jonfullmer]

thanks for being polite about it. You and I came to the same conclusion. I hate to pester forums if it's a simple mistake on my part, but it really has to be. I haven't found it yet, but I'll post when I do. I manually extracted the initrd.img and checked the vmlinuz. They clearly match:

Code: Select all

# xz -dc < initrd.img | cpio --quiet -i --make-directories
# ls -l lib/modules/
total 8
drwxr-xr-x 3 root root 4096 Jul 18 16:30 3.10.0-862.el7.x86_64
drwxr-xr-x 2 root root 4096 Jul 18 16:30 keys
# file vmlinuz 
vmlinuz: Linux kernel x86 boot executable bzImage, version 3.10.0-862.el7.x86_64 (builder@kbuilder.dev.centos.org) #1 SMP , RO-rootFS, swap_dev 0x5, Normal VGA

I guess I didn't do enough homework. Sorry.

[TrevorH]

Let us know what the eventual solution is, maybe it'll help someone else in the future.

[avij]

Additionally, anaconda 21.48.22.93 is something you would get when using CentOS 7.3. 7.5 has 21.48.22.134-1.

Perhaps you should check your TFTP server logs to make sure the correct file is being sent. You can also try mv initrd.img initrd.img.disabled to see if your boot process complains about a missing initrd.img or not.

[jonfullmer]

Okay, I figured it out. And I will go into detail about it, hoping that someone else may benefit from my wasted weeks.

First off, I was focusing on the pxeboot kernel and image as the problem. It wasn't. And yes, I verified that the PXE process was, in fact, downloading the correct files.

The problem is AFTER that.

Here's what I believe I've discovered is boot install process for a PXE install of CentOS (well, 7.x):

1) Interrupt the host boot process to PXE boot
2) PXE code performs DHCP/BootP to get an address and ask for a PXE server
3) PXE code downloads the pxelinux kernel (I'm using syslinux on my TFTP server) and the config, the pxelinux kernel provides a "boot: " prompt
4) The user chooses an option that corresponds (in the pxelinux config) to a new kernel and initrd to download, as well as additional boot line options
5) The pxeboot kernel is downloaded and run
6) The pxeboot initrd image is downloaded and mounted
7) The boot process of this new kernel/images downloads the squashfs.img file from the repo (defined in the boot line inst.repo); this is expected to be in <repo-directory>/LiveOS/
8) The squashfs.img file is "unsquashed" to reveal a new filesystem that contains one folder (called "LiveOS") which contains a "rootfs.img" file (an uncompressed ext4 filesystem file)
9) The rootfs.img file is mounted and, presumably, the kernel chroots to this filesystem
10) The install goes on its way

TrevorH helped prove to me that the problem was on my end. We were clearly using the same pxeboot files, but he was succeeding and I was not.

Avij correctly pointed out that the anaconda version matched CentOS 7.3, not 7.5. This is because the squashfs.img file in my repo was out of date (it was still 7.3). So I downloaded the new squashfs.img file. All is well with the world again.

This brings up another interesting challenge, though. I use reposync to keep my local repo in sync. Evidently, it's only including the "Packages" and "repodata" directories (as well as metadata). It does NOT sync the "LiveOS" directory. I'm not seeing an easy way to make it sync that directory.

And let's take the challenge further. I'm in a traditional enterprise environment, and outbound rsync is probably not going to be permitted. I can do HTTP/HTTPS, but not rsync. How might I keep an eye on when this squashfs file changes?

Any thoughts?

P. S. Thank you for your patience and courtesy.

[avij]

squashfs.img is only updated at point release time, ie. next time it will be updated when 7.6 gets released some day. When that happens, one way to keep yourself in sync is to download the 7.6 Everything.iso, loop mount it and then copy the contents to your local repository.

[TrevorH]

Okay, I figured it out. And I will go into detail about it, hoping that someone else may benefit from my wasted weeks.

Good, you'll be glad to hear that the other person reporting the same problem on IRC had the same situation so you solved his problem too.