Unable to boot after incomplete yum upgrade to 7.6

General support questions
Post Reply
dalu
Posts: 12
Joined: 2017/03/25 23:31:08
Location: Germany/Croatia
Contact:

Unable to boot after incomplete yum upgrade to 7.6

Post by dalu » 2018/12/07 13:09:08

Story goes,
There was a rsync backup running, I ran yum update, it was 4am I wanted to go to bed.
So yum update doesn't complete, gets stuck on kexec-tools cleanup with 134 package left to clean.

I run `yum-complete-transaction` and then `yum-complete-transaction --cleanup-only`

I re-install grub2 because that had a dependency conflict

Then package-clean ... --cleanduplicates I believe, to remove the older packages.
reboot, dead server

boot into the datacenter's rescue via pxe
mount the root partition to /mnt and the boot partition to /mnt/boot

it's a RAID LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] raid setup
so I'm talking to /dev/sda
/dev/sda4 is root
/dev/sda2 is boot

so, I "arch-chroot" into the system
remove kernels, install latest kernel so dracut regenerates
I ran grub2-install /dev/sda
no issues

I exit, unmount and reboot
unbootable system.

When I installed this system a short while ago I had to create a bios boot partition, or rather the installer created it automatically.
CentOS seems to do some boot magic which I don't understand or have no insight into.

cfdisk shows

Code: Select all

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
            1007.0 KiB  free space
   1        1024.0 KiB  BIOS boot partition
   2        1024.0 MiB  Microsoft basic data
   3        15.7 GiB    Linux swap
   4        3.6 TiB     Microsoft basic data
            1007.5 KiB  free space
-----
update:

removed rhgb quiet
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

something about plymouth failing to import network config or w/e
and a bunch of selinux errors
I'll try disabling selinux temporarily
Last edited by dalu on 2018/12/07 14:13:16, edited 3 times in total.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Unable to boot after incomplete yum upgrade to 7.6

Post by TrevorH » 2018/12/07 13:13:13

Then package-clean ... --cleanduplicates I believe, to remove the older packages.
Wrong way round, you should run it with the --removenewestdupes option to remove the new versions so that next time you run yum update it will re-update to the newer ones and make sure they are correct.
unbootable system.
You don't say how or why. What errors do you get? What does it do? Remove rhgb quiet from the kernel command line before you try again so you see all the boot messages.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

dalu
Posts: 12
Joined: 2017/03/25 23:31:08
Location: Germany/Croatia
Contact:

Re: Unable to boot after incomplete yum upgrade to 7.6

Post by dalu » 2018/12/07 13:50:03

updated original post, ping.

I didn't say because I didn't know now I'm more knowledgeable.
will post updates from now on and leave original post alone.

edit: I still have a list of duplicates. it's rather large.

dalu
Posts: 12
Joined: 2017/03/25 23:31:08
Location: Germany/Croatia
Contact:

Re: Unable to boot after incomplete yum upgrade to 7.6

Post by dalu » 2018/12/07 14:01:20

Ok the system is booting now with selinux permissive.

There is still something that I can't see what it is, failures, because it scrolls by so fast over kvm-console at boot time.

What is the current way to enable boot logging, please? I see a bunch of CentOS 5 and 6 topics dealing with logging boot but none about 7.

edit:
Only failure I'm getting, from what I was able to capture in the fast scrolling at boot time via tiny console is
abrt-coredump

But otherwise the system is booting up with selinux enforcing.

So I guess that's a [solved] for now.
edit2:
Or not. /bin/bash permission denied.
I guess I'll have to reinstall all packages.

dalu
Posts: 12
Joined: 2017/03/25 23:31:08
Location: Germany/Croatia
Contact:

Re: Unable to boot after incomplete yum upgrade to 7.6

Post by dalu » 2018/12/07 14:39:01

So now that I've reinstalled all duplicate packages..
that did exactly nothing. edit: and I believe I know why, because the datacenter's rescue system is selinux enabled. I would have to reinstall from within a running boot.

/bin/bash: Permission denied
when trying to log in via ssh

how can I re-apply all selinux policies please? Or rather the filesystem tagging.

edit2:

Code: Select all

Dec 07 15:49:23 kernel: SELinux:  Starting in permissive mode
Dec 07 15:49:38 crond[5299]: ((null)) SELinux in permissive mode, continuing (/etc/crontab)
Dec 07 15:49:38 crond[5299]: ((null)) SELinux in permissive mode, continuing (/etc/cron.d/0hourly)
Dec 07 15:49:38 crond[5299]: ((null)) SELinux in permissive mode, continuing (/etc/cron.d/raid-check)
Dec 07 15:49:38 crond[5299]: ((null)) SELinux in permissive mode, continuing (/etc/cron.d/sysstat)
Dec 07 15:50:01 crond[6007]: (*system*) NULL security context for user, but SELinux in permissive mode, continuing ()
all I get from grepping for permissive of journalctl -b

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Unable to boot after incomplete yum upgrade to 7.6

Post by TrevorH » 2018/12/07 15:13:36

Run setenforce 0 once booted up to go permissive.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

dalu
Posts: 12
Joined: 2017/03/25 23:31:08
Location: Germany/Croatia
Contact:

Re: Unable to boot after incomplete yum upgrade to 7.6

Post by dalu » 2018/12/07 15:31:35

Yes, I set it to permissive at /etc/selinux/config

But that can only be a temporary solution.

I did
restorecon -Rv /

to re-apply the securty context labels on all of the files and dirs.

Right now it's in permissive mode, because.. needs to make money. It's december, best paying month.
I'll reboot in early mornings again and see if that worked.
Until then :)

desertcat
Posts: 843
Joined: 2014/08/07 02:17:29
Location: Tucson, AZ

Re: Unable to boot after incomplete yum upgrade to 7.6

Post by desertcat » 2018/12/09 10:57:59

dalu wrote:
2018/12/07 13:09:08
"Story goes,
There was a rsync backup running, I ran yum update, it was 4am I wanted to go to bed.
So yum update doesn't complete, gets stuck on kexec-tools cleanup with 134 package left to clean.

I run `yum-complete-transaction` and then `yum-complete-transaction --cleanup-only`

I re-install grub2 because that had a dependency conflict

Then package-clean ... --cleanduplicates I believe, to remove the older packages.
reboot, dead server

boot into the datacenter's rescue via pxe
mount the root partition to /mnt and the boot partition to /mnt/boot

it's a RAID LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] raid setup
so I'm talking to /dev/sda
/dev/sda4 is root
/dev/sda2 is boot

so, I "arch-chroot" into the system
remove kernels, install latest kernel so dracut regenerates
I ran grub2-install /dev/sda
no issues

I exit, unmount and reboot
unbootable system.

When I installed this system a short while ago I had to create a bios boot partition, or rather the installer created it automatically.
CentOS seems to do some boot magic which I don't understand or have no insight into.

cfdisk shows

Code: Select all

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
            1007.0 KiB  free space
   1        1024.0 KiB  BIOS boot partition
   2        1024.0 MiB  Microsoft basic data
   3        15.7 GiB    Linux swap
   4        3.6 TiB     Microsoft basic data
            1007.5 KiB  free space
-----
update:

removed rhgb quiet
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

something about plymouth failing to import network config or w/e
and a bunch of selinux errors
I'll try disabling selinux temporarily
"Story goes...
"There was a rsync backup running, I ran yum update, it was 4am I wanted to go to bed.... "

Ah... it is currently 0356 hrs. Been sort of in your shoes -- "Oh NO!!! what did I just do?" :lol: Happy Holidays and a Safe and Prosperous New Year!

Post Reply