can't boot after yum update -y cento6.9

General support questions
chankim
Posts: 50
Joined: 2015/07/15 12:05:39

can't boot after yum update -y cento6.9

Post by chankim » 2018/04/05 03:07:34

I was using Centos 6.7 and yesterday tried to update to Centos 7.
I ran 'yum update -y' and it started downloading many packages.
I went home and found the vnc connection is closed and checked that 'yum update' process finished.
So I endtered the machine using ssh and reboot it but the machine did not repond from then.
So this morning at my work, I found the booting process is halted with the messages below.

Kernel panic - not syncing: VFS : Unable to mount root fs on unknown-block(0,0)
Pid: 1, comm: swapper Not tainted 2.6.32-696.23.1.el6.x86x64 #1
Call Trace:..

During the hard reboot, I can see 5 CentOS kernel images, I tried each, all of them shows the loading bar and displays CentOS 6.9 strangely.
(I expected to see the previous CentOS 6.7 working)
I later found with a booting with a kernel image, it is halted after Start certmonger ..
and I found that I can login to the machine using ssh at that time!

I opened /boot/grub/grub.conf and there is no initrd lines.
Below is the grub.conf file.

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,2)
# kernel /boot/vmlinuz-version ro root=/dev/sda3
# initrd /boot/initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,2)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-696.23.1.el6.x86_64)
root (hd0,2)
kernel /boot/tboot.gz logging=vga,serial,memory
module /boot/vmlinuz-2.6.32-696.23.1.el6.x86_64 ro root=UUID=b4c939f1-8d9f-45ad-b8db-ddfe4206afef intel_iommu=on rd_NO_LUKS LANG=ko_KR.UTF-8 rd_NO_MD crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_LVM rd_NO_DM rhgb quiet nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off vga=normal rdblacklist=nouveau
title CentOS (2.6.32-573.12.1.el6.x86_64)
root (hd0,2)
kernel /boot/tboot.gz logging=vga,serial,memory
module /boot/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=UUID=b4c939f1-8d9f-45ad-b8db-ddfe4206afef intel_iommu=on rd_NO_LUKS LANG=ko_KR.UTF-8 rd_NO_MD crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_LVM rd_NO_DM rhgb quiet nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off vga=normal rdblacklist=nouveau
module /boot/initramfs-2.6.32-573.12.1.el6.x86_64.img
title CentOS (2.6.32-504.30.3.el6.centos.plus.x86_64)
root (hd0,2)
kernel /boot/tboot.gz logging=vga,serial,memory
module /boot/vmlinuz-2.6.32-504.30.3.el6.centos.plus.x86_64 ro root=UUID=b4c939f1-8d9f-45ad-b8db-ddfe4206afef intel_iommu=on rd_NO_LUKS LANG=ko_KR.UTF-8 rd_NO_MD crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_LVM rd_NO_DM rhgb quiet nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off vga=normal rdblacklist=nouveau
module /boot/initramfs-2.6.32-504.30.3.el6.centos.plus.x86_64.img
title CentOS (2.6.32-504.3.3.el6.x86_64)
root (hd0,2)
kernel /boot/tboot.gz logging=vga,serial,memory
module /boot/vmlinuz-2.6.32-504.3.3.el6.x86_64 ro root=UUID=b4c939f1-8d9f-45ad-b8db-ddfe4206afef intel_iommu=on rd_NO_LUKS LANG=ko_KR.UTF-8 rd_NO_MD crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_LVM rd_NO_DM rhgb quiet nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off vga=normal rdblacklist=nouveau
module /boot/initramfs-2.6.32-504.3.3.el6.x86_64.img
title CentOS (2.6.32-358.el6.x86_64)
root (hd0,2)
kernel /boot/tboot.gz logging=vga,serial,memory
module /boot/vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=b4c939f1-8d9f-45ad-b8db-ddfe4206afef intel_iommu=on rd_NO_LUKS LANG=ko_KR.UTF-8 rd_NO_MD crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_LVM rd_NO_DM rhgb quiet nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off vga=normal
module /boot/initramfs-2.6.32-358.el6.x86_64.img

and in directory /boot, the files with name starting with 'initrd' are
initrd-2.6.32-358.el6.x86_64kdump.img initrd-2.6.32-504.30.3.el6.centos.plus.x86_64kdump.img
initrd-2.6.32-504.3.3.el6.x86_64kdump.img initrd-2.6.32-573.12.1.el6.x86_64kdump.img

Then should I insert 'initrd /boot/initrd-2.6.32-573.12.1.el6.x86_64kdump.img' in the CentOS (2.6.32-573.12.1.el6.x86_64) entry?
actually I will try that soon.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: can't boot after yum update -y cento6.9

Post by TrevorH » 2018/04/05 08:28:07

You've manually installed the nvidia .run drivers downloaded from the nvidia website and those need reinstalling after each and every kernel update or they break in exactly the manner you see. Either reinstall the .run file or, better, rerun it with --uninstall (perhaps -uninstall?) then install the ELRepo yum repo - see http://wiki.centos.org/AdditionalResources/Repositories - and install their nvidia-detect package, run it to find out which of their kmod packages it recommends then install that.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

chankim
Posts: 50
Joined: 2015/07/15 12:05:39

Re: can't boot after yum update -y cento6.9

Post by chankim » 2018/04/07 08:34:40

Hi, TrevorH, thanks again,
I found in my current setting (CentOS 6.9, by the way I want to go to CentOS 7 after this is resolved),
'cat /proc/driver/nvidia/driver' shows

NVRM version: NVIDIA UNIX x86_64 Kernel Module 352.39 Fri Aug 14 18:09:10 PDT 2015
GCC version: gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)

but I see with 'ls ~/Downloads/NVIDIA*' only (I must have dlownload whatever here)

NVIDIA-Linux-x86_64-352.63.run* NVIDIA-Linux-x86_64-361.42.run.copy* NVIDIA-Linux-x86_64-384.130.run

no 352.39 run file!. But I see under ~/Downloads,

cuda-repo-rhel6-6.5-14.x86_64.rpm cuda-repo-rhel6-7-5-local-7.5-18.x86_64.rpm

and with 'nvcc --version', I can see cuda 7.5 had been installed.

Could it be that I installed NVIDIA driver 352.39 when I installed cuda 7.5 toolkit before? ( I can't remember exactly)
I guess installing cuda installs the driver too.(oh, here https://devtalk.nvidia.com/default/topi ... r-361-28-/ I found the default driver for cuda 7.5 is nvidia driver 352.39. so my guess is right).

What would you recommend me to do? download NVIDIA-Linux-x86_64- 352.39.run file and use it to uninstall 352.39 driver?
or use rpm procedure to uninstall cuda toolkit 7.5? (and expect NVIDIA driver 352.39 to be removed together?). I can remove the package using yum remove command one by one..I have a question below.

another question is, I could remove cuda by 'sudo yum remove cuda' but this just removed cuda-7.5. I still see many cuda-7.5 related packages like cuda-nvrtc-dev, cuda-cublas-7-5,.. How can remove them all? Isn't there any 'group remove' command for cuda-7-5 related packages?
This question goes to the nvidia driver packages too. When I do 'rpm -qa | grep nvida', I see

nvidia-kmod-352.39-2.el6.x86_64
nvidia-x11-drv-390.48-1.el6_9.elrepo.x86_64
nvidia-detect-390.48-1.el6.elrepo.x86_64
yum-plugin-nvidia-1.0.2-1.el6.elrepo.noarch
nvidia-uvm-kmod-352.39-3.el6.x86_64
kmod-nvidia-390.48-1.el6_9.elrepo.x86_64

How can I uninstall above packages?

and I have another question. I guessed I should boot in text mode to remove the display driver, and I tried adding '3' at the end of 'kernel' line in grub configuration, but it's the same (stops after 'Starting certmonger ... [OK]'). What is the correct way to boot in text mode after selecting the kernel in grub? (I tried 'e' to edit the 'kernel' line and added '3', is it right?) I need this in order to reinstall the NVIDIA driver anyway as you suggested.

Please help.. Sorry for the long questions.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: can't boot after yum update -y cento6.9

Post by TrevorH » 2018/04/07 10:34:46

How did you install cuda? It's not something I've ever done but if you post details then perhaps someone else may be able to help.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

chankim
Posts: 50
Joined: 2015/07/15 12:05:39

Re: can't boot after yum update -y cento6.9

Post by chankim » 2018/04/08 08:15:14

From the existence of file ~/Downloads/cuda-repo-rhel6-7-5-local-7.5-18.x86_64.rpm,
I guess I installed it using this rpm file. So I guess the nvidia driver and cuda toolkit were all installed by rpm file.
(that yum recognizes)
Isn;t there a way to uninstall all related packages? Below are the packages installed on my computer.

ckim@stph45:~] rpm -qa | grep cuda
cuda-nvrtc-dev-7-5-7.5-18.x86_64
cuda-curand-dev-7-5-7.5-18.x86_64
cuda-misc-headers-7-5-7.5-18.x86_64
cuda-cublas-7-5-7.5-18.x86_64
cuda-cufft-7-5-7.5-18.x86_64
cuda-runtime-7-5-7.5-18.x86_64
cuda-core-7-5-7.5-18.x86_64
cuda-toolkit-7-5-7.5-18.x86_64
cuda-repo-rhel6-7-5-local-7.5-18.x86_64
cuda-command-line-tools-7-5-7.5-18.x86_64
cuda-cufft-dev-7-5-7.5-18.x86_64
cuda-samples-7-5-7.5-18.x86_64
cuda-cusolver-7-5-7.5-18.x86_64
cuda-curand-7-5-7.5-18.x86_64
cuda-7-5-7.5-18.x86_64
cuda-nvrtc-7-5-7.5-18.x86_64
cuda-cublas-dev-7-5-7.5-18.x86_64
cuda-cusolver-dev-7-5-7.5-18.x86_64
cuda-documentation-7-5-7.5-18.x86_64
cuda-cudart-7-5-7.5-18.x86_64
cuda-npp-7-5-7.5-18.x86_64
cuda-cusparse-7-5-7.5-18.x86_64
cuda-cudart-dev-7-5-7.5-18.x86_64
cuda-license-7-5-7.5-18.x86_64
cuda-cusparse-dev-7-5-7.5-18.x86_64
cuda-visual-tools-7-5-7.5-18.x86_64
cuda-driver-dev-7-5-7.5-18.x86_64
cuda-npp-dev-7-5-7.5-18.x86_64

ckim@stph45:~] rpm -qa | grep nvidia
nvidia-kmod-352.39-2.el6.x86_64
nvidia-x11-drv-390.48-1.el6_9.elrepo.x86_64
nvidia-detect-390.48-1.el6.elrepo.x86_64
yum-plugin-nvidia-1.0.2-1.el6.elrepo.noarch
nvidia-uvm-kmod-352.39-3.el6.x86_64
kmod-nvidia-390.48-1.el6_9.elrepo.x86_64

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: can't boot after yum update -y cento6.9

Post by avij » 2018/04/08 12:29:16

chankim wrote:Hi, TrevorH, thanks again,
I found in my current setting (CentOS 6.9, by the way I want to go to CentOS 7 after this is resolved),
There is no upgrade path from CentOS 6 to CentOS 7. If you want to use CentOS 7, the only supported way is to back up your data, clear your disk, install CentOS 7 and restore your data. "yum update" on CentOS 6 will only give you the latest CentOS 6.x release, currently 6.9.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: can't boot after yum update -y cento6.9

Post by TrevorH » 2018/04/08 12:31:28

ckim@stph45:~] rpm -qa | grep nvidia
nvidia-kmod-352.39-2.el6.x86_64
nvidia-x11-drv-390.48-1.el6_9.elrepo.x86_64
nvidia-detect-390.48-1.el6.elrepo.x86_64
yum-plugin-nvidia-1.0.2-1.el6.elrepo.noarch
nvidia-uvm-kmod-352.39-3.el6.x86_64
kmod-nvidia-390.48-1.el6_9.elrepo.x86_64
I am a bit surprised by this lot as it seems to show that you have both kmod-nvidia from ELRepo and a package called nvidia-kmod and I suspect they are probably both trying to supply the same functionality. If you attempt to yum remove nvidia-kmod-352.39-2.el6.x86_64 nvidia-uvm-kmod-352.39-3.el6.x86_64 and look at the list of packages it wants to take with it, is that list enormous and including all your cuda ones?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

chankim
Posts: 50
Joined: 2015/07/15 12:05:39

Re: can't boot after yum update -y cento6.9

Post by chankim » 2018/04/08 13:39:17

If you attempt to yum remove nvidia-kmod-352.39-2.el6.x86_64 nvidia-uvm-kmod-352.39-3.el6.x86_64 and look at the list of packages it wants to take with it, is that list enormous and including all your cuda ones?
I understand cuda uses nvidia driver underneath. just checking, you mean removing nvidia driver automatically removes the packages using nvidia driver? Why only kmod nvidia drivers? and what about other nvidia packages? shouldn't I remove those too?

chankim
Posts: 50
Joined: 2015/07/15 12:05:39

Re: can't boot after yum update -y cento6.9

Post by chankim » 2018/04/08 13:49:16

There is no upgrade path from CentOS 6 to CentOS 7. If you want to use CentOS 7, the only supported way is to back up your data, clear your disk, install CentOS 7 and restore your data. "yum update" on CentOS 6 will only give you the latest CentOS 6.x release, currently 6.9.
I saw https://linuxacademy.com/blog/linux/cen ... -in-place/ and https://www.lesstif.com/pages/viewpage. ... d=23757317 but also read other experts (including TrevorH) saying it's not possible to upgrade from centos 6.x to centos 7. I am just trying in-place upgrade to see if it's possible. By the way, the reason I started this is to use clang 3.9 and I found devtoolset-7(?) has llvm 3.9 (including clang 3.9). I need clang 3.9 to build CARLA ( an automotive driving simulatorr using UE4 engine). If this upgrade fails, I'll just install centos 7 but hope in-place to be possible. (with less effort incurred).

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: can't boot after yum update -y cento6.9

Post by TrevorH » 2018/04/08 14:10:44

I am just trying in-place upgrade to see if it's possible
Don't. It's broken and if it works at all, will break your machine. If you're lucky it will just refuse to run.

What I was meaning before is that you seem to have two sets of packages which both supply the nvidia drivers and that's not going to work out well. In your list you have nvidia-kmod-352.39-2.el6.x86_64 and kmod-nvidia-390.48-1.el6_9.elrepo.x86_64 which look like the same thing by a different name to me, plus they are different versions. So you have 4 packages from elrepo installed plus another 2 from somewhere else and I suspect that you should only have packages from one of those 2 sources installed.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply