System segfaults in kernel, and on chroot in rescue disk

General support questions
Post Reply
mathog
Posts: 258
Joined: 2008/07/09 23:52:06

System segfaults in kernel, and on chroot in rescue disk

Post by mathog » 2018/03/08 19:38:59

One of the users was active on a system last night when everything began segfaulting. This morning it was impossible for me to get into it through the console or a serial port and it eventually had to be shut down with the power button. On reboot it put up a Centos 6 spash screen and then shortly thereafter segfaulted.

Booted a Centos 7 DVD (all I had on hand) and went into the rescue mode. All the disks look OK (SMART values good) and the RAID (PERC H310) filesystems all fsck OK. However, doing this:

Code: Select all

chroot /mnt/sysimage
segfaults.

In other words, it behaves exactly like this:

https://access.redhat.com/solutions/386003

but I don't have a subscription so cannot read the solution there.

I don't know what the user was doing but he has root so anything is possible.

Some other posts suggest that this is the phenotype a system shows after the incorrect installation of glibc for the wrong architecture.

Anyway, it is not in great shape. The boot loader seems to be intact, it gives the option of selecting from various kernels, and it lets me edit the options. But all of them segfault at the same place.

Anybody been through this before? I think the rescue disk needs to check that all of the installed RPMs are valid, but since I cannot chroot into that system, I'm not sure how to make rpm do that.

Thanks,

David Mathog

mathog
Posts: 258
Joined: 2008/07/09 23:52:06

Re: System segfaults in kernel, and on chroot in rescue disk

Post by mathog » 2018/03/08 22:15:08

Found that the user had dropped in files

Code: Select all

/lib64/libc.so.6
/lib64/libc-2.19.so
both the same size. Removed both then did

Code: Select all

cd /lib64
ln -s libc-2.12.so libc.so.6
and rebooted. That fixed the immediate problem - it got past the Centos kernel and started putting "OK" steps on the screen until it hit

Code: Select all

Warning -- SElinux targeted policy relabel is required.  Relabeling could take a very long time, depending on file system size and speed of hard drives.
at which point it was slowly writing row after row of asterisks. If memory serves that is a 6 Tb file system. Hopefully it doesn't look in /home too!

I wonder why it needs to relabel? The only thing that would have changed from the OS's perspective is the time stamp on the soft link "libc.so.6". Unless maybe this is a side effect of the odd state it was in, where nothing would run properly. So far it has been half an hour and the system has not yet come back on line. It is physically in another building, and I waited there 20 minutes to complete, then gave up and returned to my office where I'm just waiting for its sshs to appear again.

mathog
Posts: 258
Joined: 2008/07/09 23:52:06

Re: System segfaults in kernel, and on chroot in rescue disk

Post by mathog » 2018/03/09 00:29:41

For future reference, the relabel took a bit over 2 hours.

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: System segfaults in kernel, and on chroot in rescue disk

Post by Whoever » 2018/03/09 02:53:01

mathog wrote:Found that the user had dropped in files

Code: Select all

/lib64/libc.so.6
/lib64/libc-2.19.so
Revoke that user's root/sudo privileges!

Post Reply