2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit servers crashes issue

Issues related to applications and software problems
Post Reply
black0144
Posts: 3
Joined: 2019/03/14 06:24:45

2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit servers crashes issue

Post by black0144 » 2019/03/14 06:55:21

Hi guys,

We met a tough issue recently, I'm not sure if anyone could help with it.
Here's the situation:
When we run "java -version" command on 2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit on some physical servers, the command will sometimes get stuck and not show the version info, and even cause the OS to crash.

Kernel: 2.6.32-754.9.1.el6.i68
Server: UCSC-C220-M4S
Openjdk we tried: 1.8.0.131, 1.8.0.161, 1.8.0.191

And this issue seems happen on this specific combination "2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit UCSC-C220-M4S servers". Other physical servers or Virtual Machines are not affected at all by far.

We have got strace openjdk log, but we can't find the root cause. I'll attach it to this post, if anyone is interested and would help.

Thanks.
Attachments
strace-2.6.32-754.9.1.el6.i686.log.zip
(2.76 KiB) Downloaded 65 times

User avatar
TrevorH
Forum Moderator
Posts: 26303
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit servers crashes issue

Post by TrevorH » 2019/03/14 08:23:57

Test the latest to see if whatever it is has already been fixed: 2.6.32-754.11.1.el6.x86_64
CentOS 5 died in March 2017 - migrate NOW!
CentOS 6 goes EOL sooner rather than later, get upgrading!
Full time Geek, part time moderator. Use the FAQ Luke

black0144
Posts: 3
Joined: 2019/03/14 06:24:45

Re: 2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit servers crashes issue

Post by black0144 » 2019/03/14 09:16:58

TrevorH wrote:
2019/03/14 08:23:57
Test the latest to see if whatever it is has already been fixed: 2.6.32-754.11.1.el6.x86_64
TrevorH, thanks for the suggestion, I tried updating 2.6.32-754.11.1.el6.i686 kernel for this test server and run the command. The OS still crashes after several times of running "java -version" command.
From the ssh window, I saw the final strace java -version result and OS crashed:

Code: Select all

execve("/usr/bin/java", ["/usr/bin/java", "-version"], [/* 24 vars */]) = 0
brk(0)                                  = 0x9ae7000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77a5000
readlink("/proc/self/exe", "/usr/lib/jvm/java-1.8.0-openjdk-"..., 4096) = 72
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls/i686/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls/i686/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls/i686", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/tls", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/i686/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/i686/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/i686", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/jli", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls/i686/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls/i686/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls/i686", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/tls", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/i686/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/i686/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/i686", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/sse2/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/sse2", 0xbf86891c) = -1 ENOENT (No such file or directory)
open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el6_10.i386/jre/bin/../lib/i386", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=20165, ...}) = 0
mmap2(NULL, 20165, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb77a0000
close(3)                                = 0
open("/lib/libpthread.so.0", O_RDONLY)  = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0N\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=131848, ...}) = 0
mmap2(NULL, 107044, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x5d8000
Attachments
crash.png
crash.png (147.32 KiB) Viewed 1247 times

User avatar
TrevorH
Forum Moderator
Posts: 26303
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit servers crashes issue

Post by TrevorH » 2019/03/14 09:50:34

Looks like your next step is to look at the crash dump and find out where/why/how it crashed.
CentOS 5 died in March 2017 - migrate NOW!
CentOS 6 goes EOL sooner rather than later, get upgrading!
Full time Geek, part time moderator. Use the FAQ Luke

black0144
Posts: 3
Joined: 2019/03/14 06:24:45

Re: 2.6.32-754.9.1.el6.i686 kernel CentOS6-32bit servers crashes issue

Post by black0144 » 2019/03/26 06:27:41

We found something about it, but not the root cause though:
This is the original boot option:

Code: Select all

title CentOS (2.6.32-754.9.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-754.9.1.el6.x86_64 ro root=UUID=7a6b1262-f2bd-4545-a54e-aac1dbdc7ea2 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM nopti
        initrd /initramfs-2.6.32-754.9.1.el6.x86_64.img
If we remove the nopti option, OS will not crash any more.
But this option is introduced from Meltdown CPU patch, removing this option will cause OS performance decrease.

Post Reply