[SOLVED] java using 100% cpu on CentOS/RHEL 6

Issues related to applications and software problems

[SOLVED] java using 100% cpu on CentOS/RHEL 6

Postby amol » 2012/07/02 13:57:17

I have a java app running on CentOS 6.0. It always runs in background via cron. Sometimes this app goes into wait state while using 100% cpu.

My java version is :

Code: Select all
    java version "1.6.0_17"
    OpenJDK Runtime Environment (IcedTea6 1.7.4) (rhel-1.21.b17.el6-x86_64)
    OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

Other symptoms are :

a. One thread of the process seems to be in a loop waiting for something. When traced using strace, it shows following o/p continuously :

Code: Select all
>     futex(0x7fb8000ac728, FUTEX_WAKE_PRIVATE, 1) = 0
>     futex(0x7fb8000ac754, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1340347489,> 822867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

b. It seems like the process has finished working, looking at the files it is using. Only few files are remaining. The output of 'ls /proc/pid/fd/ shows :

Code: Select all
>     lr-x------ 1 root root 64 Jun 22 13:13 0 -> pipe:[77107601]
>     l-wx------ 1 root root 64 Jun 22 13:13 1 -> pipe:[77120162]
>     l-wx------ 1 root root 64 Jun 22 13:13 2 -> /var/log/mithi/mcs/agent_account_mailstore_exceed_limit.sh.log
>     lr-x------ 1 root root 64 Jun 22 13:13 3 -> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar

More specifically, are there any known problems in running openjdk based Java processes in background on CentOS/RHEL 6?

I'm able to simulate the problem with a very simple script :

Code: Select all
#!/bin/bash

while [ 1 ]
do
   /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
   sleep 1s
done

When the above script is run for about 3 - 4 hours, one or two java processes go in hang state with cpu usage touching 100%.

Has anybody faced similar situation?
Any clues or references would be very helpful.
amol
 
Posts: 3
Joined: 2012/06/22 07:29:46

Re: java using 100% cpu on CentOS/RHEL 6

Postby pjwelsh » 2012/07/02 14:52:23

Hit by the leap-second bug?

Code: Select all
service ntpd stop
date `date +"%m%d%H%M%C%y.%S"`
service ntpd start
pjwelsh
 
Posts: 2555
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: java using 100% cpu on CentOS/RHEL 6

Postby amol » 2012/07/03 11:16:42

Doesn't seem to be the leap second bug because my problem can be simulated anytime by running following script

Code: Select all
#!/bin/bash

while [ 1 ]
do
   /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
   sleep 1s
done

However, very likely it seems to be foll bug :

https://bugzilla.kernel.org/show_bug.cgi?id=32922

Is this bug fixed in CentOS 6 kernel?

Any other ideas?
amol
 
Posts: 3
Joined: 2012/06/22 07:29:46

[SOLVED] java using 100% cpu on CentOS/RHEL 6

Postby toracat » 2012/07/03 14:54:52

However, very likely it seems to be foll bug :

https://bugzilla.kernel.org/show_bug.cgi?id=32922

Is this bug fixed in CentOS 6 kernel?

According to the referenced bugzilla, the patch was:

Code: Select all
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -1886,7 +1886,7 @@ retry:
    restart->futex.val = val;
    restart->futex.time = abs_time->tv64;
    restart->futex.bitset = bitset;
-   restart->futex.flags = flags;
+   restart->futex.flags = flags | FLAGS_HAS_TIMEOUT;
 
    ret = -ERESTART_RESTARTBLOCK;


The corresponding part of the CentOS/RHEL kernel is:

Code: Select all
        restart->futex.val = val;
        restart->futex.time = abs_time->tv64;
        restart->futex.bitset = bitset;
        restart->futex.flags = FLAGS_HAS_TIMEOUT;

        if (fshared)
                restart->futex.flags |= FLAGS_SHARED;
        if (clockrt)
                restart->futex.flags |= FLAGS_CLOCKRT;

        ret = -ERESTART_RESTARTBLOCK;


Due to the difference between the RHEL kernel and that from kernel.org, it is not easy to tell unambiguously that the patch has been applied. But it seems that way.
User avatar
toracat
Forum Moderator
 
Posts: 6696
Joined: 2006/09/03 16:37:24
Location: California, US

Re: java using 100% cpu on CentOS/RHEL 6

Postby TrevorH » 2012/07/03 15:08:03

Looks like it here too, the RH code sets FLAGS_HAS_TIMEOUT for everything then adds another bit setting on top but the important part is the FLAGS_HAS_TIMEOUT is there.
User avatar
TrevorH
Forum Moderator
 
Posts: 9167
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: [SOLVED] java using 100% cpu on CentOS/RHEL 6

Postby amol » 2012/07/27 11:43:35

The problem was solved after upgrading kernel to kernel-2.6.32-220 i.e CentOS 6.2 kernel.

Thanks all for the help.
amol
 
Posts: 3
Joined: 2012/06/22 07:29:46


Return to CentOS 6 - Software Support

Who is online

Users browsing this forum: Yahoo [Bot] and 7 guests