[SOLVED] java using 100% cpu on CentOS/RHEL 6

Issues related to applications and software problems
Post Reply
amol
Posts: 3
Joined: 2012/06/22 07:29:46

[SOLVED] java using 100% cpu on CentOS/RHEL 6

Post by amol » 2012/07/02 13:57:17

I have a java app running on CentOS 6.0. It always runs in background via cron. Sometimes this app goes into wait state while using 100% cpu.

My java version is :

[code]
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.4) (rhel-1.21.b17.el6-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)
[/code]
Other symptoms are :

a. One thread of the process seems to be in a loop waiting for something. When traced using strace, it shows following o/p continuously :

[code]
> futex(0x7fb8000ac728, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb8000ac754, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1340347489,> 822867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[/code]
b. It seems like the process has finished working, looking at the files it is using. Only few files are remaining. The output of 'ls /proc/pid/fd/ shows :

[code]
> lr-x------ 1 root root 64 Jun 22 13:13 0 -> pipe:[77107601]
> l-wx------ 1 root root 64 Jun 22 13:13 1 -> pipe:[77120162]
> l-wx------ 1 root root 64 Jun 22 13:13 2 -> /var/log/mithi/mcs/agent_account_mailstore_exceed_limit.sh.log
> lr-x------ 1 root root 64 Jun 22 13:13 3 -> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
[/code]
More specifically, are there any known problems in running openjdk based Java processes in background on CentOS/RHEL 6?

I'm able to simulate the problem with a very simple script :

[code]
#!/bin/bash

while [ 1 ]
do
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
sleep 1s
done
[/code]
When the above script is run for about 3 - 4 hours, one or two java processes go in hang state with cpu usage touching 100%.

Has anybody faced similar situation?
Any clues or references would be very helpful.

pjwelsh
Posts: 2632
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: java using 100% cpu on CentOS/RHEL 6

Post by pjwelsh » 2012/07/02 14:52:23

Hit by the [url=http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/]leap-second bug[/url]?

[code]service ntpd stop
date `date +"%m%d%H%M%C%y.%S"`
service ntpd start
[/code]

amol
Posts: 3
Joined: 2012/06/22 07:29:46

Re: java using 100% cpu on CentOS/RHEL 6

Post by amol » 2012/07/03 11:16:42

Doesn't seem to be the leap second bug because my problem can be simulated anytime by running following script

[code]
#!/bin/bash

while [ 1 ]
do
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
sleep 1s
done
[/code]
However, very likely it seems to be foll bug :

https://bugzilla.kernel.org/show_bug.cgi?id=32922

Is this bug fixed in CentOS 6 kernel?

Any other ideas?

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

[SOLVED] java using 100% cpu on CentOS/RHEL 6

Post by toracat » 2012/07/03 14:54:52

[quote]
However, very likely it seems to be foll bug :

https://bugzilla.kernel.org/show_bug.cgi?id=32922

Is this bug fixed in CentOS 6 kernel?
[/quote]
According to the referenced bugzilla, the patch was:

[code]
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -1886,7 +1886,7 @@ retry:
restart->futex.val = val;
restart->futex.time = abs_time->tv64;
restart->futex.bitset = bitset;
- restart->futex.flags = flags;
+ restart->futex.flags = flags | FLAGS_HAS_TIMEOUT;

ret = -ERESTART_RESTARTBLOCK;
[/code]

The corresponding part of the CentOS/RHEL kernel is:

[code]
restart->futex.val = val;
restart->futex.time = abs_time->tv64;
restart->futex.bitset = bitset;
restart->futex.flags = FLAGS_HAS_TIMEOUT;

if (fshared)
restart->futex.flags |= FLAGS_SHARED;
if (clockrt)
restart->futex.flags |= FLAGS_CLOCKRT;

ret = -ERESTART_RESTARTBLOCK;
[/code]

Due to the difference between the RHEL kernel and that from kernel.org, it is not easy to tell unambiguously that the patch has been applied. But it seems that way.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: java using 100% cpu on CentOS/RHEL 6

Post by TrevorH » 2012/07/03 15:08:03

Looks like it here too, the RH code sets FLAGS_HAS_TIMEOUT for everything then adds another bit setting on top but the important part is the FLAGS_HAS_TIMEOUT is there.

amol
Posts: 3
Joined: 2012/06/22 07:29:46

Re: [SOLVED] java using 100% cpu on CentOS/RHEL 6

Post by amol » 2012/07/27 11:43:35

The problem was solved after upgrading kernel to kernel-2.6.32-220 i.e CentOS 6.2 kernel.

Thanks all for the help.

Post Reply