[SOLVED] java using 100% cpu on CentOS/RHEL 6
Posted: 2012/07/02 13:57:17
I have a java app running on CentOS 6.0. It always runs in background via cron. Sometimes this app goes into wait state while using 100% cpu.
My java version is :
[code]
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.4) (rhel-1.21.b17.el6-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)
[/code]
Other symptoms are :
a. One thread of the process seems to be in a loop waiting for something. When traced using strace, it shows following o/p continuously :
[code]
> futex(0x7fb8000ac728, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb8000ac754, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1340347489,> 822867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[/code]
b. It seems like the process has finished working, looking at the files it is using. Only few files are remaining. The output of 'ls /proc/pid/fd/ shows :
[code]
> lr-x------ 1 root root 64 Jun 22 13:13 0 -> pipe:[77107601]
> l-wx------ 1 root root 64 Jun 22 13:13 1 -> pipe:[77120162]
> l-wx------ 1 root root 64 Jun 22 13:13 2 -> /var/log/mithi/mcs/agent_account_mailstore_exceed_limit.sh.log
> lr-x------ 1 root root 64 Jun 22 13:13 3 -> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
[/code]
More specifically, are there any known problems in running openjdk based Java processes in background on CentOS/RHEL 6?
I'm able to simulate the problem with a very simple script :
[code]
#!/bin/bash
while [ 1 ]
do
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
sleep 1s
done
[/code]
When the above script is run for about 3 - 4 hours, one or two java processes go in hang state with cpu usage touching 100%.
Has anybody faced similar situation?
Any clues or references would be very helpful.
My java version is :
[code]
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.4) (rhel-1.21.b17.el6-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)
[/code]
Other symptoms are :
a. One thread of the process seems to be in a loop waiting for something. When traced using strace, it shows following o/p continuously :
[code]
> futex(0x7fb8000ac728, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb8000ac754, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1340347489,> 822867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[/code]
b. It seems like the process has finished working, looking at the files it is using. Only few files are remaining. The output of 'ls /proc/pid/fd/ shows :
[code]
> lr-x------ 1 root root 64 Jun 22 13:13 0 -> pipe:[77107601]
> l-wx------ 1 root root 64 Jun 22 13:13 1 -> pipe:[77120162]
> l-wx------ 1 root root 64 Jun 22 13:13 2 -> /var/log/mithi/mcs/agent_account_mailstore_exceed_limit.sh.log
> lr-x------ 1 root root 64 Jun 22 13:13 3 -> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
[/code]
More specifically, are there any known problems in running openjdk based Java processes in background on CentOS/RHEL 6?
I'm able to simulate the problem with a very simple script :
[code]
#!/bin/bash
while [ 1 ]
do
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
sleep 1s
done
[/code]
When the above script is run for about 3 - 4 hours, one or two java processes go in hang state with cpu usage touching 100%.
Has anybody faced similar situation?
Any clues or references would be very helpful.