System hangs; potentially Java related

[novosirj]

Hi all,

I'm experiencing odd hangs and unresponsiveness reading the /proc tree for certain java processes. I am suspecting that they are causing hangs for the rest of the system as well. I'm also getting some errors about an unstable clock source (may be related to the hangs though), and stuff like that. The CPU, I/O, and memory utilization all seem well within limits. We do use cgroups and numad (just trying to think of things that are at all different about our system).

The system is a Dell PowerEdge R820 4x8 E5-4640 w/512GB RAM. Home directories are on NFS, scratch is on Lustre. There are about 25 people logged in doing various things, some of them running CPU or memory intensive processes, some just using a desktop.

Any idea where I'd start looking for trouble other than the afforementioned non-bottlenecks? I've seen it suggested that software interrupts could be the problem.

Noteworthy is that most Java process appear to be using OpenJDK. I imagine the garbage collection mechanism isn't as fancy in that one as Oracle, and some of these Java processes are many GB in heap size.

Thanks for any tips you can give me.

[Leigh507]

Hi,

These are potentially down toa bug in the TZdata package that cannot handle the leap second that was added on 30/06/2015 at 23:59:60.

You need to use the following web page which will hopefully help you through the issue...
http://www.pythian.com/blog/handling-th ... ond-linux/

Cheers,
Leigh

[novosirj]

That was probably not my issue -- I was well beyond all of those version numbers, unless I'm misunderstanding things. Fixed by magic. Oh well. :-\