java hits oom-killer in openjdk 1.8.0.144-0.b01

Issues related to applications and software problems
akuusela
Posts: 1
Joined: 2017/10/04 06:53:30

java hits oom-killer in openjdk 1.8.0.144-0.b01

Postby akuusela » 2017/10/04 07:48:53

Hi,

After updating to java-1.8.0-openjdk version 1.8.0.144-0.b01 on CentOS 7 we have experienced an irregular crash in which oom-killer terminates the java process after its memory consumption skyrockets beyond configured bounds. The jvm runs the Wildfly application server 10.1.0 or 9.0.2. Rolling back to 1.8.0.141-2.b16 is enough to avoid the issue.

Code: Select all

Oct  3 23:43:54 staging kernel: Out of memory: Kill process 2390 (java) score 783 or sacrifice child
Oct  3 23:43:54 staging kernel: Killed process 2390 (java) total-vm:12925660kB, anon-rss:6265036kB, file-rss:0kB, shmem-rss:0kB

The jvm is started with parameters -Xmx4096m and -XX:MaxMetaspaceSize=256m. I do not know exactly how quickly the memory usage ramped up but it wasn't visible in /proc/meminfo on a five minute sampling.

Regards,
Antti

LudicrousSpeed
Posts: 3
Joined: 2017/10/11 21:45:38

Re: java hits oom-killer in openjdk 1.8.0.144-0.b01

Postby LudicrousSpeed » 2017/10/11 22:19:02

I just wanted to post here and say that we are seeing the exact same issue with java running soap web services.

It is pretty random and has happened a few times now. We have had profilers on during the events and the interesting ting is memory is being consumed but when attaching visualvm to it the heap and stuff is not growing. Does anyone have more information tied to this? We will try rolling back to 141 I guess.

topiaruss
Posts: 1
Joined: 2017/10/13 12:32:46

Re: java hits oom-killer in openjdk 1.8.0.144-0.b01

Postby topiaruss » 2017/10/13 12:37:36

I consult to a UK government agency.
We hit the same issue on this version, and have rolled back to 141.b16 as a temporary fix.
The issue seems to be in non-heap memory -- metaspace, but this was hard to diagnose at any level of resolution. Just passing that on as a tip.
It is likely that a formal bug report will be posted in the near future.
--r

User avatar
TrevorH
Forum Moderator
Posts: 21002
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: java hits oom-killer in openjdk 1.8.0.144-0.b01

Postby TrevorH » 2017/10/13 13:09:56

Since this is an unmodifed RHEL package that's only been rebuilt for CentOS, you'll need to report any problems on bugzilla.redhat.com to get anything fixed. You can report it on bugs.centos.org but since CentOS only rebuild what RHEL supply, to get anything fixed it needs reporting upstream too.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

LudicrousSpeed
Posts: 3
Joined: 2017/10/11 21:45:38

Re: java hits oom-killer in openjdk 1.8.0.144-0.b01

Postby LudicrousSpeed » 2017/10/16 15:58:05

When profiling I did not see metaspace grow either. It appears to be non heap and very hard to pinpoint.

Do we need an account with Redhat to post the bug? Does anyone have a link to the process?

User avatar
TrevorH
Forum Moderator
Posts: 21002
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: java hits oom-killer in openjdk 1.8.0.144-0.b01

Postby TrevorH » 2017/10/16 16:08:07

If there's a bugzilla.redhat.com entry for this then you just need an account on bugzilla to comment on it. If it requires a Redhat support case then you need a subscription from an account with the ability to report problems (the cheap ones are "self support").
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

LudicrousSpeed
Posts: 3
Joined: 2017/10/11 21:45:38

Re: java hits oom-killer in openjdk 1.8.0.144-0.b01

Postby LudicrousSpeed » 2017/10/18 17:17:44

Thanks we have an account so I think I can make a case. I will try to get this started.