Program allocates more memory than system has (overcommit_memory=2)

Issues related to software problems
Post Reply
aleks_k
Posts: 3
Joined: 2009/07/02 19:23:59

Program allocates more memory than system has (overcommit_memory=2)

Post by aleks_k » 2009/07/02 19:50:09

The short story:
system with 3Gb RAM, 0 swap and overcommit_memory=2 allows application to allocate ~10Gb of virtual space which almost hangs the system.

The long story:
I'm running CentOS 4.7 as a Xen VM in CentOS 5.3 instance.
VM has 3Gb of memory and 2VCPUs assigned to it.
I'm trying to reproduce an occasional problem of running out of memory on another (production) server.
I have turned swap off for this purpose and created a simple java application which allocates memory (int arrays in an infinite loop untill OutOfMemoryError is thrown). On OOM the test program prints stacktrace and waits for user input from console to exit (this is done to let me check system status before test program releases allocated memory).

I also changed default value of 0 in /proc/sys/vm/overcommit_memory to 2 so that the system would not allow to request more memory than it can serve.

When I give my test program "normal" amount of memory, say 100Mb (java -Xmx100m OOMTest) it works nice and eventually throws expected OutOfMemoryError.
When I give my test program too much memory, say 9Gb (java -Xmx9000m OOMTest), it hangs after allocating around 2Gb of memory (I print a point on screen with allocation of every 10Mb).
From another console I run following diagnotic commands:
#free
total used free shared buffers cached
Mem: 3145728 3132496 13232 0 168 5812
-/+ buffers/cache: 3126516 19212
Swap: 0 0 0

#top (sorted by CPU)
PID USER PR NI VIRT RES SWAP %MEM SHR S %CPU TIME+ COMMAND
46 root 25 0 0 0 0 0.0 0 R 99.6 12:41.09 [kswapd0]
...

#top (sorted by memory)
PID USER PR NI VIRT RES SWAP %MEM SHR S %CPU TIME+ COMMAND
2678 root 17 0 9296m 2.9g 6.2g 95.9 112 S 0.0 0:51.35 /usr/local/jdk1.6.0_02/bin/java -Xmx9048m OOMTest
...

There are 2 logical (from my point of view of course) questions:
1) why can a process receive so much virtual memory with overcommit_memory set to 2?
2) why kswapd0 is taking 100% cpu even having swap off and how to avoid it?

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Program allocates more memory than system has (overcommit_me

Post by pschaff » 2009/07/02 22:47:41

Patient: Doctor, it hurts when I do this!
Doctor: Don't do that.

Sorry - couldn't resist. :-)

From my naive point of view it seems like a bug that should be reported, but then I'm not a kernel hacker.

aleks_k
Posts: 3
Joined: 2009/07/02 19:23:59

Re: Program allocates more memory than system has (overcommit_memory=2)

Post by aleks_k » 2009/07/05 21:20:47

This problem looks too serious not to have been reported since release of CentOS 4. Let me update my setup to the latest available minor version and also try on other similar VMs on other boxes I have access to and come back with results.

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Program allocates more memory than system has (overcommit_memory=2)

Post by pschaff » 2009/07/06 13:17:13

Sounds like a reasonable approach. Please post the results of your investigations.

aleks_k
Posts: 3
Joined: 2009/07/02 19:23:59

Re: Program allocates more memory than system has (overcommit_memory=2)

Post by aleks_k » 2009/07/06 20:52:40

Tried on a different VM (which is a clone of this VM but hosted on another box) and it behaved as expected:
OS didn't let run java with too much memory reserved:
# jdk1.6/bin/java -Xmx1300m OOMTest
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

After memory request was decreased to affordable amount, I got my OutOfMemoryError as expected:
# jdk1.6/bin/java -Xmx1100m OOMTest
.....................................................................................
java.lang.OutOfMemoryError: Java heap space
at OOMTest.oom(OOMTest.java:18)
at OOMTest.main(OOMTest.java:40)

So it looks like overcommit_memory=2 works as designed in my case and the change of this paramerter just didn't apply somehow to the 1st VM. I couldn't connect today to that VM through ssh (although connected fine via "xm console"), so VM itself might be out of order.

The 2nd question is still open: why kswapd0 work so hard even w/o swap available? Can this process be terminated? (kill doesn't help).

pjwelsh
Posts: 2598
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: Program allocates more memory than system has (overcommit_memory=2)

Post by pjwelsh » 2009/07/07 15:26:53

I thought that overcommit_memory=2 would give swap plus 50% (by default) of physical RAM. So, 0 + 1.5GB = 1.5GB of allocated (not vsz) memory. The 2GB hang does not look to far from correct. But, I often miss details of what is wrong.

Edit: As an addition, the "Committed_AS:" from /proc/meminfo seems to be the important an important total number.

Post Reply

Return to “CentOS 4 - Software Support”