incorrect memory usage in cento 6?

Issues related to applications and software problems
Post Reply
Abdussamad
Posts: 3
Joined: 2011/08/29 08:15:58

incorrect memory usage in cento 6?

Post by Abdussamad » 2011/08/29 08:26:02

Hello

I'd like to say right of the bat that I am familiar with how Linux reports memory usage. This is about my new centos 6 system with has just a few dozen apache processes running and is reporting 3.6GB of used memory (excluding buffers and cache).
[code]
[root@serve3 ~]# free -m
total used free shared buffers cached
Mem: 7860 5174 2686 0 126 1299
-/+ buffers/cache: 3748 4112
Swap: 10047 0 10047
[root@serve3 ~]# ps -eo rss |awk '{sum +=$1}; END {print sum/1024/1024}'
1.22688

[/code]

That's free -m output and as you can see in the second line its reporting 3.6GB of memory used with barely anything running!. The PS output shows that only 1GB or so of resident memory is being used by processes. So what's going on? Am I missing something obvious or is free buggy? Or is something else wrong with my system?

regards,
Abdussamad

Abdussamad
Posts: 3
Joined: 2011/08/29 08:15:58

Re: incorrect memory usage in cento 6?

Post by Abdussamad » 2011/08/29 13:04:34

The problem I am facing is the dentry_cache is taking up a lot memory. On the order of 3GB or more. It looks like a bug in the kernel and is the same as the one reported here:

http://comments.gmane.org/gmane.linux.redhat.release.rhel6/332

If anyone has any idea how to fix this permanently please post. I have cleared all caches using this command and that helps but is no permanent solution.

sync && echo 2 >/proc/sys/vm/drop_caches

bryanck
Posts: 5
Joined: 2011/09/04 19:15:37

Re: incorrect memory usage in cento 6?

Post by bryanck » 2011/09/04 19:27:10

I am also experiencing some oddly high memory usage numbers being reported by top and vmstat, and am getting out of memory errors in my application. FWIW, it runs fine on CentOS 5.6.

I am testing a system with Hadoop running on CentOS 6. The virtual memory usage for the Hadoop processes in top shows as 2.5 times the amount I set for the maximum heap usage for the process. If I set 4GB max heap, the process shows as 10GB virtual memory in use. For other java processes I am seeing up to 12GB virt when the max heap is set to only 2GB.

This does not seem to be simply a problem with the memory usage being reported differently, as my Hadoop jobs are running out of memory and will not run. I am using the Sun/Oracle JVM (1.6.0_26) as recommended for Hadoop.

I have applied all yum updates as of today (9/4), which included a kernel patch, but that did not solve my issue.

bryanck
Posts: 5
Joined: 2011/09/04 19:15:37

Re: incorrect memory usage in cento 6?

Post by bryanck » 2011/09/04 23:52:46

I did some more research on my issue. So RHEL 6 (and CentOS 6) includes a new version of glibc which has a new per-thread "arena" memory allocator. This will allocate a large amount of virtual memory, especially if you are running an app with many threads (Java servers tend to have many threads). Thus I was seeing crazy amounts of virtual memory being allocated for my Hadoop processes (which are Java). This can be mitigated to some extend using the MALLOC_ARENA_MAX environment variable, but in my case this was not sufficient and I will be reverting back to CentOS 5.6 until this (massive) change is sorted out.

pjwelsh
Posts: 2632
Joined: 2007/01/07 02:18:02
Location: Central IL USA

incorrect memory usage in cento 6?

Post by pjwelsh » 2011/09/05 01:58:44

Thank you very much for reporting back! I'm sure you are not the only one who is scratching your head about this. In fact, your info is throwing some caution into my CentOS 5.6 -> 6.x hardware/software change for some Tomcat installs...

bryanck
Posts: 5
Joined: 2011/09/04 19:15:37

Re: incorrect memory usage in cento 6?

Post by bryanck » 2011/09/05 06:27:21

I should clarify, the new allocator allocates extra address space per thread, but doesn't actually allocate memory. Supposedly no more memory is being used, but I am getting out of memory errors despite this claim.

bryanck
Posts: 5
Joined: 2011/09/04 19:15:37

Re: incorrect memory usage in cento 6?

Post by bryanck » 2011/09/05 07:58:22

I actually resolved my memory issues by setting MALLOC_ARENA_MAX=4, and increasing the nproc limits in limits.conf. Apparently the nproc amount has been dramatically reduced in CentOS 6 so I needed to increase that also. So I'm giving CentOS 6 another chance. ;)

Post Reply