Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?

[InitOrNot]

Hello all.

I run a web server with the LAMP stack, on CentOS 5.9 x86, inside a VMware ESXi 4.0 virtual machine (the VM is configured with VMware hardware level 7).

This CentOS server has been running great for more than 3 years, but in the last three months I've had to power cycle de VM four times because it became unresponsive, apparently because the server reached an Out-Of-Memory condition and the oom-killer function could not manage to save the situation.

The VM has only 1 GB of RAM assigned to it, which has been enough for years because it is very lightly used, and 2 GB of swap.

Here is a pastebin of the relevant logs, can anyone spot something out of the ordinary in them?

http://pastebin.com/raw.php?i=V3Ps2vNC

Why this server, which has been running trouble-free for more than three years, is now having those problems? What could I do to stop the problem from happening again?

Any help is very much appreciated!

[aks]

What limits do you set on the various memory hungry mysql daemon? Also (you don't say) are you using MyISAM or InnoDB (and have you disabled InnoDB if you're not using it)?
Generally with virtualised nodes, look at the system from OUTSIDE the vm - see what's grabbing memory and what's not at the ESX level.

[gerald_clark]

5.9 is not secure. yum update to 5.11.

[InitOrNot]

What limits do you set on the various memory hungry mysql daemon? Also (you don't say) are you using MyISAM or InnoDB (and have you disabled InnoDB if you're not using it)?
Generally with virtualised nodes, look at the system from OUTSIDE the vm - see what's grabbing memory and what's not at the ESX level.

I believe I have set MySQL to use InnoDB as default storage engine, nevertheless several databases currently in this server's MySQL service are in MyISAM format because the script used to create them specifically had the command to create them with the MyISAM engine.

This is my MySQL configuration file:

Code: Select all

$ cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
default-storage-engine = innodb
innodb_file_per_table = 1
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0

net_read_timeout = 250
net_write_timeout = 250
key_buffer_size = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
max_connections=1000

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Looking a the VM from the outside, I see nothing unusual. The VMware ESX host has plenty of free RAM, free hard disk space is not under pressure, and CPU consumption is about 20%.

When the VM reaches its "unavailable" failure state, it's CPU consumption (1 vCPU) is seen as high on the host (but the overall CPU consumption on the host, event at those moments, is no more that 30%).

[TrevorH]

How much memory the host has is immaterial unless you propose to assign the guest more RAM - it is only allowed to use what the host allows it to use and no more. Some of those mysql variables will be per thread not per server so you need to multiple the amounts specified on those by the number of concurrent threads to see how much RAM it will use. You might want to read http://dev.mysql.com/doc/refman/5.6/en/memory-use.html to see which ones those are and adjust them appropriately. There is no guarantee that the things that the OOM killer picks on are the actual culprits though it is likely that it's correct.

If you have a machine with 1GB RAM and 2GB swap and it hits OOM then it will thrash like crazy and the response time will be abysmal - it means you have commited 3GB memory and it's used all RAM and spilled over into swap and used all that too. The proportion of RAM:swap should be more like the reverse for decent response.

[InitOrNot]

How much memory the host has is immaterial unless you propose to assign the guest more RAM - it is only allowed to use what the host allows it to use and no more. Some of those mysql variables will be per thread not per server so you need to multiple the amounts specified on those by the number of concurrent threads to see how much RAM it will use. You might want to read http://dev.mysql.com/doc/refman/5.6/en/memory-use.html to see which ones those are and adjust them appropriately. There is no guarantee that the things that the OOM killer picks on are the actual culprits though it is likely that it's correct.

The point is I know the server has reached the Out-Of-Memory condition, but I am unsure as to whether the culprit is MySQL or the Apache+PHP combo. Can you spot the culprit by glancing over the logs I posted in the initial message?

Do you think enabling the sysstat daemon, and then using sar after the next "incident", will help to shed some light into the problem? Any recommendations about using sysstat + sar?

[TrevorH]

It's not possible to tell from the logs you posted since both mysqld and httpd are being killed. Even sysstat will not tell you which task is using the memory, just overall system usage. I suggest that you keep a periodic eye on 'top' to see what's the culprit, sort the list by 'M' for memory.

[InitOrNot]

Even sysstat will not tell you which task is using the memory, just overall system usage.

I've done some google searches, and it seems the pidstat command could help me to pinpoint the culprit of the OOM condition.

However, it seems the pidstat command only got into the sysstat package in version 7.1.4, whereas my CentOS 5.9 only has version 7.0.2 available:

Code: Select all

$ yum info sysstat
Loaded plugins: fastestmirror
base                                                                                                                    | 1.1 kB     00:00     
epel                                                                                                                    | 3.7 kB     00:00     
extras                                                                                                                  | 2.1 kB     00:00     
rpmforge                                                                                                                | 1.9 kB     00:00     
updates                                                                                                                 | 1.9 kB     00:00     
Available Packages
Name       : sysstat
Arch       : i386
Version    : 7.0.2
Release    : 13.el5
Size       : 184 k
Repo       : base
Summary    : The sar and iostat system monitoring commands.
URL        : http://perso.orange.fr/sebastien.godard/
License    : GPL
Description: This package provides the sar and iostat commands for Linux. Sar and
           : iostat enable system monitoring of disk, network, and other IO
           : activity.

Any idea about how could I get a more recent version of the sysstat package into my CentOS 5.9 system?

[TrevorH]

You cannot and nor should you run 5.9 in any case. Run yum update to get yourself up to date which is 5.11. The top utility can tell you about memory usage in the same way that pidstat can - both only report on what's happening 'now' not from historical data so you can't use either post-event.

[InitOrNot]

You cannot and nor should you run 5.9 in any case. Run yum update to get yourself up to date which is 5.11. The top utility can tell you about memory usage in the same way that pidstat can - both only report on what's happening 'now' not from historical data so you can't use either post-event.

Yes, upgrading to 5.11 is on the list of to-do things. But it involves paperwork and bureaucracy, so it takes time.

Meanwhile, I see the "atop" package can be useful: it seems it has the capacity to detail resource consumption per process, and also can gather historical data. What do you think about it?

Code: Select all

$ yum info atop
Loaded plugins: fastestmirror
Available Packages
Name       : atop
Arch       : i386
Version    : 1.27
Release    : 2.el5
Size       : 119 k
Repo       : epel
Summary    : An advanced interactive monitor to view the load on system and process level
URL        : http://www.atcomputing.nl/Tools/atop/
License    : GPLv2+
Description: An advanced interactive monitor for Linux-systems to view the load on
           : system-level and process-level.
           : The command atop has some major advantages compared to other
           : performance-monitors:
           :    - Resource consumption by all processes
           :    - Utilization of all relevant resources
           :    - Permanent logging of resource utilization
           :    - Highlight critical resources
           :    - Watch activity only
           :    - Watch deviations only
           :    - Accumulated process activity per user
           :    - Accumulated process activity per program
           : For more informations: http://www.atcomputing.nl/Tools/atop
           : The package does not make use of the patches available at
           : http://www.atcomputing.nl/Tools/atop/kernpatch.html