Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
Hello all.
I run a web server with the LAMP stack, on CentOS 5.9 x86, inside a VMware ESXi 4.0 virtual machine (the VM is configured with VMware hardware level 7).
This CentOS server has been running great for more than 3 years, but in the last three months I've had to power cycle de VM four times because it became unresponsive, apparently because the server reached an Out-Of-Memory condition and the oom-killer function could not manage to save the situation.
The VM has only 1 GB of RAM assigned to it, which has been enough for years because it is very lightly used, and 2 GB of swap.
Here is a pastebin of the relevant logs, can anyone spot something out of the ordinary in them?
http://pastebin.com/raw.php?i=V3Ps2vNC
Why this server, which has been running trouble-free for more than three years, is now having those problems? What could I do to stop the problem from happening again?
Any help is very much appreciated!
I run a web server with the LAMP stack, on CentOS 5.9 x86, inside a VMware ESXi 4.0 virtual machine (the VM is configured with VMware hardware level 7).
This CentOS server has been running great for more than 3 years, but in the last three months I've had to power cycle de VM four times because it became unresponsive, apparently because the server reached an Out-Of-Memory condition and the oom-killer function could not manage to save the situation.
The VM has only 1 GB of RAM assigned to it, which has been enough for years because it is very lightly used, and 2 GB of swap.
Here is a pastebin of the relevant logs, can anyone spot something out of the ordinary in them?
http://pastebin.com/raw.php?i=V3Ps2vNC
Why this server, which has been running trouble-free for more than three years, is now having those problems? What could I do to stop the problem from happening again?
Any help is very much appreciated!
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
What limits do you set on the various memory hungry mysql daemon? Also (you don't say) are you using MyISAM or InnoDB (and have you disabled InnoDB if you're not using it)?
Generally with virtualised nodes, look at the system from OUTSIDE the vm - see what's grabbing memory and what's not at the ESX level.
Generally with virtualised nodes, look at the system from OUTSIDE the vm - see what's grabbing memory and what's not at the ESX level.
-
- Posts: 10642
- Joined: 2005/08/05 15:19:54
- Location: Northern Illinois, USA
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
5.9 is not secure. yum update to 5.11.
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
I believe I have set MySQL to use InnoDB as default storage engine, nevertheless several databases currently in this server's MySQL service are in MyISAM format because the script used to create them specifically had the command to create them with the MyISAM engine.aks wrote:What limits do you set on the various memory hungry mysql daemon? Also (you don't say) are you using MyISAM or InnoDB (and have you disabled InnoDB if you're not using it)?
Generally with virtualised nodes, look at the system from OUTSIDE the vm - see what's grabbing memory and what's not at the ESX level.
This is my MySQL configuration file:
Code: Select all
$ cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
default-storage-engine = innodb
innodb_file_per_table = 1
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0
net_read_timeout = 250
net_write_timeout = 250
key_buffer_size = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
max_connections=1000
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
When the VM reaches its "unavailable" failure state, it's CPU consumption (1 vCPU) is seen as high on the host (but the overall CPU consumption on the host, event at those moments, is no more that 30%).
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
How much memory the host has is immaterial unless you propose to assign the guest more RAM - it is only allowed to use what the host allows it to use and no more. Some of those mysql variables will be per thread not per server so you need to multiple the amounts specified on those by the number of concurrent threads to see how much RAM it will use. You might want to read http://dev.mysql.com/doc/refman/5.6/en/memory-use.html to see which ones those are and adjust them appropriately. There is no guarantee that the things that the OOM killer picks on are the actual culprits though it is likely that it's correct.
If you have a machine with 1GB RAM and 2GB swap and it hits OOM then it will thrash like crazy and the response time will be abysmal - it means you have commited 3GB memory and it's used all RAM and spilled over into swap and used all that too. The proportion of RAM:swap should be more like the reverse for decent response.
If you have a machine with 1GB RAM and 2GB swap and it hits OOM then it will thrash like crazy and the response time will be abysmal - it means you have commited 3GB memory and it's used all RAM and spilled over into swap and used all that too. The proportion of RAM:swap should be more like the reverse for decent response.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
The point is I know the server has reached the Out-Of-Memory condition, but I am unsure as to whether the culprit is MySQL or the Apache+PHP combo. Can you spot the culprit by glancing over the logs I posted in the initial message?TrevorH wrote:How much memory the host has is immaterial unless you propose to assign the guest more RAM - it is only allowed to use what the host allows it to use and no more. Some of those mysql variables will be per thread not per server so you need to multiple the amounts specified on those by the number of concurrent threads to see how much RAM it will use. You might want to read http://dev.mysql.com/doc/refman/5.6/en/memory-use.html to see which ones those are and adjust them appropriately. There is no guarantee that the things that the OOM killer picks on are the actual culprits though it is likely that it's correct.
Do you think enabling the sysstat daemon, and then using sar after the next "incident", will help to shed some light into the problem? Any recommendations about using sysstat + sar?
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
It's not possible to tell from the logs you posted since both mysqld and httpd are being killed. Even sysstat will not tell you which task is using the memory, just overall system usage. I suggest that you keep a periodic eye on 'top' to see what's the culprit, sort the list by 'M' for memory.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
I've done some google searches, and it seems the pidstat command could help me to pinpoint the culprit of the OOM condition.TrevorH wrote:Even sysstat will not tell you which task is using the memory, just overall system usage.
However, it seems the pidstat command only got into the sysstat package in version 7.1.4, whereas my CentOS 5.9 only has version 7.0.2 available:
Code: Select all
$ yum info sysstat
Loaded plugins: fastestmirror
base | 1.1 kB 00:00
epel | 3.7 kB 00:00
extras | 2.1 kB 00:00
rpmforge | 1.9 kB 00:00
updates | 1.9 kB 00:00
Available Packages
Name : sysstat
Arch : i386
Version : 7.0.2
Release : 13.el5
Size : 184 k
Repo : base
Summary : The sar and iostat system monitoring commands.
URL : http://perso.orange.fr/sebastien.godard/
License : GPL
Description: This package provides the sar and iostat commands for Linux. Sar and
: iostat enable system monitoring of disk, network, and other IO
: activity.
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
You cannot and nor should you run 5.9 in any case. Run yum update to get yourself up to date which is 5.11. The top utility can tell you about memory usage in the same way that pidstat can - both only report on what's happening 'now' not from historical data so you can't use either post-event.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Re: Out-Of-Memory in LAMP server - CentOS 5.9 x86 - Why now?
Yes, upgrading to 5.11 is on the list of to-do things. But it involves paperwork and bureaucracy, so it takes time.TrevorH wrote:You cannot and nor should you run 5.9 in any case. Run yum update to get yourself up to date which is 5.11. The top utility can tell you about memory usage in the same way that pidstat can - both only report on what's happening 'now' not from historical data so you can't use either post-event.
Meanwhile, I see the "atop" package can be useful: it seems it has the capacity to detail resource consumption per process, and also can gather historical data. What do you think about it?
Code: Select all
$ yum info atop
Loaded plugins: fastestmirror
Available Packages
Name : atop
Arch : i386
Version : 1.27
Release : 2.el5
Size : 119 k
Repo : epel
Summary : An advanced interactive monitor to view the load on system and process level
URL : http://www.atcomputing.nl/Tools/atop/
License : GPLv2+
Description: An advanced interactive monitor for Linux-systems to view the load on
: system-level and process-level.
: The command atop has some major advantages compared to other
: performance-monitors:
: - Resource consumption by all processes
: - Utilization of all relevant resources
: - Permanent logging of resource utilization
: - Highlight critical resources
: - Watch activity only
: - Watch deviations only
: - Accumulated process activity per user
: - Accumulated process activity per program
: For more informations: http://www.atcomputing.nl/Tools/atop
: The package does not make use of the patches available at
: http://www.atcomputing.nl/Tools/atop/kernpatch.html
Last edited by InitOrNot on 2015/08/16 13:13:29, edited 1 time in total.