[SOLVED] 6.2: Where has all the memory gone?
Posted: 2012/01/06 19:16:16
On a fresh 6.2 install, I keep running into out of memory errors after about 24 hours of uptime.
[code]
total used free shared buffers cached
Mem: 1030888 753000 277888 0 2572 12588
-/+ buffers/cache: [b][u]737840[/u][/b] 293048
Swap: 0 0 0
[/code]
This is basically a minimal install at the moment (building it into a development system for myself) -- there is virtually [i]nothing[/i] running on this machine except for a couple daemons I'm testing (Kerberos is the main one). It starts out fine, but then overnight (when even less is going on) the buffer count mysteriously balloons to what you see above (or higher), and I have to reboot it almost every morning because nothing seems to remedy this and I can't run anything besides basic shell commands.
There was [url=https://www.centos.org/modules/newbb/viewtopic.php?topic_id=32988]a previous thread[/url] with a similar problem, but none of the remedies in that thread have worked. "echo 3 >/proc/sys/vm/drop_caches" only regains about 30M or so. dentry does not appear to be the issue in this case:
[code]
Active / Total Objects (% used) : 874616 / 888699 (98.4%)
Active / Total Slabs (% used) : 8006 / 8006 (100.0%)
Active / Total Caches (% used) : 94 / 182 (51.6%)
Active / Total Size (% used) : 27158.44K / 28482.08K (95.4%)
Minimum / Average / Maximum Object : 0.01K / 0.03K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
333576 333173 99% 0.03K 2952 113 11808K size-32
480704 480587 99% 0.02K 2368 203 9472K avtab_node
5742 5607 97% 0.34K 522 11 2088K inode_cache
8903 7399 83% 0.13K 307 29 1228K dentry
372 367 98% 2.00K 186 2 744K size-2048
9100 9072 99% 0.07K 182 50 728K sysfs_dir_cache
114 114 100% 3.00K 57 2 456K biovec-256
9492 7365 77% 0.04K 113 84 452K selinux_inode_security
6834 1059 15% 0.05K 102 67 408K buffer_head
576 380 65% 0.63K 96 6 384K ext4_inode_cache
[/code]
Like I said, there is virtually nothing running on this box. Definitely not enough to be consuming 70% of the RAM.
[code]
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Jan04 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Jan04 0:00 \_ [migration/0]
root 4 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Jan04 0:00 \_ [migration/0]
root 6 0.0 0.0 0 0 ? S Jan04 0:00 \_ [watchdog/0]
root 7 0.0 0.0 0 0 ? S Jan04 0:00 \_ [events/0]
root 8 0.0 0.0 0 0 ? S Jan04 0:00 \_ [cpuset]
root 9 0.0 0.0 0 0 ? S Jan04 0:00 \_ [khelper]
root 10 0.0 0.0 0 0 ? S Jan04 0:00 \_ [netns]
root 11 0.0 0.0 0 0 ? S Jan04 0:00 \_ [async/mgr]
root 12 0.0 0.0 0 0 ? S Jan04 0:00 \_ [pm]
root 13 0.0 0.0 0 0 ? S Jan04 0:00 \_ [sync_supers]
root 14 0.0 0.0 0 0 ? S Jan04 0:00 \_ [bdi-default]
root 15 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kintegrityd/0]
root 16 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kblockd/0]
root 17 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kacpid]
root 18 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kacpi_notify]
root 19 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kacpi_hotplug]
root 20 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ata/0]
root 21 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ata_aux]
root 22 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ksuspend_usbd]
root 23 0.0 0.0 0 0 ? S Jan04 0:00 \_ [khubd]
root 24 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kseriod]
root 25 0.0 0.0 0 0 ? S Jan04 0:00 \_ [md/0]
root 26 0.0 0.0 0 0 ? S Jan04 0:00 \_ [md_misc/0]
root 27 0.0 0.0 0 0 ? S Jan04 0:00 \_ [khungtaskd]
root 28 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kswapd0]
root 29 0.0 0.0 0 0 ? SN Jan04 0:00 \_ [ksmd]
root 30 0.0 0.0 0 0 ? S Jan04 0:00 \_ [aio/0]
root 31 0.0 0.0 0 0 ? S Jan04 0:00 \_ [crypto/0]
root 36 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kthrotld/0]
root 37 0.0 0.0 0 0 ? S Jan04 0:00 \_ [pciehpd]
root 39 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kpsmoused]
root 40 0.0 0.0 0 0 ? S Jan04 0:00 \_ [usbhid_resumer]
root 70 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kstriped]
root 270 0.0 0.0 0 0 ? S Jan04 0:00 \_ [scsi_eh_0]
root 271 0.0 0.0 0 0 ? S Jan04 0:00 \_ [scsi_eh_1]
root 280 0.0 0.0 0 0 ? S Jan04 0:00 \_ [scsi_eh_2]
root 281 0.0 0.0 0 0 ? S Jan04 0:00 \_ [vmw_pvscsi_wq_2]
root 333 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 347 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-0-8]
root 348 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 607 0.0 0.0 0 0 ? S Jan04 0:06 \_ [vmmemctl]
root 719 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 722 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 725 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 728 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 729 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 731 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 782 0.0 0.0 0 0 ? S Jan04 0:00 \_ [flush-253:3]
root 803 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/sda1-8]
root 804 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 805 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-2-8]
root 806 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 807 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-4-8]
root 808 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 809 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-1-8]
root 810 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 811 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-3-8]
root 812 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 813 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-5-8]
root 814 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 815 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-6-8]
root 816 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 850 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kauditd]
root 1 0.0 0.1 2864 1280 ? Ss Jan04 0:02 /sbin/init
root 422 0.0 0.1 2884 1128 ? S<s Jan04 0:00 /sbin/udevd -d
root 1248 0.0 0.1 3400 1784 ? S< Jan04 0:00 \_ /sbin/udevd -d
root 1249 0.0 0.1 3404 1784 ? S< Jan04 0:00 \_ /sbin/udevd -d
root 1072 0.0 0.0 12904 764 ? S<sl Jan04 0:00 auditd
root 1090 0.0 0.0 2028 516 ? Ss Jan04 0:00 /sbin/portreserve
root 1097 0.0 0.1 35536 1360 ? Sl Jan04 0:02 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
root 1131 0.0 0.0 8452 944 ? Ss Jan04 0:00 /usr/sbin/sshd
root 1279 0.0 0.3 11396 3280 ? S Jan04 0:01 \_ sshd: ck [priv]
ck 1282 0.0 0.1 11512 1772 ? S Jan04 0:04 | \_ sshd: ck@pts/1
ck 1283 0.0 0.1 5228 1860 pts/1 Ss Jan04 0:01 | \_ -bash
root 18201 0.0 0.1 5388 1476 pts/1 S 12:43 0:00 | \_ su -
root 18205 0.1 0.1 5096 1680 pts/1 S 12:43 0:00 | \_ -bash
root 18290 0.0 0.0 4872 1020 pts/1 R+ 12:53 0:00 | \_ ps auwfx
root 15244 0.0 0.3 11396 3276 ? S Jan04 0:01 \_ sshd: ck [priv]
ck 15247 0.0 0.1 11544 1744 ? S Jan04 0:04 \_ sshd: ck@pts/0
ck 15248 0.0 0.1 5228 1848 pts/0 Ss+ Jan04 0:01 \_ -bash
root 1207 0.0 0.2 12420 2464 ? Ss Jan04 0:00 /usr/libexec/postfix/master
postfix 1216 0.0 0.2 12564 2368 ? S Jan04 0:00 \_ qmgr -l -t fifo -u
postfix 18179 0.0 0.2 12496 2440 ? S 12:01 0:00 \_ pickup -l -t fifo -u
root 1217 0.0 0.1 5888 1268 ? Ss Jan04 0:02 crond
root 1226 0.0 0.2 22884 2920 ? Ss Jan04 0:01 smbd -D
root 1230 0.0 0.1 22884 1396 ? S Jan04 0:00 \_ smbd -D
root 1242 0.0 0.0 1980 508 tty2 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty2
root 1244 0.0 0.0 1980 504 tty3 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty3
root 1246 0.0 0.0 1980 508 tty4 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty4
root 1250 0.0 0.0 1980 512 tty5 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty5
root 1252 0.0 0.0 1980 512 tty6 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty6
root 1532 0.0 0.4 7532 4140 ? Ss Jan04 0:00 /usr/sbin/kadmind -P /var/run/kadmind.pid
root 1547 0.0 0.7 10312 7496 ? Ss Jan04 0:02 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
[/code]
This is a test/dev system, but suffice it to say this is making me a bit nervous to attempt any sort of production deployment.
So why does the kernel seem to be eating all the RAM?
[code]
Linux dev-box 2.6.32-220.2.1.el6.i686 #1 SMP Thu Dec 22 18:50:52 GMT 2011 i686 i686 i386 GNU/Linux
[/code]
[code]
total used free shared buffers cached
Mem: 1030888 753000 277888 0 2572 12588
-/+ buffers/cache: [b][u]737840[/u][/b] 293048
Swap: 0 0 0
[/code]
This is basically a minimal install at the moment (building it into a development system for myself) -- there is virtually [i]nothing[/i] running on this machine except for a couple daemons I'm testing (Kerberos is the main one). It starts out fine, but then overnight (when even less is going on) the buffer count mysteriously balloons to what you see above (or higher), and I have to reboot it almost every morning because nothing seems to remedy this and I can't run anything besides basic shell commands.
There was [url=https://www.centos.org/modules/newbb/viewtopic.php?topic_id=32988]a previous thread[/url] with a similar problem, but none of the remedies in that thread have worked. "echo 3 >/proc/sys/vm/drop_caches" only regains about 30M or so. dentry does not appear to be the issue in this case:
[code]
Active / Total Objects (% used) : 874616 / 888699 (98.4%)
Active / Total Slabs (% used) : 8006 / 8006 (100.0%)
Active / Total Caches (% used) : 94 / 182 (51.6%)
Active / Total Size (% used) : 27158.44K / 28482.08K (95.4%)
Minimum / Average / Maximum Object : 0.01K / 0.03K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
333576 333173 99% 0.03K 2952 113 11808K size-32
480704 480587 99% 0.02K 2368 203 9472K avtab_node
5742 5607 97% 0.34K 522 11 2088K inode_cache
8903 7399 83% 0.13K 307 29 1228K dentry
372 367 98% 2.00K 186 2 744K size-2048
9100 9072 99% 0.07K 182 50 728K sysfs_dir_cache
114 114 100% 3.00K 57 2 456K biovec-256
9492 7365 77% 0.04K 113 84 452K selinux_inode_security
6834 1059 15% 0.05K 102 67 408K buffer_head
576 380 65% 0.63K 96 6 384K ext4_inode_cache
[/code]
Like I said, there is virtually nothing running on this box. Definitely not enough to be consuming 70% of the RAM.
[code]
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Jan04 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Jan04 0:00 \_ [migration/0]
root 4 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Jan04 0:00 \_ [migration/0]
root 6 0.0 0.0 0 0 ? S Jan04 0:00 \_ [watchdog/0]
root 7 0.0 0.0 0 0 ? S Jan04 0:00 \_ [events/0]
root 8 0.0 0.0 0 0 ? S Jan04 0:00 \_ [cpuset]
root 9 0.0 0.0 0 0 ? S Jan04 0:00 \_ [khelper]
root 10 0.0 0.0 0 0 ? S Jan04 0:00 \_ [netns]
root 11 0.0 0.0 0 0 ? S Jan04 0:00 \_ [async/mgr]
root 12 0.0 0.0 0 0 ? S Jan04 0:00 \_ [pm]
root 13 0.0 0.0 0 0 ? S Jan04 0:00 \_ [sync_supers]
root 14 0.0 0.0 0 0 ? S Jan04 0:00 \_ [bdi-default]
root 15 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kintegrityd/0]
root 16 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kblockd/0]
root 17 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kacpid]
root 18 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kacpi_notify]
root 19 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kacpi_hotplug]
root 20 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ata/0]
root 21 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ata_aux]
root 22 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ksuspend_usbd]
root 23 0.0 0.0 0 0 ? S Jan04 0:00 \_ [khubd]
root 24 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kseriod]
root 25 0.0 0.0 0 0 ? S Jan04 0:00 \_ [md/0]
root 26 0.0 0.0 0 0 ? S Jan04 0:00 \_ [md_misc/0]
root 27 0.0 0.0 0 0 ? S Jan04 0:00 \_ [khungtaskd]
root 28 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kswapd0]
root 29 0.0 0.0 0 0 ? SN Jan04 0:00 \_ [ksmd]
root 30 0.0 0.0 0 0 ? S Jan04 0:00 \_ [aio/0]
root 31 0.0 0.0 0 0 ? S Jan04 0:00 \_ [crypto/0]
root 36 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kthrotld/0]
root 37 0.0 0.0 0 0 ? S Jan04 0:00 \_ [pciehpd]
root 39 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kpsmoused]
root 40 0.0 0.0 0 0 ? S Jan04 0:00 \_ [usbhid_resumer]
root 70 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kstriped]
root 270 0.0 0.0 0 0 ? S Jan04 0:00 \_ [scsi_eh_0]
root 271 0.0 0.0 0 0 ? S Jan04 0:00 \_ [scsi_eh_1]
root 280 0.0 0.0 0 0 ? S Jan04 0:00 \_ [scsi_eh_2]
root 281 0.0 0.0 0 0 ? S Jan04 0:00 \_ [vmw_pvscsi_wq_2]
root 333 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 347 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-0-8]
root 348 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 607 0.0 0.0 0 0 ? S Jan04 0:06 \_ [vmmemctl]
root 719 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 722 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 725 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 728 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 729 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 731 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kdmflush]
root 782 0.0 0.0 0 0 ? S Jan04 0:00 \_ [flush-253:3]
root 803 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/sda1-8]
root 804 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 805 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-2-8]
root 806 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 807 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-4-8]
root 808 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 809 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-1-8]
root 810 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 811 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-3-8]
root 812 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 813 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-5-8]
root 814 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 815 0.0 0.0 0 0 ? S Jan04 0:00 \_ [jbd2/dm-6-8]
root 816 0.0 0.0 0 0 ? S Jan04 0:00 \_ [ext4-dio-unwrit]
root 850 0.0 0.0 0 0 ? S Jan04 0:00 \_ [kauditd]
root 1 0.0 0.1 2864 1280 ? Ss Jan04 0:02 /sbin/init
root 422 0.0 0.1 2884 1128 ? S<s Jan04 0:00 /sbin/udevd -d
root 1248 0.0 0.1 3400 1784 ? S< Jan04 0:00 \_ /sbin/udevd -d
root 1249 0.0 0.1 3404 1784 ? S< Jan04 0:00 \_ /sbin/udevd -d
root 1072 0.0 0.0 12904 764 ? S<sl Jan04 0:00 auditd
root 1090 0.0 0.0 2028 516 ? Ss Jan04 0:00 /sbin/portreserve
root 1097 0.0 0.1 35536 1360 ? Sl Jan04 0:02 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
root 1131 0.0 0.0 8452 944 ? Ss Jan04 0:00 /usr/sbin/sshd
root 1279 0.0 0.3 11396 3280 ? S Jan04 0:01 \_ sshd: ck [priv]
ck 1282 0.0 0.1 11512 1772 ? S Jan04 0:04 | \_ sshd: ck@pts/1
ck 1283 0.0 0.1 5228 1860 pts/1 Ss Jan04 0:01 | \_ -bash
root 18201 0.0 0.1 5388 1476 pts/1 S 12:43 0:00 | \_ su -
root 18205 0.1 0.1 5096 1680 pts/1 S 12:43 0:00 | \_ -bash
root 18290 0.0 0.0 4872 1020 pts/1 R+ 12:53 0:00 | \_ ps auwfx
root 15244 0.0 0.3 11396 3276 ? S Jan04 0:01 \_ sshd: ck [priv]
ck 15247 0.0 0.1 11544 1744 ? S Jan04 0:04 \_ sshd: ck@pts/0
ck 15248 0.0 0.1 5228 1848 pts/0 Ss+ Jan04 0:01 \_ -bash
root 1207 0.0 0.2 12420 2464 ? Ss Jan04 0:00 /usr/libexec/postfix/master
postfix 1216 0.0 0.2 12564 2368 ? S Jan04 0:00 \_ qmgr -l -t fifo -u
postfix 18179 0.0 0.2 12496 2440 ? S 12:01 0:00 \_ pickup -l -t fifo -u
root 1217 0.0 0.1 5888 1268 ? Ss Jan04 0:02 crond
root 1226 0.0 0.2 22884 2920 ? Ss Jan04 0:01 smbd -D
root 1230 0.0 0.1 22884 1396 ? S Jan04 0:00 \_ smbd -D
root 1242 0.0 0.0 1980 508 tty2 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty2
root 1244 0.0 0.0 1980 504 tty3 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty3
root 1246 0.0 0.0 1980 508 tty4 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty4
root 1250 0.0 0.0 1980 512 tty5 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty5
root 1252 0.0 0.0 1980 512 tty6 Ss+ Jan04 0:00 /sbin/mingetty /dev/tty6
root 1532 0.0 0.4 7532 4140 ? Ss Jan04 0:00 /usr/sbin/kadmind -P /var/run/kadmind.pid
root 1547 0.0 0.7 10312 7496 ? Ss Jan04 0:02 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
[/code]
This is a test/dev system, but suffice it to say this is making me a bit nervous to attempt any sort of production deployment.
So why does the kernel seem to be eating all the RAM?
[code]
Linux dev-box 2.6.32-220.2.1.el6.i686 #1 SMP Thu Dec 22 18:50:52 GMT 2011 i686 i686 i386 GNU/Linux
[/code]