[quote]I'll keep this short, because I'm more or less a newbie regarding KVM. But could this be related to the balloon driver? I've never done a config that supported it, so I'm not sure how it looks from the guest OS's standpoint, or if it is enabled by default.[/quote]
Actually, this is on VMware ESXi, but...I think you just nailed it.
Had our VMware admin look into this...guess how much memory the host showed the balloon driver using? Sure enough, about 700MB. (Now [i]why[/i] it decided to balloon is anyone's guess...the host is nowhere near capacity.)
Now I'm rather annoyed, because vmmemctl doesn't indicate this usage [i]anywhere[/i] inside the guest. If it did, I wouldn't have wasted everyone's time. From the guest's point of view, the memory is just [i]not there[/i]...used, but not allocated by anything visible. I hope KVM's balloon driver is smarter than that.
[quote]
Another data point: here's a 32 bit CentOS 6 system that's been up about 22.5 days
[code]
total used free shared buffers cached
Mem: 1022248 564924 457324 0 46676 408892
-/+ buffers/cache: 109356 912892
Swap: 0 0 0
[/code]
That doesn't seem to have the same problem to me?[/quote]
No, this looks normal.
Most of your used RAM is being used as disk cache by the kernel (see the buffers/cached numbers off to the right). This memory will be reclaimed as needed for applications.
The second line of free shows your [i]real[/i] used/free memory for processes.
[SOLVED] 6.2: Where has all the memory gone?
Re: 6.2: Where has all the memory gone?
Not a waste of time. This is now in the archive and available for the benefit of future searchers.
I would guess that the balloon driver simply allocates memory to a special, otherwise unused by the guest OS, kernel buffer. So it would show up exactly this way.
I've been doing a lot of thinking about how virtual memory management should be handled in the case of a host with virtualized guests. And the only thing that makes sense to me is that one entity, the host, needs to be in control. A memory page is a memory page is a memory page, whether viewed in the context of a guest OS or the host OS. LRU means the same thing in either case. And either entity can tell how recently the page was last used. However, the host OS has a *much* broader view of *everything* occurring on the system, which might include many guests, plus the host's own apps. Only the host is in a position to make the right decisions. Only the host knows how recently a particular guest's page was used compared to all the pages across the entire system. You *don't* want too many VM cooks in the kitchen, spoiling the VM soup.
To that end, I've decided against advice that I've seen elsewhere. Namely, to give the guest OSes swap. You really don't *want* the guest kernels to be acting on their own, with only partial knowledge of the overall picture. Best not to give the guest *any* swap. Just increase its maximum ram allocation by the amount of swap that you might have been thinking about giving it. (And optionally add that amount of swap to the host.)
Extending that reasoning, you really don't want guests using excessive memory for their own private disk caching. As with RAM, a disk read is a disk read is a disk read. And you really want the *host* OS to be managing disk caching. That way, if other guests need the same blocks, they are there in the host's cache. (And only once, instead of, potentially, many times in many caches.) Also, only the host knows how much of a disk cache the entire system can really afford.
Of course, you don't want the guest to run out of memory that it really needs. So you set its maximum allocation to as much as you think it might ever legitimately need, give it no swap, and let the balloon driver keep the guest always thinking that it's starved for memory to avoid excessive disk caching at that level.
So, keeping in mind that I'm no expert at this, I would say that VMWare and the CentOS 6.2 guest kernel are doing the right thing. It's just one of those deals where you have to understand the strategy and the rationale behind it. Sort of a higher level "Why is Linux eating my RAM?" stumbling block for experts, to complement the one we already had for newbies. ;-)
-Steve
I would guess that the balloon driver simply allocates memory to a special, otherwise unused by the guest OS, kernel buffer. So it would show up exactly this way.
I've been doing a lot of thinking about how virtual memory management should be handled in the case of a host with virtualized guests. And the only thing that makes sense to me is that one entity, the host, needs to be in control. A memory page is a memory page is a memory page, whether viewed in the context of a guest OS or the host OS. LRU means the same thing in either case. And either entity can tell how recently the page was last used. However, the host OS has a *much* broader view of *everything* occurring on the system, which might include many guests, plus the host's own apps. Only the host is in a position to make the right decisions. Only the host knows how recently a particular guest's page was used compared to all the pages across the entire system. You *don't* want too many VM cooks in the kitchen, spoiling the VM soup.
To that end, I've decided against advice that I've seen elsewhere. Namely, to give the guest OSes swap. You really don't *want* the guest kernels to be acting on their own, with only partial knowledge of the overall picture. Best not to give the guest *any* swap. Just increase its maximum ram allocation by the amount of swap that you might have been thinking about giving it. (And optionally add that amount of swap to the host.)
Extending that reasoning, you really don't want guests using excessive memory for their own private disk caching. As with RAM, a disk read is a disk read is a disk read. And you really want the *host* OS to be managing disk caching. That way, if other guests need the same blocks, they are there in the host's cache. (And only once, instead of, potentially, many times in many caches.) Also, only the host knows how much of a disk cache the entire system can really afford.
Of course, you don't want the guest to run out of memory that it really needs. So you set its maximum allocation to as much as you think it might ever legitimately need, give it no swap, and let the balloon driver keep the guest always thinking that it's starved for memory to avoid excessive disk caching at that level.
So, keeping in mind that I'm no expert at this, I would say that VMWare and the CentOS 6.2 guest kernel are doing the right thing. It's just one of those deals where you have to understand the strategy and the rationale behind it. Sort of a higher level "Why is Linux eating my RAM?" stumbling block for experts, to complement the one we already had for newbies. ;-)
-Steve
-
ChairmanKaga
- Posts: 6
- Joined: 2010/02/02 18:35:17
Re: 6.2: Where has all the memory gone?
[quote]
Not a waste of time. This is now in the archive and available for the benefit of future searchers.
[/quote]
[url=http://xkcd.com/979/]True enough[/url] :)
Not a waste of time. This is now in the archive and available for the benefit of future searchers.
[/quote]
[url=http://xkcd.com/979/]True enough[/url] :)
Re: 6.2: Where has all the memory gone?
LOL. Glad the mystery is solved. What's not quite as bad, but rather more annoying are those threads in Nameless_Distro's Bugzilla consisting of a problem, followed by a slew of Me To's over the following months, with no developer response, capped off with one of those automated responses that says "It's been many months, so we assume this bug has been fixed. Please upgrade to the lastest version of the distro and report the problem again if it persists. Closing out this bug."
I've lost quite a bit of molar enamel to that one in days past.
I've lost quite a bit of molar enamel to that one in days past.
-
AlanBartlett
- Forum Moderator
- Posts: 9345
- Joined: 2007/10/22 11:30:09
- Location: ~/Earth/UK/England/Suffolk
- Contact:
Re: [SOLVED] 6.2: Where has all the memory gone?
For posterity (and on the behalf of the OP), this thread is now marked [SOLVED].