Page 1 of 1
[Solved] Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)
Posted: 2016/10/18 05:09:35
by Oisin
I have an obscure issue that I hope someone here can help with. We are trying to make our QEMU/KVM hypervisors more NUMA aware and are trying to stop KSM from merging memory across NUMA nodes. The problem is, on several of our host we cannot write to /sys/kernel/mm/ksm/merge_across_nodes, for instance.
Code: Select all
echo 0 > /sys/kernel/mm/ksm/merge_across_nodes
-bash: echo: write error: Device or resource busy
Reading works fine;
Code: Select all
cat /sys/kernel/mm/ksm/merge_across_nodes
1
Attempting to change the setting using virsh throws a similar error;
Code: Select all
virsh node-memory-tune --shm-merge-across-nodes 0
error: Unable to change memory parameters
error: failed to set shm_merge_across_nodes: Operation not permitted
The issue only effects a subset of host, though all servers should be configured fairly identically and are running fairly recent CentOS 7.2 packages installed from an internal repo. I’m unsure what could cause this.
Code: Select all
uname -a
Linux XXXXXXXXXXXXXXX 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Code: Select all
yum info qemu-kvm | grep Version
Version : 1.5.3
Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)
Posted: 2016/10/18 17:01:02
by hunter86_bg
This may sound stupid , but what's the memory usage on this host ?Any output in the logs?
For example:
Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)
Posted: 2016/10/18 23:02:35
by Oisin
The bellow is from one of out test blades.
System memory usage;
Code: Select all
free -m
total used free shared buff/cache available
Mem: 62238 15415 42002 11 4821 46462
Swap: 6999 0 6999
NUMA node VM memory usage;
Code: Select all
numastat qemu-kvm
Per-node process memory usage (in MBs)
PID Node 0 Node 1 Total
---------------- --------------- --------------- ---------------
3798 (qemu-kvm) 1253.93 10.86 1264.79
3817 (qemu-kvm) 4456.91 10.87 4467.78
3833 (qemu-kvm) 264.65 387.82 652.46
3850 (qemu-kvm) 0.01 8334.07 8334.08
---------------- --------------- --------------- ---------------
pages_sharing
Code: Select all
cat /sys/kernel/mm/ksm/pages_sharing
112172
pages_unshared
Code: Select all
cat /sys/kernel/mm/ksm/pages_unshared
26
pages_shared
Code: Select all
cat /sys/kernel/mm/ksm/pages_shared
110
ksm.service Log
Code: Select all
journalctl -u ksm.service
-- Logs begin at Mon 2016-10-17 14:25:04 ACDT, end at Wed 2016-10-19 09:23:00 ACDT. --
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging...
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging.
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging.
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging...
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopped Kernel Samepage Merging.
Oct 19 09:22:41 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging...
Oct 19 09:22:41 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging.
ksmtuned.service Log
Code: Select all
journalctl -u ksmtuned.service
-- Logs begin at Mon 2016-10-17 14:25:04 ACDT, end at Wed 2016-10-19 09:23:00 ACDT. --
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopped Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 18 15:22:55 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopped Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 19 09:23:00 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 19 09:23:00 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging (KSM) Tuning Daemon.
Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)
Posted: 2016/10/19 07:07:22
by hunter86_bg
It seems this quotation will explain this:
merge_across_nodes - specifies if pages from different numa nodes can be merged.
When set to 0, ksm merges only pages which physically
reside in the memory area of same NUMA node. That brings
lower latency to access of shared pages. Systems with more
nodes, at significant NUMA distances, are likely to benefit
from the lower latency of setting 0. Smaller systems, which
need to minimize memory usage, are likely to benefit from
the greater sharing of setting 1 (default). You may wish to
compare how your system performs under each setting, before
deciding on which to use. merge_across_nodes setting can be
changed only when there are no ksm shared pages in system:
set run 2 to unmerge pages first, then to 1 after changing
merge_across_nodes, to remerge according to the new setting.
Default: 1 (merging across nodes as in earlier releases)
Refference:
Kernel.org
It seems that you have to first unmerge your pages.
So it should be something like this:
Code: Select all
echo 2 > /sys/kernel/mm/ksm/run && sleep 300 && cat /sys/kernel/mm/ksm/pages_shared
If there are no pages shared :
Code: Select all
echo 0 > /sys/kernel/mm/ksm/merge_across_nodes && echo 1 > /sys/kernel/mm/ksm/run
Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)
Posted: 2016/10/20 01:25:22
by Oisin
Thanks, great answer! Once shared pages are unmerged it works.