[Solved] Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)

General support questions
Oisin
Posts: 5
Joined: 2016/10/18 04:29:56

[Solved] Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)

Postby Oisin » 2016/10/18 05:09:35

I have an obscure issue that I hope someone here can help with. We are trying to make our QEMU/KVM hypervisors more NUMA aware and are trying to stop KSM from merging memory across NUMA nodes. The problem is, on several of our host we cannot write to /sys/kernel/mm/ksm/merge_across_nodes, for instance.

Code: Select all

echo 0 > /sys/kernel/mm/ksm/merge_across_nodes
-bash: echo: write error: Device or resource busy


Reading works fine;

Code: Select all

cat /sys/kernel/mm/ksm/merge_across_nodes
1


Attempting to change the setting using virsh throws a similar error;

Code: Select all

virsh node-memory-tune --shm-merge-across-nodes 0
error: Unable to change memory parameters
error: failed to set shm_merge_across_nodes: Operation not permitted


The issue only effects a subset of host, though all servers should be configured fairly identically and are running fairly recent CentOS 7.2 packages installed from an internal repo. I’m unsure what could cause this.

Code: Select all

uname -a
Linux XXXXXXXXXXXXXXX 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


Code: Select all

virsh --version
1.2.17


Code: Select all

yum info qemu-kvm | grep Version
Version     : 1.5.3
Last edited by Oisin on 2016/10/20 01:25:57, edited 1 time in total.

hunter86_bg
Posts: 541
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)

Postby hunter86_bg » 2016/10/18 17:01:02

This may sound stupid , but what's the memory usage on this host ?Any output in the logs?
For example:

Code: Select all

journalctl -u ksm.service

Code: Select all

journalctl -u ksmtuned.service

Oisin
Posts: 5
Joined: 2016/10/18 04:29:56

Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)

Postby Oisin » 2016/10/18 23:02:35

The bellow is from one of out test blades.

System memory usage;

Code: Select all

free -m
              total        used        free      shared  buff/cache   available
Mem:          62238       15415       42002          11        4821       46462
Swap:          6999           0        6999


NUMA node VM memory usage;

Code: Select all

numastat qemu-kvm

Per-node process memory usage (in MBs)
PID                        Node 0          Node 1           Total
----------------  --------------- --------------- ---------------
3798 (qemu-kvm)           1253.93           10.86         1264.79
3817 (qemu-kvm)           4456.91           10.87         4467.78
3833 (qemu-kvm)            264.65          387.82          652.46
3850 (qemu-kvm)              0.01         8334.07         8334.08
----------------  --------------- --------------- ---------------


pages_sharing

Code: Select all

cat /sys/kernel/mm/ksm/pages_sharing
112172


pages_unshared

Code: Select all

cat /sys/kernel/mm/ksm/pages_unshared
26


pages_shared

Code: Select all

cat /sys/kernel/mm/ksm/pages_shared
110


ksm.service Log

Code: Select all

journalctl -u ksm.service
-- Logs begin at Mon 2016-10-17 14:25:04 ACDT, end at Wed 2016-10-19 09:23:00 ACDT. --
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging...
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging.
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging.
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging...
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopped Kernel Samepage Merging.
Oct 19 09:22:41 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging...
Oct 19 09:22:41 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging.


ksmtuned.service Log

Code: Select all

journalctl -u ksmtuned.service
-- Logs begin at Mon 2016-10-17 14:25:04 ACDT, end at Wed 2016-10-19 09:23:00 ACDT. --
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 17 14:25:32 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 17 16:21:42 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopping Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 18 15:22:50 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopped Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 18 15:22:55 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Stopped Kernel Samepage Merging (KSM) Tuning Daemon.
Oct 19 09:23:00 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Starting Kernel Samepage Merging (KSM) Tuning Daemon...
Oct 19 09:23:00 XXXXXXXXXXXXXXXXXXXXXXXX systemd[1]: Started Kernel Samepage Merging (KSM) Tuning Daemon.

hunter86_bg
Posts: 541
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)

Postby hunter86_bg » 2016/10/19 07:07:22

It seems this quotation will explain this:

merge_across_nodes - specifies if pages from different numa nodes can be merged.
When set to 0, ksm merges only pages which physically
reside in the memory area of same NUMA node. That brings
lower latency to access of shared pages. Systems with more
nodes, at significant NUMA distances, are likely to benefit
from the lower latency of setting 0. Smaller systems, which
need to minimize memory usage, are likely to benefit from
the greater sharing of setting 1 (default). You may wish to
compare how your system performs under each setting, before
deciding on which to use. merge_across_nodes setting can be
changed only when there are no ksm shared pages in system:
set run 2 to unmerge pages first, then to 1 after changing
merge_across_nodes, to remerge according to the new setting.
Default: 1 (merging across nodes as in earlier releases)


Refference: Kernel.org

It seems that you have to first unmerge your pages.

So it should be something like this:

Code: Select all

echo 2 > /sys/kernel/mm/ksm/run && sleep 300 && cat /sys/kernel/mm/ksm/pages_shared

If there are no pages shared :

Code: Select all

echo 0 > /sys/kernel/mm/ksm/merge_across_nodes && echo 1 > /sys/kernel/mm/ksm/run

Oisin
Posts: 5
Joined: 2016/10/18 04:29:56

Re: Unable to Disable KSM Merging Memory Across NUMA Nodes (Error: Device or resource busy)

Postby Oisin » 2016/10/20 01:25:22

Thanks, great answer! Once shared pages are unmerged it works.


Return to “CentOS 7 - General Support”

Who is online

Users browsing this forum: ssherman and 6 guests