CentOS slows down when copying large files

General support questions
chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: CentOS slows down when copying large files

Post by chemal » 2018/09/22 20:36:53

I'm not sure what "copying issue" you are refering to and heard is fixed in later kernels. I've never heard of this. What Linux does here is a feature not a bug. It buffers writes and then does them asynchronously to improve performance. Only problem is that per default a lot of writes are buffered and this is problematic for external and or/slow drives. If you want to unplug your external drive, all the copying you thought is done actually isn't. And if the drive is slow, the dirty buffers bind the memory they are using for a long time.

I've put the three lines mentioned above in /etc/rc.local on every desktop/laptop I use. They are even useful on servers, that have a RAID controller with lots of cache. Writes are better buffered there than in the server's RAM.

Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

Re: CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/22 21:29:44

The accepted answer in the following Stack Exchange (which suggested the same solution you offered in your previous post) mentioned that the buffering writes behavior that causes the slow down I am experiencing should be fixed in later kernels.

https://unix.stackexchange.com/question ... a-pendrive

As I said I am not sure if that is true or not. It is just something I read while researching the issue.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: CentOS slows down when copying large files

Post by chemal » 2018/09/22 23:17:07

A nice test is mentioned there. Type

dd if=/dev/zero of=foo bs=1M count=10k

in one terminal and start some large program (not used since last reboot, it must come frome disk) in another, e.g.

oowriter

It will not pop up on my CentOS 7 machine until dd is through.

Seems I'm not doing this normally. :)

You might want to try a more recent kernel (from elrepo perhaps) to check if things got better.

Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

Re: CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/23 16:15:10

chemal wrote:
2018/09/22 23:17:07
You might want to try a more recent kernel (from elrepo perhaps) to check if things got better.
I tried updating the kernel with elrepo's kernel-ml. The installation went through successfully, but the kernel boots to the CLi. I thought it might be an issue with blacklisting nouveau, since I have the nvidia driver installed. I checked the boot options in the GRUB menu for the elrepo's kernel and CentOS kernel and they are both identical. I even reinstalled kmod-nvidia after booting with the kernel-ml to make sure that whatever it does to nouveau is done properly with the new kernel installed, but I still can't get the GUI to load.

Any ideas what might be causing that?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS slows down when copying large files

Post by TrevorH » 2018/09/23 16:54:51

When you update the kernel you must reinstall the nvidia drivers. The ELRepo kmod-nvidia packages that allow the drivers to survive kernel updates only work with the distro kernels so if you were using those before you'll have to switch to the ones from nvidia themselves.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

Re: CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/23 18:17:00

TrevorH wrote:
2018/09/23 16:54:51
When you update the kernel you must reinstall the nvidia drivers. The ELRepo kmod-nvidia packages that allow the drivers to survive kernel updates only work with the distro kernels so if you were using those before you'll have to switch to the ones from nvidia themselves.
That’s a shame. I never managed to get the drivers from nvidia to work properly. Tried different solutions but it never worked. Don’t think it is worth trying it again as I can’t afford my machine going down now.

Guess I will have to stick with the district kernel and continue to rely on that dirty_bytes solution that chemal provided.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: CentOS slows down when copying large files

Post by chemal » 2018/09/24 01:37:37

Code: Select all

# don't buffer too many writes
echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

# enable completely fair queuing
echo cfq > /sys/block/sda/queue/scheduler
Adding CFQ on top seems to solve both problems. I no longer have to wait until dd is finished, oowriter starts in parallel. The remedy in later kernels is called writeback throttling (4.10+) but RH did not backport this. However, CFQ does something similar: https://patchwork.kernel.org/patch/9661671/

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS slows down when copying large files

Post by hunter86_bg » 2018/09/24 03:38:59

If CFQ is helping, then you can set it as a default scheduler, but keep in mind that is not suitable for SSDs.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: CentOS slows down when copying large files

Post by chemal » 2018/09/24 13:57:34

For consumer-grade SATA SSDs, CFQ is ok. It has SSD optimizations (see kernel docs). And it's the only way to solve problem #2 (background writes hog drive) with the distro kernel as far as I can see.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: CentOS slows down when copying large files

Post by chemal » 2018/09/24 16:27:50

Some additional tests with an NVME SSD (Samsung SM961). NVME SSDs have multiple hardware submission queues and RH has backported multiqueue I/O scheduling. So even with the default 'none' scheduler, the background writes do not monopolize the SSD. Oowriter starts in parallel to heavy dd-ing. But the number of parallel reads that come through is low, it takes a lot of time until oowriter pops up. Things are much better with the kyber scheduler.

Post Reply