CentOS slows down when copying large files

General support questions
Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/20 18:24:43

I am experiencing a very strange issue with CentOS. When I try copying large files/folders (100+GB), I find that the system tends to slow down. Initially, I thought it might be just that my system is not the highest end and that my storage is not fast enough. However, even at my school, where we have HP Z workstations and storage servers, CentOS tends to get sluggish when copying large files.

Any idea why this is happening?

To give you more information, both at home and school, I am using MATE desktop from with Dolphin as the file manager at home and Caja as the file manager at school.

Let me know if you need more information.

SCSIraidGURU
Posts: 7
Joined: 2018/09/12 17:40:31

Re: CentOS slows down when copying large files

Post by SCSIraidGURU » 2018/09/20 20:42:34

I had this happen a lot in Linux. I replaced the onboard adapters with Dlink PCIe Gig Adapters. They are usually $20. Some motherboard chipsets and Linux don't get along. The onboard NIC drivers aren't optimized. An Intel or Dlink PCIe NIC works great. Disabled the other ones onboard in the BIOS.

Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

Re: CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/20 20:49:09

It is not just a network issue. At home, this happens even when copying from a local partition to another. I think there is a deeper issue at play here.

SCSIraidGURU
Posts: 7
Joined: 2018/09/12 17:40:31

Re: CentOS slows down when copying large files

Post by SCSIraidGURU » 2018/09/20 21:12:32

What are your specs? Copying a file that large could require a lot of head activity on your drives. I usually use WD Black drives for Linux.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS slows down when copying large files

Post by TrevorH » 2018/09/20 22:22:16

Look at tuning the various parameters to do with file caching. What's happening is that it tries to keep your 100GB file in RAM since it was recently used and therefore it might be needed again soon. That has the side effect of evicting lots of useful stuff out of RAM and also expanding the caches to use a significant proportion of it and evicting running programs to swap and/or just discarding their program code (since it can be reloaded from disk). The default set up works well for most but those that do stuff like this on a regular basis need to do some tuning.

This link is for RHEL 6 but probably applies equally to 7 https://access.redhat.com/documentation ... y-tunables
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

Re: CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/21 15:04:26

TrevorH wrote:
2018/09/20 22:22:16
Look at tuning the various parameters to do with file caching. What's happening is that it tries to keep your 100GB file in RAM since it was recently used and therefore it might be needed again soon. That has the side effect of evicting lots of useful stuff out of RAM and also expanding the caches to use a significant proportion of it and evicting running programs to swap and/or just discarding their program code (since it can be reloaded from disk). The default set up works well for most but those that do stuff like this on a regular basis need to do some tuning.

This link is for RHEL 6 but probably applies equally to 7 https://access.redhat.com/documentation ... y-tunables
Hi Trevor. I have tried tinkering with these settings, but it didn't help much. In some areas, the responsiveness of the system has improved, but in other areas it became even worse when I tried lowering the swappiness and the dirty ratio values. For instance, with the default settings, it is usually the file manager and the terminal that would become a bit unresponsive and slow, but with the lower values, the mouse sometimes wouldn't even move.

The issue is way worse when I am copying one of these large files to my /home folder. I have three hard disks in my home machine. Two SSDs, one for Windows and one for CentOS, and an HDD for the work files. The /home folder is one of the partitions on the SSD. It is an EXT4 partition. Copying to that partition is where things become really noticeable. However, copying to the work HDD doesn't create a similar issue, despite the fact that it is an NTFS partition to make it compatible with both Windows and Linux.

Any ideas what might be causing this or how to improve it.

System specs:
CPU: Intel i7-4790K 8M Cache @ 4.00 GHz
RAM: 32 GB
2x 240 GB SSDs for dual-boot
1x 3 TB HDD for data storage

john2018
Posts: 43
Joined: 2018/02/17 19:09:47

Re: CentOS slows down when copying large files

Post by john2018 » 2018/09/22 09:57:12

I notice your specification is similar to ours so just thought
I'd mention we have an issue with large/multiple file copies.
I dont know if this is related but I found that copying off
large files (to test just copy everything off) to an external
drive causes centos to tell us its safe to remove
the external drive a full 5 minutes before it actually stops
copying the data.

This we classified here as catastrophic failure (especially since
we dont have any visual activity indication on the drives or case anymore).

It seems centos (linux) puts everything in ram as Trevor said but doesn't
actually keep track of when it finishes the task and notifies safe removal incorrectly.
Whether it slowed down or not I dont know as I didnt time it - but it seems
like a similar area (large file copying) on a similar system.
It could be related maybe.
Perhaps its worth a test at your end with an external drive.
(Or perhaps just running TOP or something )
Last edited by john2018 on 2018/09/23 08:50:50, edited 1 time in total.

Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

Re: CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/22 16:29:15

Thank you, John. Tbh, I didn't encounter this specific issue, but I have also had some trouble with copying from/to external drives. Mainly it is the slow down when copying large files.

The other issue I encountered with copying, however, is that copying to FAT32 flash drives never finishes. Using Dolphin, any file, even relatively small ones, would get to 100% and wait. Not sure what is causing it and I haven't tried investigating it as it is not something critical for me. I just don't using FAT32 anymore.

I have read somewhere that this copying issue is now fixed in Linux kernel (starting from 3.13) but I am not sure of course as I don't have any extra drives to test with one of the distros that have a more recent kernel.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: CentOS slows down when copying large files

Post by chemal » 2018/09/22 18:20:00

Try this:

Code: Select all

# don't buffer too many writes
echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
It means: start writing out when you have accumulated 16M of dirty buffers and stop buffering more writes if there are 48M of dirty buffers. This was once suggested by Linus himself on LKML. There also was a discussion about saner defaults, but it seems nothing came out of it.

Abdelrahman Said
Posts: 35
Joined: 2017/08/26 22:32:23

Re: CentOS slows down when copying large files

Post by Abdelrahman Said » 2018/09/22 18:45:34

chemal wrote:
2018/09/22 18:20:00
Try this:

Code: Select all

# don't buffer too many writes
echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
It means: start writing out when you have accumulated 16M of dirty buffers and stop buffering more writes if there are 48M of dirty buffers. This was once suggested by Linus himself on LKML. There also was a discussion about saner defaults, but it seems nothing came out of it.
Okay. That is getting somewhere. The system now is much more responsive. It still slows down, but it doesn't crawl like it used to do, even when copying to the /home folder.

Is there a bit of a middle ground with this setting where I can get higher copying speeds, but with better system performance?

Also, how can I make these settings permanent? I know I need to use `sysctl`, but I am not sure exactly what is it that I need? Or is it better to use these settings only when copying large files, but use the default in other cases?

I have also been just reading about tuned-adm and I was wondering if it could help with the performance in general if I set my profile to `thoroughput-performance`? Would that help with anything related to this copying issue?

Last question: Since I have seen it somewhere else that newer kernels should have this copying issue fixed (not sure if that is true or not, but I have read it), is it worth upgrading to the mainline kernel from elrepo-kernel repository? Is it okay to do so or is this something that is not recommended?

Post Reply