Deleting one very big file brought server to its knees

General support questions
Post Reply
User avatar
ben_tvpp
Posts: 62
Joined: 2018/12/04 13:28:51

Deleting one very big file brought server to its knees

Post by ben_tvpp » 2019/02/13 17:47:15

I just deleted a single file that was 60GB from a ext4 filesystem and so far it has taken 30 mins. It has also made our WordPress sites VERY slow. We have another more traditional site and this seems OK.

Whats going on and is there anyway round this?

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Deleting one very big file brought server to its knees

Post by hunter86_bg » 2019/02/13 18:34:30

How did you do that?
Most probably during the removal, the Linux kernel tried to cache the whole contents and thus you might have swapped.
Check the sar to find the swap & memory usage during the deletion.

I prefer to delete my files via:

Code: Select all

truncate -s 0 /filename
But I have never deleted such a large file on a memory-constrained machine.
Last edited by hunter86_bg on 2019/02/16 17:12:20, edited 1 time in total.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Deleting one very big file brought server to its knees

Post by TrevorH » 2019/02/13 19:39:49

What device is the underlying filesystem sitting on? If it's an SSD then have you mounted the filesystem with the 'discard' option? That would cause it to trim all sectors that are allocated to that file as it's deleted.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

ron7000
Posts: 162
Joined: 2019/01/15 20:00:28

Re: Deleting one very big file brought server to its knees

Post by ron7000 » 2019/02/15 15:37:03

I am interested in an answer to this, over the years in work environment (on servers > 256GB RAM not using CentOS) I've never had this happen specifically but sometimes when deleting files/folders of significant size, with many users using server at same time, I sometimes see a lot of flush processes when doing a top and general performance of the server is severely degraded when this is going on. I don't use any SSD's so trim should not be relevant in my case, but even if a TRIM were happening shouldn't that happen behind the scenes and have no effect? Can you not issue a command and force TRIM to happen, would that then [always] bring your server to its knees?

if I had to guess, you have 8, 16 or 32 GB or RAM and trying to delete 60gb file causes some kind of disk caching to happen that exceed RAM capacity, went back to disk i/o using swap, system got crippled waiting on disk i/o. I don't see that kind of severity because I'm running 128, 256, or 512 gb of ram.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Deleting one very big file brought server to its knees

Post by TrevorH » 2019/02/15 16:51:17

Deleting a file should just involve manipulating directory data. The file is still present on disk - that's why you never ebay a disk after just doing an rm. I think there is more going on here than we know about.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
ben_tvpp
Posts: 62
Joined: 2018/12/04 13:28:51

Re: Deleting one very big file brought server to its knees

Post by ben_tvpp » 2019/02/28 22:55:48

It was a disk problem, the raid array was dying.

Post Reply