/dev/mapper/centos-root keeps filling up, fast

[benhtn]

Hi there,
I've got CentOS Linux release 7.8.2003 (Core) running PHP & MySQL and it has been doing fine until last week (I ran yum update and took all updates, maybe related?). Suddenly the primary partition has been filling up rapidly, according to df. Once df reports full, it is unwritable - I've mounted new block storage devices and move the larger stuff off the main partition but it is still filling up - it has jumped from 85% to 100% in 24 hours, ~12GB which is nothing like the sort of amounts of data usually generated):

Code: Select all

# df -ihT
Filesystem              Type     Inodes IUsed IFree IUse% Mounted on
devtmpfs                devtmpfs   246K   388  246K    1% /dev
tmpfs                   tmpfs      249K     1  249K    1% /dev/shm
tmpfs                   tmpfs      249K   781  248K    1% /run
tmpfs                   tmpfs      249K    16  249K    1% /sys/fs/cgroup
/dev/mapper/centos-root xfs        2.3M  194K  2.1M    9% /
/dev/sdb1               ext4       1.3M   18K  1.3M    2% /mnt/block/prod1
/dev/sdc1               ext4       2.5M   492  2.5M    1% /mnt/block/prod2
/dev/sda1               xfs        500K   353  500K    1% /boot
tmpfs                   tmpfs      249K     1  249K    1% /run/user/0

However I can't find any significantly large directories on /dev/mapper/centos-root using du (nor ncdu):

Code: Select all

# du --max-depth=1 -h -x /
39M	/etc
183M	/root
8.0K	/tmp
1.8G	/var
5.4G	/usr
27M	/home
0	/media
70G	/mnt
188M	/opt
0	/srv
0	/tmp2
77G	/

I've also used `lsof | grep -i del` to see if there are deleted files being held by open processes, I see a few, e.g.

Code: Select all

sw-engine 1103              root    3u      REG              253,1           0  100759151 /tmp/.ZendSem.FmsMQx (deleted)
tuned     1106              root  DEL       REG              253,1              100862106 /tmp/#100862106
tuned     1106              root    8u      REG              253,1        4096  100862106 /tmp/#100862106 (deleted)
gmain     1106 1346         root  DEL       REG              253,1              100862106 /tmp/#100862106
gmain     1106 1346         root    8u      REG              253,1        4096  100862106 /tmp/#100862106 (deleted)
tuned     1106 1347         root  DEL       REG              253,1              100862106 /tmp/#100862106
tuned     1106 1347         root    8u      REG              253,1        4096  100862106 /tmp/#100862106 (deleted)
tuned     1106 1389         root  DEL       REG              253,1              100862106 /tmp/#100862106
tuned     1106 1389         root    8u      REG              253,1        4096  100862106 /tmp/#100862106 (deleted)
tuned     1106 1406         root  DEL       REG              253,1              100862106 /tmp/#100862106
tuned     1106 1406         root    8u      REG              253,1        4096  100862106 /tmp/#100862106 (deleted)

but the sizes are minimal right? And anyway, even rebooting is not resolving the issue.

I've recently mounted 2 block storage devices under /mnt/block/prod1 & /mnt/block/prod2. One of those is for files output by the webserver and the other is now the mysql data directory. I commented these out in /etc/fstab, and rebooted to check that files weren't being hidden "under" the mounts but they were empty.

/mnt/nfs has been mounted for a long while and receives hourly file & db backups via a cron. I've forgotten how this is mounted TBH! But it isn't in /etc/fstab.

installonly_limit=5 in /etc/yum.conf

I've read viewtopic.php?t=61945 and several other posts and (I think) elimated the obvious issues.

Help!

[benhtn]

Using I did find some large "files" apparently generated by docker.

Code: Select all

# find / -xdev -type f -size +100M -exec ls -la {} \; | sort -nk 5
-rw-------. 1 root root 2147483648 Jul  6 08:06 /var/lib/docker/devicemapper/devicemapper/metadata
-rw-------. 1 root root 107374182400 Oct 22  2018 /var/lib/docker/devicemapper/devicemapper/data

I am not using docker on this machine. I tried stopping docker and deleting them. df showed the same. After reboot they re-appeared at the same size

Code: Select all

# ls -alh /var/lib/docker/devicemapper/devicemapper/
total 12M
drwx------. 2 root root   32 Jul  8 23:44 .
drwx------. 4 root root   40 Oct 22  2018 ..
-rw-------. 1 root root 100G Jul  8 23:44 data
-rw-------. 1 root root 2.0G Jul  8 23:44 metadata

----
Update:
I uninstalled docker completely, as per the docs (https://docs.docker.com/engine/install/ ... ker-engine):

Code: Select all

# yum remove docker-ce docker-ce-cli containerd.io

And deleted the docker data dir

Code: Select all

# rm -rf /var/lib/docker

And rebooted, but no difference in df.

[benhtn]

OK, so in desparation I deleted some of the older back-up from /mnt/nfs and lo and behold:

Code: Select all

[root@localhost nfs]# df -hT
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  984M     0  984M   0% /dev
tmpfs                   tmpfs     995M     0  995M   0% /dev/shm
tmpfs                   tmpfs     995M  9.3M  986M   1% /run
tmpfs                   tmpfs     995M     0  995M   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        78G   41G   38G  52% /
/dev/sdc1               ext4       40G   12G   26G  32% /mnt/block/prod2
/dev/sdb1               ext4       20G  5.9G   13G  32% /mnt/block/prod1
/dev/sda1               xfs       497M  246M  251M  50% /boot
tmpfs                   tmpfs     199M     0  199M   0% /run/user/0

So I've done something weird/ dumb with the nfs mount that is contributing to usage on the main partition. Or maybe the nfs mount has failed and the cron has been writing to that directory regardless...?

[benhtn]

So, as anyone more observant than me will have noticed from the output of df in my first post, /mnt/nfs was not mounted at all, so of course all the backups were written to the boot partition! Checked /etc/fstab again and of course it was missing, obviously I fat fingered it out when adding the other mounts.
Case closed - hope it helps someone in future maybe...!

[lightman47]

Having been through this a couple times myself, I now have a (hidden) file named .IAmMounted on each of the drives that I mount. All my scripts check for this file before they proceed with writing to the drive.

[tunk]

Or you could use something like this:
if /bin/mountpoint -q /path/to/mountpoint

[hunter86_bg]

I use:

Code: Select all

findmnt /mountpoint &&  /path/to/script