Has to be a better way....

A 5 star hangout for overworked and underpaid system admins.
Post Reply
w00tburger
Posts: 8
Joined: 2016/09/18 05:00:50

Has to be a better way....

Post by w00tburger » 2016/09/18 05:56:10

Hello everyone! So to make a long introduction short, I have been in the IT field professionally for about 12 years. My first two years were dominated by the demands of web design, and the last 10 years had encompassed a role as a network operations technician. I have been a jack of all trades "little bit of everything, but a master of nothing".

I decided to interview with my new employer through a contracting company, and ended up here as a "system administrator". Over the last week, I have been overseeing windows and linux systems, until my boss tells me this last week that I was going to be one of their two Linux Sys admins as they are bringing on two other windows admins this upcoming week.

Without getting into too much detail, their system calls for an immediate need to put out fire on a daily basis for the next two months. A lot of this involves shifting and compressing files to meet storage demands, rebooting systems and determining causes of failure, and numerous other (what I consider BS) tasks. All of this should be automated, and it will be providing we can get additional answers (such as retention policies) so we can act accordingly.

SO. TLDR. As of right now this is the drill. "/" is running low on space for the 5th time on the 30th different server this week. The standard operation is to SSH into the server, and run this command "du / -h| sort -rn" a dozen different times in a dozen different directory's until we stumble upon a directory which is running an app that has logs with no retention plan.

Instead of this mad goose chase- can someone here help me out with a command that could break this down into something a little faster/more useful?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Has to be a better way....

Post by TrevorH » 2016/09/18 10:55:31

What I usually use is du --max-depth=1 -m -x / then repeat for the directory listed as using the most space (keep repeating, drilling down one level at a time).

Sounds like you could do with installing a monitoring system like icinga2, nagios, zabbix etc. Won't help to solve the problems but does help to alert you to impending doom before it actually hits you.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

w00tburger
Posts: 8
Joined: 2016/09/18 05:00:50

Re: Has to be a better way....

Post by w00tburger » 2016/09/18 22:22:38

I suppose hat this can be scripted into a bash script which will do the manual labor for me. I was kind of hoping there was something that I was over looking that would prevent the need to script this.

On another note, they are using Zambia, but they are only pointing to the parent directors so that leads us to searching all the sub directors and burning through the majority of our day

MartinR
Posts: 714
Joined: 2015/05/11 07:53:27
Location: UK

Re: Has to be a better way....

Post by MartinR » 2016/09/19 10:30:50

If the different directories have different owners, have a look at the quota system. Knowing which user is hogging the space may enable you to home in a bit faster.

Post Reply