weird logwatch report - can anybody explain ?

Issues related to applications and software problems
Post Reply
aegersz
Posts: 86
Joined: 2016/04/04 07:42:58

weird logwatch report - can anybody explain ?

Post by aegersz » 2018/08/08 11:13:52

Hi, i run a forum on a CentOS 6.9 30 GB VPS and check logwatch frequently but once i a while i see something BIZARRE like this:

Code: Select all

--------------------- httpd Begin ------------------------ 

 450508.69 MB transferred in 116259 responses  (1xx 0, 2xx 103076, 3xx 10374, 4xx
2559, 5xx 250) 
     22258 Images (386.32 MB),
         8 Documents (1.36 MB),
      4554 Sound files (448962.15 MB),
     87847 Content pages (1091.79 MB),
        21 Redirects (0.01 MB),
      1571 Other (67.07 MB) 
Note the: 4554 Sound files (448962.15 MB) - how could a whopping 400+ GB amount of data get transferred from a system whose disk footprint is only 30 GB:

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        30G   22G  6.4G  78% /
tmpfs           499M     0  499M   0% /dev/shm
how can this happen ? i am very curious as how to have it explained, if possible.

MartinR
Posts: 714
Joined: 2015/05/11 07:53:27
Location: UK

Re: weird logwatch report - can anybody explain ?

Post by MartinR » 2018/08/08 11:50:04

There were 103076 successful transfers. 400GB / 100k transfers = 4MB per transfer on average. 4MB for a sound file seems reasonable, it's probably the same few files being transferred may times.

aegersz
Posts: 86
Joined: 2016/04/04 07:42:58

Re: weird logwatch report - can anybody explain ?

Post by aegersz » 2018/08/08 12:10:07

Thank you MartinR, I really appreciate your explanation and also felt that it could've been the same file transferred many times but couldn't understand why.

bots and search engine crawlers index my site but this particular amount of data is HUGE.

any thoughts as to why this would happen ? (a rogue bot, perhaps ?)

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: weird logwatch report - can anybody explain ?

Post by avij » 2018/08/08 12:35:04

You could check the log files directly to see what is going on. They are probably in /var/log/httpd

One possibility is that the bot(?) that made the requests only actually downloaded the first few kilobytes of each file, but your httpd has been configured to log as if the entire file got downloaded. There is a config option for this, but I don't remember the details offhand.

aegersz
Posts: 86
Joined: 2016/04/04 07:42:58

Re: weird logwatch report - can anybody explain ?

Post by aegersz » 2018/08/08 13:12:16

thanks again !, i am actually installing the ELK stack to monitor log files so may be that will help one day.

I didn't know of such an option in appache.

i might let this one go unless any other thoughts come into play.

MartinR
Posts: 714
Joined: 2015/05/11 07:53:27
Location: UK

Re: weird logwatch report - can anybody explain ?

Post by MartinR » 2018/08/08 17:22:36

http://httpd.apache.org/docs/2.4/logs.html seems to imply that the number returned is the data actually sent. The http service in logwatch uses the http logfile group which searches for *access{_,.}log in multiple places. The figures returned are therefore those in the access log.

Have you tried getting more detail from logwatch? Try logwatch --service http --detail=high

aegersz
Posts: 86
Joined: 2016/04/04 07:42:58

Re: weird logwatch report - can anybody explain ?

Post by aegersz » 2018/08/09 02:32:28

i did, yesterday but no help on where the data is going.

i do host a little bit of music on my site and it's about 5 GB worth in total.

if only i could see what IPs transferred what .wav/.mp3 ... *sigh*.

my forum's topics are very sensitive so i get curious about glaring inconsistencies.

a lot of data moves daily and it's getting me a little paranoid.

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: weird logwatch report - can anybody explain ?

Post by Whoever » 2018/08/09 04:58:35

aegersz wrote:
2018/08/09 02:32:28


if only i could see what IPs transferred what .wav/.mp3 ... *sigh*.
Install awstats, it is in the EPEL repo.

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: weird logwatch report - can anybody explain ?

Post by avij » 2018/08/09 06:00:12

grep mp3 /var/log/httpd/access_log | more or something similar should help.

Post Reply