large uploads from server

Support for security such as Firewalls and securing linux
westconn
Posts: 10
Joined: 2011/02/21 08:00:57

large uploads from server

Post by westconn » 2018/05/30 21:34:42

i am trying to find why the server is uploading around 5Gb each day, so i can prevent it from continuing
i have tried monitoring packets, but the amount of information is defeating me, in a previous version of centos i could use bandwidth monitoring to summarize the traffic for a period by port, but i have not found how to do this for the current server

i have tried to drop all out going packets on port 53, using firewalld form information found on various websites, but so far have been unsuccessful
> systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-05-28 07:12:37 AEST; 3 days ago
Docs: man:firewalld(1)
Main PID: 19913 (firewalld)
CGroup: /system.slice/firewalld.service
└─19913 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

May 29 21:51:19 localhost.localdomain firewalld[19913]: ERROR: COMMAND_FAILED
May 29 21:51:32 localhost.localdomain firewalld[19913]: WARNING: '/usr/sbin/iptables-restore --wait=2 -n' failed: iptables-restore v1.4.21: Couldn't load match `--sport=53':No such file or directory

Error occurred at line: 2
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
May 29 21:51:32 localhost.localdomain firewalld[19913]: ERROR: COMMAND_FAILED
May 29 21:51:51 localhost.localdomain firewalld[19913]: WARNING: '/usr/sbin/iptables-restore --wait=2 -n' failed: iptables-restore v1.4.21: Couldn't load match `--sport=53':No such file or directory

Error occurred at line: 2
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
May 29 21:51:51 localhost.localdomain firewalld[19913]: ERROR: COMMAND_FAILED
May 29 21:52:21 localhost.localdomain firewalld[19913]: WARNING: '/usr/sbin/iptables-restore --wait=2 -n' failed: iptables-restore v1.4.21: Couldn't load match `--sport=53':No such file or directory

Error occurred at line: 2
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
May 29 21:52:21 localhost.localdomain firewalld[19913]: ERROR: COMMAND_FAILED
May 29 21:54:24 localhost.localdomain firewalld[19913]: WARNING: '/usr/sbin/iptables-restore --wait=2 -n' failed: iptables-restore v1.4.21: Couldn't load match `--sport=53':No such file or directory

Error occurred at line: 2
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
May 29 21:54:24 localhost.localdomain firewalld[19913]: ERROR: COMMAND_FAILED
May 29 22:28:43 localhost.localdomain firewalld[19913]: ERROR: Failed to load direct rules file '/etc/firewalld/direct.xml': INVALID_TYPE: Not a valid file: no element found: line 2, column 0
so obviously i am doing it wrong somehow, i have also tried using the firewall-config applicaton, and not even sure if port 53 is the main culprit, but it did solve a problem for me with the previous server

what information do i need to post to help?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: large uploads from server

Post by TrevorH » 2018/05/30 22:23:03

If you're suspecting that you've been hacked then you should probably take the server offline ASAP.

You could yum install iptraf-ng and then run iptraf-ng and use that to look at packet distribution etc to get an idea of what you're looking at. Be aware that if you have been hacked and the hackers have root access then they have the ability to hide the traffic from anything run on the same machine so you may only see what you're being allowed to see.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Elenax
Posts: 5
Joined: 2018/05/25 06:02:28

Re: large uploads from server

Post by Elenax » 2018/05/31 09:33:39

Don't take risk before it gets more complicated take the server offline as preliminary precaution. Its very likely there is a backdoor. SSH would be the most obvious way, so check access logs for anything that looks strange.

westconn
Posts: 10
Joined: 2011/02/21 08:00:57

Re: large uploads from server

Post by westconn » 2018/05/31 10:57:59

i do not believe i have been hacked, so much as flooded on my domain port and that the responses are at least port of my problem, which is why i was trying to drop the outgoing packets

i am attaching a log from iptraf-ng if that is of any help, had to zip, to fit for file size
iptraf-log.zip
(56.78 KiB) Downloaded 90 times
this seems to show, over a 10 minute time period, 25793031 bytes out against bytes received 794921, most of which seem to be udp on the domain port or fragments to the same ip, port unspecified destination ports and/or ipaddresses change at intervals

can i determine from the log the volume of data to each port over some extended time period

@elenax, which logs should i be looking at
i have pulled out the network cable, but still have internal access through wireless, but as this is my main email server, it is a bit hard to not have it on line, i only upgraded this server a few months ago, for similar problems and the os was too old, now everything is different

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: large uploads from server

Post by avij » 2018/05/31 11:31:22

Looks like your DNS server is used for an amplification attack. The source IP address of UDP packets is easy to spoof. You can get a bit more detail about the traffic with tcpdump, for example with tcpdump -n -c 100 port 53

Make sure your name server allows recursive DNS queries only from clients in your LAN (allow-recursion keyword) and also make sure zone transfers are limited to known hosts (allow-transfer keyword; a sensible default for this is none;).

You can also rate limit the answers. The actual rates depend on normal volume, but I have this in my own config:

Code: Select all

        rate-limit {
                responses-per-second 10;
                nxdomains-per-second 5;
                errors-per-second 5;
                exempt-clients {
                        127.0.0.1;
                        ::1;
                };
        };
YMMV, adjust numbers as appropriate.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: large uploads from server

Post by TrevorH » 2018/05/31 11:36:51

At a guess, I'd say someone is using your DNS server to perform an amplification attack against people they don't like. The way that works is they fake a packet that appears to come from an ip address and send it to your server. Your server then sends back a massively larger amount of data to the ip address the query supposedly came from. Do that from enough different public dns servers and you overwhelm the target of your attack and they drop off the internet.

Do you have to run your own DNS server? Do you need it to be publicly accessible? If not then turn it off. If you must run your own DNS server then restrict access to it to only those things that need access to it. If it must be public then you need to secure it - a quick google for "securing against dns amplification attacks" turns up lots of hits on how to do that.

Most of us use hosted DNS services for precisely this reason. It's a complex task and needs to be done properly and the necessary levels of expertise are easily and cheaply available simply by using a third party.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

westconn
Posts: 10
Joined: 2011/02/21 08:00:57

Re: large uploads from server

Post by westconn » 2018/05/31 12:02:07

i have recursion yes, but i believe i need that, as i host my own domain, i seem to remember that when recursion was no some things did not work
If your recursive DNS server has a public IP address, you MUST enable access
control to limit queries to your legitimate users. Failing to do so will
cause your server to become part of large scale DNS amplification
attacks. Implementing BCP38 within your network would greatly
reduce such attack surface
i also read up about implementing bcp38, but totally failed to understand

i tried tcpdump
tcpdump -n -c 100 port 53
tcpdump: Bluetooth link-layer type filtering not implemented
obviously i have done something wrong

which file, or where do i need to add the rate limits to?
i have implemented allow-transfer none

westconn
Posts: 10
Joined: 2011/02/21 08:00:57

Re: large uploads from server

Post by westconn » 2018/05/31 13:03:58

Most of us use hosted DNS services for precisely this reason.
i have been considering the option of just using a domain hosting service, and turning off the centos box, probably also not affected by local power outages and internet failures, but i have sort of got attached to have my own server,, even if it often gives me problems

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: large uploads from server

Post by avij » 2018/06/01 05:59:32

westconn wrote:i have recursion yes, but i believe i need that, as i host my own domain, i seem to remember that when recursion was no some things did not work
You only need recursion if the machines in your local LAN use this server as a DNS server. In this case, you need to set recursion yes; and specify the IP addresses that are allowed to send recursive queries with allow-recursion, eg. allow-recursion { localhost; 192.168.1.0/24; };

The rate limits go to the same place, ie. the options block in your /etc/named.conf

BCP38 means that if some ISP is allocated a network range like 192.0.2.0 - 192.0.2.255, the ISP should filter out any outgoing packets whose source IP address is not within this range. This is generally something that is done at a higher level at your ISP, not by individual end users. And besides, it's likely not your own ISP that is letting through those packets with forged source IP addresses, it's more likely that it's some other ISP on the other side of the world. In any case, BCP38 is not your concern, you can concentrate on other matters.

tcpdump may need an interface specifier if you have several network interfaces, eg. -i eth0

westconn
Posts: 10
Joined: 2011/02/21 08:00:57

Re: large uploads from server

Post by westconn » 2018/06/01 10:18:39

thank you to all who replied, i had spent a lo of time googling, trying to find a solution, looks like i really did not use the correct search terms

i now have recursion no, which seems to be helping a lot
my daily upload (till 8pm) is now at 0.34Gb, down from previous 7 days at an average of about 5.5Gb, with a peak earlier in the month of over 16Gb

i sort of remember that i had recursion yes trying to fix some other problem, the server not being able to resolve ips for yum etc, probably something else fixed that problem, but i thought recursion had something to do with it at the time

Post Reply