One of my servers is suspected of flooding

Support for security such as Firewalls and securing linux
username
Posts: 76
Joined: 2013/03/04 13:17:23

One of my servers is suspected of flooding

Post by username » 2018/02/01 15:54:47

HI,

One one my server got suspended and put in rescue mode. The hosting company (online.net) told me it was flooding other servers. I asked for more details but they didn't answer. If I don't succeed solving the issue and reboot the server, they will make me pay a kind of fine to unblock it again.

What sounds very weird to me is that this server is a simple LAMP running since 4 years without issues and weekly updated. Nothing has changed recently in this configuration. Is it possible that the hosting company is mistaken ?

I don't really know where to start checking... What would be the first things to do ? Changing credentials (like SSH passwords?). Or could a server be used for flooding without being compromised ? A network misconfiguration (I'm using CentOS defaults settings, nothing exotic) ? Should I reset my firewall, letting only a port opened for SSH ? Shutting down all the services like Apache, MySQL, vsVTP ?

This server runs : Apache, MySQL, phpMyAdmin, vsFTP, munin, Awstats, fail2ban, ntp, lftp, Seafile, postfix, logwatch. That's pretty much it. The main purpose of this server is to host a PHP app. It's not the main server but a "backup" server that could be used if the main one suffers from a big hardware issue for instance. The main server (is configured the same way) doesn't have any issue but is hosted at OVH.

EDIT : title typo
Last edited by username on 2018/02/02 11:24:13, edited 2 times in total.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: One of my servers is supsected of flooding

Post by TrevorH » 2018/02/01 16:00:45

It sounds like they think it has been compromised. Depending on *how* it's been compromised, you may be able to look at traffic using tools like iptraf, iftop, tcpdump and/or tshark. That may or may not tell you anything depending on how serious the compromise is - a root compromise could allow the attackers to hide their own traffic from any tools installed and run on the same machine and only be detectable from outside. If you use tcpdump/tshark to look at the traffic, make sure you exclude your own ssh session from the capture or it will loop.

Also check your logfiles though again, an attacker with root can fake those too.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: One of my servers is supsected of flooding

Post by username » 2018/02/01 18:27:42

Well, this server is prob lost. I will never know if it's safe. But I wanted to find out where was the breach. If its was caused by an exploit or a misconfiguration. If the other twin server could be attacked the same way.

I didn't see anything weird in the logs. Except the vsFTP logs were blank.

I'm running tcpdump. I took care to exclude my own IP but not my SSH port.

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: One of my servers is supsected of flooding

Post by avij » 2018/02/01 20:58:58

It may also be possible that your server is not compromised, but it is still used for DDoS purposes. One example is that if you run a public ntp server without the usual restrict default limited nomodify notrap nopeer noquery configuration option, bad people can use your server for an amplification attack by sending it UDP packets with a forged source IP address.

Examining what your server sends is the key. Something like tcpdump -c100 -n src host 192.0.2.0 and src port not 80 and host not your.home.ip.address may show something, where 192.0.2.0 is your server's IP address. Add "and src port not xx" entries as appropriate to filter out the "good" traffic from the tcpdump.

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: One of my servers is supsected of flooding

Post by Whoever » 2018/02/02 04:02:12

If you run a public DNS server, make sure that you turn on rate limiting, so that it can't be used for reflection attacks.

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: One of my servers is supsected of flooding

Post by username » 2018/02/02 10:24:40

I turned off sevral services (ntpd, httpd, mysqld, vsftpd, postfix, fail2ban, munin-node) and changed the credentials.

I ran tcpdump on both servers : the secondary one (the one that's potentially compromised) and the main one (configured the same way but hosted by another company).

After 15 min running, the pcap file on the potentially compromised server is already 20 times bigger than on the other server and prints pages and pages of logs...

I see a lot of ARP requests (my server is listed as myownserver.net)

Code: Select all

11:09:24.519042 ARP, Request who-has 62-210-215-201.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:24.534219 ARP, Request who-has 10.191.193.128 (d4:ae:52:cc:9e:39 (oui Unknown)) tell 10.191.193.1, length 46
11:09:25.510091 ARP, Request who-has 62-210-215-149.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:25.581209 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:25.715476 IP6 fe80::226:bff:feef:59ff > ff02::1: ICMP6, router advertisement, length 32
11:09:25.781358 ARP, Request who-has 62-210-215-70.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:25.816756 ARP, Request who-has 62-210-215-124.rev.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:25.922916 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:26.395592 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:26.631792 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:26.739628 ARP, Request who-has 62-210-215-177.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:26.797651 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:27.137740 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:27.675753 IP 62-210-215-57.rev.poneytelecom.eu.50966 > 62-210-215-255.poneytelecom.eu.32412: UDP, length 21
11:09:27.675767 IP 62-210-215-57.rev.poneytelecom.eu.54375 > 62-210-215-255.poneytelecom.eu.32414: UDP, length 21
11:09:27.794714 ARP, Request who-has 62-210-215-70.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:27.856034 ARP, Request who-has 62-210-215-124.rev.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:28.422553 IP 62-210-215-85.rev.poneytelecom.eu.51786 > 62-210-215-255.poneytelecom.eu.32412: UDP, length 21
11:09:28.422567 IP 62-210-215-85.rev.poneytelecom.eu.48964 > 62-210-215-255.poneytelecom.eu.32414: UDP, length 21
11:09:28.430309 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:28.643340 
11:09:28.777555 ARP, Request who-has 62-210-215-157.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:28.899797 ARP, Request who-has 62-210-215-134.rev.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:29.121367 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:29.125658 IP myownserver.net.48440 > nscache-1.online.net.domain: 53977+ PTR? 244.20.93.85.in-addr.arpa. (43)
11:09:29.617629 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
11:09:29.654716 ARP, Request who-has 10.191.193.14 (d4:ae:52:ca:11:fe (oui Unknown)) tell 10.191.193.1, length 46
11:09:29.726646 IP 45.248.56.105.56658 > myownserver.net.microsoft-ds: Flags [S], seq 1280344294, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
11:09:29.795331 ARP, Request who-has 62-210-215-70.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
11:09:30.149889 ARP, Request who-has 62-210-215-158.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
A lot of references to the ISP (online.net and their own infrastructure is called poney.telecom.eu), but maybe because they are monitoring this server :

Code: Select all

11:09:34.130833 IP myownserver.net.48506 > nscache-1.online.net.domain: 64310+ PTR? 201.215.210.62.in-addr.arpa. (45)
11:09:34.131143 IP nscache-1.online.net.domain > myownserver.net.48506: 64310 1/2/0 PTR 62-210-215-201.poneytelecom.eu. (135)
11:09:34.131322 IP myownserver.net.48304 > nscache-1.online.net.domain: 32695+ PTR? 6.16.210.62.in-addr.arpa. (42)
11:09:34.131602 IP nscache-1.online.net.domain > myownserver.net.48304: 32695 1/2/0 PTR nscache-1.online.net. (112)
11:09:34.133177 IP myownserver.net.42683 > nscache-1.online.net.domain: 11430+ PTR? 207.215.210.62.in-addr.arpa. (45)
11:09:34.133529 IP nscache-1.online.net.domain > myownserver.net.42683: 11430 1/2/0 PTR 62-210-215-207.poneytelecom.eu. (135)
11:09:34.133712 IP myownserver.net.44511 > nscache-1.online.net.domain: 48630+ PTR? 22.215.210.62.in-addr.arpa. (44)
11:09:34.134165 IP nscache-1.online.net.domain > myownserver.net.44511: 48630 1/2/0 PTR 62-210-215-22.rev.poneytelecom.eu. (137)
11:09:34.134340 IP myownserver.net.52009 > nscache-1.online.net.domain: 34791+ PTR? 192.215.210.62.in-addr.arpa. (45)

Code: Select all

11:09:24.114692 IP nscache-1.online.net.domain > myownserver.net.39652: 17931 NXDomain 0/1/0 (154)
11:09:24.114779 IP myownserver.net.35071 > nscache-1.online.net.domain: 34448+ PTR? a.4.d.2.0.0.e.f.f.f.0.0.4.5.0.5.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
11:09:24.115172 IP nscache-1.online.net.domain > myownserver.net.35071: 34448 NXDomain* 0/1/0 (149)
11:09:24.115290 IP myownserver.net.47790 > nscache-1.online.net.domain: 48432+ PTR? 0.a.0.b.8.c.f.f.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
11:09:24.115663 IP nscache-1.online.net.domain > myownserver.net.47790: 48432 NXDomain 0/1/0 (154)
11:09:24.115762 IP myownserver.net.47645 > nscache-1.online.net.domain: 32533+ PTR? c.7.f.b.8.c.f.f.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
11:09:24.116147 IP nscache-1.online.net.domain > myownserver.net.47645: 32533 NXDomain 0/1/0 (154)
11:09:24.116234 IP myownserver.net.51298 > nscache-1.online.net.domain: 23354+ PTR? c.7.f.b.8.c.e.f.f.f.2.5.e.a.6.d.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
Last edited by username on 2018/02/02 10:45:52, edited 2 times in total.

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: One of my servers is supsected of flooding

Post by username » 2018/02/02 10:39:01

avij wrote:It may also be possible that your server is not compromised, but it is still used for DDoS purposes. One example is that if you run a public ntp server without the usual restrict default limited nomodify notrap nopeer noquery configuration option, bad people can use your server for an amplification attack by sending it UDP packets with a forged source IP address.
I think I'm using the default CentOS configuration for /etc/ntp.conf :

Code: Select all

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/key

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: One of my servers is supsected of flooding

Post by TrevorH » 2018/02/02 10:47:36

There's an awful lot of DNS activity there but that may be because you haven't told tcpdump to not do dns lookups. Try using -n -nn on the tcpdump to give just ip addresses and numerical port info.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

username
Posts: 76
Joined: 2013/03/04 13:17:23

Re: One of my servers is supsected of flooding

Post by username » 2018/02/02 11:13:05

TrevorH wrote:There's an awful lot of DNS activity there but that may be because you haven't told tcpdump to not do dns lookups. Try using -n -nn on the tcpdump to give just ip addresses and numerical port info.
OK, now I'm using these parameters on both servers : tcpdump -n -nn -w filename8.pcap -i em1 not host myownserverip and src port not 80

It's weird that all this DNS activity didn't show up in the other server logs though...

Now the logs are still growing 20 times faster than on the other server

Here is a sample :

Code: Select all

12:11:34.302276 IP myownserver.net.41096 > nscache-1.online.net.domain: 28157+ PTR? 106.98.245.77.in-addr.arpa. (44)
12:11:34.302567 IP nscache-1.online.net.domain > myownserver.net.41096: 28157 NXDomain 0/1/0 (112)
12:11:34.304889 IP myownserver.net.43111 > nscache-1.online.net.domain: 58653+ PTR? 15.193.191.10.in-addr.arpa. (44)
12:11:34.305176 IP nscache-1.online.net.domain > myownserver.net.43111: 58653 NXDomain* 0/1/0 (103)
12:11:34.305356 IP myownserver.net.58959 > nscache-1.online.net.domain: 48475+ PTR? 92.193.191.10.in-addr.arpa. (44)
12:11:34.305888 IP nscache-1.online.net.domain > myownserver.net.58959: 48475 NXDomain* 0/1/0 (103)
12:11:34.306062 IP myownserver.net.60238 > nscache-1.online.net.domain: 20452+ PTR? 244.20.93.85.in-addr.arpa. (43)
12:11:34.501113 ARP, Request who-has 62-210-8-167.rev.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
12:11:34.740542 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:34.995402 IP6 fe80::226:bff:feef:59ff > ff02::1: ICMP6, router advertisement, length 32
12:11:35.018516 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:35.256387 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:35.622426 
12:11:35.659714 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:35.721344 ARP, Request who-has 62-210-215-1.poneytelecom.eu tell 212-129-49-35.rev.poneytelecom.eu, length 46
12:11:35.787548 ARP, Request who-has 62-210-215-87.rev.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
12:11:35.836612 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:35.933207 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:36.149934 ARP, Request who-has 62-210-215-70.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
12:11:36.218392 ARP, Request who-has 62-210-215-124.rev.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
12:11:36.601929 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:37.229751 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:37.360235 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:37.436457 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:37.456850 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:37.495323 IP 62-210-215-57.rev.poneytelecom.eu.50966 > 62-210-215-255.poneytelecom.eu.32412: UDP, length 21
12:11:37.495337 IP 62-210-215-57.rev.poneytelecom.eu.54375 > 62-210-215-255.poneytelecom.eu.32414: UDP, length 21
12:11:37.648992 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:38.241946 IP 62-210-215-85.rev.poneytelecom.eu.51786 > 62-210-215-255.poneytelecom.eu.32412: UDP, length 21
12:11:38.241960 IP 62-210-215-85.rev.poneytelecom.eu.48964 > 62-210-215-255.poneytelecom.eu.32414: UDP, length 21
12:11:38.375750 ARP, Request who-has 62-210-215-124.rev.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
12:11:38.395507 ARP, Request who-has 10.191.193.82 (d4:ae:52:cd:9f:18 (oui Unknown)) tell 10.191.193.1, length 46
12:11:38.644896 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:38.710410 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:38.885472 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:38.898230 ARP, Request who-has 62-210-215-218.poneytelecom.eu tell 62-210-215-1.poneytelecom.eu, length 46
12:11:38.998676 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:39.294701 IP6 fe80::250:56ff:fe01:45fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:11:39.311187 IP myownserver.net.60238 > nscache-1.online.net.domain: 20452+ PTR? 244.20.93.85.in-addr.arpa. (43)
12:11:39.311660 IP nscache-1.online.net.domain > myownserver.net.60238: 20452 ServFail 0/0/0 (43)
12:11:39.313789 IP myownserver.net.43018 > nscache-1.online.net.domain: 49792+ PTR? 39.215.210.62.in-addr.arpa. (44)
12:11:39.314377 IP nscache-1.online.net.domain > myownserver.net.43018: 49792 1/2/0 PTR www.simittechnology.com. (127)
12:11:39.314639 IP myownserver.net.45024 > nscache-1.online.net.domain: 65169+ PTR? 25.180.82.183.in-addr.arpa. (44)
12:11:39.315041 IP nscache-1.online.net.domain > myownserver.net.45024: 65169 1/2/0 PTR broadband.actcorp.in. (116)
12:11:39.315734 IP myownserver.net.48244 > nscache-1.online.net.domain: 20207+ PTR? 7.193.191.10.in-addr.arpa. (43)
12:11:39.315968 IP nscache-1.online.net.domain > myownserver.net.48244: 20207 NXDomain* 0/1/0 (102)
12:11:39.316177 IP myownserver.net.56149 > nscache-1.online.net.domain: 65358+ PTR? 6.193.191.10.in-addr.arpa. (43)
12:11:39.316551 IP nscache-1.online.net.domain > myownserver.net.56149: 65358 NXDomain* 0/1/0 (102)
12:11:39.331057 IP myownserver.net.37684 > nscache-1.online.net.domain: 10764+ PTR? 38.193.191.10.in-addr.arpa. (44)
12:11:39.331371 IP nscache-1.online.net.domain > myownserver.net.37684: 10764 NXDomain* 0/1/0 (103)
12:11:39.331623 IP myownserver.net.42121 > nscache-1.online.net.domain: 10593+ PTR? 105.43.52.196.in-addr.arpa. (44)
I also got an answer from the hosting company Online.net. I wanted to know if they had some clues about the breach. They just told me :

Code: Select all

DDOS from IP myownserverip (attack ID 1088855): protocols : tcp, targets: 123.207.54.16/32, sports: Dynamic (1024-65535), dports
So my server was attacking some others based in China.
Last edited by username on 2018/02/02 13:36:14, edited 2 times in total.

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: One of my servers is suspected of flooding

Post by avij » 2018/02/02 12:55:07

"sports: Dynamic (1024-65535), dports"

Source and destination ports .. but did you or they accidentally leave out the destination ports part?

Post Reply