Apache security issue

Support for security such as Firewalls and securing linux
Post Reply
packetman99
Posts: 13
Joined: 2017/02/23 15:26:27

Apache security issue

Post by packetman99 » 2018/07/31 06:05:00

Hi There,

I've noticed a few oddities on a webserver I'm hosting for a few customers but tracking the actual security hole seems allot more challenging then originally anticipated.

So I've started noticing these entries when running netstat.. Unfortunately I didn't gather enough data before restarting Apache. I'll get more data next time but thought to post what I have.

So this seemed suspicious since there's no reason for my webserver to connect outside to port 80. Any idea how this is done ?
tcp 0 1 x.x.x.x:41782 91.191.19.205:80 SYN_SENT
tcp 0 1 x.x.x.x:41780 91.191.19.205:80 SYN_SENT
tcp 0 1 x.x.x.x:41778 91.191.19.205:80 SYN_SENT
tcp 0 1 x.x.x.x:41768 91.191.19.205:80 SYN_SENT

lsof -Pwlni |grep 91.191.19.205
/usr/bin/ 10296 48 3u IPv4 15447859 0t0 TCP x.x.x.x:45286->91.191.19.205:80 (SYN_SENT)

PID 10296 were referring to a non existing process (/usr/bin/atd) : Sorry , I lost the lsof data for this specific process

I've also noticed a few other processes running under the apache username (syslogd, apache2) which he somehow installed under the shared memory folder (/dev/shm/.mine)

[root@eu-hosting1 shm]# pwd
/dev/shm
[root@eu-hosting1 shm]# ls -la
total 0
drwxrwxrwt 3 root root 60 Jul 31 08:02 .
drwxr-xr-x 18 root root 2780 Jul 13 18:05 ..
drwxr-xr-x 2 apache apache 280 Jun 16 23:35 .mine

[root@eu-hosting1 .mine]# ls -la
total 3620
drwxr-xr-x 2 apache apache 280 Jun 16 23:35 .
drwxrwxrwt 3 root root 60 Jul 31 08:02 ..
-rwxr-xr-x 1 apache apache 303 Jun 4 19:14 a
-rwxr-xr-x 1 apache apache 1476 Jul 25 22:35 apache2
-rw-r--r-- 1 apache apache 6 Jul 30 21:12 bash.pid
-rw-r--r-- 1 apache apache 45 Jul 30 21:12 cron.d
-rw-r--r-- 1 apache apache 15 Jul 30 21:12 dir.dir
-rwxr-xr-x 1 apache apache 15125 Feb 20 2016 e
-rwxr-xr-x 1 apache apache 838583 Feb 20 2016 f
-rwxr-xr-x 1 apache apache 281 Jun 10 00:30 r
-rwxr-xr-x 1 apache apache 1687632 May 6 16:28 syslogd
-rwxr-xr-x 1 apache apache 1125152 Jun 9 16:43 systemd
-rwxr--r-- 1 apache apache 176 Jul 30 21:12 upd
-rwxr-xr-x 1 apache apache 24 Oct 4 2017 x

Any help or pointer will be highly appreciated.

Thanks in advance,

packetman99
Posts: 13
Joined: 2017/02/23 15:26:27

Re: Apache security issue

Post by packetman99 » 2018/08/01 05:45:40

Here's a little more information. lsof gave the following information and when checking the PID with ps ax |grep 29342 it's referring to a command / executable that doesn't exist. I'm guessing this can be faked to hide the real process ? Any way of finding out how he's doing this ?

29342 ? S 0:10 /usr/bin/atd
[root@eu-hosting1 shm]# ls -la /usr/bin/atd
ls: cannot access /usr/bin/atd: No such file or directory


[root@eu-hosting1 shm]# lsof -p 29342
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
/usr/bin/ 29342 apache cwd DIR 253,1 64 104882207 /tmp
/usr/bin/ 29342 apache rtd DIR 253,1 251 64 /
/usr/bin/ 29342 apache txt REG 253,1 11408 12723837 /usr/bin/perl
/usr/bin/ 29342 apache mem REG 253,1 44520 138412117 /usr/lib64/perl5/vendor_perl/auto/Socket/Socket.so
/usr/bin/ 29342 apache mem REG 253,1 19808 150995052 /usr/lib64/perl5/auto/IO/IO.so
/usr/bin/ 29342 apache mem REG 253,1 11448 55297 /usr/lib64/libfreebl3.so
/usr/bin/ 29342 apache mem REG 253,1 2173512 87710 /usr/lib64/libc-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 144792 94872 /usr/lib64/libpthread-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 14872 94880 /usr/lib64/libutil-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 41080 94850 /usr/lib64/libcrypt-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 1139680 94854 /usr/lib64/libm-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 19776 94852 /usr/lib64/libdl-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 117680 94856 /usr/lib64/libnsl-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 106848 94874 /usr/lib64/libresolv-2.17.so
/usr/bin/ 29342 apache mem REG 253,1 1647272 100663389 /usr/lib64/perl5/CORE/libperl.so
/usr/bin/ 29342 apache mem REG 253,1 164240 87703 /usr/lib64/ld-2.17.so
/usr/bin/ 29342 apache 0r FIFO 0,9 0t0 15526304 pipe
/usr/bin/ 29342 apache 1w FIFO 0,9 0t0 15526296 pipe
/usr/bin/ 29342 apache 2w REG 253,1 14284 138460173 /var/log/httpd/error_log
/usr/bin/ 29342 apache 3u IPv4 15526309 0t0 TCP eu-hosting1.bbi.co.bw:45762->91.191.19.205:http (ESTABLISHED)
/usr/bin/ 29342 apache 27u unix 0xffff948fa1663000 0t0 15563501 socket

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: Apache security issue

Post by avij » 2018/08/01 05:48:01

You have been hacked. You should backup, wipe and reinstall your system, and make sure you can't be hacked again by installing updates regularly and disabling password logins for SSH (use keys instead).

packetman99
Posts: 13
Joined: 2017/02/23 15:26:27

Re: Apache security issue

Post by packetman99 » 2018/08/01 05:55:22

I'm trying to understand how this happened. My ssh access is pretty secure , port 22 is firewalled off to only allow my management address and I don't allow root logins. Got like a password sentence of around 16 characters using caps , lower case and plenty of special characters.

The server were also up2date before I started copying the web content across.

CentOS Linux release 7.5.1804 (Core)

I'm suspecting it's one of the Joomla or drupal sites we're hosting. Does anyone know how to find the hole and patch it up ?

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: Apache security issue

Post by avij » 2018/08/01 06:38:26

It's hard to tell. Logs may or may not have something, but the attacker may have wiped out the log entries as well.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Apache security issue

Post by TrevorH » 2018/08/01 13:07:18

I don't see any clues there that the attacker has root access. Everything I see appears to be running as the apache user.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: Apache security issue

Post by avij » 2018/08/01 13:50:59

I don't think you can create a malicious executable at /usr/bin/atd without root access.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Apache security issue

Post by TrevorH » 2018/08/01 13:58:23

Yes, that is odd since everything else is owned by the apache user. Unless it was ./usr/bin/atd though that doesn't look likely from the info we have.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

packetman99
Posts: 13
Joined: 2017/02/23 15:26:27

Re: Apache security issue

Post by packetman99 » 2018/08/02 05:36:03

Ye , it's kinda weird .. the process is owned by apache and the file /usr/bin/atd doesn't really exists. My guess is he's somehow faking it to hide the real process running in the back. Any ideas how he managed to create a folder under the /dev/shm folder which I believe is shared memory space ?

It's will be pretty simple to reload the server but only want to go this route after figuring out how he managed to do it.

Thanks in advance,

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Apache security issue

Post by TrevorH » 2018/08/02 08:12:39

/dev/shm is a tmpfs and writable by anyone by default.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply