Centos server compromised, how to patch ?

Support for security such as Firewalls and securing linux
Post Reply
tinhduong
Posts: 7
Joined: 2012/10/25 08:49:32
Contact:

Centos server compromised, how to patch ?

Post by tinhduong » 2018/07/22 06:52:15

Hi everyone,
We have a dedicated server, running Centos 6.10, shared to many user to run their website with Apache 2.2. Last week, I found that a directory of a virtual host has been compromised, and found some webshells & also perl scripts there.

Then I checked with top command, and found some abnormal processes which are running under apache user.

Code: Select all

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                        
24094 apache    20   0  364m  30m 3696 S 10.6  0.1   0:00.43 httpd                                                          
24064 apache    20   0  348m  14m 3616 S  9.3  0.1   0:03.67 httpd                                                          
10026 mysql     20   0 1971m 211m 7476 S  5.0  0.9  37:41.55 mysqld                                                         
24192 apache    20   0  366m  31m 5580 S  4.3  0.1   0:00.34 httpd                                                          
24099 apache    20   0  363m  28m 3580 S  2.0  0.1   0:00.06 httpd                                                          
24190 apache    20   0  356m  21m 5420 S  1.7  0.1   0:00.66 httpd                                                          
11088 apache    20   0 39612 3212 1100 S  1.0  0.0  91:16.90 perl                                                           
38246 apache    20   0 39612 3244 1100 S  1.0  0.0  82:55.21 perl                                                           
38247 apache    20   0 39612 3228 1100 S  1.0  0.0  83:24.10 perl                                                           
44555 apache    20   0 39612 5196 1100 S  1.0  0.0  75:11.96 perl                                                           
49051 apache    20   0 39612 5196 1100 S  1.0  0.0  64:02.18 perl                                                           
64519 apache    20   0 39612 3180 1100 S  1.0  0.0  92:04.71 perl                                                           
64520 apache    20   0 39612 3176 1100 S  1.0  0.0  92:02.29 perl                                                           
 2178 elastics  20   0 5650m 307m 1888 S  0.7  1.3   1702:10 java                                                           
11089 apache    20   0 39612 3200 1100 S  0.7  0.0  91:11.27 perl                                                           
20743 apache    20   0 39612 5196 1100 S  0.7  0.0  61:01.45 perl                                                           
26433 apache    20   0 39612 3188 1100 S  0.7  0.0  84:04.77 perl                                                           
44556 apache    20   0 39612 5196 1100 S  0.7  0.0  75:03.41 perl                                                           
49070 apache    20   0 39612 5188 1100 S  0.7  0.0  63:43.38 perl                                                           
  278 root      20   0     0    0    0 S  0.3  0.0 762:46.51 hpvsa/5                                                        
 1430 root      20   0     0    0    0 S  0.3  0.0 843:22.12 kondemand/0                                                    
 1432 root      20   0     0    0    0 R  0.3  0.0  90:40.32 kondemand/2                                                    
 1436 root      20   0     0    0    0 S  0.3  0.0 178:35.30 kondemand/6   

after tracing, I've found that attacker has add some cronjob to execute scripts under apache user, these two process started by cronjob has been killed, however, perl processes as shown above is still running.
This is the first time I have to face to security issue, so I dont know how to start to investigate.
Please guide or give me yours advices, thanks.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Centos server compromised, how to patch ?

Post by TrevorH » 2018/07/22 12:04:14

If they only had access to the apache user then you can probably recover from this. If they gained access to root then you should nuke and reinstall from scratch.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply