We have a dedicated server, running Centos 6.10, shared to many user to run their website with Apache 2.2. Last week, I found that a directory of a virtual host has been compromised, and found some webshells & also perl scripts there.
Then I checked with top command, and found some abnormal processes which are running under apache user.
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24094 apache 20 0 364m 30m 3696 S 10.6 0.1 0:00.43 httpd
24064 apache 20 0 348m 14m 3616 S 9.3 0.1 0:03.67 httpd
10026 mysql 20 0 1971m 211m 7476 S 5.0 0.9 37:41.55 mysqld
24192 apache 20 0 366m 31m 5580 S 4.3 0.1 0:00.34 httpd
24099 apache 20 0 363m 28m 3580 S 2.0 0.1 0:00.06 httpd
24190 apache 20 0 356m 21m 5420 S 1.7 0.1 0:00.66 httpd
11088 apache 20 0 39612 3212 1100 S 1.0 0.0 91:16.90 perl
38246 apache 20 0 39612 3244 1100 S 1.0 0.0 82:55.21 perl
38247 apache 20 0 39612 3228 1100 S 1.0 0.0 83:24.10 perl
44555 apache 20 0 39612 5196 1100 S 1.0 0.0 75:11.96 perl
49051 apache 20 0 39612 5196 1100 S 1.0 0.0 64:02.18 perl
64519 apache 20 0 39612 3180 1100 S 1.0 0.0 92:04.71 perl
64520 apache 20 0 39612 3176 1100 S 1.0 0.0 92:02.29 perl
2178 elastics 20 0 5650m 307m 1888 S 0.7 1.3 1702:10 java
11089 apache 20 0 39612 3200 1100 S 0.7 0.0 91:11.27 perl
20743 apache 20 0 39612 5196 1100 S 0.7 0.0 61:01.45 perl
26433 apache 20 0 39612 3188 1100 S 0.7 0.0 84:04.77 perl
44556 apache 20 0 39612 5196 1100 S 0.7 0.0 75:03.41 perl
49070 apache 20 0 39612 5188 1100 S 0.7 0.0 63:43.38 perl
278 root 20 0 0 0 0 S 0.3 0.0 762:46.51 hpvsa/5
1430 root 20 0 0 0 0 S 0.3 0.0 843:22.12 kondemand/0
1432 root 20 0 0 0 0 R 0.3 0.0 90:40.32 kondemand/2
1436 root 20 0 0 0 0 S 0.3 0.0 178:35.30 kondemand/6
after tracing, I've found that attacker has add some cronjob to execute scripts under apache user, these two process started by cronjob has been killed, however, perl processes as shown above is still running.
This is the first time I have to face to security issue, so I dont know how to start to investigate.
Please guide or give me yours advices, thanks.