Centos 7 / process inetd HIGH cpu load *Found > mining mallware * solved *

[mrmartijn]

Good day people,

Thanks in advance for tips & support. I'm not able to figure out why the process "inetd" is using so much cpu resources. Below output of top

Code: Select all

top - 14:40:11 up 25 days, 13:02,  1 user,  load average: 4.61, 4.16, 3.73
Tasks: 283 total,   2 running, 280 sleeping,   0 stopped,   1 zombie
%Cpu(s):  2.9 us,  2.9 sy, 94.1 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  3881584 total,  1013392 free,  1387396 used,  1480796 buff/cache
KiB Swap:  1048572 total,   963160 free,    85412 used.  1883772 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  436 root      20   0   65536    844    608 S 193.8  0.0  93:08.42 inetd
20163 root      20   0  157860   2364   1496 R   6.2  0.1   0:00.01 top
    1 root      20   0  199096   3328   2036 S   0.0  0.1   8:22.58 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.34 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:49.58 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S   0.0  0.0   0:04.32 migration/0
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S   0.0  0.0   8:49.77 rcu_sched
   10 root      rt   0       0      0      0 S   0.0  0.0   0:08.63 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:07.08 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:04.87 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   1:05.10 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
   18 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.70 khungtaskd
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.02 writeback
   21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd
   22 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
   23 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd
   24 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md
   30 root      20   0       0      0      0 S   0.0  0.0   5:37.47 kswapd0
   31 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd
   32 root      39  19       0      0      0 S   0.0  0.0   0:02.76 khugepaged
   33 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto
   41 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld
   43 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kmpath_rdacd
   44 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kpsmoused
   45 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ipv6_addrconf
   64 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 deferwq
   98 root      20   0       0      0      0 S   0.0  0.0   0:05.80 kauditd
  283 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff
  289 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_0
  290 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_0
  291 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_1
  292 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_1
  294 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ttm_swap
  361 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kdmflush
  362 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
  373 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kdmflush
  374 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
  387 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
  388 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfsalloc
  389 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfs_mru_cache
  390 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfs-buf/dm-0
  391 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfs-data/dm-0
  392 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfs-conv/dm-0
  393 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 xfs-cil/dm-0
  394 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfs-reclaim/dm-
  395 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfs-log/dm-0
  396 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 xfs-eofblocks/d
  397 root      20   0       0      0      0 S   0.0  0.0   6:34.12 xfsaild/dm-0
  435 root      20   0    1656     36      4 S   0.0  0.0   0:00.27 inetd
  466 root      20   0   37248   7128   6848 S   0.0  0.2  15:12.54 systemd-journal
  484 root      20   0  200776    600    600 S   0.0  0.0   0:00.00 lvmetad

OS Info >

Icon name: computer-vm
Chassis: vm
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.21.1.el7.x86_64
Architecture: x86-64

Any suggestions how to start finding what is causing this ?

edit: type in subject

Edit: Solved, server was compremissed with mining software. Data = moved, server trashed.

[avij]

I ... have a hunch, but try ps -ef | grep inetd to see which inetd it is running.

[mrmartijn]

Thanks so much for your quick reply, the outcome :

Code: Select all

# ps -ef | grep inetd
root      9328  8701  0 16:11 pts/0    00:00:00 grep --color=auto inetd
root      9618     1  0 15:04 ?        00:00:00 inetd
root      9619  9618 99 15:04 ?        01:09:26 inetd
root     14241     1  0 13:56 ?        00:00:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid

currently it's PID 9619, when I try to find which process is running (the path) I end up with

Code: Select all

# readlink /proc/9619/exe
/usr/local/bin/.~CD1363D (deleted)

Version info from xinetd

Code: Select all

Installed Packages
xinetd.x86_64                                                                                                      2:2.3.15-13.el7  

[avij]

Yeah, right. The funny thing is that the xinetd package does not provide an inetd binary. The binary in it is called xinetd. Also the /usr/local/bin path is suspicious.

I'm sorry to say, but that inetd thing you have running is not a CentOS binary. It's likely someone's cpuminer (crypto currency miner), renamed to inetd so that it would look less suspicious.

Because it was running as root, only $DEITY knows what kind of backdoors it has set up. I would suggest a back up of your data on the server, then wipe and reinstall, then secure your system better so that it would not let malicious people gain access to your server.

[hunter86_bg]

What about

Code: Select all

ls -l /proc/9619/fd

[mrmartijn]

Code: Select all

# ls -l /proc/20387/fd
total 0
lr-x------ 1 root root 64 Apr  5 16:24 0 -> /dev/null
l-wx------ 1 root root 64 Apr  5 16:24 1 -> /dev/null
lrwx------ 1 root root 64 Apr  5 16:24 10 -> anon_inode:[eventpoll]
lr-x------ 1 root root 64 Apr  5 16:24 11 -> pipe:[36314156]
l-wx------ 1 root root 64 Apr  5 16:24 12 -> pipe:[36314156]
lrwx------ 1 root root 64 Apr  5 16:24 13 -> anon_inode:[eventfd]
lr-x------ 1 root root 64 Apr  5 16:24 14 -> /dev/null
lrwx------ 1 root root 64 Apr  5 16:24 15 -> socket:[36313636]
l-wx------ 1 root root 64 Apr  5 16:24 2 -> /dev/null
lrwx------ 1 root root 64 Apr  5 16:24 3 -> socket:[36314155]
lrwx------ 1 root root 64 Apr  5 16:24 4 -> anon_inode:[eventpoll]
lr-x------ 1 root root 64 Apr  5 16:24 5 -> pipe:[36313633]
l-wx------ 1 root root 64 Apr  5 16:24 6 -> pipe:[36313633]
lr-x------ 1 root root 64 Apr  5 16:24 7 -> pipe:[36313634]
l-wx------ 1 root root 64 Apr  5 16:24 8 -> pipe:[36313634]
lrwx------ 1 root root 64 Apr  5 16:24 9 -> anon_inode:[eventfd]

another PID, killed the process but it just restarts.

DirectAdmin is running on this server (it's not mine to maintain, i Don't use direct admin, all my other servers also Centos 7 doesn't have like any inetd processes.

[TrevorH]

It's about 99.9% certain that your server has been hacked and that executable is being run by whoever hacked you. You can no longer trust that machine as you have no idea what they've done to it to ensure their continued access. You need to back up your data, reformat, reinstall the system and restore.

[mrmartijn]

I'm sincerely thankfull for your quick input and replies. Any other suggestions are still welcome of course.

I'm going to monitor outgoing traffic looking for mining pools and start the migration to another fresh server asap.

PS. Both clamav & rkhunter didn't find any suspicious files at all. Al though they both (running as root) came up with some non readable errors.

[hunter86_bg]

I'd recommend you to manually download and transfer the 'rpm' package, reinstall it and run:

Code: Select all

rpm -Va

Then observe for binary files that do not match.

[mrmartijn]

I didn't find observe any binary files that were not matching.

Meanwhile I've inspected the exe in

/proc/{pid}/exe

When opening with with a text editor I find

things like

Code: Select all

@^@stratum+tcp://^@cryptonight^@pools^@api^@cc-client^@cc-server^@userpass^@keepalive^@nicehash^@algo^@av^@background^@cpu-$
@CONFIG_UPDATED^@client_status^@current_status^@client_id^@current_pool^@current_algo_name^@cpu_brand^@external_ip^@hugepages_available^@hugepages_enabled^@double_hash_mode^@cpu_is_x64^@cpu_has_aes^@hashrate_short^@hashrate_medium^

Most defenitly a miner ...

Code: Select all

$/exe^@rsyslogd^@named^@httpd^@crond^@mysqld^@rpcbind^@auditd^@chronyd^@postfix^@inetd^@dhcpd^@ntpd^@sendmail^@atd^@nginx^

this seems to be the program names it uses to appear as a normal process.

The miner is trying to connect to 103.214.147.31:433 (which is down).

I've did a search in alle files starting with *.sh & *.py looking for things like "103.214.147.31" or "stratum" but did not find any scripts.

Moving of data (site & email) is in progresss but still would like to find out what it's diong, where it's starting from and or how to remove it.

Rest of the data in the /proc/{pid} directory

Code: Select all

# ls -all
total 0
dr-xr-xr-x   8 root root 0 Apr  6 14:46 .
dr-xr-xr-x 354 root root 0 Apr  5 12:16 ..
dr-xr-xr-x   2 root root 0 Apr  6 14:53 attr
-rw-r--r--   1 root root 0 Apr  6 15:14 autogroup
-r--------   1 root root 0 Apr  6 15:14 auxv
-r--r--r--   1 root root 0 Apr  6 15:14 cgroup
--w-------   1 root root 0 Apr  6 15:14 clear_refs
-r--r--r--   1 root root 0 Apr  6 14:53 cmdline
-rw-r--r--   1 root root 0 Apr  6 15:14 comm
-rw-r--r--   1 root root 0 Apr  6 15:14 coredump_filter
-r--r--r--   1 root root 0 Apr  6 15:14 cpuset
lrwxrwxrwx   1 root root 0 Apr  6 14:47 cwd -> /
-r--------   1 root root 0 Apr  6 15:14 environ
lrwxrwxrwx   1 root root 0 Apr  6 14:46 exe -> /usr/local/bin/.~bf564b0 (deleted)
dr-x------   2 root root 0 Apr  6 14:46 fd
dr-x------   2 root root 0 Apr  6 15:14 fdinfo
-r--------   1 root root 0 Apr  6 15:14 io
-rw-------   1 root root 0 Apr  6 15:14 limits
-rw-r--r--   1 root root 0 Apr  6 15:14 loginuid
-r--r--r--   1 root root 0 Apr  6 15:14 maps
-rw-------   1 root root 0 Apr  6 15:14 mem
-r--r--r--   1 root root 0 Apr  6 15:14 mountinfo
-r--r--r--   1 root root 0 Apr  6 15:14 mounts
-r--------   1 root root 0 Apr  6 15:14 mountstats
dr-xr-xr-x   6 root root 0 Apr  6 15:14 net
dr-x--x--x   2 root root 0 Apr  6 15:14 ns
-r--r--r--   1 root root 0 Apr  6 15:14 numa_maps
-rw-r--r--   1 root root 0 Apr  6 15:14 oom_adj
-r--r--r--   1 root root 0 Apr  6 15:14 oom_score
-rw-r--r--   1 root root 0 Apr  6 15:14 oom_score_adj
-r--r--r--   1 root root 0 Apr  6 15:14 pagemap
-r--r--r--   1 root root 0 Apr  6 15:14 personality
lrwxrwxrwx   1 root root 0 Apr  6 15:14 root -> /
-rw-r--r--   1 root root 0 Apr  6 15:14 sched
-r--r--r--   1 root root 0 Apr  6 15:14 schedstat
-r--r--r--   1 root root 0 Apr  6 15:14 sessionid
-r--r--r--   1 root root 0 Apr  6 15:14 smaps
-r--r--r--   1 root root 0 Apr  6 15:14 stack
-r--r--r--   1 root root 0 Apr  6 14:53 stat
-r--r--r--   1 root root 0 Apr  6 15:04 statm
-r--r--r--   1 root root 0 Apr  6 14:47 status
-r--r--r--   1 root root 0 Apr  6 15:14 syscall
dr-xr-xr-x  11 root root 0 Apr  6 15:14 task
-r--r--r--   1 root root 0 Apr  6 15:14 wchan

Any suggestions and or tips are still welcome.