URGENT: I think this machine gets hacked...

Support for security such as Firewalls and securing linux
bomin
Posts: 4
Joined: 2007/06/17 05:54:34

URGENT: I think this machine gets hacked...

Post by bomin » 2007/07/24 08:32:02

Hi, I got a situation here; one of my Web server has suspect activity... first, sshd. Two ssh directory can be found in /etc (sshd and ssh2). Parameters like "PermitRootLogin no" seems to be bypassed. There is no additionnal sshd executable, just the default one. I got a script that reload process automatically if not running and I can see this in system log file:
...
Jul 23 18:26:51 www1 sshd: failed
Jul 23 18:27:21 www1 sshd: Stopping sshd failed
Jul 23 18:27:21 www1 [31712]: FATAL ERROR: Creating listener failed: port 22 probably already in use!
Jul 23 18:27:21 www1 sshd: : OpenSSH_3.9p1 on i686-pc-linux-gnu
...
With a lot of brute force attemp... more than normally;
...
Jul 23 12:58:59 www1 [2603]: connection from "213.244.28.44"
Jul 23 12:58:59 www1 [2603]: Wrong password given for user 'root'.
Jul 23 12:59:00 www1 [2604]: password authentication failed. Login to account test not allowed or account non-existent.
Jul 23 12:59:01 www1 [2606]: password authentication failed. Login to account testuser not allowed or account non-existent.
Jul 23 12:59:03 www1 [2608]: password authentication failed. Login to account test1 not allowed or account non-existent.
Jul 23 12:59:04 www1 [2609]: password authentication failed. Login to account test not allowed or account non-existent.
Jul 23 12:59:06 www1 [2610]: password authentication failed. Login to account test not allowed or account non-existent.
Jul 23 12:59:07 www1 [2611]: password authentication failed. Login to account test not allowed or account non-existent.
Jul 23 12:59:09 www1 [2612]: password authentication failed. Login to account testing not allowed or account non-existent.
Jul 23 13:05:51 www1 [30380]: LoginGraceTime exceeded.
Jul 23 13:05:53 www1 [30383]: LoginGraceTime exceeded.
...

I see a lot of ftp_scanner processes running; I kill them all and they come back later (never get this before). FTP is known to be a potential security breach, so I turn it off for the moment.

I run chkrootkit and nothing has been found.

Any tips/advices will be welcome, I'm quite desperate now. Thank you.

RAID-5
Posts: 77
Joined: 2005/05/05 17:35:34
Location: Quebec

URGENT: I think this machine gets hacked...

Post by RAID-5 » 2007/07/24 09:50:56

I forgot to mention... there is a lot of sshd running process, only 1 user logged in.

Thanks

(oops, to avoid confusion, previous post was done on my friend computer)

foxb
Posts: 1924
Joined: 2006/04/20 19:03:33
Location: Montreal/QC

Re: URGENT: I think this machine gets hacked...

Post by foxb » 2007/07/24 15:48:16

Where your server is located?

Locally?

Remote?

RAID-5
Posts: 77
Joined: 2005/05/05 17:35:34
Location: Quebec

Re: URGENT: I think this machine gets hacked...

Post by RAID-5 » 2007/07/24 16:30:58

Unfortunately, the datacenter is about 320 km away from home...

I found one of my other machine with same situation (about sshd) but except that, no suspect activity like the Web server. Fortunately, this server is located at my office, I have physically access everyday... I tried to kill all sshd daemon and restart it with 'service sshd start' and same error message appear (FATAL ERROR: Creating listener failed: port 22 probably already in use!).

Thanks.

foxb
Posts: 1924
Joined: 2006/04/20 19:03:33
Location: Montreal/QC

Re: URGENT: I think this machine gets hacked...

Post by foxb » 2007/07/24 18:32:47

It is hard to say, but at least you can change ssh port and get rid of those hack attempts.

Running any detection tool on infected/hacked machine is useless you need to boot from safe source and then do the scanning.

RAID-5
Posts: 77
Joined: 2005/05/05 17:35:34
Location: Quebec

Re: URGENT: I think this machine gets hacked...

Post by RAID-5 » 2007/07/24 19:21:23

Yes, I already think at this solution, but sshd didn't restart correctly... I can login, but when entering the password the connection is closed. The worst is that sshd totally ignore parameters inside sshd_config...

I tried chkrootkit and rkhunter... any other suggestion?

Do you have any idea how he could get in? Everything was up2date and theses machine were very well firewalled (one of them was in front of a dual OpenBSD statefull firewall (first transparent) and the other server have a good IPtables script... I got up to 70 other CentOS servers, I just want to prevent this.

Thanks.

RAID-5
Posts: 77
Joined: 2005/05/05 17:35:34
Location: Quebec

Re: URGENT: I think this machine gets hacked...

Post by RAID-5 » 2007/07/24 21:21:30

I got some new infos... I found some files under /usr/local/bin

[root@www1 bin]# ls -la
total 11228
drwxr-xr-x 2 root root 4096 Jul 24 16:44 .
drwxr-xr-x 14 root root 4096 Jul 24 16:44 ..
-rw-r--r-- 1 root root 630 Jul 22 02:51 backup-web.sh
-rwxr-x--- 1 root admin 512 Mar 12 18:02 lamp-bind-instal.sh
-rwx------ 1 root root 151427 Jul 24 16:44 rkhunter
lrwxrwxrwx 1 root root 4 Jul 23 08:35 scp -> scp2
-rwxr-xr-x 1 root root 803798 Jul 23 08:35 scp2
lrwxrwxrwx 1 root root 4 Jul 9 20:48 scp.old -> scp2
lrwxrwxrwx 1 root root 5 Jul 23 08:35 sftp -> sftp2
-rwxr-xr-x 1 root root 898734 Jul 23 08:35 sftp2
lrwxrwxrwx 1 root root 5 Jul 9 20:48 sftp.old -> sftp2
lrwxrwxrwx 1 root root 12 Jul 23 08:35 sftp-server -> sftp-server2
-rwxr-xr-x 1 root root 309572 Jul 23 08:35 sftp-server2
lrwxrwxrwx 1 root root 12 Jul 9 20:48 sftp-server.old -> sftp-server2
lrwxrwxrwx 1 root root 4 Jul 23 08:35 ssh -> ssh2
-rwxr-xr-x 1 root root 2541210 Jul 23 08:35 ssh2
lrwxrwxrwx 1 root root 8 Jul 23 08:35 ssh-add -> ssh-add2
-rwxr-xr-x 1 root root 1627671 Jul 23 08:35 ssh-add2
lrwxrwxrwx 1 root root 8 Jul 9 20:48 ssh-add.old -> ssh-add2
lrwxrwxrwx 1 root root 10 Jul 23 08:35 ssh-agent -> ssh-agent2
-rwxr-xr-x 1 root root 1572220 Jul 23 08:35 ssh-agent2
lrwxrwxrwx 1 root root 10 Jul 9 20:48 ssh-agent.old -> ssh-agent2
lrwxrwxrwx 1 root root 12 Jul 23 08:35 ssh-askpass -> ssh-askpass2
lrwxrwxrwx 1 root root 12 Jul 9 20:48 ssh-askpass.old -> ssh-askpass2
-rwxr-xr-x 1 root root 2583 Jul 23 08:35 ssh-chrootmgr
-rwxr-xr-x 1 root root 15275 Jul 23 08:35 ssh-dummy-shell
lrwxrwxrwx 1 root root 11 Jul 23 08:35 ssh-keygen -> ssh-keygen2
-rwxr-xr-x 1 root root 1559891 Jul 23 08:35 ssh-keygen2
lrwxrwxrwx 1 root root 11 Jul 9 20:48 ssh-keygen.old -> ssh-keygen2
lrwxrwxrwx 1 root root 10 Jul 23 08:35 ssh-probe -> ssh-probe2
-rwxr-xr-x 1 root root 333141 Jul 23 08:35 ssh-probe2
lrwxrwxrwx 1 root root 10 Jul 9 20:48 ssh-probe.old -> ssh-probe2
-rwxr-xr-x 1 root root 7563 Jul 23 08:35 ssh-pubkeymgr
lrwxrwxrwx 1 root root 11 Jul 23 08:35 ssh-signer -> ssh-signer2
-rws--x--x 1 root root 1574866 Jul 23 08:35 ssh-signer2
lrwxrwxrwx 1 root root 11 Jul 9 20:48 ssh-signer.old -> ssh-signer2
-rwxr-x--- 1 root admin 30 Nov 6 2006 txqueulenght.sh
-rwxr-x--- 1 root admin 265 Mar 12 17:56 vmware-dependancy.sh
-rwxr-x--- 1 root admin 234 May 26 23:50 yum-update.sh


A few of theses scripts are mine except all ssh-*, sftp-* and scp-*.


I found a file under /dev too, named 'saux'... it contain ssh connection log (user@host (password) [date].


Before I see that, I change all passwords last night and since there is no more suspect process (ftp_scanner) running. If I restrict ssh connection to only one IP, I think this will patch temporary this problem, but I am not sure that sshd binary has been replaced by a hacker root kit file...

guille
Posts: 25
Joined: 2005/11/10 17:10:14
Location: Argentina

Re: URGENT: I think this machine gets hacked...

Post by guille » 2007/07/31 20:58:42

you should run a rootkit like rkhunter to check the integrity of the binaries, and if your box is infected with a trojan.

RAID-5
Posts: 77
Joined: 2005/05/05 17:35:34
Location: Quebec

Re: URGENT: I think this machine gets hacked...

Post by RAID-5 » 2007/08/01 03:35:14

Yes, it's already done. I run chkrootkit and rkhunter... do you have more to suggest?

Thanks.

computec
Posts: 1
Joined: 2007/09/04 02:00:09

Re: URGENT: I think this machine gets hacked...

Post by computec » 2007/09/04 02:21:45

Hi. Bad news...
I have 2 servers with the exact same things.
System are behind a filtering bridge, with ports 22, 25, 53 and 80 open.
All updates aplied....
I can see users login in with root valid password. All passwords are compromised.
Cannot restart sshd processes and they are not visible processes.
Checksum of sshd binaries and others dont match anymore.
I have read some of GSSAPI bug in sshd. Anyone knows if it is exploitable.
Any ideas to remove the infection?
Rkhunter says they are traces for Suckit RootKit.

Post Reply

Return to “CentOS 4 - Security Support”