[SOLVED] rsync/ssh randomly hangs when rsync executed multiple times - source machine reboot needed

[aegersz]

only one of my machines (the target machine which is also a router) is doing this but it causes the other source machine(s) to stop accepting ssh/rsync connects and i have to do a reboot on them ... they are all at CentOS 6.9 and all use ssh keys. It is not a network issue as i have the same symptoms when pulling from the intranet or the intranet.

it happens with or without the -e ssh option ... for example:

Code: Select all

rsync -zpaAPve ssh root@192.168.137.2:/etc/ /mirror/etc/

the console log:

Code: Select all

[root@3or4G bin]# getetc
receiving incremental file list
./
adjtime
          44 100%   42.97kB/s    0:00:00 (xfer#1, to-check=1457/1464)
mtab
         287 100%  280.27kB/s    0:00:00 (xfer#2, to-check=1402/1464)
webmin/system-status/
webmin/system-status/info
       26657 100%  137.74kB/s    0:00:00 (xfer#3, to-check=73/1922)
webmin/system-status/history/
webmin/system-status/history/bin
       99907 100%  424.20kB/s    0:00:00 (xfer#4, to-check=71/1922)
webmin/system-status/history/bout
      101315 100%  428.31kB/s    0:00:00 (xfer#5, to-check=70/1922)
webmin/system-status/history/cpuidle
      113745 100%  474.70kB/s    0:00:00 (xfer#6, to-check=69/1922)
webmin/system-status/history/cpuio
       99740 100%  412.72kB/s    0:00:00 (xfer#7, to-check=68/1922)
webmin/system-status/history/cpukernel
       99753 100%  364.85kB/s    0:00:00 (xfer#8, to-check=67/1922)
webmin/system-status/history/cpuuser
      100028 100%  363.14kB/s    0:00:00 (xfer#9, to-check=66/1922)
webmin/system-status/history/diskused
      175248 100%  631.52kB/s    0:00:00 (xfer#10, to-check=65/1922)
webmin/system-status/history/load
      122720 100%  437.39kB/s    0:00:00 (xfer#11, to-check=64/1922)
webmin/system-status/history/load15
      122720 100%  373.35kB/s    0:00:00 (xfer#12, to-check=63/1922)
webmin/system-status/history/load5
      122720 100%  371.03kB/s    0:00:00 (xfer#13, to-check=62/1922)
webmin/system-status/history/maxes
         101 100%    0.30kB/s    0:00:00 (xfer#14, to-check=61/1922)
webmin/system-status/history/memused
      159607 100%  478.12kB/s    0:00:00 (xfer#15, to-check=60/1922)
webmin/system-status/history/netcounts
          95 100%    0.28kB/s    0:00:00 (xfer#16, to-check=59/1922)
webmin/system-status/history/procs
      115050 100%  299.61kB/s    0:00:00 (xfer#17, to-check=58/1922)
webmin/system-status/history/rx
      197199 100%  510.81kB/s    0:00:00 (xfer#18, to-check=57/1922)
webmin/system-status/history/swapused
       99701 100%  256.22kB/s    0:00:00 (xfer#19, to-check=56/1922)
webmin/system-status/history/tx
      154714 100%  383.47kB/s    0:00:00 (xfer#20, to-check=55/1922)

sent 17151 bytes  received 54419 bytes  47713.33 bytes/sec
total size is 30184523  speedup is 421.75
[root@3or4G bin]# getetc
receiving incremental file list

sent 314 bytes  received 50926 bytes  34160.00 bytes/sec
total size is 30184523  speedup is 589.08
[root@3or4G bin]#
[root@3or4G bin]# getetc
receiving incremental file list

sent 314 bytes  received 50926 bytes  34160.00 bytes/sec
total size is 30184523  speedup is 589.08
[root@3or4G bin]# getetc
receiving incremental file list

sent 314 bytes  received 50926 bytes  102480.00 bytes/sec
total size is 30184523  speedup is 589.08
[root@3or4G bin]# getetc
ssh: connect to host 192.168.137.2 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: unexplained error (code 255) at io.c(600) [receiver=3.0.6]

.. and once it hangs, i have to reboot 192.168.137.2 ! ... any thoughts ?

[aegersz]

determined not to accept defeat, I turned off my source's firewall (iptables -F) and now it's perfect !

i'm always answering my own questions ... it's good to put pen to paper or fingers to keyboard. rsync can't be dodgy, after all this time, so i started kernel logging for certain iptables conditions and i see that i'm "under attack". shooting myself in the foot, so to speak. It's most likely because I am running the rsync's too quickly, one after the other.

I just "whitelisted" my IPHONE addresses.

the rules that caused this grief:


-A ATTK_CHECK ...


my firewall:

Code: Select all

:ATTACKED - [0:0]
:ATTK_CHECK - [0:0]
:f2b-SSH - [0:0]
-A INPUT -p tcp -m tcp --dport 22 -j f2b-SSH
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT
-A INPUT -p udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2525 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 465 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 110 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 873 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 995 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 993 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 143 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3000 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 10000 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 21,22,110,143,993,995 -m recent --update --seconds 3600 --name BANNED --rsource -j DROP
-A INPUT -p tcp -m multiport --dports 21,22,110,143,993,995 -m state --state NEW -j ATTK_CHECK
-A ATTACKED -m limit --limit 5/min -j LOG --log-prefix "IPTABLES (Rule ATTACKED): " --log-level 7
-A ATTACKED -m recent --set --name BANNED --rsource -j DROP
-A ATTK_CHECK -m recent --set --name ATTK --rsource
-A ATTK_CHECK -m recent --update --seconds 600 --hitcount 16 --name ATTK --rsource -j ATTACKED
-A ATTK_CHECK -m recent --update --seconds 60 --hitcount 6 --name ATTK --rsource -j ATTACKED
-A ATTK_CHECK -j ACCEPT
-A f2b-SSH -j RETURN

the log that gave me the clue:

Code: Select all

Jul 18 18:16:26 dev kernel: IPTABLES (Rule ATTACKED): IN=eth0 OUT= MAC=00:16:d4:ad:e2:83:74:44:01:4a:a3:3a:08:00 SRC=192.168.137.1 DST=192.168.0.180 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=32329 DF PROTO=TCP SPT=52948 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0

i'm running the rsync pull on 192.168.137.1 FROM 192.168.0.180. Lesson learned: be careful what you stick into iptables and enable iptables logging via the kernel !