OpenSSH - almost random connection resets by peer

Issues related to applications and software problems
Jimini
Posts: 5
Joined: 2017/12/08 09:52:08

OpenSSH - almost random connection resets by peer

Postby Jimini » 2017/12/08 10:21:36

Hey there,

I have a SSHd, where a bunch of Windows clients connects to via pscp (so it uses scp). These clients send very small files (10-100KB) and quit the connection after that.

A few days ago, we updated the system, including openssh-server from 6.6.1p1-35.el7_3.x86_64 to 7.4p1-12.el7_4.x86_64. The config was not changed. Since then, our logs contain thousands of the following entries every day:
fatal: ssh_packet_send_debug: Connection reset by peer

This does not affect all of the connections, only some.

So I ran openssh in debug mode for a few minutes, which led to entries like the following:
[20111]: User child is on pid 24038
[24038]: debug1: PAM: establishing credentials
[24038]: debug1: permanently_set_uid: 1002/1002
[24038]: debug3: monitor_apply_keystate: packet_set_state
[24038]: debug2: set_newkeys: mode 0
[24038]: debug1: rekey after 4294967296 blocks
[24038]: debug2: set_newkeys: mode 1
[24038]: debug1: rekey after 4294967296 blocks
[24038]: debug1: ssh_packet_set_postauth: called
[24038]: debug3: ssh_packet_set_state: done
[24038]: debug3: notify_hostkeys: key 0: ssh-rsa SHA256:[...]
[24038]: debug3: notify_hostkeys: key 1: ecdsa-sha2-nistp256 SHA256:[...]
[24038]: debug3: notify_hostkeys: sent 2 hostkeys
[24038]: debug3: send packet: type 80
[24038]: debug3: send packet: type 4
[24038]: fatal: ssh_packet_send_debug: Connection reset by peer
[24038]: debug1: do_cleanup
[24038]: debug3: PAM: sshpam_thread_cleanup entering
[24038]: debug3: mm_request_send entering: type 122
[24038]: debug3: mm_request_receive_expect entering: type 123
[24038]: debug3: mm_request_receive entering


For comparison, here are the log entries for a "sane" connection:
[16407]: User child is on pid 18352
[18352]: debug1: PAM: establishing credentials
[18352]: debug1: permanently_set_uid: 1002/1002
[18352]: debug3: monitor_apply_keystate: packet_set_state
[18352]: debug2: set_newkeys: mode 0
[18352]: debug1: rekey after 4294967296 blocks
[18352]: debug2: set_newkeys: mode 1
[18352]: debug1: rekey after 4294967296 blocks
[18352]: debug1: ssh_packet_set_postauth: called
[18352]: debug3: ssh_packet_set_state: done
[18352]: debug3: notify_hostkeys: key 0: [...]
[18352]: debug3: notify_hostkeys: key 1: [...]
[18352]: debug3: notify_hostkeys: sent 2 hostkeys
[18352]: debug3: send packet: type 80
[18352]: debug3: send packet: type 4
-> here the other connection is resetted
[18352]: debug3: send packet: type 4
[18352]: debug1: Entering interactive session for SSH2.
[18352]: debug2: fd 5 setting O_NONBLOCK
[18352]: debug2: fd 6 setting O_NONBLOCK
[18352]: debug1: server_init_dispatch
[18352]: debug3: receive packet: type 90
[...]


We use the following sshd_config (I deleted all commented lines):
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
SyslogFacility AUTHPRIV
PermitRootLogin no
AllowGroups ssh-login wheel svamon-client
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding no
UsePrivilegeSeparation sandbox # Default for new installations.
UseDNS no
MaxStartups 100:10:400
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server


Any attempt to help would be really appreciated.

Kind regards,
Jimini

pjsr2
Posts: 297
Joined: 2014/03/27 20:11:07

Re: OpenSSH - almost random connection resets by peer

Postby pjsr2 » 2017/12/08 11:28:16

OpenSSH in CentOS7.4 no longer supports ssh protocol v1. Only v2 is available.
Is your Windows pscp version using protocol version 1 or 2? Try adding option -2 to your pscp commands.

Jimini
Posts: 5
Joined: 2017/12/08 09:52:08

Re: OpenSSH - almost random connection resets by peer

Postby Jimini » 2017/12/08 11:37:10

pjsr2, thank you for your reply.

We have 250-300 clients, which send a small file via (p)scp every minute. As far as I know, the pscp command is standardized, so either every or no client should have this problem. Nevertheless, I will check this.

Kind regards,
Jimini

User avatar
TrevorH
Forum Moderator
Posts: 22340
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: OpenSSH - almost random connection resets by peer

Postby TrevorH » 2017/12/08 12:26:27

Also make sure that you don't have a a duplicate ip address on your network as that leads to exactly this sort of symptom - it's then random as to which of the 2 devices with the same ip address that the clients connect to - if they hit the right server then it works, if it hits the duplicate then it'll get a reset.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Jimini
Posts: 5
Joined: 2017/12/08 09:52:08

Re: OpenSSH - almost random connection resets by peer

Postby Jimini » 2017/12/08 13:21:16

Perhaps it is important to point out, that this problem exists since we upgraded the CentOS server a few days ago. Before, everything seemed to work flawlessly - or at least without the SSHd logging the fatal errors.

I also had a look at the clients' IP addresses - 3 clients, which resetted the connection one time, could establish a connection multiple times. For now, I can not recognize a system or a pattern in this behaviour...

Best regards,
Jimini

User avatar
TrevorH
Forum Moderator
Posts: 22340
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: OpenSSH - almost random connection resets by peer

Postby TrevorH » 2017/12/08 13:36:27

Check that nothing has been configured with the same ip address as the server. That leads to exactly these symptoms.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke