NFS not responding

General support questions including new installations
Post Reply
joemuller
Posts: 3
Joined: 2015/09/29 23:22:28

NFS not responding

Post by joemuller » 2015/09/29 23:54:58

I have a server, "cdr", that is currently set up as an NFS client. It has a single NFS mount to another local server, "depot". The problem I am seeing is that the NFS client will, on average, every 24 hours, start timing out. This can be observed by attempting an 'ls' of the mount directory - /nfs/backup.

Right before the timeout, the following entry appears in /var/log/messages:
cdr kernel: nfs: server 172.16.0.4 not responding, still trying
The client ("cdr") is running kernel-2.6.18-406.el5. Here is the fstab entry:
172.16.0.4:/export/vol2/replicated/cdr/replicated /nfs/backup nfs rsize=32768,wsize=32768 0 0
The server ("depot") is running kernel 2.6.32-573.3.1.el6.x86_64. Here is the corresponding entry in /etc/exports:
/export/vol2/replicated/cdr/replicated 172.16.6.11(rw,no_root_squash,async)
I have performed a packet capture of NFS traffic on the server side, and around 25-50% of packets appear in Wireshark as 'TCP Retransmission' and 'TCP Spurious Retransmission'. I can provide the pcap file if needed.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: NFS not responding

Post by TrevorH » 2015/09/29 23:58:31

How sure are you that there is not a duplicate ip address present?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: NFS not responding

Post by aks » 2015/09/30 17:46:37

Also check the load on the server, perhaps it's just really, really busy?

joemuller
Posts: 3
Joined: 2015/09/29 23:22:28

Re: NFS not responding

Post by joemuller » 2015/10/01 00:10:30

TrevorH wrote:How sure are you that there is not a duplicate ip address present?
I have not been able to find any duplicate IP's on the network.
aks wrote:Also check the load on the server, perhaps it's just really, really busy?
The remote server does get heavily loaded, but the other NFS exports are remaining mounted by other hosts.

Oddly enough, nfsstat -s shows 0 retransmits on the remote server ("depot").

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: NFS not responding

Post by aks » 2015/10/06 18:22:45

I have performed a packet capture of NFS traffic on the server side, and around 25-50% of packets appear in Wireshark as 'TCP Retransmission' and 'TCP Spurious Retransmission'.
At which end? Anyway that seems quite excessive.
Perhaps start here: https://ask.wireshark.org/questions/306 ... -wireshark

joemuller
Posts: 3
Joined: 2015/09/29 23:22:28

Re: NFS not responding

Post by joemuller » 2015/10/19 17:47:10

aks wrote:
I have performed a packet capture of NFS traffic on the server side, and around 25-50% of packets appear in Wireshark as 'TCP Retransmission' and 'TCP Spurious Retransmission'.
At which end? Anyway that seems quite excessive.
Perhaps start here: https://ask.wireshark.org/questions/306 ... -wireshark
The packet capture was performed on the remote NFS server. Is there a way to detect if load is having an effect on NFS? (excessive retransmits and/or reconnects)

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: NFS not responding

Post by aks » 2015/10/20 16:32:05

Yes it's the CPU utilization/load average/runq.....
If the request can't get to the CPU, it ain't gonna go anywhere quickly.

Post Reply