nfs -> Win 10 -> slow performance

Hellkeeper · Post by **Hellkeeper** » 2017/07/26 18:48:22

Hello,
I got a Web-Project with some million files (3.937.899 Files in 153.946 Directories) on my Centos 7 Server. Server <--> Win 10 Client is connected by 10Gbit.
When accessing the Proejct with Zend Studio (Eclipse) over SMB connection - everything is really terrible slow ... (Workspace rebuilds take ages)
When accessing the Project over NFS Share - it's better - but unfortunately still way from "good" ...

Already googled / experimented for several Days to get better performance - but with no real luck so far ...

NFS Stats:

Code: Select all

[root@localhost ~]# nfsstat 
Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
32788346   112        112        0          0       

Server nfs v3:
null         getattr      setattr      lookup       access       readlink     
112       0% 772720    2% 807       0% 1814450   5% 0         0% 0         0% 
read         write        create       mkdir        symlink      mknod        
1540642   4% 3247      0% 3         0% 5         0% 0         0% 0         0% 
remove       rmdir        rename       link         readdir      readdirplus  
0         0% 0         0% 13        0% 0         0% 129976    0% 28525544 86% 
fsstat       fsinfo       pathconf     commit       
29        0% 0         0% 0         0% 804       0%

I'm a bit puzzled by those 86% readdirplus Calls? Any ideas if this is normal? Or any ideas how I could achieve better performance?
I really hope someone here can help me with this Problem - as I'm out of ideas ... !

Thank you, bye from Austria
Andreas Schnederle-Wagner

Post by **TrevorH** » 2017/07/26 19:24:09

That's not a million files in one directory?

Hellkeeper · Post by **Hellkeeper** » 2017/07/26 19:56:47

of course not - lot's of Sub-Dirs within Share!

hunter86_bg · Post by **hunter86_bg** » 2017/07/30 20:12:16

You can increase the 'RPCNFSDCOUNT=' to a larger value and try again. The setting is in the "/etc/sysconfig/nfs" , but this is just a guess.
You may try with 16 and see if there will be any improvement. Check the load of the NFS server, also.

tunk · Post by **tunk** » 2017/07/30 20:51:27

Is there one network cable directly between them, or does it go via a router?
Are you sure all interfaces are setup to use 10Gbps?
What's the file system on the server (ext4, xfs, ...)?

Hellkeeper · Post by **Hellkeeper** » 2017/07/31 09:48:35

hunter86_bg:
- increasing RPCNFSDCOUNT unfortunately didn't help with performance
- Whole Server Load is OK (~0.10) - but nfsd is constantly using up ~25 - 50% CPU according to TOP

tunk:
- it's 10Gbit over Switch (Server - Switch - Workstation)
- all involved Interfaces are up and running 10Gbit
- FS is EXT4

Hellkeeper · Post by **Hellkeeper** » 2017/07/31 13:15:26

now trying with nfs41 Client for Windows (instead of v3) ... maybe this will help with performance ... will update post when I got some more information ...

Code: Select all

[root@localhost ~]# nfsstat 
Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
125555817   682        682        0          0       

Server nfs v3:
null         getattr      setattr      lookup       access       readlink     
708       0% 1626456   1% 1710      0% 5672278   4% 2         0% 0         0% 
read         write        create       mkdir        symlink      mknod        
4573552   3% 7360      0% 6         0% 11        0% 0         0% 0         0% 
remove       rmdir        rename       link         readdir      readdirplus  
1         0% 0         0% 27        0% 0         0% 735476    0% 106334837 89% 
fsstat       fsinfo       pathconf     commit       
141       0% 11        0% 11        0% 1703      0% 

Server nfs v4:
null         compound     
14        0% 6601528  99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 0         0% 94030     0% 0         0% 
create       delegpurge   delegreturn  getattr      getfh        link         
1         0% 3         0% 93163     0% 12217764 26% 10031248 21% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 9998620  21% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
94251     0% 10        0% 0         0% 0         0% 6580266  14% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
324       0% 323404    0% 132019    0% 0         0% 1         0% 2         0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
0         0% 93992     0% 94051     0% 0         0% 4         0% 0         0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
0         0% 0         0% 23962     0% 0         0% 0         0% 3         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
14        0% 17        0% 6         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 6601499  14% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 1         0% 6         0% 14        0%

Hellkeeper · Post by **Hellkeeper** » 2017/07/31 14:03:14

alright ... no luck with nfs 4.1 too ...

Working within Zend on the Share right now ... and Zend is currently frozen since a few Minutes ...

Code: Select all


[root@localhost ~]# nfsstat 
Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
130652390   686        686        0          0       

Server nfs v3:
null         getattr      setattr      lookup       access       readlink     
712       0% 1626490   1% 1710      0% 5672287   4% 2         0% 0         0% 
read         write        create       mkdir        symlink      mknod        
4573552   3% 7360      0% 6         0% 11        0% 0         0% 0         0% 
remove       rmdir        rename       link         readdir      readdirplus  
1         0% 0         0% 27        0% 0         0% 735477    0% 106334837 89% 
fsstat       fsinfo       pathconf     commit       
141       0% 11        0% 11        0% 1703      0% 

Server nfs v4:
null         compound     
14        0% 11698054 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 0         0% 113547    0% 0         0% 
create       delegpurge   delegreturn  getattr      getfh        link         
1         0% 3         0% 112685    0% 22341332 26% 18359367 21% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 18383294 21% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
113766    0% 10        0% 0         0% 0         0% 11676539 13% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
512       0% 380581    0% 303157    0% 0         0% 1         0% 2         0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
0         0% 113507    0% 113566    0% 0         0% 4         0% 0         0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
0         0% 0         0% 23962     0% 0         0% 0         0% 3         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
14        0% 17        0% 6         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 11698028 13% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 1         0% 6         0% 14        0%

kaboom · Post by **kaboom** » 2017/08/02 07:57:52

while experiencing the issue, do a sudo perf top and paste what you get

Hellkeeper · Post by **Hellkeeper** » 2017/08/08 16:02:35

Hey kaboom,

here is the "perf top" Output when such a Freeze occurs:

Code: Select all

Samples: 28K of event 'cpu-clock', Event count (approx.): 2863428068                                                                                                                                                                                            
Overhead  Shared Object                 Symbol                                                                                                                                                                                                                  
  16.11%  [kernel]                      [k] _raw_spin_unlock_irqrestore
  12.11%  [kernel]                      [k] __do_softirq
   5.82%  [kernel]                      [k] finish_task_switch
   4.18%  [kernel]                      [k] vmbus_on_event
   2.82%  [kernel]                      [k] put_cred_rcu
   2.59%  [kernel]                      [k] rcu_process_callbacks
   2.33%  [kernel]                      [k] run_timer_softirq
   1.55%  [kernel]                      [k] __slab_free
   1.37%  [kernel]                      [k] free_uid
   1.31%  [kernel]                      [k] kmem_cache_free
   1.27%  [kernel]                      [k] rcu_gp_kthread
   1.04%  [kernel]                      [k] kfree
   1.04%  [kernel]                      [k] cmpxchg_double_slab.isra.51
   1.02%  [kernel]                      [k] tick_nohz_idle_exit
   1.00%  [kernel]                      [k] rcu_idle_exit
   0.91%  [kernel]                      [k] svc_recv
   0.70%  [kernel]                      [k] find_busiest_group
   0.70%  [kernel]                      [k] memcpy
   0.67%  [kernel]                      [k] ipt_do_table
   0.65%  [kernel]                      [k] tasklet_action
   0.61%  [kernel]                      [k] tick_nohz_idle_enter
   0.56%  [kernel]                      [k] check_for_new_grace_period.isra.26
   0.51%  [kernel]                      [k] tcp_ack
   0.50%  [kernel]                      [k] nf_iterate
   0.46%  [kernel]                      [k] __nf_conntrack_find_get
   0.42%  [kernel]                      [k] rcu_process_gp_end
   0.40%  [kernel]                      [k] svc_xprt_do_enqueue
   0.40%  [kernel]                      [k] vmbus_setevent
   0.36%  [kernel]                      [k] __d_lookup
   0.36%  [kernel]                      [k] prepare_creds
   0.36%  [kernel]                      [k] _raw_spin_lock
   0.35%  [kernel]                      [k] rcu_bh_qs
   0.34%  [kernel]                      [k] get_page_from_freelist
   0.33%  [kernel]                      [k] tcp_in_window
   0.33%  [kernel]                      [k] kmem_cache_alloc
   0.33%  [kernel]                      [k] source_load
   0.32%  [kernel]                      [k] hv_begin_read
   0.31%  [kernel]                      [k] groups_free
   0.31%  [kernel]                      [k] nf_conntrack_in
   0.30%  [kernel]                      [k] tcp_packet
   0.30%  [kernel]                      [k] rndis_filter_receive
   0.28%  [kernel]                      [k] xdr_reserve_space
   0.28%  [kernel]                      [k] nfsd4_encode_fattr
   0.28%  [kernel]                      [k] svc_tcp_recvfrom
   0.27%  [kernel]                      [k] __call_rcu
   0.27%  [kernel]                      [k] netvsc_send
   0.26%  [kernel]                      [k] key_put
   0.25%  [kernel]                      [k] __schedule
   0.25%  [kernel]                      [k] _raw_spin_lock_bh
   0.25%  [kernel]                      [k] nfsd4_proc_compound
   0.24%  [kernel]                      [k] memset
   0.24%  [kernel]                      [k] fh_verify
   0.23%  [kernel]                      [k] sunrpc_cache_lookup
   0.23%  perf                          [.] dso__find_symbol
   0.23%  [kernel]                      [k] _raw_spin_lock_irqsave
   0.22%  [kernel]                      [k] _cond_resched
   0.22%  [kernel]                      [k] __slab_alloc
   0.21%  [kernel]                      [k] __kmalloc
   0.21%  [kernel]                      [k] svcauth_unix_set_client
   0.21%  [kernel]                      [k] enqueue_to_backlog
   0.21%  [kernel]                      [k] ip_finish_output
   0.21%  [kernel]                      [k] wake_up_process
   0.21%  [kernel]                      [k] tcp_recvmsg
   0.20%  [kernel]                      [k] sock_recvmsg
   0.20%  [kernel]                      [k] schedule_timeout
   0.20%  [kernel]                      [k] netvsc_channel_cb
   0.20%  [kernel]                      [k] __netif_receive_skb_core
   0.19%  [kernel]                      [k] tcp_error
   0.19%  [kernel]                      [k] arch_cpu_idle
   0.19%  [kernel]                      [k] exportfs_decode_fh
   0.18%  [kernel]                      [k] svc_tcp_data_ready
   0.18%  [kernel]                      [k] __inet_lookup_established

Code: Select all

Samples: 55K of event 'cpu-clock', Event count (approx.): 3254425621                                                                                                                                                                                            
Overhead  Shared Object                 Symbol                                                                                                                                                                                                                  
  16.83%  [kernel]                      [k] _raw_spin_unlock_irqrestore
   8.31%  [kernel]                      [k] __do_softirq
   7.09%  [kernel]                      [k] finish_task_switch
   4.21%  [kernel]                      [k] put_cred_rcu
   3.19%  [kernel]                      [k] vmbus_on_event
   2.65%  [kernel]                      [k] rcu_process_callbacks
   2.49%  [kernel]                      [k] __slab_free
   2.13%  [kernel]                      [k] free_uid
   1.99%  [kernel]                      [k] run_timer_softirq
   1.83%  [kernel]                      [k] rcu_gp_kthread
   1.65%  [kernel]                      [k] kmem_cache_free
   1.49%  [kernel]                      [k] cmpxchg_double_slab.isra.51
   1.42%  [kernel]                      [k] tick_nohz_idle_exit
   1.28%  [kernel]                      [k] kfree
   1.04%  [kernel]                      [k] find_busiest_group
   0.78%  [kernel]                      [k] svc_recv
   0.77%  [kernel]                      [k] rcu_idle_exit
   0.70%  [kernel]                      [k] memcpy
   0.69%  [kernel]                      [k] tick_nohz_idle_enter
   0.67%  [kernel]                      [k] ipt_do_table
   0.54%  [kernel]                      [k] source_load
   0.53%  [kernel]                      [k] tasklet_action
   0.52%  [kernel]                      [k] nf_iterate

Code: Select all

Samples: 177K of event 'cpu-clock', Event count (approx.): 9794270768                                                                                                                                                                                           
Overhead  Shared Object                       Symbol                                                                                                                                                                                                            
   5.14%  [kernel]                            [k] __do_softirq
   4.77%  rngd                                [.] 0x00000000000028cb
   3.86%  libX11.so.6.3.0                     [.] _XPutPixel16
   3.46%  [kernel]                            [k] _raw_spin_unlock_irqrestore
   3.09%  libX11.so.6.3.0                     [.] _XGetPixel16
   2.69%  libpython2.7.so.1.0                 [.] PyEval_EvalFrameEx
   2.19%  libc-2.17.so                        [.] __memcpy_ssse3_back
   2.01%  libc-2.17.so                        [.] __memmove_ssse3_back
   1.67%  [kernel]                            [k] vmbus_on_event
   1.58%  [kernel]                            [k] __d_lookup_rcu
   1.42%  libc-2.17.so                        [.] __memcmp_sse4_1
   1.33%  [kernel]                            [k] finish_task_switch
   1.30%  [kernel]                            [k] run_timer_softirq
   1.22%  [kernel]                            [k] copy_pte_range
   1.13%  libX11.so.6.3.0                     [.] _XSetImage
   1.05%  libc-2.17.so                        [.] _int_malloc
   1.04%  [kernel]                            [k] unmap_page_range
   0.87%  [kernel]                            [k] __do_page_fault
   0.82%  libpython2.7.so.1.0                 [.] PyUnicodeUCS4_Decode
   0.81%  [kernel]                            [k] copy_user_enhanced_fast_string
   0.79%  libpython2.7.so.1.0                 [.] lookdict_string
   0.73%  [kernel]                            [k] clear_page_c_e
   0.67%  [kernel]                            [k] rcu_process_callbacks
   0.57%  libpython2.7.so.1.0                 [.] PyUnicodeUCS4_Compare
   0.56%  [kernel]                            [k] kmem_cache_free
   0.53%  libpython2.7.so.1.0                 [.] PyDict_GetItem
   0.52%  libpython2.7.so.1.0                 [.] _PyType_Lookup
   0.50%  [kernel]                            [k] put_cred_rcu
   0.50%  [kernel]                            [k] get_page_from_freelist

Andreas

CentOS

nfs -> Win 10 -> slow performance

nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance

Re: nfs -> Win 10 -> slow performance