NFS speed on 10Gb/s RJ45

Issues related to configuring your network
Moe
Posts: 16
Joined: 2017/01/23 14:30:28

NFS speed on 10Gb/s RJ45

Post by Moe » 2018/07/19 18:39:45

Hello

I'm testing NFS on two new servers with RJ45 10Gb/s

The mount looks like this

Code: Select all

172.17.10.101:/mnt/nas/lun1/NFS-SHARE1 on /mnt/test1 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.17.10.102,local_lock=none,addr=172.17.10.101)
I'm getting around 450MB/s , writing file sizes between 4 - 40GB
( using

Code: Select all

dd if=/dev/zero of=/mnt/test1/ta2 bs=4k count=xxxxxxx 
)

which I believe is quite slow for this link

The results are similar whether used 1500 or 9000 MTU

Any help is appreciated

Regards

User avatar
TrevorH
Forum Moderator
Posts: 23164
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: NFS speed on 10Gb/s RJ45

Post by TrevorH » 2018/07/19 21:27:43

The results are similar whether used 1500 or 9000 MTU
That surprises me. Do you see a difference in cpu usage when using those 2 MTUs?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Moe
Posts: 16
Joined: 2017/01/23 14:30:28

Re: NFS speed on 10Gb/s RJ45

Post by Moe » 2018/07/21 04:03:17

TrevorH wrote:
2018/07/19 21:27:43
The results are similar whether used 1500 or 9000 MTU
That surprises me. Do you see a difference in cpu usage when using those 2 MTUs?
Actually there was a mistake, I had MTU set to 9000 on one server and not the other

However , setting MTU to 9000 on both, didn't increase much, only to about 470 MB/s

CPU utilization using 9000 is a bit lower but not much , the two servers are dual Xeon octa core, 128GB ram, so there're no bottlenecks on this side

I've noticed that larger files results in better results ( 12GB files give around 450-470MB/s , while 45-90GB files give around 560MB/s )

Also trying two threads at the same time gives about 610MB/s combined

Still I think there's still something not very correct about these numbers. or am I asking too much from RJ45 ?

User avatar
TrevorH
Forum Moderator
Posts: 23164
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: NFS speed on 10Gb/s RJ45

Post by TrevorH » 2018/07/21 11:27:21

I would start by testing using iperf3 rather than NFS. That should tell you if your network is working at full speed. With iperf3, using a 10Gbps adapter you should be seeing throughput up around 9Gbps like this:

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  68.5 GBytes  9.80 Gbits/sec    0             sender
[  4]   0.00-60.00  sec  68.5 GBytes  9.80 Gbits/sec                  receiver
Those numbers were generated over a link using a 9000 MTU. If I run the same test over a different 10Gbps link between the same 2 machines with a 1500 MTU then throughput drops from 9.8Gbps to 7.5Gbps. The difference in my case is that using iperf3, with a 1500 MTU, the cpu on the client iperf3 system is 100% of one core and when it's 9000, the same test shows cpu usage of 20%. So, test using iperf3 first and make sure the network part is OK. If it is than the next thing to check would be disk speed on the NFS server to make sure it can sustain writes of > 450MB/s.

Also, your dd command is telling you nothing because you don't have the correct set of options set. You'll want to amend it to include conv=fdatasync or oflag=direct so that you are measuring what you think you are - at the moment it's very likely that you're just measuring the speed of your RAM.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Moe
Posts: 16
Joined: 2017/01/23 14:30:28

Re: NFS speed on 10Gb/s RJ45

Post by Moe » 2018/07/21 12:35:16

I would start by testing using iperf3 rather than NFS
Yeah, I would have done so, but the NFS server doesn't have iperf installed, and it's a special server, so not sure if I will be able to run iperf on it, still checking
Those numbers were generated over a link using a 9000 MTU
How much speed ( file transfer wise ) do you get over this link ? with which protocol ? (nfs, cifs ? )
the next thing to check would be disk speed on the NFS server to make sure it can sustain writes of > 450MB/s.
This is not an issue, the disk is a raid that gives over 2000MB/s
You'll want to amend it to include conv=fdatasync or oflag=direct
Thanks, I had forgotted about the oflag=direct , will test using it and see how it goes

Appreciate your replies.

User avatar
TrevorH
Forum Moderator
Posts: 23164
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: NFS speed on 10Gb/s RJ45

Post by TrevorH » 2018/07/21 14:19:52

How much speed ( file transfer wise ) do you get over this link ? with which protocol ? (nfs, cifs ? )
My last post had numbers in it, 9.8Gbps via the link using MTU 9000, 7.8Gbps via the one with 1500.
This is not an issue, the disk is a raid that gives over 2000MB/s
That would need to be a very big RAID array with lots of separate disks in it then. If it's using SSDs then you'd be looking at using about 5 SATA attached SSDs to get that sort of numbers, if you are using spinning rust then you'll need more like 10 separate spindles to get that sort of throughput, maybe more. For example, I have a machine here with 6 x 300GB 15k rpm SAS disks attached to a Dell H730 RAID controller with 1GB cache RAM, set up in RAID 5 and the top speed of that is about 1100MB/s when reading and a lot less when writing - about 102MB/s when using a blocksize of 4k like your test and, if I change bs=4k to bs=4M, then I can get it to hit 1.1GB/s writing too. If you're using consumer grade SATA disks then you'd need a lot more than double the number of disks than I have to hit 2GB/s.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Moe
Posts: 16
Joined: 2017/01/23 14:30:28

Re: NFS speed on 10Gb/s RJ45

Post by Moe » 2018/07/21 18:00:28

My last post had numbers in it, 9.8Gbps via the link using MTU 9000, 7.8Gbps via the one with 1500.
I didn't mean iperf results, I meant in a real-life scenario, transferring real files on a shared directory via NFS, SMB , AFP..etc.
That would need to be a very big RAID array with lots of separate disks in it then
It is a big RAID indeed, over 12 SAS drives, so I'm trying to get the most throughput I can out of it.
Trying to check the OS, driver and protocol and find ways to squeeze more speed.

If you're interested in building hardware, I can share the system details with you.

With dd I usually test with 1M blocksize for local devices, 4k and 9k for tests over the network.

However I found fio to be more useful benchmarking , and found this page to have useful scripts/examples .

https://wiki.mikejung.biz/Benchmarking# ... d_Examples

Moe
Posts: 16
Joined: 2017/01/23 14:30:28

Re: NFS speed on 10Gb/s RJ45

Post by Moe » 2018/07/22 13:10:32

I managed to get iperf3 installed on the server
With MTU 9000 I'm getting

Code: Select all

[ ID][ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-69.42  sec  80.0 GBytes  9.90 Gbits/sec    6             sender
[  4]   0.00-69.42  sec  80.0 GBytes  9.90 Gbits/sec                  receiver
and with MTU 1500

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-74.36  sec  80.0 GBytes  9.24 Gbits/sec  2956             sender
[  4]   0.00-74.36  sec  80.0 GBytes  9.24 Gbits/sec                  receiver
CPU utilization is in the %40 range with both settings
So I guess we're okay on this layer

NFS wise, I'm getting around 470MB/s with MTU 9000 , if I run 2 dd threads for a 90GB file, I get around 680 MB/s ( combined ) ,if I run 4 threads I get around 750MB/s ( combined )

So I guess there's a lot of room to improve NFS wise ?

hunter86_bg
Posts: 1210
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: NFS speed on 10Gb/s RJ45

Post by hunter86_bg » 2018/07/23 15:15:53

NFS v4.1 suppprts threads, so give a try with more threads. I think that the configuration file was '/etc/sysconfig/nfs'

Moe
Posts: 16
Joined: 2017/01/23 14:30:28

Re: NFS speed on 10Gb/s RJ45

Post by Moe » 2018/07/24 10:23:41

hunter86_bg wrote:
2018/07/23 15:15:53
NFS v4.1 suppprts threads, so give a try with more threads. I think that the configuration file was '/etc/sysconfig/nfs'
Thanks, I've tried increasing the threads from 8 ( the default ) to 16 , but still getting similar

A single thread increased from 470 to 515 MB/s , while for 2 and 4 threads it actually decreased slightly ( from 680 to 660Mb/s and from 750 to 710MB/s respectively ) :o :!: :?:

Post Reply