Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Issues related to configuring your network
alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/11 18:34:30

chemal wrote:
2019/09/11 15:05:39
But in the NFSoRDMA results, there's quite a bit of difference, ...
You can experiment with the number of nfs server processes in /etc/sysconfig/nfs.
I'll have to look what my options are when I get back home.

Thanks.

*edit*
Following this series of pages: https://cromwell-intl.com/open-source/p ... ce-tuning/; my concern now is that this is going to rapidly be destructive testing for the SSDs due to the multivariate nature of the optimisating problem.

I suspect that a proper analysis would be a hypersurface for the RSO given all of the various tuning parameters that can be changed and studied for these benchmarks.

At 100 GB a pop, I can easily exceed the write endurance of the Samsung 860 EVOs just by running these tests/benchmarks.

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/13 05:58:22

chemal wrote:
2019/09/11 15:05:39
But in the NFSoRDMA results, there's quite a bit of difference, ...
You can experiment with the number of nfs server processes in /etc/sysconfig/nfs.
Apparently, the default is 8, and my head node is only a 6-core processor.

The other page that I cited said that you can't really hurt it if you specify more threads than cores, so I might try that.

System's busy with an aerodynamics CFD run that'll finish in about two days from now, so I'll try it then.

Thanks.

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/14 03:47:03

So I ended up picking up a Broadcom/Avago/LSI MegaRAID 9341-8i 12 Gbps SAS RAID HBA and put the 4x Samsung 860 EVO 1 TB SATA 6 Gbps SSDs on a RAID0 virtual drive and also put my 4x HGST 6 TB 7200 SATA 6 Gbps HDDs on another RAID0 virtual drive as well.

Here are the buffered RAID0 SSD results on local host:

Code: Select all

[root@host home]# cd /home/cluster
[root@host cluster]# time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240; rm -f 10Gfile
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 3.60042 s, 3.0 GB/s
real 3.60
user 0.00
sys 3.59
[root@host cluster]# time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240; rm -f 10Gfile
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 3.63025 s, 3.0 GB/s
real 3.63
user 0.00
sys 3.62
[root@host cluster]# time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240; rm -f 10Gfile
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 3.42913 s, 3.1 GB/s
real 3.43
user 0.00
sys 3.40
And here are the unbuffered results:

Code: Select all

[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=8k count=12500000; rm -f xxx
12500000+0 records in
12500000+0 records out
102400000000 bytes (102 GB) copied, 62.5086 s, 1.6 GB/s
real 62.51
user 3.13
sys 59.37
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=8k count=12500000; rm -f xxx
12500000+0 records in
12500000+0 records out
102400000000 bytes (102 GB) copied, 63.1949 s, 1.6 GB/s
real 63.36
user 3.20
sys 59.97
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=8k count=12500000; rm -f xxx
12500000+0 records in
12500000+0 records out
102400000000 bytes (102 GB) copied, 62.3144 s, 1.6 GB/s
real 62.48
user 3.11
sys 59.19
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=64k count=1562500; rm -f xxx
1562500+0 records in
1562500+0 records out
102400000000 bytes (102 GB) copied, 52.6125 s, 1.9 GB/s
real 52.78
user 0.48
sys 38.05
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=64k count=1562500; rm -f xxx
1562500+0 records in
1562500+0 records out
102400000000 bytes (102 GB) copied, 52.6867 s, 1.9 GB/s
real 52.85
user 0.49
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=64k count=1562500; rm -f xxx
1562500+0 records in
1562500+0 records out
102400000000 bytes (102 GB) copied, 53.1124 s, 1.9 GB/s
real 53.11
user 0.51
sys 40.18
[root@host cluster]#
I haven't tuned my NFS settings yet, so it's still only getting about 1.1 GB/s over NFSoRDMA.

Unbuffered local host writes looks pretty decent with an average of 462.35375 MB/s per drive.

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/15 01:53:31

More data:

In terms of writing that 100 GB file over NFSoRDMA, between the "tuned" NFS and "untuned" NFS (I changed the mount options and added "noatime,nodiratime") to the mount point, both on the local host and the client mounts, leaving everything else the same; writing the 100 GB file over the interconnect resulted in:

Buffered write:
1.085 GB/s "untuned" vs. 1.234 GB/s "tuned"

Unbuffered write:
1.084 GB/s "untuned" vs. 0.641 GB/s "tuned"

It is also interesting to note that if I were to create the 100 GB file first on the client's host SSD (Intel 540s Series 1 TB SATA 6 Gbps SSD) and then try and copy it to the head node's RAID0 array of SSDs, that is about HALF the speed as just creating the same 100 GB file directly onto the headnode by the client (between 0.535-0.54 GB/s).

I'm not really sure why the unbuffered write speed would fall by almost 40%. I suspect that the wearing of the drive might be starting to play a role in it, but I don't know if it is enough to explain the full 40% difference.

The "untuned" vs. "tune" NFSoRDMA reads though has these results:

Buffered reads:
4.892 GB/s vs. 6.143 GB/s

Unbuffered reads:
4.221 GB/s vs. 5.642 GB/s

That's "direct" testing, using dd and the 100 GB file.

In another test (albeit sample size of 1), where I am running my CAE/HPC application, it resulted in a 1.2% improvement in the CPU time and the total elapsed (wall clock) time between "untuned" vs. "tuned" NFS parameters.

There's probably more tuning that I CAN do with NFS, but again, I'm apt and likely to wear out the write endurance of the drive entirely via the NFS testing and tuning, which I don't want to do; so I'll have to just live with this for now.

Again, this is all with the new Broadcom/Avago/LSI MegaRAID 9341-8i 12 Gbps SAS controller with four Samsung 860 EVO 1 TB SATA 6 Gbps SSDs (in RAID0), which means that it should have plenty of bandwidth to be able to write faster than that, but as the data shows; that's not true. At least not yet.

I might be able to do some more testing, but using a RAM drive instead (tmpfs), but the size of the test file will likely have to drop since the headnode only has 64 GB of RAM. That way, I won't have to worry about the write endurance of the drive.

In other news though, I think that the reason why I might be seeing only very little change is that the data is starting to suggest that there are limitations to how fast /dev/zero can be read/generated and written to disk, which I'm not sure how much large HPC clusters/organisations test and benchmark HPC file systems.

I say this because so far, in the data that I've got, nothing seems to have cracked 1.25 GB/s write (across a four drive, RAID0 array) despite the fact that it should have plenty of bandwidth to do so.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by TrevorH » 2019/09/15 02:51:32

I suspect that the wearing of the drive might be starting to play a role in it, but I don't know if it is enough to explain the full 40% difference.
Run fstrim -v /mountpoint to have it issue a TRIM to unused portions of the filesystem on '/mountpoint'.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/15 06:03:56

TrevorH wrote:
2019/09/15 02:51:32
I suspect that the wearing of the drive might be starting to play a role in it, but I don't know if it is enough to explain the full 40% difference.
Run fstrim -v /mountpoint to have it issue a TRIM to unused portions of the filesystem on '/mountpoint'.
Thank you.

I'll have to try that.

*edit*
Tried that on the mountpoint.

Says:

Code: Select all

# fstrim -v /my_mountpoint
fstrim: /my_mountpoint: the discard operation is not supported

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/15 08:17:32

So I just finished the testing/benchmarking using the Phoronix Test Suite. (pts/fio)

Here is the setup:

Sequential write speed test only
Linux AIO
Both Direct and Indirect
Both Buffered and Unbuffered
64 kiB block size
Changed the default test directory in /usr/share/phoronix-test-suite/pts-core/static/user-config-defaults.xml to the RAID array.

Ran this on the local host of the headnode and also with one of the slave nodes against the RAID0 SSD mount.

Local host results:

Code: Select all

Buffered: No, Direct: No - 233 MB/s
Buffered: No, Direct: Yes - 234 MB/s
Buffered: Yes, Direct: No - 5255 MB/s
Buffered: Yes, Direct: Yes - 5232 MB/s
("tuned") NFSoRDMA results:

Code: Select all

Buffered: No, Direct: No - 4022 MB/s
Buffered: No, Direct: Yes - 3556 MB/s
Buffered: Yes, Direct: No - 3471 MB/s
Buffered: Yes, Direct: Yes - 3177 MB/s

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/15 15:29:00

For those that might be interested, I have new data from the Phoronix Test Suite up on their forums.

You can go check it out.

https://www.phoronix.com/forums/forum/p ... re-samples

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by hunter86_bg » 2019/09/23 13:13:49

You can now try 7.7 . Maybe the situation will be different ?

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?

Post by alpha754293 » 2019/09/25 02:37:47

hunter86_bg wrote:
2019/09/23 13:13:49
You can now try 7.7 . Maybe the situation will be different ?
Maybe.

Cluster, including headnode, is currently in production, so I'll have to wait until the next maintenance cycle to try/test that out.

Thank you.

Post Reply