Copy on file with multithread
-
- Posts: 7
- Joined: 2019/11/09 18:04:26
Re: Copy on file with multithread
Hi,
I tested more ssh's options. And here are the results :
Command : scp OPTIONS /test.img root@192.168.1.1:/dev/null
pas d'option => 145 Mo/s
-c none => 145 Mo/s
-c 3des-cbc => 23 Mo/s
-c blowfish-cbc => 87 Mo/s
-c arcfour => Lost connexion
-c aes256-ctr => 175 Mo/s
-c aes128-ctr => 210 Mo/s
Yes, if i change the Cipher then it's better.
But the maximum is 210 Mo/s and i'm looking for 1500Mo/s. I understand that if i use nfs, nbt or nc, then i can obtain 1500. But i'm looking for about ssh (native).
I tested more ssh's options. And here are the results :
Command : scp OPTIONS /test.img root@192.168.1.1:/dev/null
pas d'option => 145 Mo/s
-c none => 145 Mo/s
-c 3des-cbc => 23 Mo/s
-c blowfish-cbc => 87 Mo/s
-c arcfour => Lost connexion
-c aes256-ctr => 175 Mo/s
-c aes128-ctr => 210 Mo/s
Yes, if i change the Cipher then it's better.
But the maximum is 210 Mo/s and i'm looking for 1500Mo/s. I understand that if i use nfs, nbt or nc, then i can obtain 1500. But i'm looking for about ssh (native).
Re: Copy on file with multithread
So, as said, the problem is not with copying nor your network, it's to do with encrypting the data. If you want it faster then you don't have many choices:
1) use something other than ssh. You say it has to be "Native" but NFS is that too.
2) use a worse cipher though on your setup it appears that nothing is very good so I am guessing you have an old slow processor, possibly without built-in AES support
If you currently get 200MB/s and want to get to 1500 then you will need to make drastic changes to the way you do this. An increase of 100% throughput might be possible but one of 800%? Not likely, not without dropping encryption and transferring in plain text. Replacing your processor with a faster one that has AES native instructions might help but I doubt if it'll get you 8 times your current throughput.
1) use something other than ssh. You say it has to be "Native" but NFS is that too.
2) use a worse cipher though on your setup it appears that nothing is very good so I am guessing you have an old slow processor, possibly without built-in AES support
If you currently get 200MB/s and want to get to 1500 then you will need to make drastic changes to the way you do this. An increase of 100% throughput might be possible but one of 800%? Not likely, not without dropping encryption and transferring in plain text. Replacing your processor with a faster one that has AES native instructions might help but I doubt if it'll get you 8 times your current throughput.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
-
- Posts: 7
- Joined: 2019/11/09 18:04:26
Re: Copy on file with multithread
Thank you
You right, it's an old cpu. But i can't change it because i works with about 400 Servers that is why i'm looking for an another solution.
You right, i think that use ssh without encryption is my solution, but I use CentOS 7.7 and i think that the Chiper none is not compilate by default.I am currently working on it.
You right, it's an old cpu. But i can't change it because i works with about 400 Servers that is why i'm looking for an another solution.
You right, i think that use ssh without encryption is my solution, but I use CentOS 7.7 and i think that the Chiper none is not compilate by default.I am currently working on it.
Re: Copy on file with multithread
No, you will need to use something OTHER than ssh.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Re: Copy on file with multithread
When I transfer files I use pigz to compress the data and then pipe that through ssh
Usually more than one file, so tar, but tar also keeps permissions etc on the receiving machine. That gets more threads on the sending machine but still is single threaded on receiving machine.
Code: Select all
$ tar cf - filename(s) | pigz | ssh -oCompression=no user@remote "cd /destination/directory && tar xvzf -"
- KernelOops
- Posts: 428
- Joined: 2013/12/18 15:04:03
- Location: xfs file system
Re: Copy on file with multithread
In you want to run multiple scp commands (or other commands for that matter) then you may use the awesome good old parallel command:
Code: Select all
NAME
parallel - build and execute shell command lines from standard input in parallel
SYNOPSIS
parallel [options] [command [arguments]] < list_of_arguments
parallel [options] [command [arguments]] ( ::: arguments | :::: argfile(s) ) ...
parallel --semaphore [options] command
#!/usr/bin/parallel --shebang [options] [command [arguments]]
--
R.I.P. CentOS
--
R.I.P. CentOS
--
Re: Copy on file with multithread
There are many ways to run multiple commands. However, that helps (here) only if there are multiple files to transfer.