Copy on file with multithread

A 5 star hangout for overworked and underpaid system admins.
ChristoheB
Posts: 7
Joined: 2019/11/09 18:04:26

Re: Copy on file with multithread

Post by ChristoheB » 2019/11/12 10:34:37

Hi,

I tested more ssh's options. And here are the results :
Command : scp OPTIONS /test.img root@192.168.1.1:/dev/null
pas d'option => 145 Mo/s
-c none => 145 Mo/s
-c 3des-cbc => 23 Mo/s
-c blowfish-cbc => 87 Mo/s
-c arcfour => Lost connexion
-c aes256-ctr => 175 Mo/s
-c aes128-ctr => 210 Mo/s
Yes, if i change the Cipher then it's better.

But the maximum is 210 Mo/s and i'm looking for 1500Mo/s. I understand that if i use nfs, nbt or nc, then i can obtain 1500. But i'm looking for about ssh (native).

User avatar
TrevorH
Forum Moderator
Posts: 27156
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Copy on file with multithread

Post by TrevorH » 2019/11/12 12:03:46

So, as said, the problem is not with copying nor your network, it's to do with encrypting the data. If you want it faster then you don't have many choices:

1) use something other than ssh. You say it has to be "Native" but NFS is that too.
2) use a worse cipher though on your setup it appears that nothing is very good so I am guessing you have an old slow processor, possibly without built-in AES support

If you currently get 200MB/s and want to get to 1500 then you will need to make drastic changes to the way you do this. An increase of 100% throughput might be possible but one of 800%? Not likely, not without dropping encryption and transferring in plain text. Replacing your processor with a faster one that has AES native instructions might help but I doubt if it'll get you 8 times your current throughput.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
CentOS 5 has been EOL for nearly 3 years and should no longer be used for anything!
Full time Geek, part time moderator. Use the FAQ Luke

ChristoheB
Posts: 7
Joined: 2019/11/09 18:04:26

Re: Copy on file with multithread

Post by ChristoheB » 2019/11/12 12:22:20

Thank you

You right, it's an old cpu. But i can't change it because i works with about 400 Servers that is why i'm looking for an another solution.
You right, i think that use ssh without encryption is my solution, but I use CentOS 7.7 and i think that the Chiper none is not compilate by default.I am currently working on it.

User avatar
TrevorH
Forum Moderator
Posts: 27156
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Copy on file with multithread

Post by TrevorH » 2019/11/12 12:29:28

No, you will need to use something OTHER than ssh.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
CentOS 5 has been EOL for nearly 3 years and should no longer be used for anything!
Full time Geek, part time moderator. Use the FAQ Luke

chemal
Posts: 613
Joined: 2013/12/08 19:44:49

Re: Copy on file with multithread

Post by chemal » 2019/11/13 02:13:01

TrevorH wrote:
2019/11/12 12:29:28
No, you will need to use something OTHER than ssh.
Good old rcp gives me 910 MB/s over a 10 GBit/s IPoIB link with 54% load on a single core of a 2008 i7-920. :)

drk
Posts: 403
Joined: 2014/01/30 20:38:28

Re: Copy on file with multithread

Post by drk » 2019/11/14 03:32:58

When I transfer files I use pigz to compress the data and then pipe that through ssh

Code: Select all

$ tar cf - filename(s) | pigz | ssh -oCompression=no user@remote "cd /destination/directory && tar xvzf -"
Usually more than one file, so tar, but tar also keeps permissions etc on the receiving machine. That gets more threads on the sending machine but still is single threaded on receiving machine.

User avatar
KernelOops
Posts: 102
Joined: 2013/12/18 15:04:03
Location: xfs file system

Re: Copy on file with multithread

Post by KernelOops » 2019/11/16 06:58:48

In you want to run multiple scp commands (or other commands for that matter) then you may use the awesome good old parallel command:

Code: Select all

NAME
       parallel - build and execute shell command lines from standard input in parallel

SYNOPSIS
       parallel [options] [command [arguments]] < list_of_arguments

       parallel [options] [command [arguments]] ( ::: arguments | :::: argfile(s) ) ...

       parallel --semaphore [options] command

       #!/usr/bin/parallel --shebang [options] [command [arguments]]
--
I love my computer - all my friends live there.
--

User avatar
jlehtone
Posts: 2432
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Copy on file with multithread

Post by jlehtone » 2019/11/19 17:14:53

KernelOops wrote:
2019/11/16 06:58:48
In you want to run multiple scp commands
There are many ways to run multiple commands. However, that helps (here) only if there are multiple files to transfer.

Post Reply

Return to “CentOS Social”