Netcat can be of great help in trasferring files across network, that too in a really scalable pipeline. For bear minimum usage of file transfer, only tar and netcat utilities are required.
I tried different options and found the following superset pipeline.
sender:$ tar cf - some_directory | gzip -9 | \ pv | gpg -c | nc -q0 25000 receiver:$ nc -l -p 25000 | gpg -d | gzip -d | \ pv | tar xf -
Some Caveats
- tar and nc do not offer any security, that is why gpg is used. In a trusted network, gpg pipeline elements could be removed from both sender and receiver.
- pv could monitor the datarate very effectively. pv is not used symmetrically in the above pipe line. In the sender side it shows data rate of compressed stream whereas in the receiving side it shows the data rate of uncompressed stream. This is done deliberately so that the user could get an idea about both compressed/uncompressed data rates. By reordering gzip and pv, this could be reversed.
- Instead of using tar and gzip separately, once could combine them and use ‘cfz’ and ‘xfz’. But, this limits the differential data rate as explained above. Fine control would be possible only by splitting them. Depending upon network throughput and CPU performance, either XZ or Lzop could be used instead of Gzip. The former offers extremely good compression, whereas the latter gives very good performance.
Saurorja 