Network File Transfer using Netcat

Netcat can be of great help in trasferring files across network, that too in a really scalable pipeline. For bear minimum usage of file transfer, only tar and netcat utilities are required.

I tried different options and found the following superset pipeline.

sender:$ tar cf - some_directory | gzip -9 | \
  pv | gpg -c | nc -q0  25000
receiver:$ nc -l -p 25000 | gpg -d | gzip -d | \
  pv | tar xf -

Some Caveats

  • tar and nc do not offer any security, that is why gpg is used. In a trusted network, gpg pipeline elements could be removed from both sender and receiver.
  • pv could monitor the datarate very effectively. pv is not used symmetrically in the above pipe line. In the sender side it shows data rate of compressed stream whereas in the receiving side it shows the data rate of uncompressed stream. This is done deliberately so that the user could get an idea about both compressed/uncompressed data rates. By reordering gzip and pv, this could be reversed.
  • Instead of using tar and gzip separately, once could combine them and use ‘cfz’ and ‘xfz’. But, this limits the differential data rate as explained above. Fine control would be possible only by splitting them. Depending upon network throughput and CPU performance, either XZ or Lzop could be used instead of Gzip. The former offers extremely good compression, whereas the latter gives very good performance.

Published by Anand Sivaram (आनन्दः )

twitter.com/anand_sivaram https://www.linkedin.com/in/anandsivaram/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: