[Nolug] Fastest way to create a local mirror of a huge tree?

From: Ron Johnson <ron.l.johnson_at_cox.net>
Date: Tue, 06 Sep 2011 22:21:33 -0500
Message-ID: <4E66E33D.5040007@cox.net>

The mirror doesn't exist yet, so rsync's clever data-minimizing
algorithms aren't valid. (Also, there are lots of symlinks that need be
preserved.)

1. cp -av /data /mnt/backups/data
2. cd /data && tar -cvf – . | (cd /mnt/backups/data && tar -xpvf -)
3. rsync -avz --stats --progress /data /mnt/backups/data

In a long test, I got 32MB/s from the cp when copying files in the 200MB
to 4GB range, but 4MB/s for files in the 100KB range.

In a short test only 6MB/s from tar; maybe it was scanning through the
list of files?.

rsync bounces around from 25MB/s to 4MB/s.

So, what's your experience in the best way to create an initial mirror?

-- 
Supporting World Peace Through Nuclear Pacification
___________________
Nolug mailing list
nolug@nolug.org
Received on 09/06/11

This archive was generated by hypermail 2.2.0 : 09/07/11 EDT