Re: [Nolug] Fastest way to create a local mirror of a huge tree?

On Wed, Sep 07, 2011 at 09:36:48AM -0500, Jonathan Roberts wrote:
> I have never done this on a local machine but rsync has worked great for me
> for machine to machine syncing. On a local network, it is really fast in my
> experience. I would assume it is faster on the same box limited only by the
> hardware.

The thing about rsync is that the costs are all upfront. Unless your
file tree changes drastically between syncs (in which case, rsync is
the wrong solution), you pay almost nothing to keep the mirror(s)
incrementally synchronized. And even if your file system changes
fairly often, you can still increase the frequency of synchronizations
in most cases to amortize the costs. If your looking for a realtime/streaming
solution, then some sort of RAID might be required.

Brett

>
> Jonathan
>
>
>
> On Wed, Sep 7, 2011 at 7:43 AM, B. Estrade <estrabd@gmail.com> wrote:
>
>> On Wed, Sep 07, 2011 at 06:34:34AM -0500, Ron Johnson wrote:
>>> If I wanted to make *bad* choices, I'd go back to Windows!
>>
>> Why is dd a bad choice?
>>
>> And rsync doesn't "datamine". It uses rolling checksums and other
>> heuristics (with an incredibly high probability of success) to determine
>> what bits of a file to transmit, thus making it very efficient for
>> r"sync"'ing. That you have to initially transfer all files the first
>> time is a consequence of the case where the target mirror is in no way
>> similar to the source. Maybe there is a "do initial copy" mode, but I
>> doubt it.
>>
>> Maybe you want to ghost your machine, if so check out g4u. You won't
>> get any speed benefits from it, though.
>>
>> Bret
>>
>>>
>>> On 09/07/2011 06:21 AM, Brad Bendily wrote:
>>>>Because you can?
>>>>Isn't that what Linux is all about, choices!
>>>>
>>>>
>>>>
>>>>On Sep 7, 2011, at 5:54 AM, Ron Johnson<ron.l.johnson@cox.net> wrote:
>>>>
>>>>>Why in God's name would I dd-over-ssh on a single machine?
>>>>>
>>>>>On 09/07/2011 05:46 AM, B. Estrade wrote:
>>>>>>I like the dd (over ssh) idea.
>>>>>>
>>>>>>B. Estrade<estrabd@gmail.com>
>>>>>>On Sep 7, 2011 12:23 AM, "Jimmy Hess"<mysidia@gmail.com> wrote:
>>>>>>>On Tue, Sep 6, 2011 at 10:21 PM, Ron Johnson<ron.l.johnson@cox.net>
>>>>>>wrote:
>>>>>>>>The mirror doesn't exist yet, so rsync's clever data-minimizing
>>>>>>algorithms
>>>>>>>>aren't valid. (Also, there are lots of symlinks that need be
>>>>>>>>preserved.)
>>>>>>>>1. cp -av /data /mnt/backups/data
>>>>>>>>2. cd /data&& tar -cvf ??? . | (cd /mnt/backups/data&& tar -xpvf
>> -)
>>>>>>>>3. rsync -avz --stats --progress /data /mnt/backups/data
>>>>>>>>
>>>>>>>Netcat + XZ or gzip + CPIO or tar.
>>>>>>>
>>>>>>>nc otherhost portnumber | xz -d | cpio -idm
>>>>>>>find pathname -print | cpio -o -Hnewc | xz -1 | nc -l portnumber
>>>>>>>
>>>>>>>Then rsync to reconcile.
>>>>>>>
>>>>>>>
>>>>>>>If on a dedicated partition, consider a block-based tool that won't
>>>>>>>need to traverse the filesystem
>>>>>>>directory structure and won't need to copy unused disk blocks, e.g.
>>>>>>partimage.
>>>>>>>
>>>
>>> --
>>> Supporting World Peace Through Nuclear Pacification
>>> ___________________
>>> Nolug mailing list
>>> nolug@nolug.org
>>
>> --
>> B. Estrade <estrabd@gmail.com>
>> ___________________
>> Nolug mailing list
>> nolug@nolug.org
>>

-- 
B. Estrade <estrabd@gmail.com>
___________________
Nolug mailing list
nolug@nolug.org