Mirroring Time Machine backups off-site?

[Ashley Aitken]

Howdy All,

I would like to mirror a Time Machine backup drive off-site with daily  
updates.

I'm aware that Time Machine uses hard links to directories to do some  
of its magic.

I need to mirror the Time Machine backup to a machine running Tiger  
(which I assume doesn't know about hard links to directories?) or even  
a Linux or Windows file system.

I am thinking of using rsync but those hard links to directories have  
me worried.  I'm hoping that Time Machine's sparse bundle images for  
network backups would hide those details?

If this is would be the case, is it also possible to make a backup to  
a local disk use a sparse bundle image rather than the native file  
system?

Thanks for any suggestions, comments.

Cheers,
Ashley.

--
Ashley Aitken
Perth, Western Australia
mrhatken at mac dot com
Skype Name: MrHatken (GMT + 8 Hours!)






_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Dan Shoop]

On Sep 22, 2008, at 11:23 AM, Ashley Aitken wrote:

>
> Howdy All,
>
> I would like to mirror a Time Machine backup drive off-site with  
> daily updates.
>
> I'm aware that Time Machine uses hard links to directories to do  
> some of its magic.
>
> I need to mirror the Time Machine backup to a machine running Tiger  
> (which I assume doesn't know about hard links to directories?) or  
> even a Linux or Windows file system.
>
> I am thinking of using rsync but those hard links to directories  
> have me worried.  I'm hoping that Time Machine's sparse bundle  
> images for network backups would hide those details?
>
> If this is would be the case, is it also possible to make a backup  
> to a local disk use a sparse bundle image rather than the native  
> file system?

The problem is that it's not just hard links you need, but hard links  
of directories, which few filesystems support. Generally hard links on  
directories are exactly what filesystems try to avoid. And yes you are  
right, Tiger doesn't support directory hard links.

In addition to the problematic hard links you need a method that  
correctly preserves file times and attributes for Time Machine, which  
will be equally problematic since most tools won't preserve these on  
new volumes since the copied files are different files which implies  
different inodes and inode times.

Moreover rsync will either consume unacceptable amounts of address  
space (note that's not the same as using unacceptable amounts of  
memory, it's worse) in these sorts of operations or is more likely to  
just die. Better hope you have 64 bit machines with plenty of user  
address space -- and the memory to support that address space -- on  
both ends. Even then, I don't think rsync will be able to handle the  
hardlinks for directories.

Normally the way around this is to use dump (provided Apple taught it  
about directory hardlinks). And then this will dump the whole thing,  
no "updates", it will be necessary to transfer the whole kit and  
kaboodle each time. So that's not going to work for you either.

Problem is Time Machine is not designed for this or even a good choice  
for such backups.

I'd suggest finding a different method for backup if you want to off-
site.

Alternatively maintain a disk image of the Time Machine volume and the  
rsync the disk image file itself to another host. Of course this means  
you have to image the TM disk every day and have twice the space.

-d

------------------------------------------------------------------------
Dan Shoop
Computer Scientist
iWiring / U.S. Technical Services

shoop@...
AOL IM .................... iWiring
Nextel .................... 1-714-363-1174
Operations TOC (24/7) ..... 1-866-901-USTS
USTS Offices .............. 1-714-374-6300

For immediate response for urgent matters please speak to the Duty  
Officer
at the USTS Tactical Operations Center (above) who can reach me by  
radio.



_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Gilles Dignard]

>  Howdy All,
>
>  I would like to mirror a Time Machine backup drive off-site with
>  daily updates.
I do this all the time, though I have a trick to it.

It's a simple trick, though, and obvious once you know about the
Thermaltake BlacX SATA docking station.

http://www.thermaltakeusa.com/Products.aspx?C=1346

I have three rotating 640G drives.

Once a week I turn off TM, remove the drive from the docking station,
put it in a safety deposit box, bring back the one from the bank to
on-site undocked backup, and the on-site undocked backup goes into
the dock and TM gets turned back on.
_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Ashley Aitken]

Sorry, quick reply, in a rush.

I noted your points about hard links to directories.  That's why I was  
suggesting mirroring the sparse bundle images that TM creates for  
network backups, which I assume hide all the hard link details and all  
other file system peculiarities within the binary image, which is  
nicely chunked (bands) for incremental backup.

Your points about rsync are useful, thanks.

Cheers,
Ashley.

On 24/09/2008, at 2:53 AM, Dan Shoop wrote:

>
> On Sep 22, 2008, at 11:23 AM, Ashley Aitken wrote:
>
>>
>> Howdy All,
>>
>> I would like to mirror a Time Machine backup drive off-site with  
>> daily updates.
>>
>> I'm aware that Time Machine uses hard links to directories to do  
>> some of its magic.
>>
>> I need to mirror the Time Machine backup to a machine running Tiger  
>> (which I assume doesn't know about hard links to directories?) or  
>> even a Linux or Windows file system.
>>
>> I am thinking of using rsync but those hard links to directories  
>> have me worried.  I'm hoping that Time Machine's sparse bundle  
>> images for network backups would hide those details?
>>
>> If this is would be the case, is it also possible to make a backup  
>> to a local disk use a sparse bundle image rather than the native  
>> file system?
>
>
> The problem is that it's not just hard links you need, but hard  
> links of directories, which few filesystems support. Generally hard  
> links on directories are exactly what filesystems try to avoid. And  
> yes you are right, Tiger doesn't support directory hard links.
>
> In addition to the problematic hard links you need a method that  
> correctly preserves file times and attributes for Time Machine,  
> which will be equally problematic since most tools won't preserve  
> these on new volumes since the copied files are different files  
> which implies different inodes and inode times.
>
> Moreover rsync will either consume unacceptable amounts of address  
> space (note that's not the same as using unacceptable amounts of  
> memory, it's worse) in these sorts of operations or is more likely  
> to just die. Better hope you have 64 bit machines with plenty of  
> user address space -- and the memory to support that address space  
> -- on both ends. Even then, I don't think rsync will be able to  
> handle the hardlinks for directories.
>
> Normally the way around this is to use dump (provided Apple taught  
> it about directory hardlinks). And then this will dump the whole  
> thing, no "updates", it will be necessary to transfer the whole kit  
> and kaboodle each time. So that's not going to work for you either.
>
> Problem is Time Machine is not designed for this or even a good  
> choice for such backups.
>
> I'd suggest finding a different method for backup if you want to off-
> site.
>
> Alternatively maintain a disk image of the Time Machine volume and  
> the rsync the disk image file itself to another host. Of course this  
> means you have to image the TM disk every day and have twice the  
> space.
>
> -d
>
> ------------------------------------------------------------------------
> Dan Shoop
> Computer Scientist
> iWiring / U.S. Technical Services
>
> shoop@...
> AOL IM .................... iWiring
> Nextel .................... 1-714-363-1174
> Operations TOC (24/7) ..... 1-866-901-USTS
> USTS Offices .............. 1-714-374-6300
>
> For immediate response for urgent matters please speak to the Duty  
> Officer
> at the USTS Tactical Operations Center (above) who can reach me by  
> radio.
>
>
>

Cheers,
Ashley.

--
Ashley Aitken
Perth, Western Australia
Skype/iChat: MrHatken (GMT + 8hrs!)

_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Ashley Aitken]

Hi Gilles (et al.),

Thanks for your post.

On 24/09/2008, at 6:26 AM, Gilles Dignard wrote:

>> Howdy All,
>>
>> I would like to mirror a Time Machine backup drive off-site with
>> daily updates.
> I do this all the time, though I have a trick to it.
>
> Once a week I turn off TM, remove the drive from the docking  
> station, put it in a safety deposit box, bring back the one from the  
> bank to on-site undocked backup, and the on-site undocked backup  
> goes into the dock and TM gets turned back on.

Technically, I don't think that is really mirroring, although what I  
am trying to do it probably not as well. I want  only a daily mirror  
and I want it to be automatic (i.e. not having to swap drives and  
physically take off-site).

I think I will give my suggestion (using the sparse bundle images and  
rsync) a go and see how it goes.

What would be easier though would be if TM allowed one to have two  
backup devices which it backed up to separately / sequentially, then I  
could have one off-site and one on-site.

Cheers,
Ashley.

--
Ashley Aitken
Perth, Western Australia
Skype/iChat: MrHatken (GMT + 8hrs!)

_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Dan Shoop]

On Sep 23, 2008, at 6:26 PM, Gilles Dignard wrote:

>> Howdy All,
>>
>> I would like to mirror a Time Machine backup drive off-site with
>> daily updates.
> I do this all the time, though I have a trick to it.
>
> It's a simple trick, though, and obvious once you know about the  
> Thermaltake BlacX SATA docking station.
>
> http://www.thermaltakeusa.com/Products.aspx?C=1346
>
> I have three rotating 640G drives.
>
> Once a week I turn off TM, remove the drive from the docking  
> station, put it in a safety deposit box, bring back the one from the  
> bank to on-site undocked backup, and the on-site undocked backup  
> goes into the dock and TM gets turned back on.

Seems like DAS to me.

How is this product off-site and remote to the TM system? Sounds like  
the OP want to mirror to a drive that is off-site. DAS is always  
locally attached.

-d

------------------------------------------------------------------------
Dan Shoop
Computer Scientist
iWiring / U.S. Technical Services

shoop@...
AOL IM .................... iWiring
Nextel .................... 1-714-363-1174
Operations TOC (24/7) ..... 1-866-901-USTS
USTS Offices .............. 1-714-374-6300

For immediate response for urgent matters please speak to the Duty  
Officer
at the USTS Tactical Operations Center (above) who can reach me by  
radio.



_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[John Musbach-3]

On Mon, Sep 22, 2008 at 11:23 AM, Ashley Aitken <mrhatken@...> wrote:
>
> Howdy All,
>
> I would like to mirror a Time Machine backup drive off-site with daily
> updates.

If you are mirroring to a linux file server, you should be able to
install partimage and then run a cronjob on the linux fileserver to
use partimage to image the used space of the time machine drive every
day. This of course requires you to make a network share on the mac
for the time machine volume, also note that partimage is similar to dd
but not the same. This is so because partimage images only the used
space of a volume while dd images both used and free so partimage
images significantly save space while preserving all particularities
stored on the filesystem.



--
Best Regards,

John Musbach
_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Ashley Aitken]

Thanks for your post John.

On 25/09/2008, at 11:12 PM, John Musbach wrote:

> On Mon, Sep 22, 2008 at 11:23 AM, Ashley Aitken <mrhatken@...>  
> wrote:
>>
>> Howdy All,
>>
>> I would like to mirror a Time Machine backup drive off-site with  
>> daily
>> updates.
>
> If you are mirroring to a linux file server, you should be able to
> install partimage and then run a cronjob on the linux fileserver to
> use partimage to image the used space of the time machine drive every
> day.

So partimage sounds like it makes something like sparseimages on  
MacOSX.  This is a good idea but imaging the whole disk each day seems  
a lot of work.

I would prefer to just update what has changed.

The format of MacOSX sparse bundle images is quite interesting. It  
consists of many 8MB files within a package/folder (together they all  
form the image).

I'm wondering if rsync'ing the whole image (i.e. all the bundles) is  
necessary when I could just find out which bands have been modified  
since the last mirror and rsync those.

I wonder what would happen if bands are deleted, although in the  
context of a TM image I think it may be the case that bands are only  
modified or added, not deleted.

Cheers,
Ashley.


--
Ashley Aitken
Perth, Western Australia
Skype/iChat: MrHatken (GMT + 8hrs!)

_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Jared Earle]

That sounds like a hideous amount of bandwidth. My backup image is  
about 60GB, for instance, and I cannot begin to imagine the amount of  
upload time that would take every night.

--
Jared Earle :: iPhone-at-23x-dot-net
  http://jearle.eu/ :: http://blog.23x.net

On 25 Sep 2008, at 16:12, "John Musbach" <johnmusbach1@...> wrote:

> On Mon, Sep 22, 2008 at 11:23 AM, Ashley Aitken <mrhatken@...>  
> wrote:
>>
>> Howdy All,
>>
>> I would like to mirror a Time Machine backup drive off-site with  
>> daily
>> updates.
>
> If you are mirroring to a linux file server, you should be able to
> install partimage and then run a cronjob on the linux fileserver to
> use partimage to image the used space of the time machine drive every
> day. This of course requires you to make a network share on the mac
> for the time machine volume, also note that partimage is similar to dd
> but not the same. This is so because partimage images only the used
> space of a volume while dd images both used and free so partimage
> images significantly save space while preserving all particularities
> stored on the filesystem.
>
>
>
> --
> Best Regards,
>
> John Musbach
> _______________________________________________
> MacOSX-admin mailing list
> MacOSX-admin@...
> http://www.omnigroup.com/mailman/listinfo/macosx-admin

_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Izidor Jerebic]

Hello,

Well, I'm a bit late, but about finding out what changed: rsync has  
very efficient method which does not transfer data if the data has not  
changed (even unchanged ranges inside single file). So you can just  
rsync the whole directory and leave the optimization to rsync...

izidor

On 25.9.2008, at 17:29, Ashley Aitken wrote:

>
> I'm wondering if rsync'ing the whole image (i.e. all the bundles) is  
> necessary when I could just find out which bands have been modified  
> since the last mirror and rsync those.
>
> I wonder what would happen if bands are deleted, although in the  
> context of a TM image I think it may be the case that bands are only  
> modified or added, not deleted.
>
> Cheers,
> Ashley.
>
>
> --
> Ashley Aitken
> Perth, Western Australia
> Skype/iChat: MrHatken (GMT + 8hrs!)
>
> _______________________________________________
> MacOSX-admin mailing list
> MacOSX-admin@...
> http://www.omnigroup.com/mailman/listinfo/macosx-admin

_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Bart Silverstrim]

Izidor Jerebic wrote:
> Hello,
>
> Well, I'm a bit late, but about finding out what changed: rsync has very
> efficient method which does not transfer data if the data has not
> changed (even unchanged ranges inside single file). So you can just
> rsync the whole directory and leave the optimization to rsync...

Now, I thought this meant that if you have a one-gig file that has a
couple bits flipped around then rsync will copy over just those bits,
not the whole gig file.

But I've had cases where I launch VirtualBox and shut the machine down,
hardly changing anything on the drive image, but I keep seeing it do a
huge transfer to the USB drive when I do my backup.

Is this expected behavior?
_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Ashley Aitken]

On 30/09/2008, at 10:24 PM, Bart Silverstrim wrote:

> Izidor Jerebic wrote:
>> Hello,
>> Well, I'm a bit late, but about finding out what changed: rsync has  
>> very efficient method which does not transfer data if the data has  
>> not changed (even unchanged ranges inside single file). So you can  
>> just rsync the whole directory and leave the optimization to rsync...
>
> Now, I thought this meant that if you have a one-gig file that has a  
> couple bits flipped around then rsync will copy over just those  
> bits, not the whole gig file.

I believe that is almost correct, but it doesn't go as fine-grained as  
bits, it breaks files up into segments (can't recall how big but they  
are not bits or bytes even) and check which segments have changed and  
then copies those across.

> But I've had cases where I launch VirtualBox and shut the machine  
> down, hardly changing anything on the drive image, but I keep seeing  
> it do a huge transfer to the USB drive when I do my backup.

It's probably a case of more changing than you expect.  Would swap  
files (which could easily be a few GBs) be a part of a VirtualBox  
image?  Of course, you may also have something incorrectly specified  
in the rsync command, it's not the simplest tool to use (and I'm not  
an expert at it ;-).

Cheers,
Ashley.

--
Ashley Aitken
Perth, Western Australia
Skype/iChat: MrHatken (GMT + 8hrs!)

_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Jared Earle]

On Tue, Sep 30, 2008 at 3:24 PM, Bart Silverstrim
<bsilver@...> wrote:
> But I've had cases where I launch VirtualBox and shut the machine down,
> hardly changing anything on the drive image, but I keep seeing it do a huge
> transfer to the USB drive when I do my backup.
>
> Is this expected behavior?

VM Images are essentially disks. When you open
VMWare/Parallels/VirtualBox, the entire disk image changes as writing
to a log, waking up a VM, whatever will change the part of the disk
that tells the OS where to find stuff. When I have TimeMachine
running, I specifically tell it not to back up my Virtual Machine
files, or else you end up with dozens of 8GB files in your backups.


--
 Jared Earle :: There is no SPORK
 jearle@... :: http://jearle.eu
 Hosting :: http://cat5.org
 Blog :: http://blog.23x.net
_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Thomas Baley]

I used both for a while, as well as Vmware in large corporate environments
with thousands of servers. I have stopped using Parallels for reasons of
convenience, a couple features, support, and robustness of the firm. None of
these were real show-stoppers by themselves, though.

Currently I have eight machines running, often concurrently: Vista Ultimate
32- & 64-bit, XP Pro 32- & 64-bit, Vista Business 32- & 64-bit, Server 2003
SP 2, and Server 2008 Std, and am about to add Exchange 2003 and 2007 to
mimic one of my clients.

Everything "just works", including running Remote Desktop and lots of other
system-level software.

It's a good thing I have a 500 GB internal disk :-)

I highly recommend Vmware.

Tom Baley


On 9/30/08 10:41 , "Jared Earle" <jearle@...> wrote:

> On Tue, Sep 30, 2008 at 3:24 PM, Bart Silverstrim
> <bsilver@...> wrote:
>> But I've had cases where I launch VirtualBox and shut the machine down,
>> hardly changing anything on the drive image, but I keep seeing it do a huge
>> transfer to the USB drive when I do my backup.
>>
>> Is this expected behavior?
>
> VM Images are essentially disks. When you open
> VMWare/Parallels/VirtualBox, the entire disk image changes as writing
> to a log, waking up a VM, whatever will change the part of the disk
> that tells the OS where to find stuff. When I have TimeMachine
> running, I specifically tell it not to back up my Virtual Machine
> files, or else you end up with dozens of 8GB files in your backups.
>

Thomas R. Baley
tbaley@...
770-984-8655
404-529-4710 f






_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Izidor Jerebic]

On 30.9.2008, at 16:24, Bart Silverstrim wrote:

> Izidor Jerebic wrote:
>> Hello,
>> Well, I'm a bit late, but about finding out what changed: rsync has  
>> very efficient method which does not transfer data if the data has  
>> not changed (even unchanged ranges inside single file). So you can  
>> just rsync the whole directory and leave the optimization to rsync...
>
> Now, I thought this meant that if you have a one-gig file that has a  
> couple bits flipped around then rsync will copy over just those  
> bits, not the whole gig file.
>
> But I've had cases where I launch VirtualBox and shut the machine  
> down, hardly changing anything on the drive image, but I keep seeing  
> it do a huge transfer to the USB drive when I do my backup.
>
> Is this expected behavior?

The *network* transfer is optimized. So only a few bytes should be  
sent across the network.

But files are written in their entirety at the destination - old file  
is copied and changed in places to become new version. So local disk  
transfer is big. Also, ff you use rsync locally to copy from one disk  
to another, no optimization is performed and complete file will be  
transferred and written to disk.

There are rsync options to change the file in-place and to force  
incremental transfer. In case of local copies (no network involved)  
maybe they could be useful...

izidor

_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Bart Silverstrim]

Izidor Jerebic wrote:

>
> On 30.9.2008, at 16:24, Bart Silverstrim wrote:
>
>> Izidor Jerebic wrote:
>>> Hello,
>>> Well, I'm a bit late, but about finding out what changed: rsync has
>>> very efficient method which does not transfer data if the data has
>>> not changed (even unchanged ranges inside single file). So you can
>>> just rsync the whole directory and leave the optimization to rsync...
>>
>> Now, I thought this meant that if you have a one-gig file that has a
>> couple bits flipped around then rsync will copy over just those bits,
>> not the whole gig file.
>>
>> But I've had cases where I launch VirtualBox and shut the machine
>> down, hardly changing anything on the drive image, but I keep seeing
>> it do a huge transfer to the USB drive when I do my backup.
>>
>> Is this expected behavior?
>
> The *network* transfer is optimized. So only a few bytes should be sent
> across the network.

Aha...inconsistent behavior not too user friendly but clears up the
confusion. I wonder why it switches to a non-incremental transfer when
the transfer is local media?

No matter. I'll have to google for "rsync force incremental copy" or
something along that line.
_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin

[Scott Ribe]

> Aha...inconsistent behavior not too user friendly but clears up the
> confusion. I wonder why it switches to a non-incremental transfer when
> the transfer is local media?

Probably faster, or at least no slower. All data has to be read anyway, and
the checksum is somewhat CPU intensive, so just writing the data may be
faster. (I've done some benchmarking myself and this can show up even with
1Gbps Ethernet & slower machines.)

> No matter. I'll have to google for "rsync force incremental copy" or
> something along that line.

man rsync; it's just one of many many options.


--
Scott Ribe
scott_ribe@...
http://www.killerbytes.com/
[phone] voice


_______________________________________________
MacOSX-admin mailing list
MacOSX-admin@...
http://www.omnigroup.com/mailman/listinfo/macosx-admin