Feature Suggestions for Cross-Platform-Compatibility

[Toscano2]

Hi,

'hoping to get some feedback this time ...

I'd like to suggest and discuss some features, mainly easing
doing truly cross-platform backups.

- the appended timestamps (e.g. 'current_mirror.2008-10-07T16:43:00-07:00.data') are not compatible with
Windows. It's impossible to simply copy over/rsync on an ext3-filesystem locally rdiff-backupped data later to a Windows system without breaking the backups. (Directly using rdiff-backup between local Windows and remote Linux system cannot be used.)

So there should be an alternative commandline option (e.g. '--use-windows-compatible-timestamps' or just '--use-compatible-timestamps') which generates the timestamp without ':'/colons, e.g.: '2007-07-28T10-51-18', since the W3-time/datestamps are not meant for being used in filenames.

- the timezone could be left out, if by convention, rdiff-backup would
use/convert to UTC before writing timestamps to files, e.g. '--use-utc' to maintain backward compatibility).So problems with daylight saving time and managing servers across different timezones are resolved.

- an option like '--preserve-case' to explicitely override auto-detected settings if any case-insensitive filesystem is used (NOT via '--override-chars-to-quote').

The last two ideas would keep filenames smaller, in many cases prevent filepaths from violating length-constraints.

- finally, something as '--override-chars-to-quote-from-file' would be useful, so any non-printable characters/ranges can be specified here, too.
This would ease creating a single compatibility-file for e.g. easy Windows-Linux-Mac migration, similar to the '--include-from-file' options.

What do you think of these suggestions?

+----------------------------------------------------------------------
|This was sent by g3mtbf2kh76xy1j@... via Backup Central.
|Forward SPAM to abuse@....
+----------------------------------------------------------------------




_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@...
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

[Andrew Ferguson-3]

On Oct 10, 2008, at 8:24 AM, Toscano2 wrote:
> - the appended timestamps (e.g. 'current_mirror.
> 2008-10-07T16:43:00-07:00.data') are not compatible with
> Windows. It's impossible to simply copy over/rsync on an ext3-
> filesystem locally rdiff-backupped data later to a Windows system  
> without breaking the backups. (Directly using rdiff-backup between  
> local Windows and remote Linux system cannot be used.)

- If you use rdiff-backup on both sides, then this problem won't be  
there. Rdiff-backup is now well-tested for avoiding reserved Windows  
characters (and other reserved Windows names).
- You could also add : to the chars-to-quote yourself (and,  
presumably, the other DOS reserved characters)
- You could write a patch for rsync which translates reserved Windows  
characters and names.

> So there should be an alternative commandline option (e.g. '--use-
> windows-compatible-timestamps' or just '--use-compatible-
> timestamps') which generates the timestamp without ':'/colons, e.g.:  
> '2007-07-28T10-51-18', since the W3-time/datestamps are not meant  
> for being used in filenames.

Patches fully compatible with existing rdiff-backup usage are welcome.

> - the timezone could be left out, if by convention, rdiff-backup would
> use/convert to UTC before writing timestamps to files, e.g. '--use-
> utc' to maintain backward compatibility).So problems with daylight  
> saving time and managing servers across different timezones are  
> resolved.

I believe you can override this by adjusting the TZ variable for just  
the rdiff-backup process. e.g.,

TZ=UTC rdiff-backup ....

Let me know if that doesn't work (and you are using Python 2.3 or  
later), as it would be a bug in rdiff-backup.

> - an option like '--preserve-case' to explicitely override auto-
> detected settings if any case-insensitive filesystem is used (NOT  
> via '--override-chars-to-quote').

What's wrong with --override-chars-to-quote ?    Really, I don't  
understand why people want to do this on case-insensitive filesystems  
-- I view dataloss as infinitely more drastic than cosmetic issues  
with filenames. rdiff-backup reverses the translation during a restore.

> The last two ideas would keep filenames smaller, in many cases  
> prevent filepaths from violating length-constraints.

On Unix, rdiff-backup has proper support for handling too-long  
filenames. I see there was an email recently about this not working on  
Windows, a bug I'll try to look into after the next release.

> - finally, something as '--override-chars-to-quote-from-file' would  
> be useful, so any non-printable characters/ranges can be specified  
> here, too.
> This would ease creating a single compatibility-file for e.g. easy  
> Windows-Linux-Mac migration, similar to the '--include-from-file'  
> options.

--override-chars-to-quote is a regular expression, so you can put non-
printable characters in by escaping them (octal) ... eg:

--override-chars-to-quote "\000-\037"

will make sure that ASCII characters 0 - 31 are escaped. If you write  
a wrapper script to rdiff-backup, you can put in non-printable  
characters directly.

I would certainly consider a patch to add an --override-chars-to-quote-
from-file option, and it might be a good first patch for someone new  
to Python or the rdiff-backup code. The get_ctq_from_fsas() function  
in fs_abilities.py shows how rdiff-backup figures out which chars  
should be quoted.



Andrew




_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@...
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

[Toscano2]

Andrew Ferguson wrote:

> On Oct 10, 2008, at 8:24 AM, Toscano2 wrote:
>
> > - the appended timestamps (e.g. 'current_mirror.
> > 2008-10-07T16:43:00-07:00.data') are not compatible with
> > Windows. It's impossible to simply copy over/rsync on an ext3-
> > filesystem locally rdiff-backupped data later to a Windows system  
> > without breaking the backups. (Directly using rdiff-backup between  
> > local Windows and remote Linux system cannot be used.)
> >
>
>
> - If you use rdiff-backup on both sides, then this problem won't be  
> there. Rdiff-backup is now well-tested for avoiding reserved Windows  
> characters (and other reserved Windows names).
> - You could also add : to the chars-to-quote yourself (and,  
> presumably, the other DOS reserved characters)
> - You could write a patch for rsync which translates reserved Windows  
> characters and names.
>
>

Yes, using rdiff-backup on both sides would be best solution - it's one of the outstanding features of RDB, given that the Windows system is always available.

But with my laptop, this is not the case, the servers must be backupped regularly locally first. The remote backups I will fetch via rsync when I have time to. An rsync-patch would translate the filenames, but the rdiff-backup program would not be able to handle those converted files for recovery.


Andrew Ferguson wrote:

>
>
> > So there should be an alternative commandline option (e.g. '--use-
> > windows-compatible-timestamps' or just '--use-compatible-
> > timestamps') which generates the timestamp without ':'/colons, e.g.:  
> > '2007-07-28T10-51-18', since the W3-time/datestamps are not meant  
> > for being used in filenames.
> >
>
> Patches fully compatible with existing rdiff-backup usage are welcome.
>

ok, so I will give it a try


Andrew Ferguson wrote:

>
>
> > - the timezone could be left out, if by convention, rdiff-backup would
> > use/convert to UTC before writing timestamps to files, e.g. '--use-
> > utc' to maintain backward compatibility).So problems with daylight  
> > saving time and managing servers across different timezones are  
> > resolved.
> >
>
> I believe you can override this by adjusting the TZ variable for just  
> the rdiff-backup process. e.g.,
>
> TZ=UTC rdiff-backup ....
>
> Let me know if that doesn't work (and you are using Python 2.3 or  
> later), as it would be a bug in rdiff-backup.
>
>

good to know, at least for Linux. I will test it.


Andrew Ferguson wrote:

>
>
> > - an option like '--preserve-case' to explicitely override auto-
> > detected settings if any case-insensitive filesystem is used (NOT  
> > via '--override-chars-to-quote').
> >
>
> What's wrong with --override-chars-to-quote ?    Really, I don't  
> understand why people want to do this on case-insensitive filesystems  
> -- I view dataloss as infinitely more drastic than cosmetic issues  
> with filenames. rdiff-backup reverses the translation during a restore.
>

Basically I am with you on this issue, but in case of disaster-recovery (harddisk-crash) everything must go quickly, synchronizing RDB minor version numbers on both the backup server and -client might bring new issues (evg. upgrading python required first, or no most-current release of RDB for the linux/Windows distro available).

When the filenames are kept as they are, I only need to copy (the wanted parts of) the mirror over, without need for rdiff-backup (in most basic cases).

Personally, I use only filenames that do not collide when taken to case-insensitive filesystems. For the above procedure, the windows system is only intermediary, as an additional backup, restore is always from linux to linux.

Another poster on this mailinglist had the idea that there could be a collision-check inside RDB for the current processed directory.
If there would be collisions, a suffix could be added to the filename to indicate this, so no data is lost and this info written to logfile.


Andrew Ferguson wrote:

>
>
> > The last two ideas would keep filenames smaller, in many cases  
> > prevent filepaths from violating length-constraints.
> >
>
> On Unix, rdiff-backup has proper support for handling too-long  
> filenames. I see there was an email recently about this not working on  
> Windows, a bug I'll try to look into after the next release.
>
>
> > - finally, something as '--override-chars-to-quote-from-file' would  
> > be useful, so any non-printable characters/ranges can be specified  
> > here, too.
> > This would ease creating a single compatibility-file for e.g. easy  
> > Windows-Linux-Mac migration, similar to the '--include-from-file'  
> > options.
> >
>
>
> --override-chars-to-quote is a regular expression, so you can put non-
> printable characters in by escaping them (octal) ... eg:
>
> --override-chars-to-quote "\000-\037"
>
> will make sure that ASCII characters 0 - 31 are escaped. If you write  
> a wrapper script to rdiff-backup, you can put in non-printable  
> characters directly.
>
> I would certainly consider a patch to add an --override-chars-to-quote-
> from-file option, and it might be a good first patch for someone new  
> to Python or the rdiff-backup code. The get_ctq_from_fsas() function  
> in fs_abilities.py shows how rdiff-backup figures out which chars  
> should be quoted.
>
>

This is good news!! :D

You're right, I am completely new to both Python and rdiff-backup, but I will give it a try and do some local testing. I should use latest CVS-version, right? Where to should I submit the patches, to the mailing list?

Thank you for your feedback,
Oliver

+----------------------------------------------------------------------
|This was sent by g3mtbf2kh76xy1j@... via Backup Central.
|Forward SPAM to abuse@....
+----------------------------------------------------------------------




_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@...
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

[Andrew Ferguson-3]

On Oct 11, 2008, at 8:05 AM, Toscano2 wrote:
> good to know, at least for Linux. I will test it.

It should work on Windows, too. Windows has environment variables  
which function the same way. [1] Perhaps you can set them ad-hoc in  
a .bat script or similar?

> You're right, I am completely new to both Python and rdiff-backup,  
> but I will give it a try and do some local testing. I should use  
> latest CVS-version, right? Where to should I submit the patches, to  
> the mailing list?

Yes, please write patches against CVS and you can submit them to the  
mailing list.


thanks,
Andrew


[1] right-click My Computer -> Properties -> Advanced -> Environment  
Variables.


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@...
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki