Rsync 2.6.9 does not skip any files based on modification time

[Stefan Malte Schumacher]

Hi

I am trying to rsync some ghost images from a windows client running Windows XP to my Linux server. The problem is that rsync sends the complete files again even if nothing changed on the client side. The only way to avoid this is to use the "-c"-option but this takes nearly as long as uploading the files would.

The server is running rsync-2.6.9, /etc/rsyncd.conf looks as follows. /mnt/D/ is a FAT32 partition.

read only = false
use chroot = true
log format = %h %o %f %l %b
log file = /var/log/rsyncd.log
secrets file = /etc/rsyncd.secrets

[Backup]
path = /mnt/D/Backup
auth users = stefan
uid = stefan

[HDD-Abbilder]
path = /mnt/D/HDD-Abbilder
auth users = stefan
uid = stefan


The client is using cwRsync_2.0.10, which also uses rsync 2.6.9. I am using the patched cygwin1.dll from http://www.okisoft.co.jp/esc/utf8-cygwin/download.html in order to preserve filenames with german umlauts when uploading to the server. This is the line used to call rsync from cwrsync.cmd
rsync -rvvvv --del --modify-window=3 /cygdrive/E/HDD-Abbilder/ stefan@...::HDD-Abbilder

How can I get rsync to skip files based on modification time ?

Bye
Stefan

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Paul Slootman-5]

On Fri 08 Feb 2008, Stefan Malte Schumacher wrote:
>
> I am trying to rsync some ghost images from a windows client running Windows
> XP to my Linux server. The problem is that rsync sends the complete files
> again even if nothing changed on the client side. The only way to avoid this
> is to use the "-c"-option but this takes nearly as long as uploading the
> files would.
>
> The server is running rsync-2.6.9, /etc/rsyncd.conf looks as follows.
> /mnt/D/ is a FAT32 partition.
[...]
> used to call rsync from cwrsync.cmd
> rsync -rvvvv --del --modify-window=3 /cygdrive/E/HDD-Abbilder/
> stefan@...::HDD-Abbilder

You're correctly using --modify-window, which is otherwise the probable
solution to your problem. However, somehow rsync still considers an
update necessary. You can use -i (--itemize-changes) to let rsync show
what attributes are incorrect, triggering the update.  Once you know
that, you can try to fix it.


Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Tony Abernethy]

Are you preserving the times?

There are many combinations.
Me, I always use -a and then add stuff.
Copying or Rsyncing windows stuff always seems to have stuff
(essential to Linux/Unix) missing from the windows side.

But nothing in that parameter list looks like it would
preserve the modification time  (or any other time).

> -----Original Message-----
> From: rsync-bounces+tony=servacorp.com@...
> [mailto:rsync-bounces+tony=servacorp.com@...] On
> Behalf Of Paul Slootman
> Sent: Friday, February 08, 2008 3:03 AM
> To: rsync@...
> Subject: Re: Rsync 2.6.9 does not skip any files based on
> modification time
>
> On Fri 08 Feb 2008, Stefan Malte Schumacher wrote:
> >
> > I am trying to rsync some ghost images from a windows
> client running Windows
> > XP to my Linux server. The problem is that rsync sends the
> complete files
> > again even if nothing changed on the client side. The only
> way to avoid this
> > is to use the "-c"-option but this takes nearly as long as
> uploading the
> > files would.
> >
> > The server is running rsync-2.6.9, /etc/rsyncd.conf looks
> as follows.
> > /mnt/D/ is a FAT32 partition.
> [...]
> > used to call rsync from cwrsync.cmd
> > rsync -rvvvv --del --modify-window=3 /cygdrive/E/HDD-Abbilder/
> > stefan@...::HDD-Abbilder
>
> You're correctly using --modify-window, which is otherwise
> the probable
> solution to your problem. However, somehow rsync still considers an
> update necessary. You can use -i (--itemize-changes) to let rsync show
> what attributes are incorrect, triggering the update.  Once you know
> that, you can try to fix it.
>
>
> Paul Slootman
> --
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read:
> http://www.catb.org/~esr/faqs/smart-questions.html
>

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Paul Slootman-5]

On Fri 08 Feb 2008, Tony Abernethy wrote:

> Are you preserving the times?

Hmm, it does seem that Stefan is missing the -t option.
As I basically always do "-a", I overlooked this.


Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Stefan Malte Schumacher]

Hmm, it does seem that Stefan is missing the -t option.
As I basically always do "-a", I overlooked this.

Hello guys

Thanks for the quick replies. I just added the -tO option as well as the -i option.
The output looks this way :
<f..t..... Avalon/filename.gho>
After this rsync complains that it cannot set file names on "/Avalon/filename.gho" because the Operation is not permitted. The FAT-partition is mounted with " rw,user,nodev,nosuid,umask=000" so in theory everything the user stefan is able to - as whom rsync is doing file operations if I understand the uid-option correctly - should be possibly for rsync as well. Should I try to set the uid in the rsyncd.conf to root ? Is it possible that the chroot-option could cause this behavior ?

Bye
Stefan

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Paul Slootman-5]

On Fri 08 Feb 2008, Stefan Malte Schumacher wrote:

> >
> > Hmm, it does seem that Stefan is missing the -t option.
> > As I basically always do "-a", I overlooked this.
>
> Thanks for the quick replies. I just added the -tO option as well as the -i
> option.
> The output looks this way :
> <f..t..... Avalon/filename.gho>
> After this rsync complains that it cannot set file names on
> "/Avalon/filename.gho" because the Operation is not permitted. The

I think you mean file owner? Of file times?
"cannot set file names" seems unlikely :-)


Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Tony Abernethy]

Paul Slootman wrote:
>
> I think you mean file owner? Of file times?
> "cannot set file names" seems unlikely :-)

File  OWNER
File  GROUP

and DOS/Windows  complains
Me, I let it complain   --- much more complaining with cp than with rsync

DOS and Windows have different ideas of what file attributes are worth
preserving.
NTFS has some elaborate ACLs, but far too much trouble to do what is trivial
on *nix with users and groups.

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Fri, 2008-02-08 at 10:59 +0100, Stefan Malte Schumacher wrote:

> Thanks for the quick replies. I just added the -tO option as well as
> the -i option.
> The output looks this way :
> <f..t..... Avalon/filename.gho>
> After this rsync complains that it cannot set file names on
> "/Avalon/filename.gho" because the Operation is not permitted. The
> FAT-partition is mounted with " rw,user,nodev,nosuid,umask=000" so in
> theory everything the user stefan is able to - as whom rsync is doing
> file operations if I understand the uid-option correctly - should be
> possibly for rsync as well. Should I try to set the uid in the
> rsyncd.conf to root ? Is it possible that the chroot-option could
> cause this behavior ?

The trouble is that FAT does not store file ownership, so Linux shows
all the files on the partition as belonging to a single user and group,
by default root.  User stefan can create files due to the 777
permissions, but ownership of the files is immediately given to root,
which prevents stefan from setting their mtimes arbitrarily (since Linux
requires that a non-root user own a file in order to set its mtime
arbitrarily).

To work around this irritating incompatibility, you could set the uid in
rsyncd.conf to root, or (safer) you could mount the partition with the
"uid=stefan" mount option to make all the files owned by stefan.

Matt

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Sat, 2008-02-09 at 01:49 +0100, Stefan Malte Schumacher wrote:
> Thanks, mounting the partition with uid=stefan did the trick. Probably
> more a cygwin than a rsync-question, but
> does anybody have any suggestions how I can make rsync accept "E:
> \Bücher & Comics" as source path ?

How is rsync rejecting it?  "File has vanished"?  If so, see this
message:

http://lists.samba.org/archive/rsync/2008-January/019696.html

Matt

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Paul Johnson-11]

For these cases, I add the option --size-only.  I guarantee it will
fix it.  rsync sees files as different b/c of mismatch between fat and
linux filesystems. using -t also helps. -a does not because it tries
to do impossible things.

Hth

PJ

On 2/8/08, Matt McCutchen <matt@...> wrote:

> On Sat, 2008-02-09 at 01:49 +0100, Stefan Malte Schumacher wrote:
> > Thanks, mounting the partition with uid=stefan did the trick. Probably
> > more a cygwin than a rsync-question, but
> > does anybody have any suggestions how I can make rsync accept "E:
> > \Bücher & Comics" as source path ?
>
> How is rsync rejecting it?  "File has vanished"?  If so, see this
> message:
>
> http://lists.samba.org/archive/rsync/2008-January/019696.html
>
> Matt
>
> --
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

--
Paul E. Johnson
Professor, Political Science
1541 Lilac Lane, Room 504
University of Kansas

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Stefan Malte Schumacher]

How is rsync rejecting it?  "File has vanished"?  If so, see this
message:

http://lists.samba.org/archive/rsync/2008-January/019696.html

Matt

rsync outputs the following :

building file list ... rsync: link_stat "/cygdrive/E/B\#302\#263cher " failed:
No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]
...

It seems that the problem are not only the spaces but the Umlaute in the filename. I renamed the folder from "Bücher & Comics" to "Bücher", but the error message stays the same. I have read the link from Matt but the codepage for non-unicode apps was already set to german. FYI : I am already using the hacked cygwin1.dll.

Bye
Stefan
 

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Stefan Malte Schumacher]

It seems that the problem are not only the spaces but the Umlaute in the filename. I renamed the folder from "Bücher & Comics" to "Bücher", but the error message stays the same. I have read the link from Matt but the codepage for non-unicode apps was already set to german. FYI : I am already using the hacked cygwin1.dll.

I just renamed "Bücher & Comics" to "Buecher" and then tried to rsync again. This time it works and rsync even copies the subdirectories  - one of which is named "Bücher & Rollenspiele" correctly. So the only problem to be solved is how to get rsync to accept "Bücher & Comics" as command line parameter.

Bye
Stefan

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Sun, 2008-02-10 at 04:57 +0100, Stefan Malte Schumacher wrote:
> I just renamed "Bücher & Comics" to "Buecher" and then tried to rsync
> again. This time it works and rsync even copies the subdirectories  -
> one of which is named "Bücher & Rollenspiele" correctly. So the only
> problem to be solved is how to get rsync to accept "Bücher & Comics"
> as command line parameter.

Good.  Then my guess is that there's some weirdness in the hacked
Cygwin's handling of command-line arguments.  As a workaround, you could
try using --files-from instead of putting the path on the command line.

Matt

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Stefan Malte Schumacher]

Good.  Then my guess is that there's some weirdness in the hacked
Cygwin's handling of command-line arguments.  As a workaround, you could
try using --files-from instead of putting the path on the command line.

Hmm, it seems I need help again. I am calling again rsync with the following options :
rsync -rvtO --del --files-from=/cygdrive/C/Programme/cwRsync/rsync.txt --modify-window=3 stefan@...::Backup    

rsync.txt just contains one single line :
/cygdrive/D/Temp

rsync then outputs a list of the command line options and the following:
rsync error: syntax or usage error (code 1) at options.c(1418) [client=2.6.9]

How should I list E:\Bücher & Comics in rsync.txt ? As /cygdrive/E/Bücher & Comics or should I just use single or double quotes ?

Bye
Stefan

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Wed, 2008-02-13 at 03:59 +0100, Stefan Malte Schumacher wrote:
> Hmm, it seems I need help again. I am calling again rsync with the following options :
> rsync -rvtO --del --files-from=/cygdrive/C/Programme/cwRsync/rsync.txt --modify-window=3 stefan@...::Backup    
>
> rsync.txt just contains one single line :
> /cygdrive/D/Temp
>
> rsync then outputs a list of the command line options and the following:
> rsync error: syntax or usage error (code 1) at options.c(1418) [client=2.6.9]

With --files-from, you still have to specify a master source directory
relative to which the --files-from entries are interpreted.  In your
case, you should just use / .  Your command should be:

rsync -rvtO --del --files-from=/cygdrive/C/Programme/cwRsync/rsync.txt
\ --modify-window=3 / stefan@...::Backup

> How should I list E:\Bücher & Comics in rsync.txt ?
> As /cygdrive/E/Bücher & Comics or should I just use single or double
> quotes ?

As you said.  rsync.txt should contain these two lines:

/cygdrive/D/Temp
/cygdrive/E/Bücher & Comics

--files-from implies --relative, so by default, rsync will use the
entire source paths in the file-list and thus recreate cygdrive/D/Temp
and "/cygdrive/E/Bücher & Comics" under the destination.  You can
override this by putting a ./ in each --files-from entry; then, only the
path components after the ./ will be used.

Matt

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Stefan Malte Schumacher]

rsync -rvtO --del --files-from=/cygdrive/C/Programme/cwRsync/rsync.txt
\       --modify-window=3 / stefan@...::Backup

> How should I list E:\Bücher & Comics in rsync.txt ?
> As /cygdrive/E/Bücher & Comics or should I just use single or double
> quotes ?

As you said.  rsync.txt should contain these two lines:

/cygdrive/D/Temp
/cygdrive/E/Bücher & Comics

I just tried what you suggested Matt but unfortunately Rsync again complains about the filenames -
"Bücher & Comics" changes to "B_cher & Comics" and for the subdirectory "Bücher & Rollenspiele" I get a "file has vanished"  for each file in it and even the directory itself is not created.  If I change the name of the directory from "Bücher & Comics" to "Bucher" and call it directly from the command line instead of using --files-from "Bücher & Rollenspiele" is created and the files in it are copied. I have read the posting you read concerning the "file has vanished"-message and the language for non-unicode apps is set to german !

Bye
Stefan

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html