Xattrs and Delete over AFP

View: New views
5 Messages — Rating Filter:   Alert me  

Xattrs and Delete over AFP

by zodiac-meow :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hello group,

I have an OSX 10.3.9 Server that I'm trying to back up with rsync 3.0.2.
Since I didn't have any luck with the 10.3.9 patch and I do have 10.4 clients,
I decided to try running the backup over AFP (Apple's file sharing protocol).
I found a way to mount the AFP volume with root permissions and rsync 3.0.2
works very nicely using this on the 10.4 clients, preserving xattrs and being
good.  I understand it's not the preferred method but the rsync we're using
for 10.3.9 is version 2.6.0 and the malloc errors are killing me.  When I add
any of the --delete options (during, delay, or after) to the line it works for
a while and then starts getting this:

rsync: get_xattr_data:
lgetxattr("Path-to-the-file","com.apple.ResourceFork",0000) failed: Bad file
descriptor (9)

The 0000 is different for each file.

The rsync line I'm using is:

rsync -aHXh --stats --progress --fileflags --crtimes --delete
/Volumes/Source_Volume /Volumes/Local_Drive/Source_Mirror/

Any ideas on why the delete option would cause this to happen?

Thanks for making rsync the unbelievable tool it is.

Mike


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Xattrs and Delete over AFP

by Matt McCutchen-7 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, 2008-06-21 at 21:34 -0400, zodiac-meow@... wrote:
> I have an OSX 10.3.9 Server that I'm trying to back up with rsync 3.0.2.
> Since I didn't have any luck with the 10.3.9 patch

To be perfectly clear, do you mean you have tried Vitorio Machado's
patch?

http://lists.samba.org/archive/rsync/2008-February/020093.html

If not, that may be worth a try.

> When I add
> any of the --delete options (during, delay, or after) to the line it works for
> a while and then starts getting this:
>
> rsync: get_xattr_data:
> lgetxattr("Path-to-the-file","com.apple.ResourceFork",0000) failed: Bad file
> descriptor (9)

This may just be an oddity with the network filesystem, but that doesn't
explain why it would happen only with a --delete option, which shouldn't
significantly change rsync's access pattern to the network-mounted
source.  If you ktrace the relevant rsync process with and without
--delete, that might provide some clues about what triggers the error.

Since the error only happens with --delete and appears to involve
xattrs, you may be able to work around it by using two passes of rsync,
the first with -X but not --delete and the second with --delete but not
-X.

Matt


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (204 bytes) Download Attachment

Parent Message unknown Re: Xattrs and Delete over AFP

by zodiac-meow :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thank you for your prompt response Matt.

> On Sat, 2008-06-21 at 21:34 -0400, zodiac-meow@... wrote:
> > I have an OSX 10.3.9 Server that I'm trying to back up with rsync 3.0.2.
> > Since I didn't have any luck with the 10.3.9 patch
>
> To be perfectly clear, do you mean you have tried Vitorio Machado's
> patch?
>
> http://lists.samba.org/archive/rsync/2008-February/020093.html
>
> If not, that may be worth a try.

I think I had tried a different version.  This patch applied to the current
3.0.2 build fine except for rsync.h, which it interpreted to be a reverse
patch.  I skipped that one and tried to build on my 10.3.9 station.  The
Autoconf wasn't a high enough version so I tried to build on 10.4 using
--build=powerpc-apple-darwin7.9.0.  It built after telling me to run "make
reconfigure" but wouldn't execute on 10.3.9 due to a reference to lchown.  So
I upgraded Autoconf on 10.3.9 and it compiled with only a few warnings:

gcc -std=gnu99 -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt  -c
clientname.c -o clientname.o
clientname.c: In function `compare_addrinfo_sockaddr':
clientname.c:269: warning: long int format, int arg (arg 4)

gcc -std=gnu99 -I. -I. -framework Carbon -g -O2 -DHAVE_CONFIG_H -Wall -W
-I./popt  -c lib/sysxattrs.c -o lib/sysxattrs.o
lib/sysxattrs.c: In function `sys_fgetxattr':
lib/sysxattrs.c:277: warning: unused parameter `filedes'
lib/sysxattrs.c:277: warning: unused parameter `name'
lib/sysxattrs.c:277: warning: unused parameter `value'
lib/sysxattrs.c:277: warning: unused parameter `size'
lib/sysxattrs.c: In function `sys_lremovexattr':
lib/sysxattrs.c:389: warning: unused parameter `path'
lib/sysxattrs.c:389: warning: unused parameter `name'
gcc: -framework: linker input file unused because linking not done
gcc: Carbon: linker input file unused because linking not done



I ran the backup bouncer on it and got these results:

Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... ok
Verifying:        creation-date ... ok
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... FAIL
   Sub-test:       on directories ... FAIL
   Sub-test:          on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
   Sub-test:             on files ... FAIL
   Sub-test:              on dirs ... FAIL
FAIL
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... FAIL
   Sub-test:     lots of metadata ... FAIL
FAIL


If I understand things correctly, Apple didn't introduce an API for extended
attributes until 10.4 so they shouldn't exist on a 10.3 server. ACLs shouldn't
be expected either.  And it works on most files OK.  So far, so good.  It does
gets this error on some files and directories:

rsync: get_xattr_names: llistxattr(" New Art Dept/John:Projects",1024) failed:
Invalid argument (22)

I've narrowed this down to it being only file or directory names with a "/" in
them (which OSX translates into a ":" in terminal).  This doesn't occur when
using an rsync without the pre-Tiger patch.  This is still great news though!
So close...


> This may just be an oddity with the network filesystem, but that doesn't
> explain why it would happen only with a --delete option, which shouldn't
> significantly change rsync's access pattern to the network-mounted
> source.  If you ktrace the relevant rsync process with and without
> --delete, that might provide some clues about what triggers the error.
>
> Since the error only happens with --delete and appears to involve
> xattrs, you may be able to work around it by using two passes of rsync,
> the first with -X but not --delete and the second with --delete but not
> -X.

I eventually got it to error without the --delete option as well.  The ktrace
shows normal fetches followed by get errors like this:

  1051 rsync   CALL  getdirentries(0x3,0x1807600,0x1000,0x37941d4)
  1051 rsync   RET   getdirentries 64/0x40
  1051 rsync   CALL  lstat(0xbfffd85c,0xbfffe05c)
  1051 rsync   NAMI  "City-140-200244/140-200244-A-A to Z
Tra-693418A/44.Page.A-A to Z Trailer Sales"
  1051 rsync   RET   lstat 0
  1051 rsync   CALL  getattrlist(0xbfffe714,0xbfffd7b8,0x93848,0xc,0x1)
  1051 rsync   NAMI  "City-140-200244/140-200244-A-A to Z
Tra-693418A/44.Page.A-A to Z Trailer Sales"
  1051 rsync   RET   getattrlist 0
  1051 rsync   CALL  listxattr(0xbfffe714,0x1801800,0x400,0x1)
  1051 rsync   NAMI  "City-140-200244/140-200244-A-A to Z
Tra-693418A/44.Page.A-A to Z Trailer Sales"
  1051 rsync   RET   listxattr 44/0x2c
  1051 rsync   CALL  getxattr(0xbfffe714,0x1801800,0,0,0,0x1)
  1051 rsync   NAMI  "City-140-200244/140-200244-A-A to Z
Tra-693418A/44.Page.A-A to Z Trailer Sales"
  1051 rsync   RET   getxattr 3212/0xc8c
  1051 rsync   CALL  getxattr(0xbfffe714,0x1801800,0x1804200,0xc8c,0,0x1)
  1051 rsync   NAMI  "City-140-200244/140-200244-A-A to Z
Tra-693418A/44.Page.A-A to Z Trailer Sales"
  1051 rsync   RET   getxattr -1 errno 9 Bad file descriptor
  1051 rsync   CALL  write(0x2,0xbfffc3a0,0xb0)
  1051 rsync   GIO   fd 2 wrote 176 bytes
       "rsync: get_xattr_data: lgetxattr("City-140-200244/140-200244-A-A to Z
Tra-693418\
        A/44.Page.A-A to Z Trailer Sales","com.apple.ResourceFork",3212)
failed: Bad fil\


The inconsistency of the files it errors on lead me to believe AFP is to
blame.  It's never the same file twice.  It also looks like it fails to report
the existence of some files and directories because rsync will delete items
that are still in the source.  I think I'll file this under "B" for bad idea.

Thanks again for your help with this.

Mike


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Xattrs and Delete over AFP

by Matt McCutchen-7 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sun, 2008-06-22 at 17:28 -0400, zodiac-meow@... wrote:

> And it works on most files OK.  So far, so good.  It does
> gets this error on some files and directories:
>
> rsync: get_xattr_names: llistxattr(" New Art Dept/John:Projects",1024) failed:
> Invalid argument (22)
>
> I've narrowed this down to it being only file or directory names with a "/" in
> them (which OSX translates into a ":" in terminal).  This doesn't occur when
> using an rsync without the pre-Tiger patch.  This is still great news though!
> So close...
Good, so you have a solution that mostly works.  You could do other Mac
rsync users a favor by fixing the colon/slash issue.

> It also looks like it fails to report
> the existence of some files and directories because rsync will delete items
> that are still in the source.

I have a guess about how that is happening.  With -X, when rsync fails
to read the extended attributes of a source file (or directory), it
omits the file from the file list.  This behavior avoids removing
previously copied extended attributes from the destination file, but
with --delete, it might lead to the deletion of the destination file.
To avoid that, errors that result in the omission of files from the file
list are supposed to set the IOERR_GENERAL flag, which inhibits deletion
altogether, but this appears to have been overlooked in the case of an
extended attribute error.  An attempt at a patch is attached; Wayne,
please look into it.

Matt

[more-ioerr-general.diff]

diff --git a/flist.c b/flist.c
index b0b05dc..a99a2ee 100644
--- a/flist.c
+++ b/flist.c
@@ -1009,7 +1009,12 @@ static struct file_struct *recv_file_entry(struct file_list *flist,
  * and performing extensive checks against global options.
  *
  * Returns a pointer to the new file struct, or NULL if there was an error
- * or this file should be excluded. */
+ * or this file should be excluded.
+ *
+ * Note: Any error (here or in send_file_name) that results in the omission of
+ * an existent source file from the file list should set
+ * "io_error |= IOERR_GENERAL" to avoid deletion of the file from the
+ * destination if --delete is on. */
 struct file_struct *make_file(const char *fname, struct file_list *flist,
       STRUCT_STAT *stp, int flags, int filter_level)
 {
@@ -1027,6 +1032,7 @@ struct file_struct *make_file(const char *fname, struct file_list *flist,
  char *bp;
 
  if (strlcpy(thisname, fname, sizeof thisname) >= sizeof thisname) {
+ io_error |= IOERR_GENERAL;
  rprintf(FINFO, "skipping overly long name: %s\n", fname);
  return NULL;
  }
@@ -1336,15 +1342,19 @@ static struct file_struct *send_file_name(int f, struct file_list *flist,
  if (preserve_acls && !S_ISLNK(file->mode)) {
  sx.st.st_mode = file->mode;
  sx.acc_acl = sx.def_acl = NULL;
- if (get_acl(fname, &sx) < 0)
+ if (get_acl(fname, &sx) < 0) {
+ io_error |= IOERR_GENERAL;
  return NULL;
+ }
  }
 #endif
 #ifdef SUPPORT_XATTRS
  if (preserve_xattrs) {
  sx.xattr = NULL;
- if (get_xattr(fname, &sx) < 0)
+ if (get_xattr(fname, &sx) < 0) {
+ io_error |= IOERR_GENERAL;
  return NULL;
+ }
  }
 #endif
 



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (204 bytes) Download Attachment

Re: Xattrs and Delete over AFP

by Wayne Davison-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sun, Jun 22, 2008 at 06:09:02PM -0400, Matt McCutchen wrote:
> An attempt at a patch is attached; Wayne, please look into it.

Looks good to me.  I've applied it to the source.

..wayne..
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
LightInTheBox - Buy quality products at wholesale price