Feedback on Tombstones -04

[Mark Nottingham-2]

James,

Overall, I would very much like to see this progress -- either on the  
standards track or as informational -- ASAP, as I believe it to be  
useful, mature (with the nits below noted) and have immediate needs  
for it (albeit in internal projects, but we'd prefer not to create  
something proprietary, or to adopt still-not-stable technology).

1) "If the at:deleted-entry element does not contain a when element,  
the at: deleted-entry sharing the same atom:id as an entry MUST be  
ignored." This is too strict; if a feed has an explicit ordering  
(e.g., a marker giving semantics to lexical ordering), an  
implementation may determine which has precedence by that ordering.

2) "Atom Feed Documents MAY contain any number of at:deleted-entry  
elements, but MUST NOT contain more than one with the same ref  
attribute value." ... but later ... "An Atom feed MAY contain  
atom:entry elements and at:deleted-entry elements sharing the same  
atom:id value." The MUST NOT is too strict; it makes coincidence of  
deletions a serialisation problem (notice that it's specified in terms  
of feed documents, not feeds).

3) Are extensions elements and attributes allowed?

4) I'm still not entirely comfortable with the trash link relation;
  a) it seems to conflate the semantics of "this moved somewhere else"  
and where it moved to (i.e., the trash). There are other reasons why  
something might be moved to another feed, and
  b) the relationship with deleted-entry is still a bit fuzzy, from a  
user standpoint. Does the presence of a trash feed mean that a client  
should go to the effort of subscribing to it and removing entries that  
appear in it?

Why not, for example, allow deleted-entry to specify where an entry  
has moved to (allowing a default to be set as a link relation,  
perhaps), and let *that* feed define whether or not it's a trash bin?

5) AIUI User Experience folks say that in some cultures references to  
images of death are inappropriate; it may be good to find a more  
neutral name for the spec (I don't have a problem with it; just  
occurred to me).

Cheers,

--
Mark Nottingham     http://www.mnot.net/

[James M Snell]

Mark Nottingham wrote:
> James,
>
> Overall, I would very much like to see this progress -- either on the
> standards track or as informational -- ASAP, as I believe it to be
> useful, mature (with the nits below noted) and have immediate needs for
> it (albeit in internal projects, but we'd prefer not to create something
> proprietary, or to adopt still-not-stable technology).
>

Experimental would likely be a good choice for this.

> 1) "If the at:deleted-entry element does not contain a when element, the
> at: deleted-entry sharing the same atom:id as an entry MUST be ignored."
> This is too strict; if a feed has an explicit ordering (e.g., a marker
> giving semantics to lexical ordering), an implementation may determine
> which has precedence by that ordering.
>

Hmm.. I think it would make more sense to just require the when
attribute.  You could still apply the lexical ordering stuff but there
would be no ambiguity introduced in cases that don't use lexical ordering.

> 2) "Atom Feed Documents MAY contain any number of at:deleted-entry
> elements, but MUST NOT contain more than one with the same ref attribute
> value." ... but later ... "An Atom feed MAY contain atom:entry elements
> and at:deleted-entry elements sharing the same atom:id value." The MUST
> NOT is too strict; it makes coincidence of deletions a serialisation
> problem (notice that it's specified in terms of feed documents, not feeds).
>

Ok. I can change that to read "MUST NOT contain more than one with the
same combination of ref and when attribute values".

> 3) Are extensions elements and attributes allowed?
>

Yes. I will make this more explicit.

> 4) I'm still not entirely comfortable with the trash link relation;
>  a) it seems to conflate the semantics of "this moved somewhere else"
> and where it moved to (i.e., the trash). There are other reasons why
> something might be moved to another feed, and
>  b) the relationship with deleted-entry is still a bit fuzzy, from a
> user standpoint. Does the presence of a trash feed mean that a client
> should go to the effort of subscribing to it and removing entries that
> appear in it?
>
> Why not, for example, allow deleted-entry to specify where an entry has
> moved to (allowing a default to be set as a link relation, perhaps), and
> let *that* feed define whether or not it's a trash bin?
>

Hmmm... will have to give this some more thought.

> 5) AIUI User Experience folks say that in some cultures references to
> images of death are inappropriate; it may be good to find a more neutral
> name for the spec (I don't have a problem with it; just occurred to me).
>

Good point.  Will work on this.

- James

> Cheers,
>
> --
> Mark Nottingham     http://www.mnot.net/
>
>

[James M Snell]

Updated:
   http://www.ietf.org/internet-drafts/draft-snell-atompub-tombstones-05.txt

  * Renamed the spec
  * Removed the "trash" link relation. This can be handled separately
  * Made the when attribute required
  * Changed the "Atom processors MUST ignore any at:deleted-entry
    elements..." to "Atom processors SHOULD ignore any at:deleted-entry
    elements..."
  * Explicitly allow for extension elements and attributes
  * Changed to: "An Atom feed MAY contain any number of at:deleted-entry
    elements, but MUST NOT contain more than one with the same
    combination of ref and when attribute values."
  * Updated the example

- James

Mark Nottingham wrote:

> James,
>
> Overall, I would very much like to see this progress -- either on the
> standards track or as informational -- ASAP, as I believe it to be
> useful, mature (with the nits below noted) and have immediate needs for
> it (albeit in internal projects, but we'd prefer not to create something
> proprietary, or to adopt still-not-stable technology).
>
> 1) "If the at:deleted-entry element does not contain a when element, the
> at: deleted-entry sharing the same atom:id as an entry MUST be ignored."
> This is too strict; if a feed has an explicit ordering (e.g., a marker
> giving semantics to lexical ordering), an implementation may determine
> which has precedence by that ordering.
>
> 2) "Atom Feed Documents MAY contain any number of at:deleted-entry
> elements, but MUST NOT contain more than one with the same ref attribute
> value." ... but later ... "An Atom feed MAY contain atom:entry elements
> and at:deleted-entry elements sharing the same atom:id value." The MUST
> NOT is too strict; it makes coincidence of deletions a serialisation
> problem (notice that it's specified in terms of feed documents, not feeds).
>
> 3) Are extensions elements and attributes allowed?
>
> 4) I'm still not entirely comfortable with the trash link relation;
>  a) it seems to conflate the semantics of "this moved somewhere else"
> and where it moved to (i.e., the trash). There are other reasons why
> something might be moved to another feed, and
>  b) the relationship with deleted-entry is still a bit fuzzy, from a
> user standpoint. Does the presence of a trash feed mean that a client
> should go to the effort of subscribing to it and removing entries that
> appear in it?
>
> Why not, for example, allow deleted-entry to specify where an entry has
> moved to (allowing a default to be set as a link relation, perhaps), and
> let *that* feed define whether or not it's a trash bin?
>
> 5) AIUI User Experience folks say that in some cultures references to
> images of death are inappropriate; it may be good to find a more neutral
> name for the spec (I don't have a problem with it; just occurred to me).
>
> Cheers,
>
> --
> Mark Nottingham     http://www.mnot.net/
>
>

[Brian Smith-19]

James M Snell wrote:
>   * Renamed the spec

I think you renamed the spec to avoid the "tombstone" reference.
However, the namespace URI still has "tombstone" in it; it should be
changed as well.

>   * Removed the "trash" link relation. This can be handled separately
>   * Made the when attribute required
>   * Explicitly allow for extension elements and attributes
>   * Updated the example

These changes are all nice improvements.

>   * Changed the "Atom processors MUST ignore any at:deleted-entry
>     elements..." to "Atom processors SHOULD ignore any
>     at:deleted-entry elements..."

Instead of saying what processors should do, I think it is better to
just specify what at:deleted-by means, and let processors do whatever
they want with that information. In other words, this statement should
just be removed. This requirement doesn't fit well with AtomPub because
it is specified in terms of atom:updated; an AtomPub client will want to
compare @when to app:edited, not atom:updated.

>   * Changed to: "An Atom feed MAY contain any number of
>     at:deleted-entry elements, but MUST NOT contain more than
>     one with the same combination of ref and when attribute values."

This is an unnecessary restriction that doesn't add any value, and it
could be removed without any ill effect. It would be silly to generate
such duplicates but doing so is benign.

Like I mentioned in previous discussions, at:by and at:comment should be
removed. If it is important to keep track of that information for
deletions, then it would also be just as important to keep track of that
information for creation and editions of entries. In other words, those
things would be better specified by a separate specification that
described their usage within both at:deleted-entry AND atom:entry.

I recommend replacing the definition of the "ref" attribute with the one
from RFC 4685:

   The "ref" attribute specifies the persistent, universally unique
   identifier of the [entry that was removed].  The value MUST
   conform to the same construction and comparison rules as the value of
   the atom:id element, as defined in Section 4.2.6 of [RFC4287].
   Though the IRI might use a dereferenceable scheme, processors MUST
   NOT assume that it can be dereferenced.

It is misleading to call the when attribute an atomDateConstruct in the
RelaxNG schema, because Atom date constructs are elements, not
attributes. Similarly, I'd prefer to see @when be defined as an Atom
date construct (like atom:updated), because then we can more easily use
date-processing code that has already been written for atom:updated and
app:edited, and reuse the same Atom<->JSON mapping rule for it.

Previously, we discussed at length whether at:deleted-entry elements
should be intermixed with atom:entry elements, or before all the
atom:entry elements, and how they should be ordered. I recommend adding
clarification about these issues: "at:deleted-entry elements MAY be
appear anywhere within an atom:feed element; they may appear before,
before, between, and/or after atom:entry elements. The order of
at:deleted-entry elements is not significant."

Regards,
Brian

[James M Snell]

Brian Smith wrote:
> James M Snell wrote:
>>   * Renamed the spec
>
> I think you renamed the spec to avoid the "tombstone" reference.
> However, the namespace URI still has "tombstone" in it; it should be
> changed as well.
>

Yes, I'm planning to change that in the next rev. I wanted to make sure
the new spec title was ok first.

>>   * Removed the "trash" link relation. This can be handled separately
>>   * Made the when attribute required
>>   * Explicitly allow for extension elements and attributes
>>   * Updated the example
>
> These changes are all nice improvements.
>
>>   * Changed the "Atom processors MUST ignore any at:deleted-entry
>>     elements..." to "Atom processors SHOULD ignore any
>>     at:deleted-entry elements..."
>
> Instead of saying what processors should do, I think it is better to
> just specify what at:deleted-by means, and let processors do whatever
> they want with that information. In other words, this statement should
> just be removed. This requirement doesn't fit well with AtomPub because
> it is specified in terms of atom:updated; an AtomPub client will want to
> compare @when to app:edited, not atom:updated.
>

Yeah, I considered this and am definitely open to it.  The main
challenge is that if an entry with a newer atom:updated or app:edited
appears in a feed with an at:deleted-entry specifying the same atom:id
but a later when attribute value could still be viewed as having been
deleted if a clear precedence is not established.  The correct
interpretation is that the entry had been deleted but that it was
brought back.  I'm not certain if we can accomplish that by just
specifying the definition of at:deleted-entry.

>>   * Changed to: "An Atom feed MAY contain any number of
>>     at:deleted-entry elements, but MUST NOT contain more than
>>     one with the same combination of ref and when attribute values."
>
> This is an unnecessary restriction that doesn't add any value, and it
> could be removed without any ill effect. It would be silly to generate
> such duplicates but doing so is benign.
>

That shouldn't be a problem.  Whether this restriction is there or not
implementors will still need to write code to detect duplicates so
there's no impact to removing it.

> Like I mentioned in previous discussions, at:by and at:comment should be
> removed. If it is important to keep track of that information for
> deletions, then it would also be just as important to keep track of that
> information for creation and editions of entries. In other words, those
> things would be better specified by a separate specification that
> described their usage within both at:deleted-entry AND atom:entry.
>

I would have no problem with adding a kind of generic edit:comment
element for deletes and updates.  Such notes are frequently found in
systems such as wikis and would generally be quite useful.  Perhaps this
specification could define them in the generic way you suggest?

> I recommend replacing the definition of the "ref" attribute with the one
> from RFC 4685:
>
>    The "ref" attribute specifies the persistent, universally unique
>    identifier of the [entry that was removed].  The value MUST
>    conform to the same construction and comparison rules as the value of
>    the atom:id element, as defined in Section 4.2.6 of [RFC4287].
>    Though the IRI might use a dereferenceable scheme, processors MUST
>    NOT assume that it can be dereferenced.
>

+1.  That was actually on my list of todos but I missed it in this rev.

> It is misleading to call the when attribute an atomDateConstruct in the
> RelaxNG schema, because Atom date constructs are elements, not
> attributes. Similarly, I'd prefer to see @when be defined as an Atom
> date construct (like atom:updated), because then we can more easily use
> date-processing code that has already been written for atom:updated and
> app:edited, and reuse the same Atom<->JSON mapping rule for it.
>

Generally speaking, I agree.  The rationale for using an attribute here
is to keep the serialization as compact as possible, which I think is
important (tho others may not).

> Previously, we discussed at length whether at:deleted-entry elements
> should be intermixed with atom:entry elements, or before all the
> atom:entry elements, and how they should be ordered. I recommend adding
> clarification about these issues: "at:deleted-entry elements MAY be
> appear anywhere within an atom:feed element; they may appear before,
> before, between, and/or after atom:entry elements. The order of
> at:deleted-entry elements is not significant."
>

RFC4287 requires that a feed's metadata elements come before the listing
of entry elements. See section 4.1.1:

    The "atom:feed" element is the document (i.e., top-level) element of
    an Atom Feed Document, acting as a container for metadata and data
    associated with the feed.  Its element children consist of metadata
    elements followed by zero or more atom:entry child elements.

Therefore, the listing of at:deleted-entry elements MUST come before the
listing of atom:entry elements.  The ordering the at:deleted-entry
elements is not significant.

- James

> Regards,
> Brian
>
>

[Brian Smith-19]

James M Snell wrote:

> Brian Smith wrote:
> > Instead of saying what processors should do, I think it
> > is better to just specify what at:deleted-by means, and
> > let processors do whatever they want with that
> > information. In other words, this statement should
> > just be removed. This requirement doesn't fit well with
> > AtomPub because it is specified in terms of atom:updated;
> > an AtomPub client will want to compare @when to
> > app:edited, not atom:updated.
>
> Yeah, I considered this and am definitely open to it.
> The main  challenge is that if an entry with a newer
> atom:updated or app:edited appears in a feed with an
> at:deleted-entry specifying the same atom:id
> but a later when attribute value could still be
> viewed as having been deleted if a clear precedence
> is not established. The correct interpretation is
> that the entry had been deleted but that it was
> brought back.  I'm not certain if we can accomplish
> that by just specifying the definition of at:deleted-entry.

The case I care about is where I have deleted an entry from an AtomPub
collection and then un-delete it, so atom:updated shouldn't be updated,
but app:edited should be updated (atom:updated < deleted-entry/@when <
app:edited).

Why not have at:deleted-entry contain app:edited and/or atom:updated
instead of @when? Then say that timestamps should be compared using the
same date construct (compare only app:edited to app:edited, and
atom:updated to atom:updated). AtomPub collections would use app:edited
and most other collections would use atom:updated, just like they do
with atom:entry already today.

> > Like I mentioned in previous discussions, at:by and
> > at:comment should be removed.

> I would have no problem with adding a kind of generic edit:comment
> element for deletes and updates.  Such notes are frequently found in
> systems such as wikis and would generally be quite useful.  
> Perhaps this specification could define them in the generic way
> you suggest?

I think there is more chance of succeeding if the specification is kept
as narrowly-focused as possible.

> Therefore, the listing of at:deleted-entry elements MUST come
> before the listing of atom:entry elements.
> The ordering the at:deleted-entry elements is not significant.

That is fine with me. But, how about wrapping the at:deleted-entry
elements in a wrapper element:

<at:deleted-entries>
   <at:deleted-entry .../>
   <at:deleted-entry .../>
   <at:deleted-entry .../>
</at:deleted-entries>

This would allow a feed to indicate that it supports this feature, even
when no entries have been deleted:

    <at:deleted-entries/>

Knowing that the collection provides these tombstones would all clients
to skip crawling the collection to look for deleted entries.

Regards,
Brian

[James M Snell]

Brian Smith wrote:
> [snip]
> Why not have at:deleted-entry contain app:edited and/or atom:updated
> instead of @when? Then say that timestamps should be compared using the
> same date construct (compare only app:edited to app:edited, and
> atom:updated to atom:updated). AtomPub collections would use app:edited
> and most other collections would use atom:updated, just like they do
> with atom:entry already today.
>

That would be entirely possible.  Will stew on that a bit.

>>> Like I mentioned in previous discussions, at:by and
>>> at:comment should be removed.
>
>> I would have no problem with adding a kind of generic edit:comment
>> element for deletes and updates.  Such notes are frequently found in
>> systems such as wikis and would generally be quite useful.  
>> Perhaps this specification could define them in the generic way
>> you suggest?
>
> I think there is more chance of succeeding if the specification is kept
> as narrowly-focused as possible.
>

Agreed, which is why I originally was focusing on one specific, and
narrow issue.  :-)

>> Therefore, the listing of at:deleted-entry elements MUST come
>> before the listing of atom:entry elements.
>> The ordering the at:deleted-entry elements is not significant.
>
> That is fine with me. But, how about wrapping the at:deleted-entry
> elements in a wrapper element:
>
> <at:deleted-entries>
>    <at:deleted-entry .../>
>    <at:deleted-entry .../>
>    <at:deleted-entry .../>
> </at:deleted-entries>
>
> This would allow a feed to indicate that it supports this feature, even
> when no entries have been deleted:
>

I don't really see the value of this.  Either there are at:deleted-entry
elements or not.  If there aren't any, it really doesn't matter if the
feed supports the feature.

- James

>     <at:deleted-entries/>
>
> Knowing that the collection provides these tombstones would all clients
> to skip crawling the collection to look for deleted entries.
>
> Regards,
> Brian
>
>

[Eric Scheid]

On 23/4/08 3:46 AM, "Brian Smith" <brian@...> wrote:

> Why not have at:deleted-entry contain app:edited and/or atom:updated
> instead of @when? Then say that timestamps should be compared using the
> same date construct (compare only app:edited to app:edited, and
> atom:updated to atom:updated). AtomPub collections would use app:edited
> and most other collections would use atom:updated, just like they do
> with atom:entry already today.

+1 : elegant, simple

> This would allow a feed to indicate that it supports this feature, even
> when no entries have been deleted:
>
>   <at:deleted-entries/>
>
> Knowing that the collection provides these tombstones would all clients
> to skip crawling the collection to look for deleted entries.

+1 also

e.

[Mark Stahl]

One issue that we've encountered with tombtones is that most services don't keep them forever.     (Otherwise, the list of tombstones only ever grows.)  

This leads to problems with clients that periodically poll for changes - they don't know if the set of tombstones is complete since the last time they polled. 

I'd suggest adding an element at:deleted-since, so the server can let the client know the tomebstone window in effect.  Building on the proposal for at;deleted-entries, I'd suggest something like this: 

<at:deleted-entries>
  <at:deleted-since>2003-12-13T18:30:02.25Z</at:deleted-since>
  <at:deleted-entry .../>
  <at:deleted-entry .../>
  <at:deleted-entry .../>
</at:deleted-entries>

Another useful piece of information to the client trying to sync is a hint as to the typical tomebstone lifetime.  But I'm not sure if it belongs in this spec. (Perhaps it's more appropriate in a service document?)

-- mark

On Tue, Apr 22, 2008 at 5:15 PM, Eric Scheid <eric.scheid@...> wrote:

On 23/4/08 3:46 AM, "Brian Smith" <brian@...> wrote:

> Why not have at:deleted-entry contain app:edited and/or atom:updated
> instead of @when? Then say that timestamps should be compared using the
> same date construct (compare only app:edited to app:edited, and
> atom:updated to atom:updated). AtomPub collections would use app:edited
> and most other collections would use atom:updated, just like they do
> with atom:entry already today.

+1 : elegant, simple

> This would allow a feed to indicate that it supports this feature, even
> when no entries have been deleted:
>
>   <at:deleted-entries/>
>
> Knowing that the collection provides these tombstones would all clients
> to skip crawling the collection to look for deleted entries.

+1 also

e.

--
Test driving http://five.sentenc.es/

[Brian Smith-19]

Mark Stahl wrote:
> One issue that we've encountered with tombtones is that most services
> don't
> keep them forever.     (Otherwise, the list of tombstones only ever
> grows.)

For most systems, keeping track of deleted items on the back end is not a
problem. The only problem is that you don't want the feeds to be bloated
with an ever-increasing number of tombstones, right?

> This leads to problems with clients that periodically poll for changes -
> they don't know if the set of tombstones is complete since the last time
> they polled.

> I'd suggest adding an element at:deleted-since, so the server can let the
> client know the tomebstone window in effect.

To solve this problem, I implemented a simple feed paging extension. The
first page of the collection feed (at the collection URI) contains a
"feed-changes" link. The feed-changes link points to a feed that is only
guaranteed to contain all the information (e.g. entries and tombstones) that
was changed since the first page of the collection feed was fetched. The
clients can use this to answer the question "what are all the changes that
occurred since the last time I synced with the server?"

The first page of the "feed-changes" feed contains another "feed-changes"
link that works in an analogous fashion. Clients can keep in sync with the
collection by chasing the "feed-changes" links. The "feed-changes" feeds can
also use RFC 5005 paging; to get a complete sync, one has to page through
all the pages (using "next") before chasing the "feed-changes" link from the
first page.

My suggestion is to finish up the at:deleted-entry mechanism that is as
simple as possible, and then define a separate AtomPub synchronization
protocol on top of it, based on the "feed-changes" mechanism I just
described, where the at:deleted-entry mechanism would be a required element
of the "feed-changes" feeds. What do you think?

> Another useful piece of information to the client trying to sync is a hint
> as to the typical tomebstone lifetime.  But I'm not sure if it belongs in
> this spec. (Perhaps it's more appropriate in a service document?)

It is a good idea to keep this information in the feed, because this is
useful in Atom feeds that are not AtomPub feeds.

Regards,
Brian

[James Holderness]

Mark Stahl wrote:
> This leads to problems with clients that periodically poll for
> changes - they don't know if the set of tombstones is complete
> since the last time they polled.
>
> I'd suggest adding an element at:deleted-since, so the server
> can let the client know the tomebstone window in effect.

Assuming you did know whether the set of tombstones was complete or not, how
would this effect your behaviour as a client? In other words, what would you
do differently if you knew the set of tombstones was incomplete?

Regards
James

[Mark Nottingham-2]

In a paged feed, I just put the tombstones on the page that's  
appropriate; e.g., if the entries are sorted by date (and most are),  
the tombstone goes on the relevant page (i.e., using the time of  
deletion).


On 08/05/2008, at 2:19 AM, Mark Stahl wrote:

> One issue that we've encountered with tombtones is that most  
> services don't keep them forever.     (Otherwise, the list of  
> tombstones only ever grows.)
>
> This leads to problems with clients that periodically poll for  
> changes - they don't know if the set of tombstones is complete since  
> the last time they polled.
>
> I'd suggest adding an element at:deleted-since, so the server can  
> let the client know the tomebstone window in effect.  Building on  
> the proposal for at;deleted-entries, I'd suggest something like this:
>
> <at:deleted-entries>
>   <at:deleted-since>2003-12-13T18:30:02.25Z</at:deleted-since>
>   <at:deleted-entry .../>
>   <at:deleted-entry .../>
>   <at:deleted-entry .../>
> </at:deleted-entries>
>
> Another useful piece of information to the client trying to sync is  
> a hint as to the typical tomebstone lifetime.  But I'm not sure if  
> it belongs in this spec. (Perhaps it's more appropriate in a service  
> document?)
>
> -- mark
>
>
> On Tue, Apr 22, 2008 at 5:15 PM, Eric Scheid <eric.scheid@...
> > wrote:
>
> On 23/4/08 3:46 AM, "Brian Smith" <brian@...> wrote:
>
> > Why not have at:deleted-entry contain app:edited and/or atom:updated
> > instead of @when? Then say that timestamps should be compared  
> using the
> > same date construct (compare only app:edited to app:edited, and
> > atom:updated to atom:updated). AtomPub collections would use  
> app:edited
> > and most other collections would use atom:updated, just like they do
> > with atom:entry already today.
>
> +1 : elegant, simple
>
> > This would allow a feed to indicate that it supports this feature,  
> even
> > when no entries have been deleted:
> >
> >   <at:deleted-entries/>
> >
> > Knowing that the collection provides these tombstones would all  
> clients
> > to skip crawling the collection to look for deleted entries.
>
> +1 also
>
> e.
>
>
>
>
> --
> Test driving http://five.sentenc.es/

--
Mark Nottingham     http://www.mnot.net/

[Mark Stahl]

On Wed, May 7, 2008 at 2:19 PM, Brian Smith <brian@...> wrote:

Mark Stahl wrote:

One issue that we've encountered with tombtones is that most services don't
keep them forever.     (Otherwise, the list of tombstones only ever grows.)

For most systems, keeping track of deleted items on the back end is not a problem. The only problem is that you don't want the feeds to be bloated with an ever-increasing number of tombstones, right?

Typically, the limitation on the number of tombstones reflects a limitation in the amount of storage available in a backend service.  If tombstone lifetimes are infinite, the storage required for an atom pub collection can grow to an ever increasing size, even if the number of elements allowed in that collection is bounded.   Most services I talk about doing AtomPub purge tombstones after some time (for example, 30 days.)   So I'm faced with the problem today.  Advertising the tombstone window would solve a lot of problems. 

This leads to problems with clients that periodically poll for changes -
they don't know if the set of tombstones is complete since the last time
they polled.

I'd suggest adding an element at:deleted-since, so the server can let the
client know the tomebstone window in effect.

To solve this problem, I implemented a simple feed paging extension. The first page of the collection feed (at the collection URI) contains a "feed-changes" link. The feed-changes link points to a feed that is only guaranteed to contain all the information (e.g. entries and tombstones) that was changed since the first page of the collection feed was fetched. The clients can use this to answer the question "what are all the changes that occurred since the last time I synced with the server?"

The first page of the "feed-changes" feed contains another "feed-changes" link that works in an analogous fashion. Clients can keep in sync with the collection by chasing the "feed-changes" links. The "feed-changes" feeds can also use RFC 5005 paging; to get a complete sync, one has to page through all the pages (using "next") before chasing the "feed-changes" link from the first page.

This addresses the large feed document issue, but is still premised on the idea that you store all tombstones indefinitely, so it doesn't solve the problem I'm attempting to solve. 

My suggestion is to finish up the at:deleted-entry mechanism that is as simple as possible, and then define a separate AtomPub synchronization protocol on top of it, based on the "feed-changes" mechanism I just described, where the at:deleted-entry mechanism would be a required element of the "feed-changes" feeds. What do you think?

I'm willing to separate the problem of a sync spec from tombstones, so I guess it depends on whether you consider it important that the tombstone spec have the capability to indicate that the set of tombstones provided may not be complete.  I tend towards thinking that it's important enough to be included in the main spec, but YMMV, and I'd be happy to have at:deleted-entry today so we can build on it. 

Another useful piece of information to the client trying to sync is a hint
as to the typical tomebstone lifetime.  But I'm not sure if it belongs in
this spec. (Perhaps it's more appropriate in a service document?)

It is a good idea to keep this information in the feed, because this is useful in Atom feeds that are not AtomPub feeds.

The reason I hesitate to mention this is it seems to be heading more and more towards a sync spec.  And I'm not trying to solve the whole sync problem with this spec, but create tools that will be used to solve sync. 

Regards,
Brian

--
Test driving http://five.sentenc.es/

[Mark Stahl]

On Wed, May 7, 2008 at 2:40 PM, James Holderness <j4_james@...> wrote:

Mark Stahl wrote:

This leads to problems with clients that periodically poll for
changes - they don't know if the set of tombstones is complete
since the last time they polled.

I'd suggest adding an element at:deleted-since, so the server
can let the client know the tomebstone window in effect.

Assuming you did know whether the set of tombstones was complete or not, how would this effect your behaviour as a client? In other words, what would you do differently if you knew the set of tombstones was incomplete?

James,

The issue is sync, and the diffrence is between incremental and full sync. 

Assume a client has previously downloaded a complete copy of a collection at time T1.  Some time later, T2, the client retrieves the updated feed.  Assume entries and tombstones are both in app:edited order, as suggested by Mark Nottingham.  If the client knows that the server is still holding tombstones of all entries deleted since T1, then it can correctly update it's local copy of the collection to be in sync with the server. Otherwise, it must retrieve the full collection and manually calculate the missing deletes in order to sync. 

-- m

--
Test driving http://five.sentenc.es/

[Mark Stahl]

MarkN,

The idea of interleving entries and deleted-entries in edited order sounds like a good pattern.  It's particularly useful  for clients attempting to retrieve the most recent changes.  Combining this idea with the idea of a tombstone "lifetime", then, one could construct a feed that looks something like this:

<feed>
   <entry> ... updated at T ... </entry>
   <at:deleted-entry> ... deleted at T-1... </at:deleted-entry>
   <at:deleted-entry> ... deleted at T-2... </at:deleted-entry>
   <entry> ... updated at T-3 ... </entry>
   <at:deleted-since>T-4</at:deleted-since>
   <entry> ... updated at T-5 ... </entry>
   <entry> ... updated at T-6 ... </entry>
   ...
</feed>

I would not recommend this order be required.  But it would be very useful for clients attempting to learn of changes to the feed since the last time they accessed it.

-- m

On Wed, May 7, 2008 at 8:45 PM, Mark Nottingham <mnot@...> wrote:

In a paged feed, I just put the tombstones on the page that's appropriate; e.g., if the entries are sorted by date (and most are), the tombstone goes on the relevant page (i.e., using the time of deletion).

On 08/05/2008, at 2:19 AM, Mark Stahl wrote:

One issue that we've encountered with tombtones is that most services don't keep them forever.     (Otherwise, the list of tombstones only ever grows.)

This leads to problems with clients that periodically poll for changes - they don't know if the set of tombstones is complete since the last time they polled.

I'd suggest adding an element at:deleted-since, so the server can let the client know the tomebstone window in effect.  Building on the proposal for at;deleted-entries, I'd suggest something like this:

<at:deleted-entries>
 <at:deleted-since>2003-12-13T18:30:02.25Z</at:deleted-since>
 <at:deleted-entry .../>
 <at:deleted-entry .../>
 <at:deleted-entry .../>
</at:deleted-entries>

Another useful piece of information to the client trying to sync is a hint as to the typical tomebstone lifetime.  But I'm not sure if it belongs in this spec. (Perhaps it's more appropriate in a service document?)

-- mark


On Tue, Apr 22, 2008 at 5:15 PM, Eric Scheid <eric.scheid@...> wrote:

On 23/4/08 3:46 AM, "Brian Smith" <brian@...> wrote:

> Why not have at:deleted-entry contain app:edited and/or atom:updated
> instead of @when? Then say that timestamps should be compared using the
> same date construct (compare only app:edited to app:edited, and
> atom:updated to atom:updated). AtomPub collections would use app:edited
> and most other collections would use atom:updated, just like they do
> with atom:entry already today.

+1 : elegant, simple

> This would allow a feed to indicate that it supports this feature, even
> when no entries have been deleted:
>
>   <at:deleted-entries/>
>
> Knowing that the collection provides these tombstones would all clients
> to skip crawling the collection to look for deleted entries.

+1 also

e.




--
Test driving