Achieving changed-only http polling

[nick_b]

I'm using PollingHttpMessageReceiver to obtain web content, but unfortunately it retrieves content regardless of if it has changed.  Do I have to roll my own AbstractPollingMessageReceiver implementation if I only want to only route content as it is changed?

I was thinking there are actually 2-3 ways to do this;

1) In the connector using last-modified / if-modified-since, as a browsers do and keep last poll time in a class-level static.

2) Filter in a custom bridge-like UMO that would look at http header or checksum content to inspect for changes.

3) More universal approach like Mule filters or routers to act as a generic message-changed gate.

#1/2 seem awfully bespoke for the requirement.  #3 looks attractive and something one may expect to see in the Mule docs, but maybe not technically feasible.  Any thoughts?

If #1, would a patch to PollingHttpMessageReceiver be useful to anyone?  If so, it would probably involve adding pollChangesOnly property to the HTTP endpoint, like the pollingFrequency property already there.

Ta, Nick

--------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email

[Andrew Perepelytsya]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Nick,

I think patching the http receiver is an easy and good-enough solution. One note, however. For consistency with other polling receivers you may want to use something like 'contentAge' property, which would route a message only if 'now-lastModified is > contentAge'.

Depending on your requirements, you may also consider implementing an HTTP HEAD request to get headers only from the server without an actual content.

Andrew

On 8/19/07, Nick Bower <nick.bower@...> wrote:

I'm using PollingHttpMessageReceiver to obtain web content, but unfortunately it retrieves content regardless of if it has changed.  Do I have to roll my own AbstractPollingMessageReceiver implementation if I only want to only route content as it is changed?

I was thinking there are actually 2-3 ways to do this;

1) In the connector using last-modified / if-modified-since, as a browsers do and keep last poll time in a class-level static.

2) Filter in a custom bridge-like UMO that would look at http header or checksum content to inspect for changes.

3) More universal approach like Mule filters or routers to act as a generic message-changed gate.

#1/2 seem awfully bespoke for the requirement.  #3 looks attractive and something one may expect to see in the Mule docs, but maybe not technically feasible.  Any thoughts?

If #1, would a patch to PollingHttpMessageReceiver be useful to anyone?  If so, it would probably involve adding pollChangesOnly property to the HTTP endpoint, like the pollingFrequency property already there.

Ta, Nick

--------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email

[nick_b]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Could find the other polling receivers, but followed your advice anyway.  Patch is here;

http://mule.mulesource.org/jira/browse/MULE-2172

This will give the ability to frequently monitor large resources for changes without actually downloading them using conditional GETs.


Andrew Perepelytsya wrote:

Nick,

I think patching the http receiver is an easy and good-enough solution. One note, however. For consistency with other polling receivers you may want to use something like 'contentAge' property, which would route a message only if 'now-lastModified is > contentAge'.

Depending on your requirements, you may also consider implementing an HTTP HEAD request to get headers only from the server without an actual content.

Andrew

On 8/19/07, Nick Bower <nick.bower@...> wrote:

I'm using PollingHttpMessageReceiver to obtain web content, but unfortunately it retrieves content regardless of if it has changed.  Do I have to roll my own AbstractPollingMessageReceiver implementation if I only want to only route content as it is changed?

I was thinking there are actually 2-3 ways to do this;

1) In the connector using last-modified / if-modified-since, as a browsers do and keep last poll time in a class-level static.

2) Filter in a custom bridge-like UMO that would look at http header or checksum content to inspect for changes.

3) More universal approach like Mule filters or routers to act as a generic message-changed gate.

#1/2 seem awfully bespoke for the requirement.  #3 looks attractive and something one may expect to see in the Mule docs, but maybe not technically feasible.  Any thoughts?

If #1, would a patch to PollingHttpMessageReceiver be useful to anyone?  If so, it would probably involve adding pollChangesOnly property to the HTTP endpoint, like the pollingFrequency property already there.

Ta, Nick

--------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email