Predefined/Saved queries

[Sergey Chernyshev-3]

I'm starting to use Ask queries for RSS and iCal support on the site and one of the issues I'm having is huge URLs that get generated when queries are significantly large.

These URLs can't really be imported into RSS readers or iCal tools (Outlook, Google Calendar) which defeats the purpose. My current workaround is to run cron jobs to dump them into the files, but that's not really good solution as I can't use variables in these queries, for example.

One of the potential solutions I see is to save queries as articles in some (Ask) namespace and use page names as shortcuts for these queries.

Markus, Denny, I'll appreciate your thoughts on how hard will it be to add this query article reading functionality it to #ask parser function and Special:Ask code - I just want to understand what it might take for me or other developer to implement this. Also, I think it's worth discussing which aspects of the query should be saved and which should still be specified when this query is used, e.g. which properties to display, how to format (RSS, iCal) and etc.

I can definitely give various examples based on TechPresentations experience, like topic RSS feeds, event series iCal feeds and so on.

Will appreciate any feedback on this.

Thank you,

         Sergey


--
Sergey Chernyshev
http://www.sergeychernyshev.com/
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@...
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel

[Markus Krötzsch-2]

On Dienstag, 10. Juni 2008, Sergey Chernyshev wrote:

> I'm starting to use Ask queries for RSS and iCal support on the site and
> one of the issues I'm having is huge URLs that get generated when queries
> are significantly large.
>
> These URLs can't really be imported into RSS readers or iCal tools
> (Outlook, Google Calendar) which defeats the purpose. My current workaround
> is to run cron jobs to dump them into the files, but that's not really good
> solution as I can't use variables in these queries, for example.
>
> One of the potential solutions I see is to save queries as articles in some
> (Ask) namespace and use page names as shortcuts for these queries.
>
> Markus, Denny, I'll appreciate your thoughts on how hard will it be to add
> this query article reading functionality it to #ask parser function and
> Special:Ask code - I just want to understand what it might take for me or
> other developer to implement this. Also, I think it's worth discussing
> which aspects of the query should be saved and which should still be
> specified when this query is used, e.g. which properties to display, how to
> format (RSS, iCal) and etc.

I generally like the idea. It would also open new ways for caching queries or
for having some form of access control for query creation (maybe some wikis
do not want anyone to make large queries). But I do not see all the details
yet.

I would like saved queries to specify only the query as such, without
specifics regarding format or printouts. At least that would make it easier
for us to cache results of saved queries (since the cache would always have
the same format, storing only a list of results). It would then be easy to
reuse a saved query in #ask by just replacing the saved query string by
something like [[Query:Name of query]] (so saved queries would work
like "dynamic categories"). One could even combine saved queries with
additional query parameters in this way! For example:

{{#ask: [[Query:French cities]] [[Population::>1000]] }}

where [[Query:French cities]] is the query "[[Category:City]] [[located
in::France]]". This would give you some ways of refining a saved query in
different ways, and one could still benefit from caching (since we would work
directly with the French cities insternally, and not look at all cities
again).

Caching would of course be somewhat independent from saving queries: one could
save a query and just have it as a syntactic shortcut, or one could have it
pre-computed (maybe offline, maybe using some useful update strategy during
editing). The latter case allows things to get slightly out-of-date, but
would provide very stable performance even on huge wikis.


So let us think about how to do that:

* We need a new namespace of some name (seeing the potential use of queries in
#ask, I suggest not to use "Ask:" as a namespace). Otherwise this is a piece
of cake.

* We need to define how users would enter queries that they would like to be
saved. I think each query-page would hold one query and arbitrary bits of
documentation. To enter the query, users would use the syntax as in #ask.
There are some options for the details:
** We use a special property such as [[query:: [[Category:City]] ... ]].
** We use a new parser hook such as {{#query:  [[Category:City]] ... }}.
** We overload #ask for pages in that namespace, i.e.
{{#ask:  [[Category:City]] ... }} would work differently on those pages (it
could still show the results, but it would also be required to be unique, and
its query part would be saved).


* We (or at least I) need to consider how to store queries in the database. I
can treat them similar to property values that can be very long (Type:Text).
As long as there is no caching, the query string would thus be processed in a
purely syntactic manner: you use it in a query, and SMW replaces it with its
real query string.

* I can use the page id of the query page to identify the queries internally,
e.g. to store the information about them being cached (including location in
DB, date of last refresh), or for certain statistic data (result count, used
in query optimisation).


I think that is almost an implementation plan. Let us decide quickly on the
open questions, and I shall have that done (at least without the caching)
before the next release.

Best regards,

Markus

>
> I can definitely give various examples based on TechPresentations
> experience, like topic RSS feeds, event series iCal feeds and so on.
>
> Will appreciate any feedback on this.
>
> Thank you,
>
>          Sergey



--
Markus Krötzsch
Semantic MediaWiki    http://semantic-mediawiki.org
http://korrekt.org    markus@...


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@...
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel

[Jeff Thompson-5]

Markus Krötzsch wrote:
 > For example:
 >
 > {{#ask: [[Query:French cities]] [[Population::>1000]] }}
 >
 > where [[Query:French cities]] is the query "[[Category:City]] [[located
 > in::France]]".

In a previous very illuminating message, you said "It is very easy to map
an #ask query to an OWL class description (the query then is the task of
finding all elements of the class description)."  So, in the same
way that [[Category:City]] matches all pages which assert [[Category:City]],
each page has an implicit assertion [[elementOf::French cities]], where
"French cities" is a query page, but this page is really a page about
a class description.

 > * We need to define how users would enter queries that they would like to be
 > saved. I think each query-page would hold one query and arbitrary bits of
 > documentation. To enter the query, users would use the syntax as in #ask.
 > There are some options for the details:
 > ** We use a special property such as [[query:: [[Category:City]] ... ]].
 > ** We use a new parser hook such as {{#query:  [[Category:City]] ... }}.
 > ** We overload #ask for pages in that namespace, i.e.
 > {{#ask:  [[Category:City]] ... }} would work differently on those pages (it
 > could still show the results, but it would also be required to be unique, and
 > its query part would be saved).

I much prefer [[query:: [[Category:City]] ... ]].  I would like to be able
treat a query (class description) like other pages, and ask SWM questions
about the queries in the wiki.  So treating "query" like any other property is good.

Also, eventually SMW could support alternative properties than "query" to
describe the class (of query results).  For example, a property that allows
an alternate syntax, or some XML from an OWL class description.

 > * We (or at least I) need to consider how to store queries in the database. I
 > can treat them similar to property values that can be very long (Type:Text).
 > As long as there is no caching, the query string would thus be processed in a
 > purely syntactic manner: you use it in a query, and SMW replaces it with its
 > real query string.

Wouldn't you store the list of pages in the same way that you store them for
a category?  (Isn't the list of pages in a category cached?)

- Jeff




-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@...
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel

[Sergey Chernyshev-3]

Sorry for late response - I've been busy with Linked Data Planet conference and other stuff. Please see my comments below.

On Wed, Jun 11, 2008 at 4:47 AM, Markus Krötzsch <markus@...> wrote:

{{#ask: [[Query:French cities]] [[Population::>1000]] }}

where [[Query:French cities]] is the query "[[Category:City]] [[located
in::France]]". This would give you some ways of refining a saved query in
different ways, and one could still benefit from caching (since we would work
directly with the French cities insternally, and not look at all cities
again).

I was thinking about deriving good cache algorithm from queries separately from this issue, but if saved queries can help with cache, it can be very helpful.

Caching would of course be somewhat independent from saving queries: one could
save a query and just have it as a syntactic shortcut, or one could have it
pre-computed (maybe offline, maybe using some useful update strategy during
editing). The latter case allows things to get slightly out-of-date, but
would provide very stable performance even on huge wikis.

Yes, it's still not very clear how this cache will be invalidated when pages get changed, but performance can be more important in some cases.
 

So let us think about how to do that:

* We need a new namespace of some name (seeing the potential use of queries in
#ask, I suggest not to use "Ask:" as a namespace). Otherwise this is a piece
of cake.

Yes, "Ask" is a good candidate - pages in this namespace should probably also automatically display results of the query in some default format.
 

* We need to define how users would enter queries that they would like to be
saved. I think each query-page would hold one query and arbitrary bits of
documentation. To enter the query, users would use the syntax as in #ask.
There are some options for the details:
** We use a special property such as [[query:: [[Category:City]] ... ]].
** We use a new parser hook such as {{#query:  [[Category:City]] ... }}.
** We overload #ask for pages in that namespace, i.e.
{{#ask:  [[Category:City]] ... }} would work differently on those pages (it
could still show the results, but it would also be required to be unique, and
its query part would be saved).

There is another approach - Semantic Forms and Widgets extensions just use <include>/<noinclude> tags to identify what gets displayed and what gets used by Form/Widget engine. It doesn't make much sense to transclude pages in Ask namespace so this makes sense.

Also, it makes no sense to have multiple queries defined on one page in Ask namespace so using a parser function ({{#query}}) or property style ([[query::...]]) might bring some unexpected results. And definitely overriding a #ask is a bad idea since documentation piece of the page might want to include some arbitrary query results.
 

* We (or at least I) need to consider how to store queries in the database. I
can treat them similar to property values that can be very long (Type:Text).
As long as there is no caching, the query string would thus be processed in a
purely syntactic manner: you use it in a query, and SMW replaces it with its
real query string.

If we use method I suggested, then query text is going to be just page text with <include>/<noinclude> rules applied. You can also store parsed query code in database or cache if you think it makes sense.

* I can use the page id of the query page to identify the queries internally,
e.g. to store the information about them being cached (including location in
DB, date of last refresh), or for certain statistic data (result count, used
in query optimisation).

Yep, queries being pages is a good thing. The only problem I ran into is that using ask query in documentation piece of the page that defines that query requires refresh of the page after it's saved because at the parsing stage where you need to use pages content for query (in my case widget code), it's obviously not saved yet.
 

I think that is almost an implementation plan. Let us decide quickly on the
open questions, and I shall have that done (at least without the caching)
before the next release.

That would be great! I hope my late response will still have you consider my comments.

          Sergey

Best regards,

Markus

>
> I can definitely give various examples based on TechPresentations
> experience, like topic RSS feeds, event series iCal feeds and so on.
>
> Will appreciate any feedback on this.
>
> Thank you,
>
>          Sergey



--

Markus Krötzsch
Semantic MediaWiki    http://semantic-mediawiki.org
http://korrekt.org    markus@...

--
Sergey Chernyshev
http://www.sergeychernyshev.com/
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@...
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel

[cnit]

>     So let us think about how to do that:
>
>     * We need a new namespace of some name (seeing the potential use
>     of queries in
>     #ask, I suggest not to use "Ask:" as a namespace). Otherwise this
>     is a piece
>     of cake.
>
>
> Yes, "Ask" is a good candidate - pages in this namespace should
> probably also automatically display results of the query in some
> default format.

I think Markus has tought not to use "Ask:" namespace but something like
"Query:" instead. Doesn't make much difference, though.

>
> There is another approach - Semantic Forms and Widgets extensions just
> use <include>/<noinclude> tags to identify what gets displayed and
> what gets used by Form/Widget engine. It doesn't make much sense to
> transclude pages in Ask namespace so this makes sense.
>
>
> If we use method I suggested, then query text is going to be just page
> text with <include>/<noinclude> rules applied. You can also store
> parsed query code in database or cache if you think it makes sense.
>

Will such include work with multipage queries ("further results")?
I am also very much for short links to queries and to their further results.
Dmitriy

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@...
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel