query performance issues in 1.1

[Michael Gratton-3]

Hi,

After running 1.0-rc1 for a long long time, I finally upgraded to 1.1
the other day. But I was a bit worried to find that the performance of
queries seems to be an order of magnitude worse.

This particular database contains one really large collection (~28k
docs) of about 10 different types of documents. With 1.0-rc1 I could
execute an XPath query like "/foo", where the element `foo' was indexed
and start getting result back within a few tens of seconds. Running 1.1
with the exact same configuration takes a minute and a half.

I know the collection is rather large, but splitting it up will only
delay the problem becoming as bad for a bit longer - the number of
documents grows daily. I've tried both upgrade methods as described in
the Upgrading How-To, dropped and recreated all indexes, tried both the
XML-RPC client and running the server embedded - all with no joy.

Is there something obvious here I'm missing? Any suggestions
appreciated.

Thanks,
/Mike

--
Michael Gratton <michael@...>    
Quuxo Software <http://web.quuxo.com/>

[Natalia Shilenkova]

On Feb 4, 2008 8:15 AM, Michael Gratton <michael@...> wrote:

>
> Hi,
>
> After running 1.0-rc1 for a long long time, I finally upgraded to 1.1
> the other day. But I was a bit worried to find that the performance of
> queries seems to be an order of magnitude worse.
>
> This particular database contains one really large collection (~28k
> docs) of about 10 different types of documents. With 1.0-rc1 I could
> execute an XPath query like "/foo", where the element `foo' was indexed
> and start getting result back within a few tens of seconds. Running 1.1
> with the exact same configuration takes a minute and a half.
>
> I know the collection is rather large, but splitting it up will only
> delay the problem becoming as bad for a bit longer - the number of
> documents grows daily. I've tried both upgrade methods as described in
> the Upgrading How-To, dropped and recreated all indexes, tried both the
> XML-RPC client and running the server embedded - all with no joy.
>
> Is there something obvious here I'm missing? Any suggestions
> appreciated.

I would recommend to check indexer configuration first, to make sure
that indexer type and pattern are correct. Database configuration (it
can be retrieved by executing bin/xindice rd -c /db/system/SysConfig
-n database.xml) should include something like:
<index class="org.apache.xindice.core.indexer.NameIndexer" name="test"
pattern="foo" />
Here I assume that you need NameIndexer because of the sample query
("/foo") that you provided.

Also, can you see if there are any exceptions in the log?

Another thing to try would be to delete the index and run the same
query again. If the query execution time does not increase, then the
performance issues  are likely caused by query engine not using the
index. The reason for that probably can be found in configuration or
logs.

If that doesn't help, I may need more information about your
collection to run some tests.

Regards,
Natalia

> Thanks,
> /Mike
>
> --
> Michael Gratton <michael@...>
> Quuxo Software <http://web.quuxo.com/>
>

[Michael Gratton-3]

On Mon, 2008-02-04 at 19:10 -0500, Natalia Shilenkova wrote:
> I would recommend to check indexer configuration first, to make sure
> that indexer type and pattern are correct.

Ahh, thanks for that. All the indexes were using ValueIndexer when some
should have been using NameIndexer.

I guess I missed it because the old indexer was split up back when 0.9
was released. I wonder why that wasn't a problem with 1.0-rc1? Looking
back at that old instance's configuration, the indexes all used
ValueIndexer as well. Odd.

Anyway, thanks for you help. Should I put something on the wiki about
it? Does anyone use 0.9/1.0 any more?

/Mike

--
Michael Gratton <michael@...>    
Quuxo Software <http://web.quuxo.com/>

[Natalia Shilenkova]

On Feb 5, 2008 8:11 AM, Michael Gratton <michael@...> wrote:

>
> On Mon, 2008-02-04 at 19:10 -0500, Natalia Shilenkova wrote:
> > I would recommend to check indexer configuration first, to make sure
> > that indexer type and pattern are correct.
>
> Ahh, thanks for that. All the indexes were using ValueIndexer when some
> should have been using NameIndexer.
>
> I guess I missed it because the old indexer was split up back when 0.9
> was released. I wonder why that wasn't a problem with 1.0-rc1? Looking
> back at that old instance's configuration, the indexes all used
> ValueIndexer as well. Odd.
>
> Anyway, thanks for you help. Should I put something on the wiki about
> it? Does anyone use 0.9/1.0 any more?

You're welcome and please feel free to add any information you
consider useful to the wiki. Any kind of documentation is greatly
appreciated. Even if no one uses old version anymore it could help
troubleshooting index problems.

Regards,
Natalia