Re: Alternative to Compass/Searchable plugin

If I understood most of what you wrote, I'm *very* interested in looking at your solution :-)

BarZ

Konstantyn Smirnov wrote:

Hi all

I am developing an application now, and the core component is the search
engine, because I need to make 80+ mio documents full-text-searchable. So I
scanned the web and Nabble :) and invested like 1.5 months in Compass
framework + Searchable plugin.

That was the reason for me to ask lots of stupid questions to Maurice here
and Shannon on his 'home'-forum :)

Now I came to conclusion, that due to the nature of my app, the Compass and
Co were not acceptable, despite being great framework and plugin.

The core requirements of my project are the following:

1. ability of managing 80+ different searchable 'domain' classes which are
constantly changing on the weekly basis --> therefore the index must be
changed as well. Here must be mentioned, that those 'classes' shall be
modifiable by not-technical stuff.

2. Server restart shall be avoided at all cost, the app must be available
24-7, or even 60-24-7 :)

Compass is not really suitable for both of those, that's why I decided to
prototype a direct use of Lucene API.
Actually, if you think about it, it's not a bad idea at all! Lucene and
Groovy/Grails share a single common point, that it powerfull enough to bring
a 10-years-old search-engine and a pretty young and fancy dynamic framework.

And the point is Lucene's ability to index and search through documents with
different number of fields in the same index!

That perfectly matches to Groovy's meta-programming concepts, which allows
to add properties in run-time.

And exactly here lays the main drawback of Compass (from the PoV of my app):
as Compass is built with Hibernate in mind, it lets the user play ONLY with
'static' OSEM mappings. If you need to change the structure, you have no
other ways, than either to restart the server or to rebuild the compass
instance.

That's exactly the behaviour one would expect from Hibernate and underlying
RDBMS: in order to modify table you would need to:
- run some kind of migrate script to apply new DML statements and keep the
older data consistent
- change hbm-mappings off-line, then restart server and deploy the new
version.

Native Lucene system doesn't have those limitations, but the Compass does!

Another inherent constraint of the Searchable plugin is it's focus on domain
classes only. That means, that is I wanna have a domain class, which shall
not be persisted with GORM and be put somewhere in src/, than such class
will not be made "searchable" automagically.

I don't even want to mention a routine of adding a new searchable class into
the app..

Deeply submerged in such considerations I spent like 2 days and wrote 3 core
classes + IndexServive + Controller + some tests, and now I have a custom
solution which replaces almost all Compass' functionalities I used before!
I allows me to
- register classes (in run-time) with a given package pattern, create an
empty index dir for them
- change class/index structure on the fly
- inject searching and indexing methods, like save(), delete(), load() etc
- Grails-pagination of hits-lists is supported
- search in multiple indieces simulatenously
- boost indexing performance by applying different paramaters depending on
domain class' structure and nature

The only thing I really miss now is the marvelous QueryBuilder, but this
solvable, as Compass and Searchable plugin are open-sourced :)

If some-one is interested in my solution, we can think about generalizing it
and issue a plugin or whatever

Hope, you didn't get tired reading this post ;)
 

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email