Alternative to Compass/Searchable plugin

[Konstantyn Smirnov]

Hi all

I am developing an application now which allowes users to run full-text searches against 80+ mio documents. I scanned the web and Nabble :) and invested like 1.5 months in integrationg of Compass framework + Searchable plugin.

That was the reason for me to ask lots of stupid questions to Maurice here and Shannon on his 'home'-forum :)

Now I came to conclusion, that due to the nature of my app, the Compass and Co were not acceptable, despite being great framework and plugin.

The core requirements of my project are the following:

1. ability of managing 80+ different searchable 'domain' classes which are constantly changing on the weekly basis --> therefore the index must be changed as well. Here must be mentioned, that those 'classes' shall be modifiable by not-technical stuff.

2. Server restart shall be avoided at all cost, the app must be available 24-7, or even 60-24-7 :)

Compass is not really suitable for both of those, that's why I decided to prototype a direct use of Lucene API.

Actually, if you think about it, it's not a bad idea at all! Lucene and Groovy/Grails share a single common point, that it powerfull enough to bring together a 10-years-old search-engine and a pretty young and fancy dynamic framework.

And the point is Lucene's ability to index and search through documents with different number of fields in the same index!

That perfectly matches to Groovy's meta-programming concepts, which allows to add/remove properties in run-time.

And exactly here lays the main drawback of Compass (from the PoV of my app): as Compass is built with Hibernate in mind, it lets the user play ONLY with 'static' OSEM mappings. If you need to change the structure, you have no other ways, than either to restart the server or to rebuild the compass instance.

That's exactly the behaviour one would expect from Hibernate and underlying RDBMS: in order to modify table you would need to:
- run some kind of migrate script to apply new DML statements and keep the older data consistent
- change hbm-mappings off-line, then restart server and deploy the new version.

Native Lucene system doesn't have those limitations, but the Compass does!

Another inherent constraint of the Searchable plugin is it's focus on domain classes only. That means, that is I wanna have a domain class, which shall not be persisted with GORM and be put somewhere in src/, than such class will not be made "searchable" automagically.

I don't even want to mention a routine of adding a new searchable class into the app..

Deeply submerged in such considerations I spent like 2 days and wrote 3 core classes + IndexServive + Controller + some tests, and now I have a custom solution which replaces almost all Compass' functionalities I used before!

I allows me to
- register classes (in run-time) with a given package pattern, create an empty index dir for them
- change class/index structure on the fly
- inject searching and indexing methods, like save(), delete(), load() etc
- Grails-pagination of hits-lists is supported
- search in multiple indieces simulatenously
- boost indexing performance by applying different paramaters depending on domain class' structure and nature

The only thing I really miss now is the marvelous QueryBuilder, but this is solvable, as both Compass and Searchable plugin are open-sourced :)

If someone is interested in my solution, we can think about generalizing it and issue a plugin or whatever

Hope, you didn't get tired reading this post ;)

[Barzilai Spinak-2]

If I understood most of what you wrote, I'm *very* interested in looking
at your solution :-)

BarZ

Konstantyn Smirnov wrote:

> Hi all
>
> I am developing an application now, and the core component is the search
> engine, because I need to make 80+ mio documents full-text-searchable. So I
> scanned the web and Nabble :) and invested like 1.5 months in Compass
> framework + Searchable plugin.
>
> That was the reason for me to ask lots of stupid questions to Maurice here
> and Shannon on his 'home'-forum :)
>
> Now I came to conclusion, that due to the nature of my app, the Compass and
> Co were not acceptable, despite being great framework and plugin.
>
> The core requirements of my project are the following:
>
> 1. ability of managing 80+ different searchable 'domain' classes which are
> constantly changing on the weekly basis --> therefore the index must be
> changed as well. Here must be mentioned, that those 'classes' shall be
> modifiable by not-technical stuff.
>
> 2. Server restart shall be avoided at all cost, the app must be available
> 24-7, or even 60-24-7 :)
>
> Compass is not really suitable for both of those, that's why I decided to
> prototype a direct use of Lucene API.
>
> Actually, if you think about it, it's not a bad idea at all! Lucene and
> Groovy/Grails share a single common point, that it powerfull enough to bring
> a 10-years-old search-engine and a pretty young and fancy dynamic framework.
>
> And the point is Lucene's ability to index and search through documents with
> different number of fields in the same index!
>
> That perfectly matches to Groovy's meta-programming concepts, which allows
> to add properties in run-time.
>
> And exactly here lays the main drawback of Compass (from the PoV of my app):
> as Compass is built with Hibernate in mind, it lets the user play ONLY with
> 'static' OSEM mappings. If you need to change the structure, you have no
> other ways, than either to restart the server or to rebuild the compass
> instance.
>
> That's exactly the behaviour one would expect from Hibernate and underlying
> RDBMS: in order to modify table you would need to:
> - run some kind of migrate script to apply new DML statements and keep the
> older data consistent
> - change hbm-mappings off-line, then restart server and deploy the new
> version.
>
> Native Lucene system doesn't have those limitations, but the Compass does!
>
> Another inherent constraint of the Searchable plugin is it's focus on domain
> classes only. That means, that is I wanna have a domain class, which shall
> not be persisted with GORM and be put somewhere in src/, than such class
> will not be made "searchable" automagically.
>
> I don't even want to mention a routine of adding a new searchable class into
> the app..
>
> Deeply submerged in such considerations I spent like 2 days and wrote 3 core
> classes + IndexServive + Controller + some tests, and now I have a custom
> solution which replaces almost all Compass' functionalities I used before!
>
> I allows me to
> - register classes (in run-time) with a given package pattern, create an
> empty index dir for them
> - change class/index structure on the fly
> - inject searching and indexing methods, like save(), delete(), load() etc
> - Grails-pagination of hits-lists is supported
> - search in multiple indieces simulatenously
> - boost indexing performance by applying different paramaters depending on
> domain class' structure and nature
>
> The only thing I really miss now is the marvelous QueryBuilder, but this
> solvable, as Compass and Searchable plugin are open-sourced :)
>
> If some-one is interested in my solution, we can think about generalizing it
> and issue a plugin or whatever
>
> Hope, you didn't get tired reading this post ;)
>  

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Dustin Whitney]

My team looked at solr as an alternative to the searchable plugin.  http://lucene.apache.org/solr/  I have to say, that I think the searchable plugin is rad, but maybe you'll find solr more useful.

-Dustin

On Fri, May 16, 2008 at 12:38 AM, Barzilai Spinak <barcho@...> wrote:

If I understood most of what you wrote, I'm *very* interested in looking at your solution :-)

BarZ

Konstantyn Smirnov wrote:

Hi all

I am developing an application now, and the core component is the search
engine, because I need to make 80+ mio documents full-text-searchable. So I
scanned the web and Nabble :) and invested like 1.5 months in Compass
framework + Searchable plugin.

That was the reason for me to ask lots of stupid questions to Maurice here
and Shannon on his 'home'-forum :)

Now I came to conclusion, that due to the nature of my app, the Compass and
Co were not acceptable, despite being great framework and plugin.

The core requirements of my project are the following:

1. ability of managing 80+ different searchable 'domain' classes which are
constantly changing on the weekly basis --> therefore the index must be
changed as well. Here must be mentioned, that those 'classes' shall be
modifiable by not-technical stuff.

2. Server restart shall be avoided at all cost, the app must be available
24-7, or even 60-24-7 :)

Compass is not really suitable for both of those, that's why I decided to
prototype a direct use of Lucene API.
Actually, if you think about it, it's not a bad idea at all! Lucene and
Groovy/Grails share a single common point, that it powerfull enough to bring
a 10-years-old search-engine and a pretty young and fancy dynamic framework.

And the point is Lucene's ability to index and search through documents with
different number of fields in the same index!

That perfectly matches to Groovy's meta-programming concepts, which allows
to add properties in run-time.

And exactly here lays the main drawback of Compass (from the PoV of my app):
as Compass is built with Hibernate in mind, it lets the user play ONLY with
'static' OSEM mappings. If you need to change the structure, you have no
other ways, than either to restart the server or to rebuild the compass
instance.

That's exactly the behaviour one would expect from Hibernate and underlying
RDBMS: in order to modify table you would need to:
- run some kind of migrate script to apply new DML statements and keep the
older data consistent
- change hbm-mappings off-line, then restart server and deploy the new
version.

Native Lucene system doesn't have those limitations, but the Compass does!

Another inherent constraint of the Searchable plugin is it's focus on domain
classes only. That means, that is I wanna have a domain class, which shall
not be persisted with GORM and be put somewhere in src/, than such class
will not be made "searchable" automagically.

I don't even want to mention a routine of adding a new searchable class into
the app..

Deeply submerged in such considerations I spent like 2 days and wrote 3 core
classes + IndexServive + Controller + some tests, and now I have a custom
solution which replaces almost all Compass' functionalities I used before!
I allows me to
- register classes (in run-time) with a given package pattern, create an
empty index dir for them
- change class/index structure on the fly
- inject searching and indexing methods, like save(), delete(), load() etc
- Grails-pagination of hits-lists is supported
- search in multiple indieces simulatenously
- boost indexing performance by applying different paramaters depending on
domain class' structure and nature

The only thing I really miss now is the marvelous QueryBuilder, but this
solvable, as Compass and Searchable plugin are open-sourced :)

If some-one is interested in my solution, we can think about generalizing it
and issue a plugin or whatever

Hope, you didn't get tired reading this post ;)
 

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email

[Seymour Cakes-2]

This has been done by many before Searchable became the defacto search plugin. Even I myself wrote a simple Lucene plugin, but I later abandon it in favor of Searchable.

Alternative to Searchable is a good thing IMHO.

On Thu, May 15, 2008 at 4:53 PM, Konstantyn Smirnov <injecteer@...> wrote:

Hi all

I am developing an application now, and the core component is the search
engine, because I need to make 80+ mio documents full-text-searchable. So I
scanned the web and Nabble :) and invested like 1.5 months in Compass
framework + Searchable plugin.

That was the reason for me to ask lots of stupid questions to Maurice here
and Shannon on his 'home'-forum :)

Now I came to conclusion, that due to the nature of my app, the Compass and
Co were not acceptable, despite being great framework and plugin.

The core requirements of my project are the following:

1. ability of managing 80+ different searchable 'domain' classes which are
constantly changing on the weekly basis --> therefore the index must be
changed as well. Here must be mentioned, that those 'classes' shall be
modifiable by not-technical stuff.

2. Server restart shall be avoided at all cost, the app must be available
24-7, or even 60-24-7 :)

Compass is not really suitable for both of those, that's why I decided to
prototype a direct use of Lucene API.

Actually, if you think about it, it's not a bad idea at all! Lucene and
Groovy/Grails share a single common point, that it powerfull enough to bring
a 10-years-old search-engine and a pretty young and fancy dynamic framework.

And the point is Lucene's ability to index and search through documents with
different number of fields in the same index!

That perfectly matches to Groovy's meta-programming concepts, which allows
to add properties in run-time.

And exactly here lays the main drawback of Compass (from the PoV of my app):
as Compass is built with Hibernate in mind, it lets the user play ONLY with
'static' OSEM mappings. If you need to change the structure, you have no
other ways, than either to restart the server or to rebuild the compass
instance.

That's exactly the behaviour one would expect from Hibernate and underlying
RDBMS: in order to modify table you would need to:
- run some kind of migrate script to apply new DML statements and keep the
older data consistent
- change hbm-mappings off-line, then restart server and deploy the new
version.

Native Lucene system doesn't have those limitations, but the Compass does!

Another inherent constraint of the Searchable plugin is it's focus on domain
classes only. That means, that is I wanna have a domain class, which shall
not be persisted with GORM and be put somewhere in src/, than such class
will not be made "searchable" automagically.

I don't even want to mention a routine of adding a new searchable class into
the app..

Deeply submerged in such considerations I spent like 2 days and wrote 3 core
classes + IndexServive + Controller + some tests, and now I have a custom
solution which replaces almost all Compass' functionalities I used before!

I allows me to
- register classes (in run-time) with a given package pattern, create an
empty index dir for them
- change class/index structure on the fly
- inject searching and indexing methods, like save(), delete(), load() etc
- Grails-pagination of hits-lists is supported
- search in multiple indieces simulatenously
- boost indexing performance by applying different paramaters depending on
domain class' structure and nature

The only thing I really miss now is the marvelous QueryBuilder, but this
solvable, as Compass and Searchable plugin are open-sourced :)

If some-one is interested in my solution, we can think about generalizing it
and issue a plugin or whatever

Hope, you didn't get tired reading this post ;)
--
View this message in context: http://www.nabble.com/Alternative-to-Compass-Searchable-plugin-tp17248352p17248352.html
Sent from the grails - user mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[Konstantyn Smirnov]

Yep, Solr is interesting indeed and I considered using it as well, but I don't have another month for learning & integrating Solr.

It has lot's of functionality I won't ever need, but I will have to configure it somehow anyway.

Also, the operation mode of Solr is described as a "standalone enterprise search server" which uses a "web-services like API", meaning that it can really become a bottleneck in a hi-load envirnoment.

On the other hand my 3-classes-big module works within the application, effectively removing marshalling overhead.

Than IMHO, it's better to have an extremely simple wrapper around Lucene API which does exactly what I need, instead of integrating a full-blown framework with lot's of not required stuff.

Last but not least comes the FUN of developing such a thing in Groovy :) The system behaves almost exactly the way I expected.
My longest debug session (due to ClassLoader probs) lasted for 1 hour, so most of the time I spent in adding new features and test-cases.

2 Barzilai: shall I send the stuff per email or share somehow?

[Barzilai Spinak-2]

Is it written as a plugin?
Is there any possibility for you to put it in a zip and hang it from
some url so everybody can benefit?
At this point, I'm just gathering lots of ideas and tips on how to write
plugins that spice-up different aspects of my Grails artefacts, so your
solution may be interesting to look at, even if I end up using something
else, or writing my own stuff (not only with respect to indexing and
searching but also different ideas in persistence)

BarZ

Konstantyn Smirnov wrote:

> Yep, Solr is interesting indeed and I considered using it as well, but I
> don't have another month for learning & integrating Solr.
>
> It has lot's of functionality I won't ever need, but I will have to
> configure it somehow anyway.
>
> Also, the operation mode of Solr is described as a "standalone enterprise
> search server" which uses a "web-services like API", meaning that it can
> really become a bottleneck in a hi-load envirnoment.
>
> On the other hand my 3-classes-big module works within the application,
> effectively removing marshalling overhead.
>
> Than IMHO, it's better to have an extremely simple wrapper around Lucene API
> which does exactly what I need, instead of integrating a full-blown
> framework with lot's of not required stuff.
>
> Last but not least comes the FUN of developing such a thing in Groovy :) The
> system behaves almost exactly the way I expected.
> My longest debug session (due to ClassLoader probs) lasted for 1 hour, so
> most of the time I spent in adding new features and test-cases.
>
> 2 Barzilai: shall I send the stuff per email or share somehow?
>
>  

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Konstantyn Smirnov]

here you go.

src.rar

It's not a plugin, just 3 core classes along with some test-cases.

The index structure is flat, no parent-children relations. Query API is still missing.

By default, all fields (exept for those in Document.groovy) in all classes must be tokenized and stored, no boost and other fancy stuff.

[Konstantyn Smirnov]

here's a slightly updated version
proj.rar

[chrislusf]

Maybe you should look at DBSight, a database search based on lucene. It uses direct SQL to crawl database rather than relying on some specific framework. Indexing, Re-Indexing, Incremental Indexing are all easy to do and efficient. It'll get you started in several minutes. Free version can create size unlimited index now. Chris

[Konstantyn Smirnov]

chrislusf wrote:

Maybe you should look at DBSight, a database search based on lucene.

It uses direct SQL to crawl database rather than relying on some specific framework. Indexing, Re-Indexing, Incremental Indexing are all easy to do and efficient.

It'll get you started in several minutes. Free version can create size unlimited index now.

Chris

thanks for the info, I looked through the site.
Sounds interesting, but not for my case.

I'll describe briefly what my module + Lucene must be able to do (and does now) within a Grails app:

1. Create new *domain* classes and modify them in run-time, and prepare the index structures for them, all within 2 minutes time-frame
2. Simulateneously in several threads (I had 7 once) index the raw data within 6-8 sec per 10 MB data file. Each index's minimal size is 2.5 GB ~ 150000 document, but it can be up to 13 GB ~ 8+ Mio docs.
3. Simulateneously search in up to 20 indexes in various combinations with search times up to 800 ms

No need for internet crawling and different presentation formats.

Can DBSight serve as a good and performant replacement for that?

[chrislusf]

1. No. DBSight can not handle that, and I suppose no existing tools can handle your case.

2. My experience is that in multi-threaded indexing mode, 20G data can take 4 hours to index. Your 6~8sec/10MB looks in the range.

3. This should be possible.

Maybe you would need a producer~consumer model to separate the indexing and searching process to separate machines, because either one will slow down the machine a lot.

--
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
DBSight customer, a shopping comparison site, (anonymous per request) got 2.6 Million Euro funding!

On Tue, Jun 10, 2008 at 3:37 PM, Konstantyn Smirnov <injecteer@...> wrote:

chrislusf wrote:
>
> Maybe you should look at  http://www.dbsight.net DBSight , a database

> search based on lucene.
>
> It uses direct SQL to crawl database rather than relying on some specific
> framework. Indexing, Re-Indexing, Incremental Indexing are all easy to do
> and efficient.
>
> It'll get you started in several minutes. Free version can create size
> unlimited index now.
>
> Chris
>

thanks for the info, I looked through the site.
Sounds interesting, but not for my case.

I'll describe briefly what my module + Lucene must be able to do (and does
now) within a Grails app:

1. Create new *domain* classes and modify them in run-time, and prepare the
index structures for them, all within 2 minutes time-frame
2. Simulateneously in several threads (I had 7 once) index the raw data
within 6-8 sec per 10 MB data file. Each index's minimal size is 2.5 GB ~
150000 document, but it can be up to 13 GB ~ 8+ Mio docs.
3. Simulateneously search in up to 20 indexes in various combinations with
search times up to 800 ms

No need for internet crawling and different presentation formats.

Can DBSight serve as a good and performant replacement for that?




--
View this message in context: http://www.nabble.com/Alternative-to-Compass-Searchable-plugin-tp17248352p17766301.html

Sent from the grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[Konstantyn Smirnov]

chrislusf wrote:

1. No. DBSight can not handle that, and I suppose no existing tools can
handle your case.

except for the one I wrote for my project :)
I need to acomplish another couple of weird tasks there.

chrislusf wrote:

Maybe you would need a producer~consumer model to separate the indexing and
searching process to separate machines, because either one will slow down
the machine a lot.

yep, that's one of the standard scenarios we considered

chrislusf wrote:

2. My experience is that in multi-threaded indexing mode, 20G data can take
4 hours to index. Your 6~8sec/10MB looks in the range.

3. This should be possible.

if so, I'll better stick with my *framework* consisting of 3 classes only :)

Not the least role plays the fact, that those classes a Groovy ones, which makes them perfectly suitable for a grails app.

[Stefan-93]

Speaking of searchable: Do we have a simple method to index PDf files  
related to
or referenced by domain classes?

Or may indexing of referenced resources be implemented later on?

Am 11.06.2008 um 02:32 schrieb Konstantyn Smirnov:

> chrislusf wrote:
>>
>> 1. No. DBSight can not handle that, and I suppose no existing tools  
>> can
>> handle your case.
>>
>
> except for the one I wrote for my project :)
> I need to acomplish another couple of weird tasks there.
>
>
> chrislusf wrote:
>>
>> Maybe you would need a producer~consumer model to separate the  
>> indexing
>> and
>> searching process to separate machines, because either one will  
>> slow down
>> the machine a lot.
>>
> yep, that's one of the standard scenarios we considered
>
>
>
> chrislusf wrote:
>>
>> 2. My experience is that in multi-threaded indexing mode, 20G data  
>> can
>> take
>> 4 hours to index. Your 6~8sec/10MB looks in the range.
>>
>> 3. This should be possible.
>>
>
> if so, I'll better stick with my *framework* consisting of 3 classes  
> only :)
>
> Not the least role plays the fact, that those classes a Groovy ones,  
> which
> makes them perfectly suitable for a grails app.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Konstantyn Smirnov]

Stefan-93 wrote:

Speaking of searchable: Do we have a simple method to index PDf files  
related to
or referenced by domain classes?

/

You'll have to map a pdf-file into a Lucene document or into your domain class.

In both cases you'll have to parse the file using some sort of adapter. Those adapters can be found in lucene-based frameworks like Nutch, but it's also easy to develop your own parser within a couple of hours using Apache POI, if you need just PDF-domain conversion.

The resulting instance of your domain class can be indexed as root or non-root object by Searchable.

[Brad Whitaker]

I attended a presentation on Solr last week that made a very credible case that Solr has very high performance and is easily scalable. Solr uses a single "master" that builds the index and any number of "slave" machines that serve queries. The presenter indicated that the Netflix site supports all of their queries with 5 slave machines, but the main point was that performance is easily scaled simply by adding more slaves. A separate "producer~consumer model" that was mentioned in another message in this thread is a key aspect of the Solr architecture.

Regarding the API, Solr supports "XML/HTTP and JSON APIs" but there is also a Java client API available through Solrj (http://wiki.apache.org/solr/Solrj).  

Konstantyn Smirnov wrote:

Also, the operation mode of Solr is described as a "standalone enterprise search server" which uses a "web-services like API", meaning that it can really become a bottleneck in a hi-load envirnoment.

[Konstantyn Smirnov]

Brad Whitaker wrote:

Solr uses a single "master" that builds the index and any number of "slave" machines that serve queries. The presenter indicated that the Netflix site supports all of their queries with 5 slave machines, but the main point was that performance is easily scaled simply by adding more slaves.

Couldn't it be done with a load-balanced cluster of tomcats sharing the same phisycal index storage or even using it's own index replica?
I'm pretty sure, that for my application Solr or anything else Lucene-based won't bring any performance gains...

[Brad Whitaker]

Konstantyn Smirnov wrote:

Brad Whitaker wrote:

Solr uses a single "master" that builds the index and any number of "slave" machines that serve queries. The presenter indicated that the Netflix site supports all of their queries with 5 slave machines, but the main point was that performance is easily scaled simply by adding more slaves.

Couldn't it be done with a load-balanced cluster of tomcats sharing the same phisycal index storage or even using it's own index replica?
I'm pretty sure, that for my application Solr or anything else Lucene-based won't bring any performance gains...

I agree that there is no reason to think that Solr would perform any better than what you have implemented, and as you mentioned previously you are avoiding the overhead associated with marshalling data between processes. And yes, you can undoubtedly use a load-balanced cluster of tomcats. My understanding of Solr is that each slave has its own local storage, data is moved from master to slave using rsync, and that Solr includes an administration interface for managing the nodes and updates.