Alternative to Compass/Searchable plugin

Hi all

I am developing an application now which allowes users to run full-text searches against 80+ mio documents. I scanned the web and Nabble :) and invested like 1.5 months in integrationg of Compass framework + Searchable plugin.

That was the reason for me to ask lots of stupid questions to Maurice here and Shannon on his 'home'-forum :)

Now I came to conclusion, that due to the nature of my app, the Compass and Co were not acceptable, despite being great framework and plugin.

The core requirements of my project are the following:

1. ability of managing 80+ different searchable 'domain' classes which are constantly changing on the weekly basis --> therefore the index must be changed as well. Here must be mentioned, that those 'classes' shall be modifiable by not-technical stuff.

2. Server restart shall be avoided at all cost, the app must be available 24-7, or even 60-24-7 :)

Compass is not really suitable for both of those, that's why I decided to prototype a direct use of Lucene API.

Actually, if you think about it, it's not a bad idea at all! Lucene and Groovy/Grails share a single common point, that it powerfull enough to bring together a 10-years-old search-engine and a pretty young and fancy dynamic framework.

And the point is Lucene's ability to index and search through documents with different number of fields in the same index!

That perfectly matches to Groovy's meta-programming concepts, which allows to add/remove properties in run-time.

And exactly here lays the main drawback of Compass (from the PoV of my app): as Compass is built with Hibernate in mind, it lets the user play ONLY with 'static' OSEM mappings. If you need to change the structure, you have no other ways, than either to restart the server or to rebuild the compass instance.

That's exactly the behaviour one would expect from Hibernate and underlying RDBMS: in order to modify table you would need to:
- run some kind of migrate script to apply new DML statements and keep the older data consistent
- change hbm-mappings off-line, then restart server and deploy the new version.

Native Lucene system doesn't have those limitations, but the Compass does!

Another inherent constraint of the Searchable plugin is it's focus on domain classes only. That means, that is I wanna have a domain class, which shall not be persisted with GORM and be put somewhere in src/, than such class will not be made "searchable" automagically.

I don't even want to mention a routine of adding a new searchable class into the app..

Deeply submerged in such considerations I spent like 2 days and wrote 3 core classes + IndexServive + Controller + some tests, and now I have a custom solution which replaces almost all Compass' functionalities I used before!

I allows me to
- register classes (in run-time) with a given package pattern, create an empty index dir for them
- change class/index structure on the fly
- inject searching and indexing methods, like save(), delete(), load() etc
- Grails-pagination of hits-lists is supported
- search in multiple indieces simulatenously
- boost indexing performance by applying different paramaters depending on domain class' structure and nature

The only thing I really miss now is the marvelous QueryBuilder, but this is solvable, as both Compass and Searchable plugin are open-sourced :)

If someone is interested in my solution, we can think about generalizing it and issue a plugin or whatever

Hope, you didn't get tired reading this post ;)