RE: possible index corruption

[Alexander Veremyev]

Hi George,

flock() is used to manage concurrent index updating. It's used since it's available on all systems. That's the only reason to use it instead of something else (ex. Semaphore, Shared Memory and IPC Functions extension).

I don't have universal solution for this now, but could try to make it for your environment.
Can you enable semaphores (compile php with the --enable-sysvsem option) for your php installation?


With best regards,
   Alexander Veremyev.
 

> -----Original Message-----
> From: gherson [mailto:gherson@...]
> Sent: Thursday, January 03, 2008 5:46 PM
> To: fw-formats@...
> Subject: Re: [fw-formats] possible index corruption
>
>
> Thanks, Johannes.  You found the problem.  I'd read the limitations but
> hadn't studied them carefully enough.  From your link:
>
> '29.2.7.2. Supported Filesystems
>
> 'Zend_Search_Lucene uses flock() to provide concurrent searching, index
> updating and optimization.
>
> 'According to the PHP documentation, "flock() will not work on NFS and
> many
> other networked file systems".
>
> 'Do not use networked file systems with Zend_Search_Lucene.'
>
> http://us2.php.net/manual/en/function.flock.php says "When using a
> multithreaded server API like ISAPI you may not be able to rely on flock()
> to protect files against other PHP scripts running in parallel threads of
> the same server instance!"
>
> One of the two web servers involved in my testing has an ISAPI, not an
> Apache2, server API.
>
> Question:  is there any way to use Zend_Search_Lucene on a server whose
> api
> is ISAPI?
>
> thank you,
> George Herson
>
>
> Johannes Schmidt-4 wrote:
> >
> > Hi George,
> >
> > it's just a (fast) guess:
> > http://framework.zend.com/manual/en/zend.search.lucene.index-
> creation.html#zend.search.lucene.index-creation.limitations
> >
> > Regards,
> > Johannes
> >
> > gherson wrote:
> >> I'm developing a Zend_Search_Lucene-powered text search feature for my
> >> intranet.  I'm currently rebuilding my index ~daily to get past
> symptoms:
> >>
> >> 1. Luke becomes unable to open the index, instead returning, e.g.,
> >> "\\storage01\Store_01\Sites_02\INDEXES\MainIndex\_l.cfs (The system
> >> cannot
> >> find the file specified)".
> >>
> >> 2. Line
> >> Zend_Search_Lucene::open("$IndexPath/MainIndex");
> >> hangs.
> >>
> >> If I had to guess I'd say my problem starts when >1 PHP scripts try
> >> accessing the index simultaneously (which I read is allowed).
> >>
> >> The only thing special before the index open command is
> >> Zend_Search_Lucene_Analysis_Analyzer::setDefault(new
> >> Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive());
> >>
> >> Platform is WinXP Pro, PHP Version 5.2.4, "Apache/2.2.6 (Win32) DAV/2",
> >> Zend
> >> Framework 1.0.2.
> >>
> >> TIA,
> >> George Herson
> >
> >
>
> --
> View this message in context: http://www.nabble.com/possible-index-
> corruption-tp14587058s16154p14598127.html
> Sent from the Zend MFS mailing list archive at Nabble.com.
>
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date:
> 02.01.2008 11:29
>

No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.17.13/1214 - Release Date: 08.01.2008 13:38
 

[gherson]

Thanks alot for the offer Alexander but I've been directed to use Apache so unless something else develops that won't be necessary.

For the next person, I'll mention my IHMOs that
o While it exists, the ISAPI limitation should be explicitly mentioned in the   http://framework.zend.com/manual/en/zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.limitations section.  Tracking down  hangs resulting from corruption can be such a time waster.
o The default search should err on the side of finding too much rather than too little, so I'd have

Zend_Search_Lucene_Analysis_Analyzer::setDefault(new
Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive());
as the default to allow case-insensitive and numerically inclusive search.

Thanks for ZSL, it's working great at the moment.

George Herson

----- Original Message ----
From: Alexander Veremyev <alexander.v@...>
To: gherson <gherson@...>; fw-formats@...
Sent: Thursday, January 10, 2008 12:58:36 PM
Subject: RE: [fw-formats] possible index corruption


Hi George,

flock() is used to manage concurrent index updating. It's used since
 it's available on all systems. That's the only reason to use it instead
 of something else (ex. Semaphore, Shared Memory and IPC Functions
 extension).

I don't have universal solution for this now, but could try to make it
 for your environment.
Can you enable semaphores (compile php with the --enable-sysvsem
 option) for your php installation?


With best regards,
   Alexander Veremyev.
 

> -----Original Message-----
> From: gherson [mailto:gherson@...]
> Sent: Thursday, January 03, 2008 5:46 PM
> To: fw-formats@...
> Subject: Re: [fw-formats] possible index corruption
>
>
> Thanks, Johannes.  You found the problem.  I'd read the limitations
 but
> hadn't studied them carefully enough.  From your link:
>
> '29.2.7.2. Supported Filesystems
>
> 'Zend_Search_Lucene uses flock() to provide concurrent searching,
 index
> updating and optimization.
>
> 'According to the PHP documentation, "flock() will not work on NFS
 and
> many
> other networked file systems".
>
> 'Do not use networked file systems with Zend_Search_Lucene.'
>
> http://us2.php.net/manual/en/function.flock.php says "When using a
> multithreaded server API like ISAPI you may not be able to rely on
 flock()
> to protect files against other PHP scripts running in parallel
 threads of
> the same server instance!"
>
> One of the two web servers involved in my testing has an ISAPI, not
 an
> Apache2, server API.
>
> Question:  is there any way to use Zend_Search_Lucene on a server
 whose

> api
> is ISAPI?
>
> thank you,
> George Herson
>
>
> Johannes Schmidt-4 wrote:
> >
> > Hi George,
> >
> > it's just a (fast) guess:
> > http://framework.zend.com/manual/en/zend.search.lucene.index-
> creation.html#zend.search.lucene.index-creation.limitations
> >
> > Regards,
> > Johannes
> >
> > gherson wrote:
> >> I'm developing a Zend_Search_Lucene-powered text search feature

 for my
> >> intranet.  I'm currently rebuilding my index ~daily to get past
> symptoms:
> >>
> >> 1. Luke becomes unable to open the index, instead returning, e.g.,
> >> "\\storage01\Store_01\Sites_02\INDEXES\MainIndex\_l.cfs (The
 system
> >> cannot
> >> find the file specified)".
> >>
> >> 2. Line
> >> Zend_Search_Lucene::open("$IndexPath/MainIndex");
> >> hangs.
> >>
> >> If I had to guess I'd say my problem starts when >1 PHP scripts
 try
> >> accessing the index simultaneously (which I read is allowed).
> >>
> >> The only thing special before the index open command is
> >> Zend_Search_Lucene_Analysis_Analyzer::setDefault(new
> >>
 Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive());
> >>
> >> Platform is WinXP Pro, PHP Version 5.2.4, "Apache/2.2.6 (Win32)
 DAV/2",

> >> Zend
> >> Framework 1.0.2.
> >>
> >> TIA,
> >> George Herson
> >
> >
>
> --
> View this message in context: http://www.nabble.com/possible-index-
> corruption-tp14587058s16154p14598127.html
> Sent from the Zend MFS mailing list archive at Nabble.com.
>
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date:
> 02.01.2008 11:29
>

No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.17.13/1214 - Release Date:
 08.01.2008 13:38