Search index warnings live server but not local

[Jack Sleight]

Hi,
I have a script that builds the lucene search index. When I run it locally it's all fine, when I run it on the live server I get a lot of these warnings:

Notice: iconv() [function.iconv]: Detected an illegal character in input string in ...1.5.2/library/Zend/Search/Lucene/Analysis/Analyzer/Common/Text.php on line 56

My local server is Windows, running PHP 5.2.4, the live server is CentOS running PHP 5.1.6

I'm assuming it's some sort of character set/encoding issue, but I've got no idea what's causing it. Any ideas?
Thanks,

[Jack Sleight]

Sorry, should have given the manual a better read, fixed it now.
http://framework.zend.com/manual/en/zend.search.lucene.best-practice.html#zend.search.lucene.best-practice.encoding

Jack Sleight wrote:

Hi,
I have a script that builds the lucene search index. When I run it locally it's all fine, when I run it on the live server I get a lot of these warnings:

Notice: iconv() [function.iconv]: Detected an illegal character in input string in ...1.5.2/library/Zend/Search/Lucene/Analysis/Analyzer/Common/Text.php on line 56

My local server is Windows, running PHP 5.2.4, the live server is CentOS running PHP 5.1.6

I'm assuming it's some sort of character set/encoding issue, but I've got no idea what's causing it. Any ideas?
Thanks,

[Matthew Weier O'Phinney-3]

-- Jack Sleight <jack.sleight@...> wrote
(on Thursday, 26 June 2008, 12:06 PM +0100):

> I have a script that builds the lucene search index. When I run it locally it's
> all fine, when I run it on the live server I get a lot of these warnings:
>
> Notice: iconv() [function.iconv]: Detected an illegal character in input string
> in ...1.5.2/library/Zend/Search/Lucene/Analysis/Analyzer/Common/Text.php on
> line 56
>
> My local server is Windows, running PHP 5.2.4, the live server is CentOS
> running PHP 5.1.6
>
> I'm assuming it's some sort of character set/encoding issue, but I've got no
> idea what's causing it. Any ideas?

I've noticed the same on a couple of servers I run as well. The good
news is that it doesn't appear to affect indexing. Perhaps Alex can
chime in and indicate what PHP settings may be involved?

--
Matthew Weier O'Phinney
Software Architect       | matthew@...
Zend Framework           | http://framework.zend.com/

[Jack Sleight]

Hi Matthew,
Well I figured out what was causing it, some of the fields contained ü characters, and the encoding was UTF-8, but on the live server it didn't think the values sent to the Zend_Search_Lucene_Field factory methods were UTF-8 encoded. I resolved it by specifying UTF-8 as the third argument. No idea why it didn't do that on my local version, I guess it's something to do with the locales.

If this is the same reason it's happening for you then you may wish to double check that it's not affecting indexing, because it was for me. Take the following field value:

aaaa bbbb cccc dddd uuüu xxxx zzzz

For me, the ü was obviously causing the error, but only the words before the one containing ü got indexed. A search for xxxx or zzzz would return nothing.

Matthew Weier O'Phinney wrote:

-- Jack Sleight jack.sleight@... wrote
(on Thursday, 26 June 2008, 12:06 PM +0100):
  

I've noticed the same on a couple of servers I run as well. The good
news is that it doesn't appear to affect indexing. Perhaps Alex can
chime in and indicate what PHP settings may be involved?