Problems with foreign characters (portuguese)

[fgomes]

I'm trying to use the latest version of programd with aiml files with portuguese characters, and it don't understand the  patterns with foreign characters, and also reply with bad character encoding. Is it necessary to configure the character encoding somewhere? I'm using <?xml version="1.0" encoding="UTF-8"?> in the aiml files, also tried with <?xml version="1.0" encoding="ISO-8859-1"?>, both with the same result.
I have similar problems also with pyAIML, so it can be a bigger problem :-(
Thanks in advance for all your help!

Best regards

Fernando

[Noel Bush]

Hi Fernando,

You should be able to use any encoding you want to.  The key, however,
is to make sure that you are actually saving the file in the encoding
you specify in the <?xml?> processing instruction.  Just changing it to
say "UTF-8" or "ISO-8859-1" or "KOI8-R" or whatever will not change the
actual encoding of the file.  This is something you need to control from
your text editor.  It can be tricky to deal with this issue, since
there's no way to just look at a file and tell what charset/encoding it
is using.

However, if you do match up the declared encoding with the actual
encoding, I think you will find that Program D works properly.  At
least, it has done well with all character sets / encodings that I've
tried it with.

Noel

fgomes wrote:

> I'm trying to use the latest version of programd with aiml files with
> portuguese characters, and it don't understand the  patterns with foreign
> characters, and also reply with bad character encoding. Is it necessary to
> configure the character encoding somewhere? I'm using <?xml version="1.0"
> encoding="UTF-8"?> in the aiml files, also tried with <?xml version="1.0"
> encoding="ISO-8859-1"?>, both with the same result.
> I have similar problems also with pyAIML, so it can be a bigger problem :-(
> Thanks in advance for all your help!
>
> Best regards
>
> Fernando

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[fgomes]

Hi Noel, thanks for the reply!

I've tried to convert the aiml file to UTF-8 using PSPad, also specifying UTF-8 in the <?xml?> header, but the problem persist. When I say "Olá" (this is an atomic pattern for the bot - <pattern>OLÁ</pattern>), I have the following log in the console:
[2007-03-13 08:54:02,224] WARN: No match found for path "Ol ".

So it seems that the á character is being replaced, and the pattern doesn't match. Also the reply to the question "tempo" gets the following answer:

"YourBot> O tempo Ú tudo a todo momento e em nenhum lugar."

The Ú character should be the é character.

Do you have any other tip? How can I be shure is the aiml file is really converted to utf-8 encoding? I'm using ProgramD 4.6, pre-compiled, in windows with java 1.5.0_06 SDK.

Thanks in advance

Fernando

Noel Bush wrote:

Hi Fernando,

You should be able to use any encoding you want to.  The key, however,
is to make sure that you are actually saving the file in the encoding
you specify in the <?xml?> processing instruction.  Just changing it to
say "UTF-8" or "ISO-8859-1" or "KOI8-R" or whatever will not change the
actual encoding of the file.  This is something you need to control from
your text editor.  It can be tricky to deal with this issue, since
there's no way to just look at a file and tell what charset/encoding it
is using.

However, if you do match up the declared encoding with the actual
encoding, I think you will find that Program D works properly.  At
least, it has done well with all character sets / encodings that I've
tried it with.

Noel

fgomes wrote:
> I'm trying to use the latest version of programd with aiml files with
> portuguese characters, and it don't understand the  patterns with foreign
> characters, and also reply with bad character encoding. Is it necessary to
> configure the character encoding somewhere? I'm using <?xml version="1.0"
> encoding="UTF-8"?> in the aiml files, also tried with <?xml version="1.0"
> encoding="ISO-8859-1"?>, both with the same result.
> I have similar problems also with pyAIML, so it can be a bigger problem :-(
> Thanks in advance for all your help!
>
> Best regards
>
> Fernando
_______________________________________________
programd mailing list
[e-mail]
http://aitools.org/mailman/listinfo/programd

[Noel Bush]

You might want to try figuring out what the default encoding is on your
platform, and indicating that in the xml pi.  It might be something like
"Windows-1251", for instance.

I am not familiar with PSPad.  There isn't a definitive way to determine
the encoding in use in a text file, but if you can try to use something
like iconv you can try converting the file from what you think is its
current encoding to something else, say UTF-8, and see if the result
looks right when you load that in an editor.

fgomes wrote:

> Hi Noel, thanks for the reply!
>
> I've tried to convert the aiml file to UTF-8 using PSPad, also specifying
> UTF-8 in the <?xml?> header, but the problem persist. When I say "Olá" (this
> is an atomic pattern for the bot - <pattern>OLÁ</pattern>), I have the
> following log in the console:
> [2007-03-13 08:54:02,224] WARN: No match found for path "Ol ".
>
> So it seems that the á character is being replaced, and the pattern doesn't
> match. Also the reply to the question "tempo" gets the following answer:
>
> "YourBot> O tempo Ú tudo a todo momento e em nenhum lugar."
>
> The Ú character should be the é character.
>
> Do you have any other tip? How can I be shure is the aiml file is really
> converted to utf-8 encoding? I'm using ProgramD 4.6, pre-compiled, in
> windows with java 1.5.0_06 SDK.
>
> Thanks in advance
>
> Fernando
>
> Noel Bush wrote:
>> Hi Fernando,
>>
>> You should be able to use any encoding you want to.  The key, however,
>> is to make sure that you are actually saving the file in the encoding
>> you specify in the <?xml?> processing instruction.  Just changing it to
>> say "UTF-8" or "ISO-8859-1" or "KOI8-R" or whatever will not change the
>> actual encoding of the file.  This is something you need to control from
>> your text editor.  It can be tricky to deal with this issue, since
>> there's no way to just look at a file and tell what charset/encoding it
>> is using.
>>
>> However, if you do match up the declared encoding with the actual
>> encoding, I think you will find that Program D works properly.  At
>> least, it has done well with all character sets / encodings that I've
>> tried it with.
>>
>> Noel
>>
>> fgomes wrote:
>>> I'm trying to use the latest version of programd with aiml files with
>>> portuguese characters, and it don't understand the  patterns with foreign
>>> characters, and also reply with bad character encoding. Is it necessary
>>> to
>>> configure the character encoding somewhere? I'm using <?xml version="1.0"
>>> encoding="UTF-8"?> in the aiml files, also tried with <?xml version="1.0"
>>> encoding="ISO-8859-1"?>, both with the same result.
>>> I have similar problems also with pyAIML, so it can be a bigger problem
>>> :-(
>>> Thanks in advance for all your help!
>>>
>>> Best regards
>>>
>>> Fernando
>> _______________________________________________
>> programd mailing list
>> programd@...
>> http://aitools.org/mailman/listinfo/programd
>>
>>
>

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[fgomes]

On Tuesday 13 March 2007 09:03, fgomes wrote:

Just to update my previous message, this seems to be some windows related
problem, I put the same aiml files on a linux PC running the same version of
ProgramD and it works without any encoding problem... There is an adittional
diference, I'm using Java 1.6 on the linux machine and 1.5 on the Windows XP
machine... At least I have the bot talking ;-)

Fernando

> Hi Noel, thanks for the reply!
>
> I've tried to convert the aiml file to UTF-8 using PSPad, also specifying
> UTF-8 in the <?xml?> header, but the problem persist. When I say "Olá"
> (this is an atomic pattern for the bot - <pattern>OLÁ</pattern>), I have
> the following log in the console:
> [2007-03-13 08:54:02,224] WARN: No match found for path "Ol ".
>
> So it seems that the á character is being replaced, and the pattern doesn't
> match. Also the reply to the question "tempo" gets the following answer:
>
> "YourBot> O tempo Ú tudo a todo momento e em nenhum lugar."
>
> The Ú character should be the é character.
>
> Do you have any other tip? How can I be shure is the aiml file is really
> converted to utf-8 encoding? I'm using ProgramD 4.6, pre-compiled, in
> windows with java 1.5.0_06 SDK.
>
> Thanks in advance
>
> Fernando
>
> Noel Bush wrote:
> > Hi Fernando,
> >
> > You should be able to use any encoding you want to.  The key, however,
> > is to make sure that you are actually saving the file in the encoding
> > you specify in the <?xml?> processing instruction.  Just changing it to
> > say "UTF-8" or "ISO-8859-1" or "KOI8-R" or whatever will not change the
> > actual encoding of the file.  This is something you need to control from
> > your text editor.  It can be tricky to deal with this issue, since
> > there's no way to just look at a file and tell what charset/encoding it
> > is using.
> >
> > However, if you do match up the declared encoding with the actual
> > encoding, I think you will find that Program D works properly.  At
> > least, it has done well with all character sets / encodings that I've
> > tried it with.
> >
> > Noel
> >
> > fgomes wrote:
> >> I'm trying to use the latest version of programd with aiml files with
> >> portuguese characters, and it don't understand the  patterns with
> >> foreign characters, and also reply with bad character encoding. Is it
> >> necessary to
> >> configure the character encoding somewhere? I'm using <?xml
> >> version="1.0" encoding="UTF-8"?> in the aiml files, also tried with
> >> <?xml version="1.0" encoding="ISO-8859-1"?>, both with the same result.
> >> I have similar problems also with pyAIML, so it can be a bigger problem
> >>
> >> :-(
> >>
> >> Thanks in advance for all your help!
> >>
> >> Best regards
> >>
> >> Fernando
> >
> > _______________________________________________
> > programd mailing list
> > programd@...
> > http://aitools.org/mailman/listinfo/programd

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd

[Noel Bush]

Yes, Linux uses UTF-8 everywhere by default, so it's a lot easier to
deal with this kind of stuff.  If you are using a localized version of
Windows, it's going to default to some Microsoft character set for that
language, which may be something like what I mentioned below.

Fernando Gomes wrote:

> On Tuesday 13 March 2007 09:03, fgomes wrote:
>
> Just to update my previous message, this seems to be some windows related
> problem, I put the same aiml files on a linux PC running the same version of
> ProgramD and it works without any encoding problem... There is an adittional
> diference, I'm using Java 1.6 on the linux machine and 1.5 on the Windows XP
> machine... At least I have the bot talking ;-)
>
> Fernando
>
>> Hi Noel, thanks for the reply!
>>
>> I've tried to convert the aiml file to UTF-8 using PSPad, also specifying
>> UTF-8 in the <?xml?> header, but the problem persist. When I say "Olá"
>> (this is an atomic pattern for the bot - <pattern>OLÁ</pattern>), I have
>> the following log in the console:
>> [2007-03-13 08:54:02,224] WARN: No match found for path "Ol ".
>>
>> So it seems that the á character is being replaced, and the pattern doesn't
>> match. Also the reply to the question "tempo" gets the following answer:
>>
>> "YourBot> O tempo Ú tudo a todo momento e em nenhum lugar."
>>
>> The Ú character should be the é character.
>>
>> Do you have any other tip? How can I be shure is the aiml file is really
>> converted to utf-8 encoding? I'm using ProgramD 4.6, pre-compiled, in
>> windows with java 1.5.0_06 SDK.
>>
>> Thanks in advance
>>
>> Fernando
>>
>> Noel Bush wrote:
>>> Hi Fernando,
>>>
>>> You should be able to use any encoding you want to.  The key, however,
>>> is to make sure that you are actually saving the file in the encoding
>>> you specify in the <?xml?> processing instruction.  Just changing it to
>>> say "UTF-8" or "ISO-8859-1" or "KOI8-R" or whatever will not change the
>>> actual encoding of the file.  This is something you need to control from
>>> your text editor.  It can be tricky to deal with this issue, since
>>> there's no way to just look at a file and tell what charset/encoding it
>>> is using.
>>>
>>> However, if you do match up the declared encoding with the actual
>>> encoding, I think you will find that Program D works properly.  At
>>> least, it has done well with all character sets / encodings that I've
>>> tried it with.
>>>
>>> Noel
>>>
>>> fgomes wrote:
>>>> I'm trying to use the latest version of programd with aiml files with
>>>> portuguese characters, and it don't understand the  patterns with
>>>> foreign characters, and also reply with bad character encoding. Is it
>>>> necessary to
>>>> configure the character encoding somewhere? I'm using <?xml
>>>> version="1.0" encoding="UTF-8"?> in the aiml files, also tried with
>>>> <?xml version="1.0" encoding="ISO-8859-1"?>, both with the same result.
>>>> I have similar problems also with pyAIML, so it can be a bigger problem
>>>>
>>>> :-(
>>>>
>>>> Thanks in advance for all your help!
>>>>
>>>> Best regards
>>>>
>>>> Fernando
>>> _______________________________________________
>>> programd mailing list
>>> programd@...
>>> http://aitools.org/mailman/listinfo/programd
>
> _______________________________________________
> programd mailing list
> programd@...
> http://aitools.org/mailman/listinfo/programd

_______________________________________________
programd mailing list
programd@...
http://aitools.org/mailman/listinfo/programd