Test Simplified Chinese translation in UTF-8 against SM-1.4.15-rc1

[Miro Konecny]

>> I know that you, Tomas, have had some concerns about why we can't go
>> all UTF-8 for the web page displays and outgoing mails, but it also
>> solves some other problems such as writing mails in more than one
>> language. I'm still for the idea to convert all
>> translations to UTF-8 at once.
>
>I didn't have concerns about going to utf-8 and I was running utf-8
>version of SquirrelMail since 1.2.something. I have only left that
>decision to translators instead of enforcing it myself. Romanian, Slovak
>and Chinese translators showed you and Fedora how to switch SquirrelMail
>to utf-8.

I'd strongly recommend to go ahead and convert all translations to UTF-8.
Since it wasn't enforced at the squirrelmail project level, various ad-hoc
solutions are being implemented at other levels. For instance, many Linux
distributions convert translations themselves and release SM packages
fully converted to UTF-8. There are various HOWTOs and/or scripts on
the web which aim at the same direction, but their quality is sometimes
very questionable and they might divert users into wrong solutions.

Now, the sad part: converted packages can write/reply/forward all emails
regardless of charset and language. Standard SM packages only work
in one legacy charset per language, which is sometimes not enough even
for national email. In short - the standard version has problems with
replies to messages from converted SMs, the opposite situation works
fine.

I don't think there are any concerns with going to utf-8 - our experience
shows that it works much better than legacy iso-8859-2 charset.

Thus I believe it would make very much sense to get rid of all those
ad-hoc solutions and cleanly fix the problem at the squirrelmail project level.

Regards,
Miro







-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Fredrik Jervfors-4]

>> I know that you, Tomas, have had some concerns about why we can't go
>> all UTF-8 for the web page displays and outgoing mails, but it also
>> solves some other problems such as writing mails in more than one
>> language. I'm still for the idea to convert all translations to UTF-8 at
>> once.
>
> I didn't have concerns about going to utf-8 and I was running utf-8
> version of SquirrelMail since 1.2.something. I have only left that
> decision to translators instead of enforcing it myself.

I see. I think that we shouldn't leave that decision to the translators,
but make it a rule instead. This of course means to update the
documentation and the existing translations. I'll let the World know
before making such changes though, and give people a possibility to state
opinions about it as well.

> Romanian, Slovak and Chinese translators showed you and Fedora how to
> switch SquirrelMail to utf-8.

I did those changes ("setup.php" etc.) myself per request from those teams
(they didn't provide any patches), so I guess I showed them. ;)

> Chinese issues are caused by:
> 1. Option to translate special folder names
> (http://www.topolis.lt/screen-shots/folders-i18n.png) that I have opposed
> but still implemented in 1.5.x.

I understand that problem, i.e. that the IMAP specification only allows
translation of the INBOX and conflicts may occur if other folders are
translated as well. I guess I should take a look at the SquirrelMail 1.5
code for translating other folders and if needed update it. There are many
parameters to consider with that kind of functionality in the IMAP client.

> 2. Broken mime messages. Most likely caused
> by emails generated in Foxmail mailer, if Chinese spammers are using it
> and not forging Foxmail user-agent string. Single part emails can be
> fixed. Some plugin just have to modify message cache and sync charset
> headers same way html_mail plugin syncs broken text/html headers. And some
> SquirrelMail scripts will have to use message cache instead of refetching
> message information from IMAP.

It's a bit complex, but not impossible, to "fix" mails from broken
mailers. (In a perfect world workarounds like that shouldn't be needed,
but it might be a tad bit unrealistic to fix all broken mailers in the
World.)

> 3. Broken special folder names are most likely caused by admin setting
> special folders incorrectly or lack of mbstring support in older PHP setup
> and older SquirrelMail installation.

This I don't understand right now, but maybe if I'll dig into the code and
have a look around.

> Frank's phpinfo() output shows that his mbstring settings are not broken.
> His PHP 5.1.6 setup should have all mbstring languages enabled and
> gb2312 should be supported.

Good for him. :)

Sincerely,
Fredrik

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Fredrik Jervfors-4]

>>> I know that you, Tomas, have had some concerns about why we can't go
>>> all UTF-8 for the web page displays and outgoing mails, but it also
>>> solves some other problems such as writing mails in more than one
>>> language. I'm still for the idea to convert all translations to UTF-8
>>> at once.
>>
>> I didn't have concerns about going to utf-8 and I was running utf-8
>> version of SquirrelMail since 1.2.something. I have only left that
>> decision to translators instead of enforcing it myself. Romanian,
>> Slovak
>> and Chinese translators showed you and Fedora how to switch SquirrelMail
>> to utf-8.
>
> I'd strongly recommend to go ahead and convert all translations to UTF-8.
> Since it wasn't enforced at the squirrelmail project level, various
> ad-hoc solutions are being implemented at other levels. For instance, many
> Linux distributions convert translations themselves and release SM
> packages fully converted to UTF-8. There are various HOWTOs and/or scripts
> on the web which aim at the same direction, but their quality is sometimes
> very questionable and they might divert users into wrong solutions.
>
> Now, the sad part: converted packages can write/reply/forward all emails
> regardless of charset and language. Standard SM packages only work in one
> legacy charset per language, which is sometimes not enough even for
> national email. In short - the standard version has problems with replies
> to messages from converted SMs, the opposite situation works fine.
>
> I don't think there are any concerns with going to utf-8 - our experience
> shows that it works much better than legacy iso-8859-2 charset.
>
> Thus I believe it would make very much sense to get rid of all those
> ad-hoc solutions and cleanly fix the problem at the squirrelmail project
> level.

I agree, but right now the SquirrelMail's Project's policy is what it is,
i.e. that it's up to the translators. I'm all for changing it though, but
in that case I'll start a discussion about that in a new thread. (See my
reply to Tomas in this thread as well.)

Sincerely,
Fredrik

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Miro Konecny]

>> What kind of IMAP server do you use ?
>> Are you able to create any folders with Chinese name - i.e. other
>> than trash/sent/draft etc?
>
>It's dovecot-1.0-1.2.rc15.el5. I can create other folders/subfolders
>in Chinese without problem. They're displayed as expected.

Hmm, it seems that within config.php you need to use folder names
encoded in utf7-imap.

Use this link to get utf7-imap equivalents of your chinese name for Sent/Draft/Trash:

http://www.mail-archive.com/squirrelmail-users@.../msg26395/utf7-imap.php

Then paste the result into config.php  - it will display correctly.

Regards,
Miro




-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Frank Wang-3]

>>> What kind of IMAP server do you use ?

>>> Are you able to create any folders with Chinese name - i.e. other
>>> than trash/sent/draft etc?
>>
>>It's dovecot-1.0-1.2.rc15.el5. I can create other folders/subfolders
>>in Chinese without problem. They're displayed as expected.
>
> Hmm, it seems that within config.php you need to use folder names
> encoded in utf7-imap.
>
> Use this link to get utf7-imap equivalents of your chinese name for
> Sent/Draft/Trash:
>
> http://www.mail-archive.com/squirrelmail-users@.../msg26395/utf7-imap.php
>
> Then paste the result into config.php  - it will display correctly.

It works! Now I think utf-8 version is better than the gb2312 one from
Chinese translation prospective. I agree utf-8 will bring other benefit as
well, but I still need to do a little bit more interoperability test with
the popular mail providers here to make sure, as the users are most likely
to interact with them other than the one I've tested. I'll finish that
ASAP.
Thanks!

Regards,
Frank




-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Frank Wang-3]

> It works! Now I think utf-8 version is better than the gb2312 one from
> Chinese translation prospective. I agree utf-8 will bring other benefit as
> well, but I still need to do a little bit more interoperability test with
> the popular mail providers here to make sure, as the users are most likely
> to interact with them other than the one I've tested. I'll finish that
> ASAP.
I've finished some test against 9 popular email providers in China.
They're 126.com, 163.com, sina.com, suhu.com, yahoo.cn, tom.cn, yeah.net,
gmail.com and qq.com. Unfortunately, nearly half of them has difficulty
interact with UTF-8 SM-1.4.15-rc1. They're sina.com, 163.com, yeah.net and
qq.com. The mails from them are all displayed corrupted in the new SM. I
noticed that their subject in headers are containing '=?gbk?' in common,
and their subject/body/attachment can be displayed correctly if browser
encoding set to gb2312/gbk/gb18030. This probably means gbk encoding is
not converted to utf-8 correctly. For yahoo.cn,
only the Chinese attachment name sent to SM is corrupted, the body and
subject are alright. I'll attach two sample headers here.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Miro Konecny]

>I've finished some test against 9 popular email providers in China.
>They're 126.com, 163.com, sina.com, suhu.com, yahoo.cn, tom.cn, yeah.net,
>gmail.com and qq.com. Unfortunately, nearly half of them has difficulty
>interact with UTF-8 SM-1.4.15-rc1. They're sina.com, 163.com, yeah.net and
>qq.com. The mails from them are all displayed corrupted in the new SM. I
>noticed that their subject in headers are containing '=?gbk?' in common,
>and their subject/body/attachment can be displayed correctly if browser
>encoding set to gb2312/gbk/gb18030. This probably means gbk encoding is
>not converted to utf-8 correctly.

Yes - the decode library does not have a file for decoding gbk

Assuming gb18030 is similar to gbk (is that true?) you can make it work
using the following hack:

go into directory   functions/decode
copy gb18030.php  into  gbk.php
edit gbk.php and replace text "function charset_decode_gb18030"  
    with "function charset_decode_gbk"

With this modification, your sample subject headers display correctly.

Anyway, the solid fix would be to add sane gbk.php into the decode library...

Regards,
Miro


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Fredrik Jervfors-4]

>> I've finished some test against 9 popular email providers in China.
>> They're 126.com, 163.com, sina.com, suhu.com, yahoo.cn, tom.cn,
>> yeah.net, gmail.com and qq.com. Unfortunately, nearly half of them has
>> difficulty interact with UTF-8 SM-1.4.15-rc1. They're sina.com, 163.com,
>> yeah.net and qq.com. The mails from them are all displayed corrupted in
>> the new SM. I noticed that their subject in headers are containing
>> '=?gbk?' in common,
>> and their subject/body/attachment can be displayed correctly if browser
>> encoding set to gb2312/gbk/gb18030. This probably means gbk encoding is
>>  not converted to utf-8 correctly.
>
> Yes - the decode library does not have a file for decoding gbk
>
> Assuming gb18030 is similar to gbk (is that true?)

As far as I can tell, GBK is a sub-set of GB18030.

> you can make it work
> using the following hack:
>
> go into directory   functions/decode copy gb18030.php  into  gbk.php edit
> gbk.php and replace text "function charset_decode_gb18030" with "function
> charset_decode_gbk"
>
> With this modification, your sample subject headers display correctly.
>
> Anyway, the solid fix would be to add sane gbk.php into the decode
> library...

Since GB18030 covers GBK, the solution you propose should work. I'll
implement it in the repository and get back to the list when it's done.

Sincerely,
Fredrik

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n

[Miro Konecny]

>Since GB18030 covers GBK, the solution you propose should work. I'll
>implement it in the repository and get back to the list when it's done.

Hi Fredrik,

  we've discussed this with Frank off-list and he stated that there are
some differences:

"The mapping table from GB18030-2000 to Unicode is backward-compatible with the
mapping table from GB 2312-1980 to Unicode, however, the GBK to Unicode table has
a few differences. GBK contains characters which were not defined in Unicode 2.1,
but were added in later versions of Unicode."

http://developers.sun.com/dev/gadc/technicalpublications/articles/gb18030.html

Anyway, both recode and iconv support GBK natively, so we've
tested a cleaner solution - calling those with charset "GBK". I've attached
the file gbk.php which implements this in the other thread, so just take
that one, check for any errors and if you find it OK please commit that.

Regards,
Miro



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
-----
squirrelmail-i18n mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
Information about translations: http://squirrelmail.org/wiki/LanguageTranslation
List address: squirrelmail-i18n@...
List archives: http://news.gmane.org/gmane.mail.squirrelmail.internationalization
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-i18n