Source L10n for extensions (ChatZilla, venkman and others)

[Robert Kaiser]

Hi,

[Posted to dev-l10n as well as ChatZilla and venkman groups and
owners/peers; please follow up to the dev-l10n group/list with
discussion, reactions and followups, so everyone affected can read all
of this discussion in one single place]

You might have seen my recent rush for ChatZilla and venkman source
L10n, and I admit I probably was too eager to get something going that I
stepped on a few people's toes. I want to start a discussion with this
post now to get this all solved the hopefully right way, with
participation of everyone directly affected by this.


I've been informed that what we need for any addition to the L10n CVS
mechanism is a proposal on what's expected to work, and what is not, and
who's expected to do what (Axel called that "concrete agreement" or
"concrete contract" in bug comments).
This means that everyone should know how it affects 'hourly' builds,
nightlies, release builds, and language packs and what release manager,
extension author, and localizers are supposed to do in the process for
the added locale strings of those extensions - in this case ChatZilla.
The contract/agreement should also tell what is supposed to turn the
build red, what turns the build green, when something gets shipped, when
not - everything one needs to know about how our processes will/do work.


I will NOT post this contract here because we obviously need the
extension author(s)/owner(s) to agree. I'm posting my intentions here
and a way how I propose we can move forward on this. This should be a
base for discussion, not something anyone reasonable has already agreed
with - just to make this clear from the beginning.


My main target here is to enable us to ship SeaMonkey 2 in a fully
localized fashion, including ChatZilla and possibly venkman, but not
force localizers to provide those extensions in their language.
As DOM inspector and reporter have other ways of providing L10n which
work in a way that we can ship in a way where a user get them to display
in their language, my planning has no priority for them but should
include figuring out a system that could be applied to them later
(possibly only after the 1.9 cycle that SeaMonkey 2 is based on).

So, the highest priority for me is to get localized SeaMonkey 2 (trunk)
working with optionally localized ChatZilla (and possibly venkman).
If it's easy and nice to do, we should figure out and possibly enable
some way of L10n for XPI packages for those extensions as well as
SeaMonkey builds created with --enable-ui-locale and langpacks.

Localized SeaMonkey 2 is created via repackaging, which takes an en-US
build and replaces its various localizable content with resources for a
different language. Unfortunately we don't have machines running yet
that would do this in an automated fashion, so my first attempt was for
getting --enable-ui-locale working and I figured out a way that would
make some kind of localized XPIs possible.
While the locations of the en-US and actual L10n files are the same for
all those ways of L10n, the Makefile and XPI creation changes might be
more or less different. In this way, my first take on this problem
probably wasn't ideal - but at least the locations of the ChatZilla
files in CVS are correct. So far, so good.
I'll probably work on getting one of my private trunk tinderboxen to run
that repackaging stuff so I can get some idea of what changes we need to
get this working correctly.

I don't know for sure yet, but what we might need there is to have
localizations for the extensions in different files from the extension,
or at least have the non-en-US localizations separately from the main
extension.
Interestingly, ChatZilla currently distributes separate language XPIs to
supplement their extension with add-on locales. They even invented a
locale versioning method to make sure users are having the correct
language pack. Gecko-1.9-based extension manager now can cope with that
itself due to supporting extension dependencies. And this process has
its L10n files separate from the extension itself, which is what
repackaging and even AUS2 might want or prefer to work correctly.

So, to me, it seems that the way to go for those extensions is something
Axel called "dependent langpacks" - which is more or less what ChatZilla
already uses, but with extension dependency added into it (which is
ignored on older applications, so ChatZilla langpacks can easily add
them and still be compatible with older apps).
What this means is that the original extension packages will probably
all include the en-US locale as previously, and additionally we'll have
every existing additional language as a separate extension (both in
SeaMonkey and as XPIs).

How does that idea/proposal sound?
Something you feel comfortable with having?
If not, what alternatives do you see?
What additional processes do we need to go with that?
How can we arrive on a contract/agreement like described in the first
section of this post, maybe based on this solution?

If we arrive at a point that extension owners/peers and L10n people feel
comfortable enough with, I'd be happy to spent some time looking into
how we can make it a reality and work on the needed changes to
Makefiles, etc. - but first, let's get to a point where we agree on how
to do it.
Sorry again for being to eager to get something done in my first
attempt. I felt I needed to get something moving before some doors might
close and we might not get a solution in the 1.9 cycle which we need for
SeaMonkey2 though. I hope we can now come to a solution that will work
for all of us and make our processes smoother all in all.

Robert Kaiser

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Robert Kaiser]

Gijs Kruitbosch wrote:
> So, in response to KaiRo's proposals, there's been some discussion,
> and it seems to me that for ChatZilla, the best option is what's
> described below.

 From what I see, you, Axel and me seem to agree on the solution, which
is very nice to see :)

> - It *could* have a dependency relation set to the ChatZilla
> extension, ensuring that extension updates and the like will go
> smoothly - IF AND ONLY IF this does not break earlier versions of
> Firefox (ie, 1.0, which didn't know about extension dependencies). It
> seems that it doesn't break such versions, but it'd be nice to be
> sure.

I was assured by people who should know (Mossop) that it works this way.
But I agree with you on that we should test this (actually everything we
are doing here) before making an actual public release of that rework.

> If you would build ChatZilla as part of SeaMonkey, I'm not really sure
> what goes on in the deep bowels of the Mozilla build system. But, as
> far as I can tell, this should still use approximately the same
> process, in the sense that there is a separate "add-on" which contains
> only the localization, and a separate GUID and authors/contributors.

Right. Ideally, I'd like the build system to use/generate packages that
are the same as what is being released elsewhere.


Robert Kaiser

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Robert Kaiser]

Rimas Kudelis wrote:
> However, I really don't understand what the problem is in simply
> packaging all available translations together with the extension, like
> it is done with Lightning. Would anyone enlighten me on that?

It just gets _very_ huge if you have a bigger number of localizations.
Lightning or DOMI could be sooo much smaller in download if they would
not need to do that.

And actually, I don't want to take the penalty of shipping every
available localizations to all users of SeaMonkey 2, as we already have
enough people complaining about the size of SeaMonkey 1.1.x as-is.
My primary concern is still the suite, but I think it would be good to
be using more or less the same solution for both the in-suite extensions
and the separate XPIs.

Perhaps we can find a solution of installing both an extension and its
fitting language pack at once through AMO - I think that would be the
nicest solution.

Robert Kaiser

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Gijs Kruitbosch ("Hannibal")]

Robert Kaiser wrote:

> Rimas Kudelis wrote:
>> However, I really don't understand what the problem is in simply
>> packaging all available translations together with the extension, like
>> it is done with Lightning. Would anyone enlighten me on that?
>
> It just gets _very_ huge if you have a bigger number of localizations.
> Lightning or DOMI could be sooo much smaller in download if they would
> not need to do that.
>
> And actually, I don't want to take the penalty of shipping every
> available localizations to all users of SeaMonkey 2, as we already have
> enough people complaining about the size of SeaMonkey 1.1.x as-is.
> My primary concern is still the suite, but I think it would be good to
> be using more or less the same solution for both the in-suite extensions
> and the separate XPIs.
>
> Perhaps we can find a solution of installing both an extension and its
> fitting language pack at once through AMO - I think that would be the
> nicest solution.
>
> Robert Kaiser
>

"What he said."

The trade-off for adding all these localization is not trivial - for
ChatZilla's jar file, the en-US locale is about 10-15% of the total
jarfile. Which means if you add 10 or so locales, half the jar (and
hence xpi) is localizations (90% of which the user will not need).
That's not really reasonable.

Only installing some of them from one single download is not possible at
this time, to my knowledge. While you can put multiple add-ons in one
xpi, I don't think that the current add-ons UI lets you pick which of
those you want to install, so doing that would not solve our problem. It
is probably too late in the game for the 1.9 cycle to add UI for that.
And besides, in those cases we would still need the separate
l10n-specific xpi files which were proposed in my earlier post (so your
idea could be a later addition to this).


As far as AMO is concerned - if we could find a solution for that stuff,
that'd be great. I'm not sure how easy it would be - you would have to
ask the AMO folks.

~ Gijs
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Benjamin Smedberg]

Gijs Kruitbosch wrote:

>> Perhaps we can find a solution of installing both an extension and its
>> fitting language pack at once through AMO - I think that would be the
>> nicest solution.

This is doable. The extension manager already supports "multi-item packages"
where multiple extensions are bundled into a single XPI. Really all we need
at that point is to have AMO generate a multi-item package of CZ-core plus
the relevant CZ-langpack.

--BDS
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Robert Kaiser]

Benjamin Smedberg wrote:

> Gijs Kruitbosch wrote:
>
>>> Perhaps we can find a solution of installing both an extension and its
>>> fitting language pack at once through AMO - I think that would be the
>>> nicest solution.
>
> This is doable. The extension manager already supports "multi-item packages"
> where multiple extensions are bundled into a single XPI. Really all we need
> at that point is to have AMO generate a multi-item package of CZ-core plus
> the relevant CZ-langpack.

That really sounds good - and the good thing is that AMO is not bound by
1.9 build cycle freezes :)

As CZ still supports a range of older apps as hosts, we just need to
care who does understand that concept and who doesn't, but that can
probably be done on the AMO side of the medal.

Robert Kaiser

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Rimas Kudelis-3]

Gijs Kruitbosch rašė:
> The trade-off for adding all these localization is not trivial - for
> ChatZilla's jar file, the en-US locale is about 10-15% of the total
> jarfile. Which means if you add 10 or so locales, half the jar (and
> hence xpi) is localizations (90% of which the user will not need).
> That's not really reasonable.

First, I was talking about standalone extension here. With SeaMonkey
it's easier, as you know a priori which locale the user wants to use for
Cz when he downloads SM. So, for the SM case, I think it's absolutely OK
to only include one locale.

> Only installing some of them from one single download is not possible at
> this time, to my knowledge. While you can put multiple add-ons in one
> xpi, I don't think that the current add-ons UI lets you pick which of
> those you want to install, so doing that would not solve our problem. It
> is probably too late in the game for the 1.9 cycle to add UI for that.

Is it?

> And besides, in those cases we would still need the separate
> l10n-specific xpi files which were proposed in my earlier post (so your
> idea could be a later addition to this).

Yes. What I'm concerned about is not the way extensions are packaged,
but rather the way they are distributed. I believe that the less
seeking/reading the user has to do to download what he wants, the better.

> As far as AMO is concerned - if we could find a solution for that stuff,
> that'd be great. I'm not sure how easy it would be - you would have to
> ask the AMO folks.

One thing that I'm not sure about is how AMO would find out which locale
I actually want. Will it ask me, or will it make some decisions based on
my IP or my UA language, or what?

RQ
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Axel Hecht]

Benjamin Smedberg wrote:

> Gijs Kruitbosch wrote:
>
>>> Perhaps we can find a solution of installing both an extension and its
>>> fitting language pack at once through AMO - I think that would be the
>>> nicest solution.
>
> This is doable. The extension manager already supports "multi-item packages"
> where multiple extensions are bundled into a single XPI. Really all we need
> at that point is to have AMO generate a multi-item package of CZ-core plus
> the relevant CZ-langpack.

Or we make AMO create multi-item install links, InstallTrigger to the
victory. (Can I run, pretty please? I know I said the xpevil word.)
Which would need to be tested, I don't know how that mixes with
dependencies.

Of course, on top of that, the actual discovery and exposure of
localizations of extensions could use some improvement.

Axel
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Gijs Kruitbosch ("Hannibal")]

Axel Hecht wrote:

> Benjamin Smedberg wrote:
>> Gijs Kruitbosch wrote:
>>
>>>> Perhaps we can find a solution of installing both an extension and its
>>>> fitting language pack at once through AMO - I think that would be the
>>>> nicest solution.
>>
>> This is doable. The extension manager already supports "multi-item
>> packages"
>> where multiple extensions are bundled into a single XPI. Really all we
>> need
>> at that point is to have AMO generate a multi-item package of CZ-core
>> plus
>> the relevant CZ-langpack.
>
> Or we make AMO create multi-item install links, InstallTrigger to the
> victory. (Can I run, pretty please? I know I said the xpevil word.)
> Which would need to be tested, I don't know how that mixes with
> dependencies.
>
> Of course, on top of that, the actual discovery and exposure of
> localizations of extensions could use some improvement.
>
> Axel

Yeah. Apparently I didn't respond to Benjamin's point - I'm very much
aware there are various technical solutions for presenting a locale
extension along with the extension itself - but the real problem, as far
as I'm concerned, is if/how AMO is going to find these add-ons.

~ Gijs
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Axel Hecht]

<...>, zip, basically, yes.

Let's make a post on which contracts we currently have. I might be
biased (I am), but I'm missing some release-management parts in the
discussion so far.

With domi and reporter, we currently expose the "contract" via client.mk.

reporter is in LOCALES_browser, and it's not optional.

domi is not in LOCALES_browser, and it's so optional it hurts.

What do these contracts fix? Here are the questions that I can answer,
or make up answers for, depending on client.mk:

- Does this thing need to get localized?
- Who's responsible for bustages?
- Who's responsible for QA?
- Who's responsible for sign-off for shipping?

REPORTER-style contract

reporter is not really an extension in its own right anyway, we really
just use it as an optional package. Thus it's not that bad that the
extensions contract ID isn't perfect.
reporter is part of Firefox, and it's up to the localizer to make sure
it works in her or his localization. The localizer needs to fix the
bustages, needs to organize QA, and signs-off on the localization for
release as part of the sign-off process for the complete localization.

DOMI-style contract

DOMI lives as a regular extension, built as shipped as one big blob. For
the most part, it's up to the DOMI owners to make sure that that
extension doesn't bust Firefox. Localizers sign off on their
localization at the time they make a change and request to be added
back. Not shipping a DOMI localization at release time requires an
explicit action to pull a localization, whereas in the reporter-style,
the localization would just not have been added to the shipped-locales file.


One question that we could ask in addition would be:

- When is a localization built?

Like, we could only start to build localizations of an extension after
it is frozen. That would take out the part of "oh, you changed
something, and now the localization needs to... what?" We'd basically
would move, in firefox-speak, move from cvs to shipped-locales, without
bothering about all-locales at any point in time.
That would require a way for localizers to locally test, though, and to
check if something has a bustage.

These are my experiences and thoughts from a more release- and
process-driven perspective.

Axel
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Robert Kaiser]

Axel Hecht wrote:
> <...>, zip, basically, yes.
>
> Let's make a post on which contracts we currently have. I might be
> biased (I am), but I'm missing some release-management parts in the
> discussion so far.

Thanks for bringing those in, I left those out at first because I
thought they probably depend on what the technical solution is - and we
need the module owners in this.

What I'd like to have for SeaMonkey is the following:

> - Does this thing need to get localized?

No, it should be optional.

> - Who's responsible for bustages?
> - Who's responsible for QA?
> - Who's responsible for sign-off for shipping?

Once a localizers opts into doing it and has a first working L10n, I'd
like to hand him over the full responsibility on all of those and the
L10n of the extension will be treated like it was non-optional and a
part of the SeaMonkey L10n for this language. Upon request of the
SeaMonkey L10n owner or the extension L10n owner, the L10n of the
extension can be excluded from builds again.
Sign-off for shipping should be done for full SeaMonkey, which makes it
the responsibility of the SeaMonkey localizer if it's included. I only
want to have one person to talk to when it comes to shipping a SeaMonkey
release. Upon the responsibility of this person, we'd also be able to
exclude the extension L10n from the release for bustage or bad quality.
That's probably also what I had in mind when creating that all-locales
file for ChatZilla.

For shipping the external add-on XPIs, I think the extension owner/peers
need to tell what they have in mind here, the SeaMonkey team of course
can't force a process on them.


So, to take your terms, I basically like to have a reporter-style
contract for the SeaMonkey case with a mechanism for making the
localization of the extension optional and handle it as an extension in
this way.
This is only one side of the medal though, as extension owners want to
independently ship their add-on in different ways, so we need their side
as well and formulate it into a common contract, even if that one might
be two-fold.

Robert Kaiser

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Axel Hecht]

Robert Kaiser wrote:

> Axel Hecht wrote:
>> <...>, zip, basically, yes.
>>
>> Let's make a post on which contracts we currently have. I might be
>> biased (I am), but I'm missing some release-management parts in the
>> discussion so far.
>
> Thanks for bringing those in, I left those out at first because I
> thought they probably depend on what the technical solution is - and we
> need the module owners in this.
>
> What I'd like to have for SeaMonkey is the following:
>
>> - Does this thing need to get localized?
>
> No, it should be optional.
>
>> - Who's responsible for bustages?
>> - Who's responsible for QA?
>> - Who's responsible for sign-off for shipping?
>
> Once a localizers opts into doing it and has a first working L10n, I'd
> like to hand him over the full responsibility on all of those and the
> L10n of the extension will be treated like it was non-optional and a
> part of the SeaMonkey L10n for this language. Upon request of the
> SeaMonkey L10n owner or the extension L10n owner, the L10n of the
> extension can be excluded from builds again.
> Sign-off for shipping should be done for full SeaMonkey, which makes it
> the responsibility of the SeaMonkey localizer if it's included. I only
> want to have one person to talk to when it comes to shipping a SeaMonkey
> release. Upon the responsibility of this person, we'd also be able to
> exclude the extension L10n from the release for bustage or bad quality.
> That's probably also what I had in mind when creating that all-locales
> file for ChatZilla.
>
> For shipping the external add-on XPIs, I think the extension owner/peers
> need to tell what they have in mind here, the SeaMonkey team of course
> can't force a process on them.
>
>
> So, to take your terms, I basically like to have a reporter-style
> contract for the SeaMonkey case with a mechanism for making the
> localization of the extension optional and handle it as an extension in
> this way.
> This is only one side of the medal though, as extension owners want to
> independently ship their add-on in different ways, so we need their side
> as well and formulate it into a common contract, even if that one might
> be two-fold.

I'm not sure if this is right, but let's assume for the moment that it is.

extensions/irc would thus not be part of LOCALES_suite, and localizers
that do want to localize chatzilla had to use MOZ_CO_MODULES to get it?

There would be joint deal between suite/locales/shipped-locales and
extensions/irc/locales/all-locales, too, to formulate the shipping of,
say, chatzilla-ab-CD.jar?

Axel
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Robert Kaiser]

Axel Hecht wrote:
> extensions/irc would thus not be part of LOCALES_suite, and localizers
> that do want to localize chatzilla had to use MOZ_CO_MODULES to get it?

It seems that both currently available options might not be what I
really want to have there :(

> There would be joint deal between suite/locales/shipped-locales and
> extensions/irc/locales/all-locales, too, to formulate the shipping of,
> say, chatzilla-ab-CD.jar?

Possibly. But I haven't even figured out the shipped-locales stuff yet
though... I guess anything present in shipped-locales has to be in the
parallel all-locales anyways, right?

Robert Kaiser

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Benjamin Smedberg]

Gijs Kruitbosch wrote:

> Yeah. Apparently I didn't respond to Benjamin's point - I'm very much
> aware there are various technical solutions for presenting a locale
> extension along with the extension itself - but the real problem, as far
> as I'm concerned, is if/how AMO is going to find these add-ons.

Which add-ons? Extension locale packs are not regular extensions that you
would browse for apart from the extension their localizing... they would be
tightly associated in the database.

--BDS
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Gijs Kruitbosch ("Hannibal")]

Benjamin Smedberg wrote:

> Gijs Kruitbosch wrote:
>
>> Yeah. Apparently I didn't respond to Benjamin's point - I'm very much
>> aware there are various technical solutions for presenting a locale
>> extension along with the extension itself - but the real problem, as far
>> as I'm concerned, is if/how AMO is going to find these add-ons.
>
> Which add-ons? Extension locale packs are not regular extensions that you
> would browse for apart from the extension their localizing... they would be
> tightly associated in the database.
>
> --BDS

Right, but that'd take considerable effort on the AMO part of things, on
something which we're just now cooking up as some kind of "standard" or
whatever you want to call it. (in fact, I'm positive 'standard' is not
the right name, but it's past midnight and my brain is like mashed
potatoes at the moment... sorry about that)

I don't think this will be done by the time Fx 3 releases, for instance
- and possibly not even when SeaMonkey 2 releases. Heck, I'm not even
sure the AMO people would be willing to add all this complexity at all.

And then, I'm not saying that is a Huge Problem (tm), I'm just being
cautious in promising this nice storyline about AMO magically finding
the right bits for you, because I have no clue on AMO's
readiness/willingness/suitableness to actually do this. Maybe they're
wildly enthousiastic about it and it will get implemented within the
next week, maybe they'll stamp the bug as WONTFIX as soon as it gets
filed, maybe it'll grow to be one of those bugs which sits around for
several years before anything substantial happens. I don't want to give
false promises.

~ Gijs
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Robert Kaiser]

Gijs Kruitbosch wrote:
> Benjamin Smedberg wrote:
>> Which add-ons? Extension locale packs are not regular extensions that you
>> would browse for apart from the extension their localizing... they
>> would be
>> tightly associated in the database.
>
> Right, but that'd take considerable effort on the AMO part of things, on
> something which we're just now cooking up as some kind of "standard" or
> whatever you want to call it.

True that it will take some effort for AMO, but you should be aware that
this is actually not a completely new thing we're cooking up, Benjamin
and others had envisioned such a system for some time, that's one reason
why we have those extension dependencies and extension locale packs
already supported in Toolkit 1.9 EM.
Oh, and how do you know that AMO is not going to take the effort of
supporting this?

> I don't think this will be done by the time Fx 3 releases

I think it actually could be done, but we'd need AMO people to tell if
and how it can, it's wron for us to assume they will or will not.
The good thing is that AMO is not bound to Firefox feature and string
freezes in the way the Gecko core is, so they can still work on this
right up until the last week or so before Firefox 3 and even after that
(though I guess for a short time right around the release they will be
overloaded with traffic and don't want additional risk).

Anyways, I think we need to actually have something available they can
use for testing the new feature if we want them to implement it.

Robert Kaiser

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger