Component releases, proposed solution

[Jukka Zitting]

Hi,

There's been a lot of discussion about release strategy and
componentization lately, and I've been trying to figure out how to fit
all the different requirements into a single solution for Jackrabbit
1.5+. Here's what I have in mind...

First, the main requirements for componentization:

* The codebase should be reasonably modular based on exposed
functionality and required dependencies
* Component release cycles should be independent except as constrained
by functional dependencies

But also:

* It should be easy to download (or checkout) and build all of Jackrabbit
* A Jackrabbit installation should have a clear version number that
identifies the exact contents of the installation

To cover all of these requirements I propose:

* We keep a single release cycle for all of Jackrabbit (including a
single trunk,branches,tags structure, single Jira project, etc.)
* Each component within Jackrabbit trunk (and branches) has an
independent version number that is only increased based on actual
changes in the component
* Thus a "Jackrabbit release" contains a collection of different
component versions that are known to work well together
* When making a release, only those components that have been modified
(and thus have a new version number) are deployed to the Maven
repository
* Release notes will contain a list of included component versions and
the changes to each component

The effects of this policy would be:

* Downstream projects like Sling that use direct Maven dependencies to
Jackrabbit components have an easier time tracking actual changes in
the components
* The concepts "Jackrabbit release" and "Jackrabbit version" are still
simple to grasp and use for example when reporting or reproducing
issues
* There's still a single trunk where development of all components occurs

Applied to the Jackrabbit 1.4.x release cycle, this would have given
us the following releases:

* Jackrabbit 1.4.1, including core 1.4.1
* Jackrabbit 1.4.2, including core 1.4.2
* Jackrabbit 1.4.3, including jcr-commons 1.4.1 and jcr-rmi 1.4.1
* Jackrabbit 1.4.4, including core 1.4.3
* Jackrabbit 1.4.5, including core 1.4.4
* Jackrabbit 1.4.6, including core 1.4.5

Also, each release would have contained updated versions of aggregate
components like jackrabbit-webapp and jackrabbit-jca, whose version
numbers would thus follow the top level version number.

WDYT?

BR,

Jukka Zitting

[Alexander Klimetschek]

On Tue, Jul 22, 2008 at 12:51 PM, Jukka Zitting <jukka.zitting@...> wrote:
> Applied to the Jackrabbit 1.4.x release cycle, this would have given
> us the following releases:
>
> * Jackrabbit 1.4.1, including core 1.4.1
> * Jackrabbit 1.4.2, including core 1.4.2
> * Jackrabbit 1.4.3, including jcr-commons 1.4.1 and jcr-rmi 1.4.1
> * Jackrabbit 1.4.4, including core 1.4.3
> * Jackrabbit 1.4.5, including core 1.4.4
> * Jackrabbit 1.4.6, including core 1.4.5

Generally I agree, but I know that something like jackrabbit 1.4.6
containing a 1.4.5 core jar would be very confusing when users report
a problem. Couldn't we make an exception that the most important
component jackrabbit-core always gets the same version number as the
overall release - which would imply that sometimes core gets a version
number increase without an actual code change.

Regards,
Alex

--
Alexander Klimetschek
alexander.klimetschek@...

[Marcel Reutegger]

Hi,

I'm not sure about all the implications such a change has, but I like the idea
to have an overall jackrabbit release version.

It would certainly make downloading and using the latest jackrabbit version a
lot easier.

regards
  marcel

Jukka Zitting wrote:

> Hi,
>
> There's been a lot of discussion about release strategy and
> componentization lately, and I've been trying to figure out how to fit
> all the different requirements into a single solution for Jackrabbit
> 1.5+. Here's what I have in mind...
>
> First, the main requirements for componentization:
>
> * The codebase should be reasonably modular based on exposed
> functionality and required dependencies
> * Component release cycles should be independent except as constrained
> by functional dependencies
>
> But also:
>
> * It should be easy to download (or checkout) and build all of Jackrabbit
> * A Jackrabbit installation should have a clear version number that
> identifies the exact contents of the installation
>
> To cover all of these requirements I propose:
>
> * We keep a single release cycle for all of Jackrabbit (including a
> single trunk,branches,tags structure, single Jira project, etc.)
> * Each component within Jackrabbit trunk (and branches) has an
> independent version number that is only increased based on actual
> changes in the component
> * Thus a "Jackrabbit release" contains a collection of different
> component versions that are known to work well together
> * When making a release, only those components that have been modified
> (and thus have a new version number) are deployed to the Maven
> repository
> * Release notes will contain a list of included component versions and
> the changes to each component
>
> The effects of this policy would be:
>
> * Downstream projects like Sling that use direct Maven dependencies to
> Jackrabbit components have an easier time tracking actual changes in
> the components
> * The concepts "Jackrabbit release" and "Jackrabbit version" are still
> simple to grasp and use for example when reporting or reproducing
> issues
> * There's still a single trunk where development of all components occurs
>
> Applied to the Jackrabbit 1.4.x release cycle, this would have given
> us the following releases:
>
> * Jackrabbit 1.4.1, including core 1.4.1
> * Jackrabbit 1.4.2, including core 1.4.2
> * Jackrabbit 1.4.3, including jcr-commons 1.4.1 and jcr-rmi 1.4.1
> * Jackrabbit 1.4.4, including core 1.4.3
> * Jackrabbit 1.4.5, including core 1.4.4
> * Jackrabbit 1.4.6, including core 1.4.5
>
> Also, each release would have contained updated versions of aggregate
> components like jackrabbit-webapp and jackrabbit-jca, whose version
> numbers would thus follow the top level version number.
>
> WDYT?
>
> BR,
>
> Jukka Zitting
>

[Jukka Zitting]

Hi,

On Tue, Jul 22, 2008 at 3:30 PM, Alexander Klimetschek <aklimets@...> wrote:
> Generally I agree, but I know that something like jackrabbit 1.4.6
> containing a 1.4.5 core jar would be very confusing when users report
> a problem. Couldn't we make an exception that the most important
> component jackrabbit-core always gets the same version number as the
> overall release - which would imply that sometimes core gets a version
> number increase without an actual code change.

I guess we could do that. And in fact in any case the core version
would need to be upgraded whenever any of the core dependencies
(jcr-commons, spi-*, etc.) are modified, so the number of "extra" core
version increases would probably remain quite rare.

BR,

Jukka Zitting

[beddoes]

Hi,

We went down this path in a rather large project and it didn't end up
working out. The follow up on the mailing lists from people who weren't
using a dependency manager and were continually confused by version
numbers was a real pain.

In the end we went back to a single version number for the entire
project and did a synchronized release of the entire thing when enough
significant changes had been accumulated. Naturally some people choose
to run bleeding edge builds of some components to get around their
immediate problems before the next synchronized release.

We use Ivy for dependency management so it makes little difference to us
if each jar is numbered independently or the same as the overall
package, we're still able to pull down what X.Y.Z should be as a whole.

So I personally sit on the fence on this one but thought I'd share an
experience which will hopefully help you guys make up your mind.

Bradley

Jukka Zitting wrote:

> Hi,
>
> On Tue, Jul 22, 2008 at 3:30 PM, Alexander Klimetschek <aklimets@...> wrote:
>> Generally I agree, but I know that something like jackrabbit 1.4.6
>> containing a 1.4.5 core jar would be very confusing when users report
>> a problem. Couldn't we make an exception that the most important
>> component jackrabbit-core always gets the same version number as the
>> overall release - which would imply that sometimes core gets a version
>> number increase without an actual code change.
>
> I guess we could do that. And in fact in any case the core version
> would need to be upgraded whenever any of the core dependencies
> (jcr-commons, spi-*, etc.) are modified, so the number of "extra" core
> version increases would probably remain quite rare.
>
> BR,
>
> Jukka Zitting

[Esteban Franqueiro-3]

>> * Jackrabbit 1.4.1, including core 1.4.1
>> * Jackrabbit 1.4.2, including core 1.4.2
>> * Jackrabbit 1.4.3, including jcr-commons 1.4.1 and jcr-rmi 1.4.1
>> * Jackrabbit 1.4.4, including core 1.4.3
>> * Jackrabbit 1.4.5, including core 1.4.4
>> * Jackrabbit 1.4.6, including core 1.4.5
>
> Generally I agree, but I know that something like jackrabbit 1.4.6
> containing a 1.4.5 core jar would be very confusing when users report
> a problem. Couldn't we make an exception that the most important
> component jackrabbit-core always gets the same version number as the
> overall release - which would imply that sometimes core gets a version
> number increase without an actual code change.

I agree with Alex here. But I think that the Jackrabbit version should be the same as the core
version, so that we should release a new version only when there are changes in core. Why releasing
a whole new version for a simple, low impact change in commons? If a change is big enough that it
warrants a change in versions, it probably also affects core.
My AR$2.
Regards,

Esteban Franqueiro
esteban.franqueiro@...


Notice:  This email message, together with any attachments, may contain information  of  BEA Systems,  Inc.,  its subsidiaries  and  affiliated entities,  that may be confidential,  proprietary,  copyrighted  and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.

[Carsten Ziegeler]

Alexander Klimetschek wrote:

> On Tue, Jul 22, 2008 at 12:51 PM, Jukka Zitting <jukka.zitting@...> wrote:
>> Applied to the Jackrabbit 1.4.x release cycle, this would have given
>> us the following releases:
>>
>> * Jackrabbit 1.4.1, including core 1.4.1
>> * Jackrabbit 1.4.2, including core 1.4.2
>> * Jackrabbit 1.4.3, including jcr-commons 1.4.1 and jcr-rmi 1.4.1
>> * Jackrabbit 1.4.4, including core 1.4.3
>> * Jackrabbit 1.4.5, including core 1.4.4
>> * Jackrabbit 1.4.6, including core 1.4.5
>
> Generally I agree, but I know that something like jackrabbit 1.4.6
> containing a 1.4.5 core jar would be very confusing when users report
> a problem. Couldn't we make an exception that the most important
> component jackrabbit-core always gets the same version number as the
> overall release - which would imply that sometimes core gets a version
> number increase without an actual code change.
>

Ok, I guess you all will like this suggestion :)

What about using a completly different versioning for the release, like
jackrabbit 2008-07 or 1.4.200807 :) Or something completly different
which keeps you free of using the same version numbers for core and the
release itself.
Somehow would feel strange to me, that you mandate to use the same
version number for core and the release but not for other essential parts.

Carsten
--
Carsten Ziegeler
cziegeler@...

[Jukka Zitting]

Hi,

On Tue, Jul 22, 2008 at 5:47 PM, Carsten Ziegeler <cziegeler@...> wrote:
> What about using a completly different versioning for the release, like
> jackrabbit 2008-07 or 1.4.200807 :) Or something completly different which
> keeps you free of using the same version numbers for core and the release
> itself.

Or we could use a different versioning scheme for the components...

Alternatively, how about if the modified component always got the top
level version number? That way we'd have gaps in the component version
numbers, but it would always be easy to correlate the component
version number with the top-level release that first contained it.

For example, for Jackrabbit 1.4 that would be:

* Jackrabbit 1.4.1, including core 1.4.1
* Jackrabbit 1.4.2, including core 1.4.2
* Jackrabbit 1.4.3, including jcr-commons 1.4.3 and jcr-rmi 1.4.3
* Jackrabbit 1.4.4, including core 1.4.4
* Jackrabbit 1.4.5, including core 1.4.5
* Jackrabbit 1.4.6, including core 1.4.6

BR,

Jukka Zitting

[Thomas Müller-2]

Hi,

> Alternatively, how about if the modified component always got the top
> level version number? That way we'd have gaps in the component version
> numbers, but it would always be easy to correlate the component
> version number with the top-level release that first contained it.

+1 for the 'gap version schema'.

Regards,
Thomas

[Carsten Ziegeler]

Jukka Zitting wrote:

> Hi,
>
> On Tue, Jul 22, 2008 at 5:47 PM, Carsten Ziegeler <cziegeler@...> wrote:
>> What about using a completly different versioning for the release, like
>> jackrabbit 2008-07 or 1.4.200807 :) Or something completly different which
>> keeps you free of using the same version numbers for core and the release
>> itself.
>
> Or we could use a different versioning scheme for the components...
>
> Alternatively, how about if the modified component always got the top
> level version number? That way we'd have gaps in the component version
> numbers, but it would always be easy to correlate the component
> version number with the top-level release that first contained it.
>
> For example, for Jackrabbit 1.4 that would be:
>
> * Jackrabbit 1.4.1, including core 1.4.1
> * Jackrabbit 1.4.2, including core 1.4.2
> * Jackrabbit 1.4.3, including jcr-commons 1.4.3 and jcr-rmi 1.4.3
> * Jackrabbit 1.4.4, including core 1.4.4
> * Jackrabbit 1.4.5, including core 1.4.5
> * Jackrabbit 1.4.6, including core 1.4.6
>

Hmm, but wouldn't this mean that let's say Jackrabbit 1.4.4 include
jcr-commons 1.4.3 given that commons hasn't changed in the meantime?

Carsten
--
Carsten Ziegeler
cziegeler@...

[Jukka Zitting]

Hi,

On Tue, Jul 22, 2008 at 7:37 PM, Carsten Ziegeler <cziegeler@...> wrote:
> Hmm, but wouldn't this mean that let's say Jackrabbit 1.4.4 include
> jcr-commons 1.4.3 given that commons hasn't changed in the meantime?

Correct. But that probably wouldn't be too troublesome, as if the user
can actually pinpoint the problem to jcr-commons 1.4.3 (i.e. she looks
below the top-level packaging or dependency), then the problem would
already have occurred in the Jackrabbit 1.4.3 release.

BR,

Jukka Zitting

[Felix Meschberger-2]

Hi Jukka,

Thanks for bringing this up in this concise form. I basically agree with
what you propose. Yet a big question turns around in my head: What
exactly _is_ Jackrabbit ?

Well, Jackrabbit is a JCR 1.0 implementation you might say. Quite true.
But this falls short, because this only relates to jackrabbit-core
(along with the API and commons libraries required).

So, you might say Jackrabbit is a project providing a JCR 1.0
implementation and other stuff related to JCR in general and Jackrabbit
in particular. Some such stuff coming to mind is SPI, RMI, Object
Content Mapping, Web Application, two flavors of WebDAV and more ...

<sidenote>SPI and RMI actually are in some sense JCR 1.0 implementations
on their own right</sidenote>

So, what _is_ Jackrabbit ? Or maybe: What is _the_ Jackrabbit Release ?
I can't tell. And maybe it is not important right now.

Right now, it is important to say:

   * Each component has its own versioning scheme and release
     cycle.
   * Components maybe released at the same time if there exist
     dependencies.
   * Whether a component is released at a minor or micro version
     number depends only on the changes contained in the
     component.
   * Jackrabbit Release(s) comprising multiple components have
     their own version number themselves. Whether minor or micro
     versions are released has to be decided on a release by
     release level.
   * Jackrabbit Release(s) do _not_ trigger component releases.
     Rather they just "package" the latest component releases.

The "Jackrabbit release" does not need to contain the latest nitty
gritty feature. Rather it must be easily usable out of the box to test
drive it or to just have something proven and tested (over time not just
with automated tests).

As corrolary to these notes, I would add (yes these will be
controversial, I know):

   * Branches are used for development and not for releases.
   * Releases are (almost) only cut from trunk
   * In case of an important bugfix to be applied to an old
     release, a branch will be created from the respective
     release tag and a release number with qualifier (fourth
     place in the version number, the major/minor/micro
     version numbers are the same) is created. E.g. 1.4.2.1433
     where 1433 might be the JIRA issue containing the comments
     leading to this release and 1.4.2 is the release (tag)
     from which the bug fix release was branched.


Why this ?

Trunk should always be as stable as possible. Releases are built from
tags based on trunk. This further simplifies the release procedure as
there is no more branching and nore more bug fix back porting.

Major new developments -- the implementation of the foreseen JCR 2.0 ACL
functionality comes to mind -- should be done on development branches
and merged back into trunk as the developer(s) see fit aka ready for
release. Regular development still takes place on trunk, of course.

This simplifies the world for our users: We just tell them take the
latest Jackrabbit release or take trunk. Both will work. It complicates
matters a bit for those developers implementing major features having to
work on a branch instead of trunk. But these are a minority compared to
all the other developers and -- most importantly -- our users.

WDYT?


Regards
Felix


Jukka Zitting schrieb:

> Hi,
>
> There's been a lot of discussion about release strategy and
> componentization lately, and I've been trying to figure out how to fit
> all the different requirements into a single solution for Jackrabbit
> 1.5+. Here's what I have in mind...
>
> First, the main requirements for componentization:
>
> * The codebase should be reasonably modular based on exposed
> functionality and required dependencies
> * Component release cycles should be independent except as constrained
> by functional dependencies
>
> But also:
>
> * It should be easy to download (or checkout) and build all of Jackrabbit
> * A Jackrabbit installation should have a clear version number that
> identifies the exact contents of the installation
>
> To cover all of these requirements I propose:
>
> * We keep a single release cycle for all of Jackrabbit (including a
> single trunk,branches,tags structure, single Jira project, etc.)
> * Each component within Jackrabbit trunk (and branches) has an
> independent version number that is only increased based on actual
> changes in the component
> * Thus a "Jackrabbit release" contains a collection of different
> component versions that are known to work well together
> * When making a release, only those components that have been modified
> (and thus have a new version number) are deployed to the Maven
> repository
> * Release notes will contain a list of included component versions and
> the changes to each component
>
> The effects of this policy would be:
>
> * Downstream projects like Sling that use direct Maven dependencies to
> Jackrabbit components have an easier time tracking actual changes in
> the components
> * The concepts "Jackrabbit release" and "Jackrabbit version" are still
> simple to grasp and use for example when reporting or reproducing
> issues
> * There's still a single trunk where development of all components occurs
>
> Applied to the Jackrabbit 1.4.x release cycle, this would have given
> us the following releases:
>
> * Jackrabbit 1.4.1, including core 1.4.1
> * Jackrabbit 1.4.2, including core 1.4.2
> * Jackrabbit 1.4.3, including jcr-commons 1.4.1 and jcr-rmi 1.4.1
> * Jackrabbit 1.4.4, including core 1.4.3
> * Jackrabbit 1.4.5, including core 1.4.4
> * Jackrabbit 1.4.6, including core 1.4.5
>
> Also, each release would have contained updated versions of aggregate
> components like jackrabbit-webapp and jackrabbit-jca, whose version
> numbers would thus follow the top level version number.
>
> WDYT?
>
> BR,
>
> Jukka Zitting
>

[Thomas Müller-2]

Hi,

> Major new developments -- the implementation of the foreseen JCR 2.0 ACL
> functionality comes to mind -- should be done on development branches and
> merged back into trunk as the developer(s) see fit aka ready for release.
> Regular development still takes place on trunk, of course.

If done correctly, major new development can be done in the trunk
without risk. How to do that: Use a (static final boolean) constant to
enable / disable a certain new feature or new code path. It is used in
the DataStore implementation. At first, it was disabled (only enabled
when I run tests), and now it is enabled by default:

class InternalValue {
   public static final boolean USE_DATA_STORE =
        Boolean.valueOf(System.getProperty("org.jackrabbit.useDataStore",
"true")).booleanValue();
}

and then, everywhere where it is required, do this:
if (USE_DATA_STORE) {
  // new code
} else {
  // old code
}

As an added advantage, it's easy to find out if this feature is
responsible for a problem, by disabling the system property.

>  * Releases are (almost) only cut from trunk
>  * Branches are used for development and not for releases.
> Trunk should always be as stable as possible. Releases are built from tags
> based on trunk. This further simplifies the release procedure as there is no
> more branching and nore more bug fix back porting.

That's a good idea.

Regards,
Thomas

[Felix Meschberger-2]

Hi,

Thomas Müller schrieb:
> Hi,
>
>> Major new developments -- the implementation of the foreseen JCR 2.0 ACL
>> functionality comes to mind -- should be done on development branches and
>> merged back into trunk as the developer(s) see fit aka ready for release.
>> Regular development still takes place on trunk, of course.
>
> If done correctly, major new development can be done in the trunk
> without risk.

Sure, if it only concerns a single location or limited set of locations.
If it touches multiple places, this is not that easy.

> How to do that: Use a (static final boolean) constant to
> enable / disable a certain new feature or new code path. It is used in
> the DataStore implementation. At first, it was disabled (only enabled
> when I run tests), and now it is enabled by default:
>
> class InternalValue {
>    public static final boolean USE_DATA_STORE =
>         Boolean.valueOf(System.getProperty("org.jackrabbit.useDataStore",
> "true")).booleanValue();
> }
>
> and then, everywhere where it is required, do this:
> if (USE_DATA_STORE) {
>   // new code
> } else {
>   // old code
> }
>
> As an added advantage, it's easy to find out if this feature is
> responsible for a problem, by disabling the system property.

I basically like the idea but have my reservations with respect to
system property use: It makes it difficult in embedded situations and
even almost makes it impossible to use in app server or servlet
container environments. For example: you want to have two repositories
in the same Java VM, one with  and one without data store....

If is really just for testing, a "public" field is bad, because this
quickly becomes (de-facto) API. And a branch may still be better suited.

Probably better -- also for visibility -- would be to have a
functionality to provide this kind of setup as part of the Repository
descriptors. (And yes, I cannot imagine how to implement this in the
InternalValue class you presented. Maybe the use case is slightly wrong
in that the InternalValue should just use a "store", which in turn
either uses the new data store code or the old code and has to be setup
somehow)

Regards
Felix

[Jukka Zitting]

Hi,

On Wed, Jul 23, 2008 at 9:35 AM, Felix Meschberger <fmeschbe@...> wrote:
> As corrolary to these notes, I would add (yes these will be controversial, I
> know):
>
>  * Branches are used for development and not for releases.
>  * Releases are (almost) only cut from trunk
>  * In case of an important bugfix to be applied to an old
>    release, a branch will be created from the respective
>    release tag [...]

-1 I'm more a fan of release branches than development branches. Why?

a) It makes it easy to produce patch releases containing just bug
fixes. Such maintenance releases are important as they can be easily
applied in production with low risk of regressions and zero need for
configuration changes.

b) Release branches only require "one-way merging" from trunk. This is
quite simple and pretty well supported by Subversion. Development
branches however require "two-way merging", first to keep up with
other changes in trunk and then to merge changes back to trunk. The
merge tracking feature in Subversion 1.5 makes this easier, but it's
still tricky.

c) I think it's important that we have all development happening in
the same tree. Any extensive development effort that lasts longer than
a few weeks will require coordination with other developers, and the
easiest way to do that is to keep it in trunk.

Our trunk is (and should be) the place where bleeding edge development
happens. We use branches so that trunk can continue to evolve while a
feature release is being stabilized or maintenance releases are being
prepared.

BR,

Jukka Zitting

[Thomas Müller-2]

Hi,

>> If done correctly, major new development can be done in the trunk
>> without risk.
>
> Sure, if it only concerns a single location or limited set of locations. If
> it touches multiple places, this is not that easy.

USE_DATA_STORE is used in multiple places, I don't think it's a big
problem. And at some point the setting can be removed (I think it
could be removed by now).

> I basically like the idea but have my reservations with respect to system
> property use: It makes it difficult in embedded situations and even almost
> makes it impossible to use in app server or servlet container environments.

This system property is only enabled for testing. If disabled by
default, it can be enabled to test an unreleased feature. If enabled
by default, it can be used as a 'kill switch' to test if this
functionality is responsible for a problem. I don't see a use case
where you want to start multiple repositories with different settings.

> For example: you want to have two repositories in the same Java VM, one with
>  and one without data store....

I don't see a use case for that currently. The system property is not
something that people need to or should use (unless they want to
explicitly test).

> If is really just for testing, a "public" field is bad, because this quickly
> becomes (de-facto) API. And a branch may still be better suited.

It is a final field, and initialized only once. The system property is
not final, but if clearly documented I don't think there is a risk
people use it in production. This flag is used instead of branching,
and given the disadvantages of branching I think it is worth thinking
about it.

> Probably better -- also for visibility -- would be to have a functionality
> to provide this kind of setup as part of the Repository descriptors.

In that case there is a bigger risk that people use it in production I
think. Also, it couldn't be used in all components: only in those that
have access to the repository.

> Maybe the use case is slightly wrong in that the InternalValue
> should just use a "store", which in turn either uses the new data store code
> or the old code and has to be setup somehow)

It is doing that: it is eighter using BLOBValue (the old mechanism) or
BLOBIn* (the new mechanism). InternalValue acts as the factory. Of
course you could add another class that acts as the factory, but it
wouldn't really simplify things.

Regards,
Thomas