Open source ivy files project?

[Archie Cobbs]

Hi,

I've been using (and liking) Ivy for a while now and have some thoughts on
how the state of the Ivy world might be improved. I'd appreciate any
feedback...

At work we have created our own Ivy repository. This is normal and works
fine.

However, building this repository is tedious and error-prone. For each Java
package we want to use, I have to go through the same tedious steps:

   1. Go download jfoobar-1.2.3.zip from http://www.jfoobar.com
   2. Examine jfoobar to understand:
      1. Which JARs are required, which are optional, etc.
      2. What is an appropriate set of ivy configurations to define
      based on previous step
      3. Determine which JARs in jfoobar are really part of jfoobar
      and which are simply required libraries included from some other project
   3. Create an ivy.xml for jfoobar that defines the configurations and
   dependencies from step #2 (plus homepage, copyright, etc.)
   4. Create a new ant project that performs the following steps:
      1. Unpack jfoobar-1.2.3.zip
      2. Extract the appropriate JARs (renaming them to remove
      revision numbers)
      3. Extract the Javadocs and put into a javadoc.zip
      4. Extract the sources and put into a sources.zip
      5. Publishes the new ivy module to our local repository
      5. Recursively perform this entire process for all dependencies
   found in step #2
   6. Execute ant project from step #4

The real work is in steps #2 and #3. The tendency due to laziness is to just
have a "default" configuration and dump all the JARs (whether part of
jfoobar or not) into it. The result is, as before, a enormous CLASSPATH
containing multiple versions of the same dependent libraries over and over
again. I.e., nothing much has changed since before ivy.

If instead you really try to pick apart all the extra libraries, create
separate modules for them, etc. you are rewarded with a combinatorial
explosion due to step #5. But at least you then have CLASSPATH sanity...

So here's my dream: I want there to be an open source project somewhere out
on the web that captures the end result of performing the above steps for
any and all Java projects that exist. Imagine a project that does for Ivy
what JPackage.org does for RPM.

Some key goals of this would be:

   1. Ivy definitions for zillions of Java projects already created and
   made available to Ivy users everywhere
   2. Open source: multiple contributors, each maintaining the particular
   packages they know well
   3. Does not require storing any JAR, ZIP, or TGZ files on the website
   itself, only a capture of the above steps' logic
   4. For each package, we get a *standard* definition of the
   configurations available for that package
   5. An easy way to configure my local Ivy to use this information

I think goal #3 is important. This web site should contain meta-data, not
copies of archives available elsewhere (but yes to MD5 checksums, etc.)

Regarding step #5 there are a couple of possibilities...

   1. There could be an easy way to use this info to automatically
   build/update your own, private repository (specifying exactly which projects
   you care about).
   2. This is a little fancier... some way to simply include this website
   in your Ivy configuration using a (new) resolver. This resolver would
   download the corresponding instructions (perhaps just build.xml and
   ivy.xml files), build the Ivy module using ant, publish it to the
   local machine (in a new type of "cache repository"), and then find the
   module in the local "cache repository".

This idea is only half-baked. But it seems like something is definitely
needed. An unmaintained ivyrep and maven-only ibiblio are not cutting it for
me.

Thoughts?

Thanks,
-Archie

--
Archie L. Cobbs

[antoinell]

Hello Archie,

I see two issues here. One is to find volunteers contributing for a work
of description. The other is to find a balance between very simple
configurations and very complicated ones.

There are potentially a lot of different dimensions along which one
could define configurations. For instance support for JDK 1.4, 1.5, 1.6,
support for different operating systems, for different human languages, ...

When I create descriptors at work, I have decided that all projects
which are hosted at sourceforge, for instance junit, have sourceforge as
organisation. This is arbitrary, but the legal status of the different
sourceforge projects is not clear to me.

So people creating descriptors will have to make a lot of decisions for
which there are no clear guidelines.

I remember that at one point in time I was reading the emails of the
repository list of apache, and for various reasons I have seen cases
where people were asking the republication of an existing POM. This can
happen to to an ivy repository. I am not sure how this should be handled.

Regards,
Antoine


Archie Cobbs wrote:

> Hi,
>
> I've been using (and liking) Ivy for a while now and have some thoughts on
> how the state of the Ivy world might be improved. I'd appreciate any
> feedback...
>
> At work we have created our own Ivy repository. This is normal and works
> fine.
>
> However, building this repository is tedious and error-prone. For each Java
> package we want to use, I have to go through the same tedious steps:
>
>    1. Go download jfoobar-1.2.3.zip from http://www.jfoobar.com
>    2. Examine jfoobar to understand:
>       1. Which JARs are required, which are optional, etc.
>       2. What is an appropriate set of ivy configurations to define
>       based on previous step
>       3. Determine which JARs in jfoobar are really part of jfoobar
>       and which are simply required libraries included from some other project
>    3. Create an ivy.xml for jfoobar that defines the configurations and
>    dependencies from step #2 (plus homepage, copyright, etc.)
>    4. Create a new ant project that performs the following steps:
>       1. Unpack jfoobar-1.2.3.zip
>       2. Extract the appropriate JARs (renaming them to remove
>       revision numbers)
>       3. Extract the Javadocs and put into a javadoc.zip
>       4. Extract the sources and put into a sources.zip
>       5. Publishes the new ivy module to our local repository
>       5. Recursively perform this entire process for all dependencies
>    found in step #2
>    6. Execute ant project from step #4
>
> The real work is in steps #2 and #3. The tendency due to laziness is to just
> have a "default" configuration and dump all the JARs (whether part of
> jfoobar or not) into it. The result is, as before, a enormous CLASSPATH
> containing multiple versions of the same dependent libraries over and over
> again. I.e., nothing much has changed since before ivy.
>
> If instead you really try to pick apart all the extra libraries, create
> separate modules for them, etc. you are rewarded with a combinatorial
> explosion due to step #5. But at least you then have CLASSPATH sanity...
>
> So here's my dream: I want there to be an open source project somewhere out
> on the web that captures the end result of performing the above steps for
> any and all Java projects that exist. Imagine a project that does for Ivy
> what JPackage.org does for RPM.
>
> Some key goals of this would be:
>
>    1. Ivy definitions for zillions of Java projects already created and
>    made available to Ivy users everywhere
>    2. Open source: multiple contributors, each maintaining the particular
>    packages they know well
>    3. Does not require storing any JAR, ZIP, or TGZ files on the website
>    itself, only a capture of the above steps' logic
>    4. For each package, we get a *standard* definition of the
>    configurations available for that package
>    5. An easy way to configure my local Ivy to use this information
>
> I think goal #3 is important. This web site should contain meta-data, not
> copies of archives available elsewhere (but yes to MD5 checksums, etc.)
>
> Regarding step #5 there are a couple of possibilities...
>
>    1. There could be an easy way to use this info to automatically
>    build/update your own, private repository (specifying exactly which projects
>    you care about).
>    2. This is a little fancier... some way to simply include this website
>    in your Ivy configuration using a (new) resolver. This resolver would
>    download the corresponding instructions (perhaps just build.xml and
>    ivy.xml files), build the Ivy module using ant, publish it to the
>    local machine (in a new type of "cache repository"), and then find the
>    module in the local "cache repository".
>
> This idea is only half-baked. But it seems like something is definitely
> needed. An unmaintained ivyrep and maven-only ibiblio are not cutting it for
> me.
>
> Thoughts?
>
> Thanks,
> -Archie
>
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Adrian Sandor]

Archie Cobbs wrote:
> So here's my dream: I want there to be an open source project somewhere out
> on the web that captures the end result of performing the above steps for
> any and all Java projects that exist.

I'm with you Archie :)
Also see
http://www.nabble.com/Add-maven2-resolver--to15361586.html#a15363362

Antoine has some good points too.
While in my local repository I can do things the way I want, and just go
and make changes when I need, in a public repository it's much more
complicated.
But there are solutions to all the problems, when enough people decide
to work together.

Adrian

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Xavier Hanin]

On Wed, Apr 2, 2008 at 12:48 AM, Archie Cobbs <archie@...> wrote:

> Hi,
>
> I've been using (and liking) Ivy for a while now and have some thoughts on
> how the state of the Ivy world might be improved. I'd appreciate any
> feedback...
>
> At work we have created our own Ivy repository. This is normal and works
> fine.
>
> However, building this repository is tedious and error-prone. For each
> Java
> package we want to use, I have to go through the same tedious steps:
>
>   1. Go download jfoobar-1.2.3.zip from http://www.jfoobar.com
>   2. Examine jfoobar to understand:
>      1. Which JARs are required, which are optional, etc.
>      2. What is an appropriate set of ivy configurations to define
>      based on previous step
>      3. Determine which JARs in jfoobar are really part of jfoobar
>      and which are simply required libraries included from some other
> project
>   3. Create an ivy.xml for jfoobar that defines the configurations and
>   dependencies from step #2 (plus homepage, copyright, etc.)
>   4. Create a new ant project that performs the following steps:
>      1. Unpack jfoobar-1.2.3.zip
>      2. Extract the appropriate JARs (renaming them to remove
>      revision numbers)
>      3. Extract the Javadocs and put into a javadoc.zip
>      4. Extract the sources and put into a sources.zip
>      5. Publishes the new ivy module to our local repository
>      5. Recursively perform this entire process for all dependencies
>   found in step #2
>   6. Execute ant project from step #4
>
> The real work is in steps #2 and #3. The tendency due to laziness is to
> just
> have a "default" configuration and dump all the JARs (whether part of
> jfoobar or not) into it. The result is, as before, a enormous CLASSPATH
> containing multiple versions of the same dependent libraries over and over
> again. I.e., nothing much has changed since before ivy.
>
> If instead you really try to pick apart all the extra libraries, create
> separate modules for them, etc. you are rewarded with a combinatorial
> explosion due to step #5. But at least you then have CLASSPATH sanity...
>
> So here's my dream: I want there to be an open source project somewhere
> out
> on the web that captures the end result of performing the above steps for
> any and all Java projects that exist. Imagine a project that does for Ivy
> what JPackage.org does for RPM.
>
> Some key goals of this would be:
>
>   1. Ivy definitions for zillions of Java projects already created and
>   made available to Ivy users everywhere
>   2. Open source: multiple contributors, each maintaining the particular
>   packages they know well
>   3. Does not require storing any JAR, ZIP, or TGZ files on the website
>   itself, only a capture of the above steps' logic
>   4. For each package, we get a *standard* definition of the
>   configurations available for that package
>   5. An easy way to configure my local Ivy to use this information
>
> I think goal #3 is important. This web site should contain meta-data, not
> copies of archives available elsewhere (but yes to MD5 checksums, etc.)
>
> Regarding step #5 there are a couple of possibilities...
>
>   1. There could be an easy way to use this info to automatically
>   build/update your own, private repository (specifying exactly which
> projects
>   you care about).
>   2. This is a little fancier... some way to simply include this website
>   in your Ivy configuration using a (new) resolver. This resolver would
>   download the corresponding instructions (perhaps just build.xml and
>   ivy.xml files), build the Ivy module using ant, publish it to the
>   local machine (in a new type of "cache repository"), and then find the
>   module in the local "cache repository".
>
> This idea is only half-baked. But it seems like something is definitely
> needed. An unmaintained ivyrep and maven-only ibiblio are not cutting it
> for
> me.
>
> Thoughts?

I think steps #2 and #4 will be less and less necessary in the future. Most
projects are now available on a public maven  repository. This means that
you can at least get their jar already unpacked and available on the web,
and most of the time you can also get their sources and javadocs zipped. So
the real problem IMO is metadata and repositories cleaness/stability. As I
already said, I'm far from opposed to see this kind of public Ivy repository
emerge, I just think it's too early for the Ivy committer team to get
involved, while we still have quite a lot of work before releasing Ivy 2.0.0
final and IvyDE 1.3.0 final. But I speak only for myself.

Xavier


>
>
> Thanks,
> -Archie
>
> --
> Archie L. Cobbs
>



--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/

[Gilles Scokart]

Speaking for me, I agree that the maven repository is sometimes crapy.
 But how would you do better?   If you think you can, please do.

Note anyway that it will never replace an internal repository that
will always be much more secure (security = availability , integrity
and confidentiality).




On 02/04/2008, Xavier Hanin <xavier.hanin@...> wrote:

> On Wed, Apr 2, 2008 at 12:48 AM, Archie Cobbs <archie@...> wrote:
>
>  > Hi,
>  >
>  > I've been using (and liking) Ivy for a while now and have some thoughts on
>  > how the state of the Ivy world might be improved. I'd appreciate any
>  > feedback...
>  >
>  > At work we have created our own Ivy repository. This is normal and works
>  > fine.
>  >
>  > However, building this repository is tedious and error-prone. For each
>  > Java
>  > package we want to use, I have to go through the same tedious steps:
>  >
>  >   1. Go download jfoobar-1.2.3.zip from http://www.jfoobar.com
>  >   2. Examine jfoobar to understand:
>  >      1. Which JARs are required, which are optional, etc.
>  >      2. What is an appropriate set of ivy configurations to define
>  >      based on previous step
>  >      3. Determine which JARs in jfoobar are really part of jfoobar
>  >      and which are simply required libraries included from some other
>  > project
>  >   3. Create an ivy.xml for jfoobar that defines the configurations and
>  >   dependencies from step #2 (plus homepage, copyright, etc.)
>  >   4. Create a new ant project that performs the following steps:
>  >      1. Unpack jfoobar-1.2.3.zip
>  >      2. Extract the appropriate JARs (renaming them to remove
>  >      revision numbers)
>  >      3. Extract the Javadocs and put into a javadoc.zip
>  >      4. Extract the sources and put into a sources.zip
>  >      5. Publishes the new ivy module to our local repository
>  >      5. Recursively perform this entire process for all dependencies
>  >   found in step #2
>  >   6. Execute ant project from step #4
>  >
>  > The real work is in steps #2 and #3. The tendency due to laziness is to
>  > just
>  > have a "default" configuration and dump all the JARs (whether part of
>  > jfoobar or not) into it. The result is, as before, a enormous CLASSPATH
>  > containing multiple versions of the same dependent libraries over and over
>  > again. I.e., nothing much has changed since before ivy.
>  >
>  > If instead you really try to pick apart all the extra libraries, create
>  > separate modules for them, etc. you are rewarded with a combinatorial
>  > explosion due to step #5. But at least you then have CLASSPATH sanity...
>  >
>  > So here's my dream: I want there to be an open source project somewhere
>  > out
>  > on the web that captures the end result of performing the above steps for
>  > any and all Java projects that exist. Imagine a project that does for Ivy
>  > what JPackage.org does for RPM.
>  >
>  > Some key goals of this would be:
>  >
>  >   1. Ivy definitions for zillions of Java projects already created and
>  >   made available to Ivy users everywhere
>  >   2. Open source: multiple contributors, each maintaining the particular
>  >   packages they know well
>  >   3. Does not require storing any JAR, ZIP, or TGZ files on the website
>  >   itself, only a capture of the above steps' logic
>  >   4. For each package, we get a *standard* definition of the
>  >   configurations available for that package
>  >   5. An easy way to configure my local Ivy to use this information
>  >
>  > I think goal #3 is important. This web site should contain meta-data, not
>  > copies of archives available elsewhere (but yes to MD5 checksums, etc.)
>  >
>  > Regarding step #5 there are a couple of possibilities...
>  >
>  >   1. There could be an easy way to use this info to automatically
>  >   build/update your own, private repository (specifying exactly which
>  > projects
>  >   you care about).
>  >   2. This is a little fancier... some way to simply include this website
>  >   in your Ivy configuration using a (new) resolver. This resolver would
>  >   download the corresponding instructions (perhaps just build.xml and
>  >   ivy.xml files), build the Ivy module using ant, publish it to the
>  >   local machine (in a new type of "cache repository"), and then find the
>  >   module in the local "cache repository".
>  >
>  > This idea is only half-baked. But it seems like something is definitely
>  > needed. An unmaintained ivyrep and maven-only ibiblio are not cutting it
>  > for
>  > me.
>  >
>  > Thoughts?
>
>
> I think steps #2 and #4 will be less and less necessary in the future. Most
>  projects are now available on a public maven  repository. This means that
>  you can at least get their jar already unpacked and available on the web,
>  and most of the time you can also get their sources and javadocs zipped. So
>  the real problem IMO is metadata and repositories cleaness/stability. As I
>  already said, I'm far from opposed to see this kind of public Ivy repository
>  emerge, I just think it's too early for the Ivy committer team to get
>  involved, while we still have quite a lot of work before releasing Ivy 2.0.0
>  final and IvyDE 1.3.0 final. But I speak only for myself.
>
>  Xavier
>
>
>
>  >
>  >
>  > Thanks,
>  > -Archie
>  >
>  > --
>  > Archie L. Cobbs
>  >
>
>
>
>
> --
>  Xavier Hanin - Independent Java Consultant
>  http://xhab.blogspot.com/
>  http://ant.apache.org/ivy/
>  http://www.xoocode.org/
>

--
Gilles Scokart

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Archie Cobbs]

Thanks for all the comments so far regarding this idea. The first step was
to get a discussion going and so far that's worked :-)

Some more thoughts... please let me know what you think makes sense and what
doesn't...

*Organizations.* Regarding choice of "organization" for each module: why not
use the name of the project as the organization for all of the ivy modules?
That is what we do internally in our "enterprise repository".

This seems obvious to me, but maybe I'm misunderstanding what "organization"
is supposed to be for. I've always thought of it as referring to the people
that created the *module* (not the *artifacts*).

If you don't do it that way, you eliminate the possibility of multiple
packagers of the same stuff, which would be bad (e.g., think about how many
different Linux distributions there are).

*Volunteers. *Regarding finding volunteers: there is no problem here. Either
there will be enough interest and volunteers and the project will succeed,
or there won't and it will die. This is the way open source is supposed to
work!

Also: regarding the question of whether Ivy developers should be spending
their time on this idea. Answer: what a flattering idea :-) This project
would however be separate from Ivy. Ivy developers would of course be
welcome and encouraged but there would hopefully be plenty of other "random"
people involved (like me). The best would be if a developer of each java
project was maintaining their own project's meta-data.

*Configurations. *Regarding defining configurations: I think the general
approach with each module should be to start simple, then refine over time,
and plan an "additive" evolution. That is to say, it's much less disruptive
for a module to add a new configuration than it is to change or remove an
existing one. So the general evolution should be that new, more precise
configurations are added over time as necessary/desired.

Example: you have "production" and "test" and want to refine along the JDK14
vs JDK15 dimension. Then you leave "production" and "test" alone but add
"production-jdk14", "test-jdk14", "production-jdk15", and "test-jdk15". This
way things stay backward compatible.

*Public vs. Internal.* While it's true that an internal private repository
is always going to be safer and more secure than a public one, I think we
can be smart here, and it doesn't have to be "all or none", i.e., you can
choose where along the spectrum you want to live.

First, the meta-data instructions that knows how to retrieve the archive
from wherever would be required to confirm integrity via checksums. This
eliminates one source of potential "poisoning".

Secondly, this project would really be composed of two separate things:

   1. Meta-data/instructions describing how to retrieve and/or build each
   module's ivy.xml file and artifacts (with integrity checksums for all
   external resources)
   2. A new ivy resolver that knows how to use the meta-data in #1 to
   resolve ivy modules

So for example there's no reason a paranoid enterprise couldn't use the
resolver in #2 but replace #1 with their own meta-data, etc.

*Name. *We'd need a name for this project. I suggest "Ivy Roundup".

*SVN Repository.* The project would have its own SVN repository.

This would contain meta-data (in whatever form) describing how, for each
module, to retrieve and/or build the ivy.xml and artifacts in that module.
This meta-data could be as simple as an ant build.xml, or perhaps an XML
file in a simple domain-specific language (<extract url="http://.../foobar.zip"
file="dist/foobar.jar"/>) or whatever.

If a source project includes its own ivy.xml file, it can be used directly
(possibly with changes applied via XSLT). Otherwise the likely common case
would be that the ivy.xml is just included directly in the meta-data.

We would optimize for projects published via maven, where we can derive some
or all of our meta-data from the maven meta-data (<maven-project
url="..."/>).

The SVN repository would also contain Java code that defines an ivy resolver
that knows how to utilize the above meta-data to resolve Ivy modules
(someday if there is great success this could be merged into ivy itself).
This resolver would be configurable so that it could work with propely
formed meta-data from any source, not just this project's version of it.

*Website. *The project web site must provide an easy way to visually inspect
the (styled HTML) meta-data for each module using a web browser (perhaps
using direct SVN HTTP access). Also it should have a simple search
capability.

Now what is the end result of this project, i.e., the product produced by
this project? What exactly do you download and how do you use it?

One possibility is that the output of this project would be an ivy module,
containing the meta-data and resolver.

The project would also have its own public Ivy repository, but containing
only one module: itself.

To make use of this, users would then:

   1. Add the project's ivy repository to their settings
   2. Use ivy to download the project module normally
   3. Update their ivy settings to use the project's resolver, configured
   to point at the project's website

Of course steps #1 and #2 could be done manually as well.

Thanks,
-Archie

P.S. I apologize for the duplicate original email (got tricked by gmail
email aliasing problem)

--
Archie L. Cobbs

[Xavier Hanin]

On Wed, Apr 2, 2008 at 5:30 PM, Archie Cobbs <archie@...> wrote:

> Thanks for all the comments so far regarding this idea. The first step was
> to get a discussion going and so far that's worked :-)
>
> Some more thoughts... please let me know what you think makes sense and
> what
> doesn't...
>
> *Organizations.* Regarding choice of "organization" for each module: why
> not
> use the name of the project as the organization for all of the ivy
> modules?
> That is what we do internally in our "enterprise repository".
>
> This seems obvious to me, but maybe I'm misunderstanding what
> "organization"
> is supposed to be for. I've always thought of it as referring to the
> people
> that created the *module* (not the *artifacts*).
>
> If you don't do it that way, you eliminate the possibility of multiple
> packagers of the same stuff, which would be bad (e.g., think about how
> many
> different Linux distributions there are).

I think I would try to follow maven 2 conventions, just for the ease of use
for people coming from maven2, and because I think they are pretty good when
followed. Obviously apache commons-* would be renamed since they don't
follow the convention. For java projects, I think using the main package
name of the main module in the project makes sense most of the time.

*Volunteers. *Regarding finding volunteers: there is no problem here. Either
> there will be enough interest and volunteers and the project will succeed,
> or there won't and it will die. This is the way open source is supposed to
> work!

This sounds like a good approach. I was afraid you were just asking for the
project to be done by itself.

>
>
> Also: regarding the question of whether Ivy developers should be spending
> their time on this idea. Answer: what a flattering idea :-) This project
> would however be separate from Ivy. Ivy developers would of course be
> welcome and encouraged but there would hopefully be plenty of other
> "random"
> people involved (like me). The best would be if a developer of each java
> project was maintaining their own project's meta-data.

As I said earlier, if a community raise and develop this, it's more than
welcome, and I'll be glad to participate if I find some spare time.

>
>
> *Configurations. *Regarding defining configurations: I think the general
> approach with each module should be to start simple, then refine over
> time,
> and plan an "additive" evolution. That is to say, it's much less
> disruptive
> for a module to add a new configuration than it is to change or remove an
> existing one. So the general evolution should be that new, more precise
> configurations are added over time as necessary/desired.
>
> Example: you have "production" and "test" and want to refine along the
> JDK14
> vs JDK15 dimension. Then you leave "production" and "test" alone but add
> "production-jdk14", "test-jdk14", "production-jdk15", and "test-jdk15".
> This
> way things stay backward compatible.

This is a very nice way to handle metadata evolution, I already proposed
this kind of metadata backward compatibility for public repository during
discussions which happened at the beginning of Ivy's incubation. Better
support for deprecated configurations (which are supported since very early
versions of Ivy, but not used) would help in this area too. If we want to
ensure real backward compatibility, it would be nice to have a tool to
validate that the new metadata is actually backward compatible with the old
one.

>
> *Public vs. Internal.* While it's true that an internal private repository
> is always going to be safer and more secure than a public one, I think we
> can be smart here, and it doesn't have to be "all or none", i.e., you can
> choose where along the spectrum you want to live.
>
> First, the meta-data instructions that knows how to retrieve the archive
> from wherever would be required to confirm integrity via checksums. This
> eliminates one source of potential "poisoning".
>
> Secondly, this project would really be composed of two separate things:
>
>   1. Meta-data/instructions describing how to retrieve and/or build each
>   module's ivy.xml file and artifacts (with integrity checksums for all
>   external resources)
>   2. A new ivy resolver that knows how to use the meta-data in #1 to
>   resolve ivy modules
>
> So for example there's no reason a paranoid enterprise couldn't use the
> resolver in #2 but replace #1 with their own meta-data, etc.

And they could build their metadata from what is provided online, so I
clearly see  as a very interesting design.

>
>
> *Name. *We'd need a name for this project. I suggest "Ivy Roundup".
>
> *SVN Repository.* The project would have its own SVN repository.
>
> This would contain meta-data (in whatever form) describing how, for each
> module, to retrieve and/or build the ivy.xml and artifacts in that module.
> This meta-data could be as simple as an ant build.xml, or perhaps an XML
> file in a simple domain-specific language (<extract url="http://
> .../foobar.zip"
> file="dist/foobar.jar"/>) or whatever.
>
> If a source project includes its own ivy.xml file, it can be used directly
> (possibly with changes applied via XSLT). Otherwise the likely common case
> would be that the ivy.xml is just included directly in the meta-data.
>
> We would optimize for projects published via maven, where we can derive
> some
> or all of our meta-data from the maven meta-data (<maven-project
> url="..."/>).

I really like this idea of leveraging maven repo when it makes sense, and
doing more work only when necessary.

>
>
> The SVN repository would also contain Java code that defines an ivy
> resolver
> that knows how to utilize the above meta-data to resolve Ivy modules
> (someday if there is great success this could be merged into ivy itself).
> This resolver would be configurable so that it could work with propely
> formed meta-data from any source, not just this project's version of it.


Sounds nice.


> *Website. *The project web site must provide an easy way to visually
> inspect
> the (styled HTML) meta-data for each module using a web browser (perhaps
> using direct SVN HTTP access). Also it should have a simple search
> capability.

There's something very important missing in your list (but probably not in
your head :-)) is the source of communication for the community. A mailing
list is nice, especially if you have a good web front end for it (like with
google groups). This list would allow people to discuss how they think a
module should be added, what's the best name for a project's organization,
and so on.

To add or update data in the repository, I think there should be a tool, not
just a svn ci of whatever you've down. The tool (or server) would be
responsible for validating that what you do is correct: valid xml, valid
metadata, all dependencies are available, BC on metadata update, ... I think
this is the only way to get a community maintained repository clean, and as
such is *very* important to have from the beginning.

>
> Now what is the end result of this project, i.e., the product produced by
> this project? What exactly do you download and how do you use it?
>
> One possibility is that the output of this project would be an ivy module,
> containing the meta-data and resolver.
>
> The project would also have its own public Ivy repository, but containing
> only one module: itself.
>
> To make use of this, users would then:
>
>   1. Add the project's ivy repository to their settings
>   2. Use ivy to download the project module normally
>   3. Update their ivy settings to use the project's resolver, configured
>   to point at the project's website
>
> Of course steps #1 and #2 could be done manually as well.

That sounds nice and simple enough.

Xavier

>
>
> Thanks,
> -Archie
>
> P.S. I apologize for the duplicate original email (got tricked by gmail
> email aliasing problem)
>
> --
> Archie L. Cobbs
>

--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/

[Archie Cobbs]

On Wed, Apr 2, 2008 at 11:05 AM, Xavier Hanin <xavier.hanin@...>
wrote:

> > *Organizations.* Regarding choice of "organization" for each module: why
> > not
> > use the name of the project as the organization for all of the ivy
> > modules?
> > That is what we do internally in our "enterprise repository".
> >
> > This seems obvious to me, but maybe I'm misunderstanding what
> > "organization"
> > is supposed to be for. I've always thought of it as referring to the
> > people
> > that created the *module* (not the *artifacts*).
> >
> > If you don't do it that way, you eliminate the possibility of multiple
> > packagers of the same stuff, which would be bad (e.g., think about how
> > many
> > different Linux distributions there are).
>
> I think I would try to follow maven 2 conventions, just for the ease of
> use
> for people coming from maven2, and because I think they are pretty good
> when
> followed. Obviously apache commons-* would be renamed since they don't
> follow the convention. For java projects, I think using the main package
> name of the main module in the project makes sense most of the time.

I apologize for my ignorance... Two questions:

   1. I know little about maven. What are the "maven 2 conventions"?
   2. What happens in ivy when two different repositories in your
   settings publish the same organization, name, and version of a module?

Thanks,
-Archie

--
Archie L. Cobbs

[Xavier Hanin]

On Wed, Apr 2, 2008 at 6:22 PM, Archie Cobbs <archie@...> wrote:

> On Wed, Apr 2, 2008 at 11:05 AM, Xavier Hanin <xavier.hanin@...>
> wrote:
>
> > > *Organizations.* Regarding choice of "organization" for each module:
> why
> > > not
> > > use the name of the project as the organization for all of the ivy
> > > modules?
> > > That is what we do internally in our "enterprise repository".
> > >
> > > This seems obvious to me, but maybe I'm misunderstanding what
> > > "organization"
> > > is supposed to be for. I've always thought of it as referring to the
> > > people
> > > that created the *module* (not the *artifacts*).
> > >
> > > If you don't do it that way, you eliminate the possibility of multiple
> > > packagers of the same stuff, which would be bad (e.g., think about how
> > > many
> > > different Linux distributions there are).
> >
> > I think I would try to follow maven 2 conventions, just for the ease of
> > use
> > for people coming from maven2, and because I think they are pretty good
> > when
> > followed. Obviously apache commons-* would be renamed since they don't
> > follow the convention. For java projects, I think using the main package
> > name of the main module in the project makes sense most of the time.
>
>
> I apologize for my ignorance... Two questions:
>
>   1. I know little about maven. What are the "maven 2 conventions"?

To use package names convention:
http://maven.apache.org/guides/mini/guide-naming-conventions.html
http://docs.codehaus.org/display/MAVENUSER/Choosing+a+Group+ID


>
>   2. What happens in ivy when two different repositories in your
>   settings publish the same organization, name, and version of a module?

It depends on your settings, but most probably the first one will be used
and the second one ignored. The couple organization#module MUST identify a
module uniquely. A module's revision is identified by organization#module +
revision + extra attributes if any. But for a public repository I wouldn't
use extra attributes, so sticking with pure organization#module;revision as
identifier should be a good choice.

Xavier


>
>
> Thanks,
> -Archie
>
> --
> Archie L. Cobbs
>



--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/

[Archie Cobbs]

On Wed, Apr 2, 2008 at 11:40 AM, Xavier Hanin <xavier.hanin@...>
wrote:

> >   2. What happens in ivy when two different repositories in your
> >   settings publish the same organization, name, and version of a module?
>
> It depends on your settings, but most probably the first one will be used
> and the second one ignored. The couple organization#module MUST identify a
> module uniquely. A module's revision is identified by organization#module
> +
> revision + extra attributes if any. But for a public repository I wouldn't
> use extra attributes, so sticking with pure organization#module;revision
> as
> identifier should be a good choice.
>

OK, so organization+module MUST be unique... but how do you expect to
enforce your "MUST"?? Are you in control of all the Ivy repositories of the
world? Or, is everyone expected to create their own private repository and
never share?

I think the "organization" attribute has to refer to the ivy module
packager, not the originator of the code. Otherwise, the system seems
totally broken to me.

The only way you can expect organization+module to be unique is to require
either:

   1. Only the Foo organization can ever publish an ivy module with
   organization="foo"; OR
   2. There is some global authority designating who is the one true
   authorized group that is allowed to put organization="foo" in their ivy.xml
   files

And option #2 is not an option unless you want to get into the business of
ivy dictatorship.

Examples: #1 is approach taken by Java package naming; #2 is approach taken
by DNS domain system.

Let's take an example, e.g. Hibernate. If someone (other than the Hibernate
developers) creates an ivy module with organization="hibernate.org"
name="hibernate" then a priori this module is going to be incompatible with
the same thing created by someone else.

Since there's no one true Ivy repository (RIP ivyrep) this is bound to
happen: all the ivy repositories out there are going to be incompatible. And
there's no clear way for me to specify that "I'd like the Hibernate module
from repository A, but I'd like the Spring module from repository B".

I'd suggest Ivy should encourage a decentralized model that is more
compatible with the reality of the Internet. To me this seems like a pretty
fundamental issue that needs to be examined for Ivy to succeed.

-Archie

--
Archie L. Cobbs

[Archie Cobbs]

Thinking about this more, I think it boils down to this question: do we
assume the entity creating software jfoobar is the same as the entity
creating the ivy.xml file for jfoobar?

In an ideal world, yes I agree: this would be true, and then there is no
question who the "organization" is.

However, in the real world, I don't see many projects shipping ivy.xml files
in their jfoobar.zip distributions....

For Ivy to get more popular and usable, there has to be a way for people to
just "plug & play" into an existing ivy distribution network of some kind
(like the one I'm proposing). Like CPAN is for perl, etc.

But for such networks to exist, Ivy has to support "3rd party" packaging.
Otherwise, there will have to be a "one true source" of ivy.xml files... and
who will that be? I don't see anyone stepping up to the plate at the moment.

And for 3rd party packaging to work (see below), we need to be able to
separate the "producing" organization from the "packaging" organization.

So you either have to make "organization" the packager.. or, add another
resolution dimension ("packager"?)

-Archie

On Wed, Apr 2, 2008 at 12:22 PM, Archie Cobbs <archie@...> wrote:

> On Wed, Apr 2, 2008 at 11:40 AM, Xavier Hanin <xavier.hanin@...>
> wrote:
>
> > >   2. What happens in ivy when two different repositories in your
> > >   settings publish the same organization, name, and version of a
> > module?
> >
> > It depends on your settings, but most probably the first one will be
> > used
> > and the second one ignored. The couple organization#module MUST identify
> > a
> > module uniquely. A module's revision is identified by
> > organization#module +
> > revision + extra attributes if any. But for a public repository I
> > wouldn't
> > use extra attributes, so sticking with pure organization#module;revision
> > as
> > identifier should be a good choice.
> >
>
> OK, so organization+module MUST be unique... but how do you expect to
> enforce your "MUST"?? Are you in control of all the Ivy repositories of the
> world? Or, is everyone expected to create their own private repository and
> never share?
>
> I think the "organization" attribute has to refer to the ivy module
> packager, not the originator of the code. Otherwise, the system seems
> totally broken to me.
>
> The only way you can expect organization+module to be unique is to require
> either:
>
>    1. Only the Foo organization can ever publish an ivy module with
>    organization="foo"; OR
>    2. There is some global authority designating who is the one true
>    authorized group that is allowed to put organization="foo" in their ivy.xml
>    files
>
> And option #2 is not an option unless you want to get into the business of
> ivy dictatorship.
>
> Examples: #1 is approach taken by Java package naming; #2 is approach
> taken by DNS domain system.
>
> Let's take an example, e.g. Hibernate. If someone (other than the
> Hibernate developers) creates an ivy module with organization="
> hibernate.org" name="hibernate" then a priori this module is going to be
> incompatible with the same thing created by someone else.
>
> Since there's no one true Ivy repository (RIP ivyrep) this is bound to
> happen: all the ivy repositories out there are going to be incompatible. And
> there's no clear way for me to specify that "I'd like the Hibernate module
> from repository A, but I'd like the Spring module from repository B".
>
> I'd suggest Ivy should encourage a decentralized model that is more
> compatible with the reality of the Internet. To me this seems like a pretty
> fundamental issue that needs to be examined for Ivy to succeed.
>
> -Archie
>
> --
> Archie L. Cobbs
>

--
Archie L. Cobbs

[Adrian Sandor]

Archie Cobbs wrote:

> Thinking about this more, I think it boils down to this question: do we
> assume the entity creating software jfoobar is the same as the entity
> creating the ivy.xml file for jfoobar?
>
> In an ideal world, yes I agree: this would be true, and then there is no
> question who the "organization" is.
>
> However, in the real world, I don't see many projects shipping ivy.xml files
> in their jfoobar.zip distributions....
>
> For Ivy to get more popular and usable, there has to be a way for people to
> just "plug & play" into an existing ivy distribution network of some kind
> (like the one I'm proposing). Like CPAN is for perl, etc.
>
> But for such networks to exist, Ivy has to support "3rd party" packaging.
> Otherwise, there will have to be a "one true source" of ivy.xml files... and
> who will that be? I don't see anyone stepping up to the plate at the moment.
>
> And for 3rd party packaging to work (see below), we need to be able to
> separate the "producing" organization from the "packaging" organization.
>
> So you either have to make "organization" the packager.. or, add another
> resolution dimension ("packager"?)
>
> -Archie
>  

I see your point, and I have some comments:
- The org. and module don't have to be unique *in the whole universe*,
but just in each repository; different repositories can have the same
module with the same org. but different packagers (and probably
different ivy descriptors)
- There may be 2 modules with the exact same name but different
organizations (creators); if the packager is the same, then how can you
distinguish them, other than by the org.?
- Anyway, I think it's a good idea to add an optional packager
attribute, and perhaps a packaging version (so that you can fix a
problem in the descriptor without overwriting it and without changing
the module revision), then we should think about how ivy will choose a
module when it's available with different packagers

Btw, I'm not an ivy developer, but just a tiny contributor.

Adrian

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...