Open source ivy files project?

[Xavier Hanin]

On Mon, Apr 7, 2008 at 4:56 PM, Archie Cobbs <archie@...> wrote:

> On Mon, Apr 7, 2008 at 2:56 AM, Xavier Hanin <xavier.hanin@...>
> wrote:
>
> > About the caching of build artifacts, caching doesn't occur in
> > findArtifactRef, so you should add some cache code before opening the
> > stream
> > on the builder artifact to get it cached.
> >
>
> I'm not completely sure I understand how the caching of JARs is supposed
> to
> work...
>
> I understand why the builder XML file would not be cached, but I don't
> understand why the JAR files are not cached. It seems like the caching of
> JAR files should be handled the same way as before, by the existing
> superclass code.
>
> It does look like the ivy.xml file is being cached however.
>
> Any pointers would be appreciated.

Sorry, no pointers to give, jar caching should work if you don't bypass the
default download method implementation in BasicResolver. Don't have time to
look back to your code, but add a breakpoint in the download method and see
how it works with your resolver, and you should see what's wrong about that.

>
>
>
> > Another thing to clear out: security. By downloading and executing ant
> > scripts, you give a lot of power to each module download. Hence checking
> > that downloaded scripts are trustable is very important. By using a
> custom
> > xml language and transforming it, security would be important at
> > installation time (Ivy + roundup resolver), and less after (well, code
> > download needs to be secure too for other reasons, but it's not specific
> > to
> > your resolver).
> >
>
> Yes, I've thought about this. I see a couple of possible answers:
>
>   1. We could argue that downloading an ant file from an URL is no less
>   secure that downloading a JAR from an URL. The latter is exactly what
> you're
>   doing when you use URLRepository pointed at (for example) the maven
>   repository: you are downloading code that you will execute, and you are
>   placing your trust in the maintainer of the website to not do anything
> evil.
>   The security is in the user controlling the URL. The same is true when
> you
>   configure and use a BuilderRepository.
>   2. We could add support for a "dumbed down" XML format that was safer
>   than a generic build.xml file and use XSLT to style it. Then the user
> could
>   (optionally) configure the resolver with allowAntBuildXML="false" or
>   whatever if they wanted to ensure no executable builder code was
> downloaded.
>   The "builder XML" would just get styled into build.xml after
> downloading. If
>   they left allowAntBuildXML="true" then we would fall back to the
>   existing "build.xml" method if no XML file was found.
>
> Do you "buy" the argument in #1?

Well, the problem is not strictly the same. In one case the resolver is
actually executing the ant script. In the other case, Ivy is only
downloading bytes, and you decide to execute them. But you can take care of
executing them only in a sandbox, or in a virtual machine with no rights, or
whatever. Ivy is not responsible for what you do with the downloaded
artifacts. I don't mean there is no security risk though, I just say it's
different.


The problem I see with #2 is that now we
> are adding a requirement for xalan (or equivalent) to be present when ivy
> runs. This doesn't work e.g. for the situation where ivy itself is used to
> download xalan :-)

Or you can use something else than xslt, like some java code. It's not
pretty, but we already do this frequently in Ivy: rewrite in Java what some
libraries already do, simply because we don't want to hit the chicken and
egg problem. If the language is simple, writing some java code to parse it
and write the ant script can be pretty straightforward.

>
>
> In any case, it is my intention to develop the XSLT required for #2
> anyway,
> because it will be useful for building the "roundup" online repository.
>
> Thinking about the future... of course I would hope that some form of this
> "builder resolver" code can be included in ivy itself. This would be an
> important step because it would help motivate people to start contributing
> to this idea of a community "ivy roundup" builder repository, and we can
> all
> start using and sharing a common pool of ivy meta-data.
>
> What other issues need to be addressed before this could happen?

Mainly test and documentation. By test I mean some automatic test, based on
junit, and using a repo packaged with Ivy sources (probably accessed with
file: URLs), as we do for almost everything in Ivy. This is the only way to
get something included in Ivy core, because once included it has to be
maintained, and maintaining stuff without automatic tests is a pain.

Then you also need to convince the committer team about the resolver
usefulness. I like the idea, and I think I'll defend the inclusion if I see
at least a beginning of growth of a public repo using this resolver.

Xavier


>
>
> Thanks,
> -Archie
>
> --
> Archie L. Cobbs
>



--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/

[Archie Cobbs]

On Mon, Apr 7, 2008 at 10:31 AM, Xavier Hanin <xavier.hanin@...> wrote:

The problem I see with #2 is that now we
> are adding a requirement for xalan (or equivalent) to be present when ivy
> runs. This doesn't work e.g. for the situation where ivy itself is used to
> download xalan :-)

Or you can use something else than xslt, like some java code. It's not
pretty, but we already do this frequently in Ivy: rewrite in Java what some
libraries already do, simply because we don't want to hit the chicken and
egg problem. If the language is simple, writing some java code to parse it
and write the ant script can be pretty straightforward.

I just realized that this is not really a problem, because we can delay the XSLT step until when we run "ant" (i.e., make that step part of the build.xml we generate), and <xslt> is a "core" ant task.

> What other issues need to be addressed before this could happen?

Mainly test and documentation. By test I mean some automatic test, based on
junit, and using a repo packaged with Ivy sources (probably accessed with
file: URLs), as we do for almost everything in Ivy. This is the only way to
get something included in Ivy core, because once included it has to be
maintained, and maintaining stuff without automatic tests is a pain.

Yes I agree unit tests should be a requirement.

Regarding security, I've made an initial version of the XSLT. See attached files for (3) sample "builder XML" for TestNG, (2) the XSLT, and (3) sample build.xml output.

There is a tradeoff between security and flexibility. I decided to allow a handful of basic ant tasks like <move>, <copy>, <zip>, <unzip>, etc. but otherwise everything is "controlled".

Let me know if you think I made the right tradeoffs here.

Thanks,
-Archie

--
Archie L. Cobbs

<?xml version="1.0" encoding="ISO-8859-1"?>

<!-- $Id$ -->
<builder-module>

    <!-- Property definitions are allowed -->
    <property name="name" value="testng"/>
    <property name="version" value="5.7"/>
    <property name="zipname" value="${name}-${version}"/>

    <!--
        The <resource> tag specifies some file to download, where to unpack it,
        and its expected checksum.

        The type (zip, tar, etc.) is determined automatically from the
        suffix, but it can also be explicitly overridden via "type"
        attribute, which can be one of: "none" (i.e., don't unpack),
        "zip", "tar", "tar.gz" (alias "tgz"), or "tar.bz2"

        The "dest" attribute specifies a subdirectory for unpacking the archive.

        The "url" and "sha1" attributes are self-explanatory and both required.

        Nested tags are optional; they become children of the <unzip>, etc.
    -->
    <resource dest="archive" url="http://testng.org/${zipname}.zip"
      sha1="2ea19275dc17453306f8bb780fe6ef6e9af7756b">
        <include name="${zipname}/src/main/**/*"/>
        <include name="${zipname}/src/jdk15/**/*"/>
        <include name="${zipname}/javadocs/**/*"/>
        <include name="${zipname}/*.jar"/>
    </resource>

    <!--
        Instructions that are reponsible for putting each artifact in place,
        where "in place" means copying it to "artifacts/[type]s/[artifact].[ext]".

        The only tags allowed as children of <build> are:

            <copy> <jar> <mkdir> <move> <tar>
            <unjar> <untar> <unwar> <unzip>
            <war> <zip>
    -->
    <build>

        <!-- jar  -->
        <move file="archive/${zipname}/${zipname}-jdk15.jar" tofile="artifacts/jars/${name}.jar"/>

        <!-- source -->
        <zip destfile="artifacts/sources/${name}.zip">
            <fileset dir="archive/${zipname}/src/main">
                <include name="**/*.java"/>
            </fileset>
            <fileset dir="archive/${zipname}/src/jdk15">
                <include name="**/*.java"/>
            </fileset>
        </zip>

        <!-- javadoc -->
        <zip destfile="artifacts/javadocs/javadoc.zip">
            <fileset dir="archive/${zipname}/javadocs"/>
        </zip>

    </build>

</builder-module>


<?xml version="1.0" encoding="UTF-8"?>

<!-- $Id$ -->
<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

    <xsl:output encoding="UTF-8" method="xml" indent="yes" media-type="text/xml"/>

    <xsl:template match="/builder-module">
        <xsl:comment> GENERATED FILE - DO NOT EDIT </xsl:comment>
        <xsl:comment> Generated by $Id$ </xsl:comment>
        <project name="builder-project" default="build">
            <xsl:apply-templates select="property"/>
            <target name="resources">
                <xsl:apply-templates select="resource"/>
            </target>
            <target name="build" depends="resources">
                <mkdir dir="artifacts/jars"/>
                <mkdir dir="artifacts/sources"/>
                <mkdir dir="artifacts/javadocs"/>
                <!-- ...add some other common artifact types here... -->
                <xsl:apply-templates select="build/*"/>
            </target>
        </project>
    </xsl:template>

    <!-- Properties -->
    <xsl:template match="/builder-module/property">
        <xsl:copy-of select="."/>
    </xsl:template>

    <!-- The allowed build actions -->
    <xsl:template match="/builder-module/build/copy"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/jar"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/mkdir"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/move"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/tar"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/unjar"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/untar"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/unwar"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/unzip"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/war"><xsl:copy-of select="."/></xsl:template>
    <xsl:template match="/builder-module/build/zip"><xsl:copy-of select="."/></xsl:template>

    <!-- Resource definitions -->
    <xsl:template match="/builder-module/resource">

        <!-- Determine type of archive (automatically or via "type" attribute) -->
        <xsl:variable name="type">
            <xsl:choose>
                <xsl:when test="@type">
                    <xsl:value-of select="@type"/>
                </xsl:when>
                <xsl:when test="substring(@url, string-length(@url) - 3) = '.zip'">zip</xsl:when>
                <xsl:when test="substring(@url, string-length(@url) - 3) = '.tar'">tar</xsl:when>
                <xsl:when test="substring(@url, string-length(@url) - 3) = '.tgz'">tar.gz</xsl:when>
                <xsl:when test="substring(@url, string-length(@url) - 6) = '.tar.gz'">tar.gz</xsl:when>
                <xsl:when test="substring(@url, string-length(@url) - 7) = '.tar.bz2'">tar.bz2</xsl:when>
                <xsl:otherwise>
                    <xsl:message terminate="yes">
                        <xsl:value-of select="concat('ERROR: cannot determine type of archive: ', @url, ' ')"/>
                        <xsl:value-of select="'Please set the "type" attribute to one of: none, zip, tar, tar.gz, or tar.bz2. '"/>
                    </xsl:message>
                </xsl:otherwise>
            </xsl:choose>
        </xsl:variable>

        <!-- Download resource -->
        <xsl:variable name="getdir" select="concat('directory.', generate-id())"/>
        <xsl:variable name="fileprop" select="concat('filename.', generate-id())"/>
        <xsl:variable name="filename" select="concat('${', $fileprop, '}')"/>
        <xsl:variable name="filepath" select="concat($getdir, '/', $filename)"/>
        <basename property="{$fileprop}" file="{@url}"/>
        <mkdir dir="{$getdir}"/>
        <get src="{@url}" dest="{$filepath}" verbose="true" usetimestamp="true"/>

        <!-- Verify checksum -->
        <xsl:variable name="csumprop" select="concat('checksum.', generate-id())"/>
        <xsl:variable name="csumvalue" select="concat('${', $csumprop, '}')"/>
        <checksum file="{$filepath}" algorithm="SHA" verifyproperty="{$csumprop}" property="{@sha1}"/>
        <fail message="checksum verification for {$filename} failed!">
            <condition>
                <isfalse value="{$csumvalue}"/>
            </condition>
        </fail>
        <echo message="successfully verified checksum of {$filename}"/>

        <!-- Unpack -->
        <mkdir dir="{@dest}"/>
        <xsl:choose>
            <xsl:when test="$type = 'none'">
                <move file="{$filepath}" todir="{@dest}"/>
            </xsl:when>
            <xsl:when test="$type = 'zip'">
                <unzip src="{$filepath}" dest="{@dest}">
                    <fileset dir=".">
                        <xsl:copy-of select="*"/>
                    </fileset>
                </unzip>
            </xsl:when>
            <xsl:when test="starts-with($type, 'tar')">
                <xsl:variable name="compression">
                    <xsl:choose>
                        <xsl:when test="$type = 'tar.gz'">gzip</xsl:when>
                        <xsl:when test="$type = 'tar.bz2'">bzip2</xsl:when>
                        <xsl:otherwise>none</xsl:otherwise>
                    </xsl:choose>
                </xsl:variable>
                <untar src="{$filepath}" dest="{@dest}">
                    <fileset dir=".">
                        <xsl:copy-of select="*"/>
                    </fileset>
                </untar>
            </xsl:when>
            <xsl:otherwise>
                <xsl:message terminate="yes">ERROR: internal error: impossible case </xsl:message>
            </xsl:otherwise>
        </xsl:choose>

        <!-- Delete download directory -->
        <delete dir="{$getdir}"/>
    </xsl:template>

    <!-- Ignore anything unexpected -->
    <xsl:template match="@*|node()"/>
</xsl:transform>


<?xml version="1.0" encoding="UTF-8"?>
<!-- GENERATED FILE - DO NOT EDIT -->
<!-- Generated by $Id$ -->
<project name="builder-project" default="build">
  <property name="name" value="testng"/>
  <property name="version" value="5.7"/>
  <property name="zipname" value="${name}-${version}"/>
  <target name="resources">
    <basename property="filename.id2245134" file="file:///home/archie/home.svn/hogwash/oss/testng/5.7/src/sources/testng-5.7.zip"/>
    <mkdir dir="directory.id2245134"/>
    <get src="file:///home/archie/home.svn/hogwash/oss/testng/5.7/src/sources/testng-5.7.zip" dest="directory.id2245134/${filename.id2245134}" verbose="true" usetimestamp="true"/>
    <checksum file="directory.id2245134/${filename.id2245134}" algorithm="SHA" verifyproperty="checksum.id2245134" property="2ea19275dc17453306f8bb780fe6ef6e9af7756b"/>
    <fail message="checksum verification for ${filename.id2245134} failed!">
      <condition>
        <isfalse value="${checksum.id2245134}"/>
      </condition>
    </fail>
    <echo message="verified checksum of ${filename.id2245134}"/>
    <mkdir dir="archive"/>
    <unzip src="directory.id2245134/${filename.id2245134}" dest="archive">
      <fileset dir=".">
        <include name="${zipname}/src/main/**/*"/>
        <include name="${zipname}/src/jdk15/**/*"/>
        <include name="${zipname}/javadocs/**/*"/>
        <include name="${zipname}/*.jar"/>
      </fileset>
    </unzip>
    <delete dir="directory.id2245134"/>
  </target>
  <target name="build" depends="resources">
    <mkdir dir="artifacts/jars"/>
    <mkdir dir="artifacts/sources"/>
    <mkdir dir="artifacts/javadocs"/>
    <move file="archive/${zipname}/${zipname}-jdk15.jar" tofile="artifacts/jars/${name}.jar"/>
    <zip destfile="artifacts/sources/${name}.zip">
            <fileset dir="archive/${zipname}/src/main">
                <include name="**/*.java"/>
            </fileset>
            <fileset dir="archive/${zipname}/src/jdk15">
                <include name="**/*.java"/>
            </fileset>
        </zip>
    <zip destfile="artifacts/javadocs/javadoc.zip">
            <fileset dir="archive/${zipname}/javadocs"/>
        </zip>
  </target>
</project>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Archie Cobbs]

I've added a JUnit test and documentation for the new builder resolver in the attached patch.

The caching problem is fixed, and XSLT is used to ensure valid checksums and restrict ant functionality.

I've also just barely started an online repository: http://code.google.com/p/ivyroundup/

Let me know what you think.

Thanks,
-Archie

2008/4/7 Archie Cobbs <archie@...>:

On Mon, Apr 7, 2008 at 10:31 AM, Xavier Hanin <xavier.hanin@...> wrote:

The problem I see with #2 is that now we
> are adding a requirement for xalan (or equivalent) to be present when ivy
> runs. This doesn't work e.g. for the situation where ivy itself is used to
> download xalan :-)

Or you can use something else than xslt, like some java code. It's not
pretty, but we already do this frequently in Ivy: rewrite in Java what some
libraries already do, simply because we don't want to hit the chicken and
egg problem. If the language is simple, writing some java code to parse it
and write the ant script can be pretty straightforward.

I just realized that this is not really a problem, because we can delay the XSLT step until when we run "ant" (i.e., make that step part of the build.xml we generate), and <xslt> is a "core" ant task.

> What other issues need to be addressed before this could happen?

Mainly test and documentation. By test I mean some automatic test, based on
junit, and using a repo packaged with Ivy sources (probably accessed with
file: URLs), as we do for almost everything in Ivy. This is the only way to
get something included in Ivy core, because once included it has to be
maintained, and maintaining stuff without automatic tests is a pain.

Yes I agree unit tests should be a requirement.

Regarding security, I've made an initial version of the XSLT. See attached files for (3) sample "builder XML" for TestNG, (2) the XSLT, and (3) sample build.xml output.

There is a tradeoff between security and flexibility. I decided to allow a handful of basic ant tasks like <move>, <copy>, <zip>, <unzip>, etc. but otherwise everything is "controlled".

Let me know if you think I made the right tradeoffs here.

Thanks,
-Archie

--
Archie L. Cobbs

--
Archie L. Cobbs


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Xavier Hanin]

2008/4/7 Archie Cobbs <archie@...>:

> On Mon, Apr 7, 2008 at 10:31 AM, Xavier Hanin <xavier.hanin@...>
> wrote:
>
> > The problem I see with #2 is that now we
> > > are adding a requirement for xalan (or equivalent) to be present when
> > ivy
> > > runs. This doesn't work e.g. for the situation where ivy itself is
> > used to
> > > download xalan :-)
> >
> > Or you can use something else than xslt, like some java code. It's not
> > pretty, but we already do this frequently in Ivy: rewrite in Java what
> > some
> > libraries already do, simply because we don't want to hit the chicken
> > and
> > egg problem. If the language is simple, writing some java code to parse
> > it
> > and write the ant script can be pretty straightforward.
> >
>
> I just realized that this is not really a problem, because we can delay
> the XSLT step until when we run "ant" (i.e., make that step part of the
> build.xml we generate), and <xslt> is a "core" ant task.
>
> > What other issues need to be addressed before this could happen?
> >
> > Mainly test and documentation. By test I mean some automatic test, based
> > on
> > junit, and using a repo packaged with Ivy sources (probably accessed
> > with
> > file: URLs), as we do for almost everything in Ivy. This is the only way
> > to
> > get something included in Ivy core, because once included it has to be
> > maintained, and maintaining stuff without automatic tests is a pain.
> >
>
> Yes I agree unit tests should be a requirement.
>
> Regarding security, I've made an initial version of the XSLT. See attached
> files for (3) sample "builder XML" for TestNG, (2) the XSLT, and (3) sample
> build.xml output.
>
> There is a tradeoff between security and flexibility. I decided to allow a
> handful of basic ant tasks like <move>, <copy>, <zip>, <unzip>, etc. but
> otherwise everything is "controlled".
>
> Let me know if you think I made the right tradeoffs here.
>

I really like it! It's simple to understand, powerful enough yet under
control. One thing that would be nice to ensure a good deal of security
would be to allow only relative paths in the file related operations. This
would ensure that the module builder has only access to a very limited part
of the filesystem.

I also like the fact that you make sha1 mandatory on the resource element.
The drawback is that it means that we won't be able to download something
which has no available sha1.

One thing which really needs to be done IMHO is the maven import operation.
Many artifacts are available on maven repository, so I think good support
for importing artifacts from maven would be great. If we provide the org,
name and revision values to the build file or the xslt, something as simple
as <m2resource qualifiers="default,source,javadoc" /> could be enough when
we use the same org/name as maven groupId/artifactId. In other cases it
could be <m2resource groupId="com.acme.foo" artifactId="foo-bar"
qualifiers="default,source,javadoc" />. Maybe we should put qualifiers as
subtags to allow defining expected sha1 for each artifact. This would be
converted in a download + checksum check + move to artifacts directory
operation. What do you think?

Xavier

>
>
> Thanks,
> -Archie
>
> --
> Archie L. Cobbs
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/

[Archie Cobbs]

On Tue, Apr 8, 2008 at 2:16 AM, Xavier Hanin <xavier.hanin@...> wrote:

> I really like it! It's simple to understand, powerful enough yet under
> control. One thing that would be nice to ensure a good deal of security
> would be to allow only relative paths in the file related operations. This
> would ensure that the module builder has only access to a very limited
> part
> of the filesystem.
>

Yes I thought of that as well but was not sure I could squash every possible
case. For example, I thought of disallowing leading slashes, but that
doesn't work, because I could use a property to subvert this by saying:

  <property name="hack" value="/boot"/>
  ...
  <move file="foobar" to="${hack}/vmlinux"/>

On the other hand, we could just get rid of <property> support... though
that would make maintaining the builder.xml files more tedious. For example,
one thing I like about the current system is that the builder.xml files for
different revisions of a module can be identical (except for SHA1), thanks
to the use of ${ivy.builder.revision}, etc.

There are other possible problems with the children tags of <move>, <copy>,
etc. such as the use of resource collections... ?

In other words I'm definitely willing to try to tighten up the security
aspects but we need to think carefully through all of the possible cases or
else it won't be worth the trouble (can't float a boat having only 95% of
the leaks fixed).

Another possible direction to take would be cryptographic signatures on the
builder.xml files, similar to how digitally signed RPMs work. Maybe longer
term, as this is more of a maintenance pain and places a burden on the user
to find, evaluate, and import trusted keys.

I also like the fact that you make sha1 mandatory on the resource element.
> The drawback is that it means that we won't be able to download something
> which has no available sha1.
>

Not a problem... the person who creates the builder.xml file can be
responsible for computing the checksum manually if necessary.

> One thing which really needs to be done IMHO is the maven import
> operation.
> Many artifacts are available on maven repository, so I think good support
> for importing artifacts from maven would be great. If we provide the org,
> name and revision values to the build file or the xslt, something as
> simple
> as <m2resource qualifiers="default,source,javadoc" /> could be enough when
> we use the same org/name as maven groupId/artifactId. In other cases it
> could be <m2resource groupId="com.acme.foo" artifactId="foo-bar"
> qualifiers="default,source,javadoc" />. Maybe we should put qualifiers as
> subtags to allow defining expected sha1 for each artifact. This would be
> converted in a download + checksum check + move to artifacts directory
> operation. What do you think?
>

I agree completely and this should be very easy to do... unfortunately I
have zero knowledge of maven. I do have lots of knowledge of XSLT though so
if someone could walk me through what steps need to be done for a
<m2resource> tag then I'll be happy to write it up in the XSLT.

Thanks,
-Archie

--
Archie L. Cobbs

[Xavier Hanin]

On Tue, Apr 8, 2008 at 6:20 PM, Archie Cobbs <archie@...> wrote:

> On Tue, Apr 8, 2008 at 2:16 AM, Xavier Hanin <xavier.hanin@...>
> wrote:
>
> > I really like it! It's simple to understand, powerful enough yet under
> > control. One thing that would be nice to ensure a good deal of security
> > would be to allow only relative paths in the file related operations.
> This
> > would ensure that the module builder has only access to a very limited
> > part
> > of the filesystem.
> >
>
> Yes I thought of that as well but was not sure I could squash every
> possible
> case. For example, I thought of disallowing leading slashes, but that
> doesn't work, because I could use a property to subvert this by saying:
>
>  <property name="hack" value="/boot"/>
>  ...
>  <move file="foobar" to="${hack}/vmlinux"/>
>
> On the other hand, we could just get rid of <property> support... though
> that would make maintaining the builder.xml files more tedious. For
> example,
> one thing I like about the current system is that the builder.xml files
> for
> different revisions of a module can be identical (except for SHA1), thanks
> to the use of ${ivy.builder.revision}, etc.
>
> There are other possible problems with the children tags of <move>,
> <copy>,
> etc. such as the use of resource collections... ?
>
> In other words I'm definitely willing to try to tighten up the security
> aspects but we need to think carefully through all of the possible cases
> or
> else it won't be worth the trouble (can't float a boat having only 95% of
> the leaks fixed).

Yes, maybe this should be kept for later, because it's not easy to achieve.
One way to do it would be to use our own ant tasks instead of standard
tasks, those tasks extending the standard one and checking that everything
is done in sandbox only. But then we'd have to use custom tasks in Ant,
which require one more step in install...


> Another possible direction to take would be cryptographic signatures on
> the
> builder.xml files, similar to how digitally signed RPMs work. Maybe longer
> term, as this is more of a maintenance pain and places a burden on the
> user
> to find, evaluate, and import trusted keys.

Indeed.

>
>
> I also like the fact that you make sha1 mandatory on the resource element.
> > The drawback is that it means that we won't be able to download
> something
> > which has no available sha1.
> >
>
> Not a problem... the person who creates the builder.xml file can be
> responsible for computing the checksum manually if necessary.

You're absolutely right!

>
>
>
> > One thing which really needs to be done IMHO is the maven import
> > operation.
> > Many artifacts are available on maven repository, so I think good
> support
> > for importing artifacts from maven would be great. If we provide the
> org,
> > name and revision values to the build file or the xslt, something as
> > simple
> > as <m2resource qualifiers="default,source,javadoc" /> could be enough
> when
> > we use the same org/name as maven groupId/artifactId. In other cases it
> > could be <m2resource groupId="com.acme.foo" artifactId="foo-bar"
> > qualifiers="default,source,javadoc" />. Maybe we should put qualifiers
> as
> > subtags to allow defining expected sha1 for each artifact. This would be
> > converted in a download + checksum check + move to artifacts directory
> > operation. What do you think?
> >
>
> I agree completely and this should be very easy to do... unfortunately I
> have zero knowledge of maven. I do have lots of knowledge of XSLT though
> so
> if someone could walk me through what steps need to be done for a
> <m2resource> tag then I'll be happy to write it up in the XSLT.

If it's only to download artifacts, it's very easy. All you need is to
download the artifacts using a pattern like this:
[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]

The classifier needs to be provided in the m2resource tag, except for the
main artifact, which has no classifier.

For instance here are the files available for Ivy itself on maven repo:
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.0.0-beta2/

As you can see there's the default artifact (the jar), an artifact for
'sources' classifier, and one for 'javadoc' classifier. Maybe we should also
be able to specify the ext for each artifact, and also the file path to use
once downloaded (relative to artifacts dir). Maybe something like:
<m2resource groupId="org.apache.ivy" artifactId="ivy" version="2.0.0-beta2">
  <artifact ext="jar" tofile="jars/ivy.jar"
sha1="43188890f8eb2a105665d62c4bda4b24703568ee" />
  <artifact qualifier="sources" ext="jar" tofile="sources/ivy.zip"
sha1="6ab99abd8a02a961a5054b0359b9ae75f2ae2972" />
  <artifact qualifier="javadoc" ext="jar" tofile="javadocs/javadoc.zip"
sha1="6b3cf2877a1c79adb181fa218b99f3b20ceabbe5" />
</m2resource>

Not sure of the exact syntax, but you get the idea.

Xavier


>
> Thanks,
> -Archie
>
> --
> Archie L. Cobbs
>



--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/

[Archie Cobbs]

On Tue, Apr 8, 2008 at 4:07 PM, Xavier Hanin <xavier.hanin@...> wrote:

> I agree completely and this should be very easy to do... unfortunately I
> have zero knowledge of maven. I do have lots of knowledge of XSLT though
> so if someone could walk me through what steps need to be done for a
> <m2resource> tag then I'll be happy to write it up in the XSLT.

If it's only to download artifacts, it's very easy. All you need is to
download the artifacts using a pattern like this:
[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]

The classifier needs to be provided in the m2resource tag, except for the
main artifact, which has no classifier.

For instance here are the files available for Ivy itself on maven repo:
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.0.0-beta2/

As you can see there's the default artifact (the jar), an artifact for
'sources' classifier, and one for 'javadoc' classifier. Maybe we should also
be able to specify the ext for each artifact, and also the file path to use
once downloaded (relative to artifacts dir). Maybe something like:
<m2resource groupId="org.apache.ivy" artifactId="ivy" version="2.0.0-beta2">
 <artifact ext="jar" tofile="jars/ivy.jar"
sha1="43188890f8eb2a105665d62c4bda4b24703568ee" />
 <artifact qualifier="sources" ext="jar" tofile="sources/ivy.zip"
sha1="6ab99abd8a02a961a5054b0359b9ae75f2ae2972" />
 <artifact qualifier="javadoc" ext="jar" tofile="javadocs/javadoc.zip"
sha1="6b3cf2877a1c79adb181fa218b99f3b20ceabbe5" />
</m2resource>

Not sure of the exact syntax, but you get the idea.

OK I think this should be easy to do... see updated patch (attached) which includes <m2resource>, ivy documentation, and new schema validation.

This does bring up a larger question I want to ask first: are we sure we really want to create and maintain an ivy module in ivyroundup for every package in the maven2 repository?

Obviously, ivy is better than maven :-) but what happens if we don't? Someone could always just configure one builder resolver to point at ivyroundup and the other ibiblio resovler at the maven2 repo.

But let's assume we do want to do this. For this to be feasible, we need to ensure the maintenance of all these modules is not a huge burden, and that they add some kind of value.

A definite requirement would a custom ant task in the ivyroundup project that automatically populates the ivyroundup repository with all the maven2 projects by downloading every pom and meta-data file from the maven2 repo and processing them automatically.

I'm willing to write this but I'd need some help/advice on what exactly to do since I'm still learning about maven.

Once this is done, over time people can go through and improve/refine the auto-generated ivy.xml files... assuming we have some volunteers who are interested in specific projects, etc.

Does that make sense?

In any case, I've populated the Ivy RoundUp web site with a few modules, including one maven2 one (commons-email). Please take a look and let me know if it looks reasonable to you so far.

Thanks,
-Archie

--
Archie L. Cobbs


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Archie Cobbs]

Xavier et.al.,

Please take a look at the latest Builder resolver patch
(here<http://code.google.com/p/ivyroundup/wiki/files/builder-resolver.patch.gz>)
and let me know what else is needed before this is suitable for inclusion in
Ivy. Hopefully it's pretty close (the documentation gives a good overview).

I'd also appreciate any comments on the roundup web
site<http://code.google.com/p/ivyroundup>and am looking for any
interested contributors/maintainers to join the
project.

Thanks,
-Archie

--
Archie L. Cobbs

[Xavier Hanin]

On Wed, Apr 9, 2008 at 9:14 PM, Archie Cobbs <archie@...> wrote:

> On Tue, Apr 8, 2008 at 4:07 PM, Xavier Hanin <xavier.hanin@...>
> wrote:
>
> > > I agree completely and this should be very easy to do... unfortunately
> > I
> > > have zero knowledge of maven. I do have lots of knowledge of XSLT
> > though
> > > so if someone could walk me through what steps need to be done for a
> > > <m2resource> tag then I'll be happy to write it up in the XSLT.
> >
> > If it's only to download artifacts, it's very easy. All you need is to
> > download the artifacts using a pattern like this:
> >
> > [organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]
> >
> > The classifier needs to be provided in the m2resource tag, except for
> > the
> > main artifact, which has no classifier.
> >
> > For instance here are the files available for Ivy itself on maven repo:
> > http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.0.0-beta2/
> >
> > As you can see there's the default artifact (the jar), an artifact for
> > 'sources' classifier, and one for 'javadoc' classifier. Maybe we should
> > also
> > be able to specify the ext for each artifact, and also the file path to
> > use
> > once downloaded (relative to artifacts dir). Maybe something like:
> > <m2resource groupId="org.apache.ivy" artifactId="ivy"
> > version="2.0.0-beta2">
> >  <artifact ext="jar" tofile="jars/ivy.jar"
> > sha1="43188890f8eb2a105665d62c4bda4b24703568ee" />
> >  <artifact qualifier="sources" ext="jar" tofile="sources/ivy.zip"
> > sha1="6ab99abd8a02a961a5054b0359b9ae75f2ae2972" />
> >  <artifact qualifier="javadoc" ext="jar" tofile="javadocs/javadoc.zip"
> > sha1="6b3cf2877a1c79adb181fa218b99f3b20ceabbe5" />
> > </m2resource>
> >
> > Not sure of the exact syntax, but you get the idea.
> >
>
> OK I think this should be easy to do... see updated patch (attached) which
> includes <m2resource>, ivy documentation, and new schema validation.
>

I've quickly reviewed the patch, and it seems to be exactly what I was
thinking.

>
>
> This does bring up a larger question I want to ask first: are we sure we
> really want to create and maintain an ivy module in ivyroundup for every
> package in the maven2 repository?
>
> Obviously, ivy is better than maven :-) but what happens if we don't?
> Someone could always just configure one builder resolver to point at
> ivyroundup and the other ibiblio resovler at the maven2 repo.
>

If people do this, we will probably end up with one of the problems of maven
2 repo: inconsistent names. If people need to use ivyroundup with maven2,
they will probably create dependencies in roundup to maven2 repo, and so you
quickly get inconsistent names. I believe to have something clean we need
both tools to ease the import of modules from maven2 repo to ivy roundup AND
community check of the quality of what is imported.

>
> But let's assume we do want to do this. For this to be feasible, we need
> to ensure the maintenance of all these modules is not a huge burden, and
> that they add some kind of value.
>
> A definite requirement would a custom ant task in the ivyroundup project
> that automatically populates the ivyroundup repository with all the maven2
> projects by downloading every pom and meta-data file from the maven2 repo
> and processing them automatically.
>

For many cases, this is fine. But as I said, if we don't want to replaicate
the same problems as the maven2 repo, we can't do things that automatically:
we need human or community checks of what is imported. For some modules,
when we know creators provide good metadata, we could fully automate the
import of new versions. In other cases, we'd need to change the organization
or module name, and keep the rest. In worst cases, only the jar could be
reusable, while we'd still need to find sources and javadocs ourselves. This
is quite a huge work, and as such require a community of motivated people.


>
> I'm willing to write this but I'd need some help/advice on what exactly to
> do since I'm still learning about maven.
>
You don't have much to know to do this: Ivy is already able to convert poms
to ivy files. So the main requirement is to understand that groupId is
organization in Ivy, and artifactId is module name in Ivy. Then there is the
conversion of dots in slashes in the repository for groupId, and the concept
of qualifier, which is mainly used for javadoc and sources, and which is not
stored in metadata: you have no way to know which qualified artifacts are
available except by listing files in the module revision directory.

>
>
> Once this is done, over time people can go through and improve/refine the
> auto-generated ivy.xml files... assuming we have some volunteers who are
> interested in specific projects, etc.
>
I think we at least need to avoid having the same inconsistencies in names
from the beginning. Having clear rules for module names and orgs (like
following the package name convention) is the only way to avoid the problems
you have when you search the maven repo for a module and end up with what
seem to be the good answer with different organizations, and different
versions. If we mimic maven2 repo, I see no added value. Obviously, doing
this kind of work takes time...

>
>
> Does that make sense?
>
> In any case, I've populated the Ivy RoundUp web site with a few modules,
> including one maven2 one (commons-email). Please take a look and let me know
> if it looks reasonable to you so far.
>
I'm already quite lost with the modules, which demonstrate the need to clean
the names before doing the import. I see commons modules in their own
organization (exactly as in maven2 repo) and commons-email in
org.apache.commons organization (which makes better sense to me). This once
again shows the difficulty to do something really better than maven2 repo.

Xavier

>
>
> Thanks,
> -Archie
>
> --
> Archie L. Cobbs
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/

[Gilles Scokart]

Maven has naming conventions [1], [2].

The problem is that those conventions apeared  with maven 2.  maven 1
didn't had this and the maven repository actully contains an export of
the maven 1 repository.  That's why the naming convention didn't
seemed to be always followed.

Note however, that they give advices to fix that [3].


[1] http://maven.apache.org/guides/mini/guide-naming-conventions.html
[2] http://maven.apache.org/maven-conventions.html
[3] http://maven.apache.org/guides/mini/guide-relocation.html


On 16/04/2008, Xavier Hanin <xavier.hanin@...> wrote:

> On Wed, Apr 9, 2008 at 9:14 PM, Archie Cobbs <archie@...> wrote:
>
>  > On Tue, Apr 8, 2008 at 4:07 PM, Xavier Hanin <xavier.hanin@...>
>  > wrote:
>  >
>  > > > I agree completely and this should be very easy to do... unfortunately
>  > > I
>  > > > have zero knowledge of maven. I do have lots of knowledge of XSLT
>  > > though
>  > > > so if someone could walk me through what steps need to be done for a
>  > > > <m2resource> tag then I'll be happy to write it up in the XSLT.
>  > >
>  > > If it's only to download artifacts, it's very easy. All you need is to
>  > > download the artifacts using a pattern like this:
>  > >
>  > > [organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]
>  > >
>  > > The classifier needs to be provided in the m2resource tag, except for
>  > > the
>  > > main artifact, which has no classifier.
>  > >
>  > > For instance here are the files available for Ivy itself on maven repo:
>  > > http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.0.0-beta2/
>  > >
>  > > As you can see there's the default artifact (the jar), an artifact for
>  > > 'sources' classifier, and one for 'javadoc' classifier. Maybe we should
>  > > also
>  > > be able to specify the ext for each artifact, and also the file path to
>  > > use
>  > > once downloaded (relative to artifacts dir). Maybe something like:
>  > > <m2resource groupId="org.apache.ivy" artifactId="ivy"
>  > > version="2.0.0-beta2">
>  > >  <artifact ext="jar" tofile="jars/ivy.jar"
>  > > sha1="43188890f8eb2a105665d62c4bda4b24703568ee" />
>  > >  <artifact qualifier="sources" ext="jar" tofile="sources/ivy.zip"
>  > > sha1="6ab99abd8a02a961a5054b0359b9ae75f2ae2972" />
>  > >  <artifact qualifier="javadoc" ext="jar" tofile="javadocs/javadoc.zip"
>  > > sha1="6b3cf2877a1c79adb181fa218b99f3b20ceabbe5" />
>  > > </m2resource>
>  > >
>  > > Not sure of the exact syntax, but you get the idea.
>  > >
>  >
>  > OK I think this should be easy to do... see updated patch (attached) which
>  > includes <m2resource>, ivy documentation, and new schema validation.
>  >
>
> I've quickly reviewed the patch, and it seems to be exactly what I was
>  thinking.
>
>
>  >
>  >
>  > This does bring up a larger question I want to ask first: are we sure we
>  > really want to create and maintain an ivy module in ivyroundup for every
>  > package in the maven2 repository?
>  >
>  > Obviously, ivy is better than maven :-) but what happens if we don't?
>  > Someone could always just configure one builder resolver to point at
>  > ivyroundup and the other ibiblio resovler at the maven2 repo.
>  >
>
>
> If people do this, we will probably end up with one of the problems of maven
>  2 repo: inconsistent names. If people need to use ivyroundup with maven2,
>  they will probably create dependencies in roundup to maven2 repo, and so you
>  quickly get inconsistent names. I believe to have something clean we need
>  both tools to ease the import of modules from maven2 repo to ivy roundup AND
>  community check of the quality of what is imported.
>
>
>
>  >
>  > But let's assume we do want to do this. For this to be feasible, we need
>  > to ensure the maintenance of all these modules is not a huge burden, and
>  > that they add some kind of value.
>  >
>  > A definite requirement would a custom ant task in the ivyroundup project
>  > that automatically populates the ivyroundup repository with all the maven2
>  > projects by downloading every pom and meta-data file from the maven2 repo
>  > and processing them automatically.
>  >
>
> For many cases, this is fine. But as I said, if we don't want to replaicate
>  the same problems as the maven2 repo, we can't do things that automatically:
>  we need human or community checks of what is imported. For some modules,
>  when we know creators provide good metadata, we could fully automate the
>  import of new versions. In other cases, we'd need to change the organization
>  or module name, and keep the rest. In worst cases, only the jar could be
>  reusable, while we'd still need to find sources and javadocs ourselves. This
>  is quite a huge work, and as such require a community of motivated people.
>
>
>
>  >
>  > I'm willing to write this but I'd need some help/advice on what exactly to
>  > do since I'm still learning about maven.
>  >
>
> You don't have much to know to do this: Ivy is already able to convert poms
>  to ivy files. So the main requirement is to understand that groupId is
>  organization in Ivy, and artifactId is module name in Ivy. Then there is the
>  conversion of dots in slashes in the repository for groupId, and the concept
>  of qualifier, which is mainly used for javadoc and sources, and which is not
>  stored in metadata: you have no way to know which qualified artifacts are
>  available except by listing files in the module revision directory.
>
>
>  >
>  >
>  > Once this is done, over time people can go through and improve/refine the
>  > auto-generated ivy.xml files... assuming we have some volunteers who are
>  > interested in specific projects, etc.
>  >
>
> I think we at least need to avoid having the same inconsistencies in names
>  from the beginning. Having cl