Artifactory retrieving artifacts outside of "walled" repository

View: New views
7 Messages — Rating Filter:   Alert me  

Artifactory retrieving artifacts outside of "walled" repository

by blalor :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Artifactory retrieving artifacts outside of "walled" repository

Morning, all.  I'm working on setting up an Artifactory repostiory within my workgroup.  I want to have a single "blessed" or "walled" repository that contains all of the artifacts needed for a given circumstance, and I want to deny maven clients using that repository from ever going outside of that allowed set of artifacts.  I started this effort a few months ago with just a static repository structure served up by Apache httpd.  I'm working with some existing legacy (ie. broken by those who don't understand Maven) projects where developers have shipped a local ~/.m2/repository directory around to each other and always built using offline mode. 

So, rather than set up a proper <repositories/> element in each project's pom.xml, I'm trying to override *all* repositories with <mirrors/> in ~/.m2/settings.xml.  I started by creating a localRepository in Artifactory:

        <localRepository>
            <key>myapp-frozen</key>
            <description>Local repository used for building MyApp</description>
            <handleReleases>true</handleReleases>
            <handleSnapshots>true</handleSnapshots>
        </localRepository>

I imported the artifacts from my old static repository into the myapp-frozen repository in Artifactory.  Then I added the following <mirror/> to ~/.m2/settings.xml:

        <mirror>
            <id>maven-proxy</id>
            <name>Maven-Proxy Mirror</name>
            <url>http://zoot/artifactory/myapp-frozen</url>
            <mirrorOf>*</mirrorOf>
        </mirror>

So, previously, with the httpd-served repository, if the maven client tried to request an artifact not in the repository, it would fail.  Now, using Artifactory and the myapp-frozen repository, if a new artifact is requested (I noticed this when upgrading to a new version of Maven on my workstation), Artifactory happily runs off and retrieves it.  Here are snippets from artifactory.log and request.log at the time the requests were made:

request.log:
"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.pom HTTP/1.1" 200 4849

"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.pom.sha1 HTTP/1.1" 200 40

"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.jar HTTP/1.1" 200 59010

"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.jar.sha1 HTTP/1.1" 200 40

artifactory.log:
[INFO ] (HttpRepo.java:128{10}) - repo1: Retrieving 'http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.pom'

[INFO ] (HttpRepo.java:128{10}) - repo1: Retrieving 'http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.jar'

So, why is Artifactory retrieving resources from repo1 when I'm asking for them explicitly from myapp-frozen?

Thanks,
Brian


--
Brian Lalor
Fidelity Life Insurance Company (FILI) - Systems & Technology
Merrimack, NH
603-791-8612
brian.lalor@...



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory retrieving artifacts outside of "walled" repository

by freddy33 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

This is a known bug of Maven. Please refer to http://www.jfrog.org/confluence/display/RTF/Configuring+Maven
By the way, in maven 2.0.8 and above there is a mirror-any feature.
Did someone used it? Is it solving the bug?
Another issue you have to be careful with is plugin that can be executed without pom.xml file, like archetype. In this case repository settings in profile are not loaded and so maven is going to repo1. The workaround is to have a dummy pom.xml where you are executing the plugin.

Good luck.

On Thu, May 22, 2008 at 3:30 PM, Lalor, Brian <Brian.Lalor@...> wrote:

Morning, all.  I'm working on setting up an Artifactory repostiory within my workgroup.  I want to have a single "blessed" or "walled" repository that contains all of the artifacts needed for a given circumstance, and I want to deny maven clients using that repository from ever going outside of that allowed set of artifacts.  I started this effort a few months ago with just a static repository structure served up by Apache httpd.  I'm working with some existing legacy (ie. broken by those who don't understand Maven) projects where developers have shipped a local ~/.m2/repository directory around to each other and always built using offline mode. 

So, rather than set up a proper <repositories/> element in each project's pom.xml, I'm trying to override *all* repositories with <mirrors/> in ~/.m2/settings.xml.  I started by creating a localRepository in Artifactory:

        <localRepository>
            <key>myapp-frozen</key>
            <description>Local repository used for building MyApp</description>
            <handleReleases>true</handleReleases>
            <handleSnapshots>true</handleSnapshots>
        </localRepository>

I imported the artifacts from my old static repository into the myapp-frozen repository in Artifactory.  Then I added the following <mirror/> to ~/.m2/settings.xml:

        <mirror>
            <id>maven-proxy</id>
            <name>Maven-Proxy Mirror</name>
            <url>http://zoot/artifactory/myapp-frozen</url>
            <mirrorOf>*</mirrorOf>
        </mirror>

So, previously, with the httpd-served repository, if the maven client tried to request an artifact not in the repository, it would fail.  Now, using Artifactory and the myapp-frozen repository, if a new artifact is requested (I noticed this when upgrading to a new version of Maven on my workstation), Artifactory happily runs off and retrieves it.  Here are snippets from artifactory.log and request.log at the time the requests were made:

request.log:
"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.pom HTTP/1.1" 200 4849

"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.pom.sha1 HTTP/1.1" 200 40

"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.jar HTTP/1.1" 200 59010

"GET /artifactory/myapp-frozen/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.jar.sha1 HTTP/1.1" 200 40

artifactory.log:
[INFO ] (HttpRepo.java:128{10}) - repo1: Retrieving 'http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.pom'

[INFO ] (HttpRepo.java:128{10}) - repo1: Retrieving 'http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-war-plugin/2.1-alpha-1/maven-war-plugin-2.1-alpha-1.jar'

So, why is Artifactory retrieving resources from repo1 when I'm asking for them explicitly from myapp-frozen?

Thanks,
Brian


--
Brian Lalor
Fidelity Life Insurance Company (FILI) - Systems & Technology
Merrimack, NH
603-791-8612
brian.lalor@...



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users




--
http://freddy33.blogspot.com/
http://www.jfrog.org/
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory retrieving artifacts outside of"walled" repository

by blalor :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message


From: artifactory-users-bounces@... [mailto:artifactory-users-bounces@...] On Behalf Of Frederic Simon
Sent: Thursday, May 22, 2008 8:52 AM
To: artifactory-users@...
Subject: Re: [Artifactory-users] Artifactory retrieving artifacts outside of"walled" repository

This is a known bug of Maven. Please refer to http://www.jfrog.org/confluence/display/RTF/Configuring+Maven
By the way, in maven 2.0.8 and above there is a mirror-any feature.
Did someone used it? Is it solving the bug?
Another issue you have to be careful with is plugin that can be executed without pom.xml file, like archetype. In this case repository settings in profile are not loaded and so maven is going to repo1. The workaround is to have a dummy pom.xml where you are executing the plugin. 
 
Perhaps I wasn't clear.  I'm using Maven v2.0.9 in this particular instance.  Poor POM management is leading to Maven trying to retrieve newer versions of plugins that aren't in my "frozen" repository.

It's also Artifactory (v1.2.5) that is retrieving the artifacts not in the "myapp-frozen" Artifactory repository.  The client is trying to retrieve a non-existant artifact from an Artifactory localRepository, and Artifactory (mistakenly?) goes ahead and retreives it from repo1 instead of returning a 404.  Maven is doing what it's told and only querying Artifactory.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory retrieving artifacts outside of"walled" repository

by freddy33 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

OK, sorry did not see the log was from Artifactory!
Some questions:
- Does "myapp-frozen" contains the 2.1-alpha-1 version of the war plugin?
- If no, your goall is to stop the client from trying to get it?
- Maven is trying to retrieve this version because it's in  the maven-metadata.xml file. This file is aggregated by Artifactory from the list of accessible repos given for a virtual repo name.

So what I suggest:
- Make sure the maven-metadata.xml file in myapp-frozen does not contains version that should not be there (who that you don't want).
- By default the "virtual repository" represented by the url http://zoot/artifactory/myapp-frozen is the local repo "myapp-frozen" PLUS all the remote repository. So, to change this behavior you need to create a virtual repository of your own which contains only the local and remote repositories you trust and are frozen. Please check http://www.jfrog.org/confluence/display/RTF/Configuring+Artifactory#ConfiguringArtifactory-VirtualRepositories

Hope this should solve the issue.

On Thu, May 22, 2008 at 4:58 PM, Lalor, Brian <Brian.Lalor@...> wrote:

From: artifactory-users-bounces@... [mailto:artifactory-users-bounces@...] On Behalf Of Frederic Simon
Sent: Thursday, May 22, 2008 8:52 AM
To: artifactory-users@...
Subject: Re: [Artifactory-users] Artifactory retrieving artifacts outside of"walled" repository

This is a known bug of Maven. Please refer to http://www.jfrog.org/confluence/display/RTF/Configuring+Maven
By the way, in maven 2.0.8 and above there is a mirror-any feature.
Did someone used it? Is it solving the bug?
Another issue you have to be careful with is plugin that can be executed without pom.xml file, like archetype. In this case repository settings in profile are not loaded and so maven is going to repo1. The workaround is to have a dummy pom.xml where you are executing the plugin. 
 
Perhaps I wasn't clear.  I'm using Maven v2.0.9 in this particular instance.  Poor POM management is leading to Maven trying to retrieve newer versions of plugins that aren't in my "frozen" repository.

It's also Artifactory (v1.2.5) that is retrieving the artifacts not in the "myapp-frozen" Artifactory repository.  The client is trying to retrieve a non-existant artifact from an Artifactory localRepository, and Artifactory (mistakenly?) goes ahead and retreives it from repo1 instead of returning a 404.  Maven is doing what it's told and only querying Artifactory.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users




--
http://freddy33.blogspot.com/
http://www.jfrog.org/
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory retrieving artifacts outside of"walled" repository

by Yoav Landman :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

As, Fred explained this is the behavior of a direct local repo request that is considered as an implicit virtual repo containing the local repo and all other remote caches. This is by design to keep backward compatibility with the behavior before the itroduction of virtual repositories. You can read more on this here: http://www.jfrog.org/jira/browse/RTFACT-306

Though it can be worked around by creating a dedicated virtual repo, with the upcoming release we are thinking towards chaging this behavior to the behavior you expected to see, which seems to be what most people expect anyway.

Yoav

On Thu, May 22, 2008 at 5:33 PM, Frederic Simon <freds@...> wrote:
OK, sorry did not see the log was from Artifactory!
Some questions:
- Does "myapp-frozen" contains the 2.1-alpha-1 version of the war plugin?
- If no, your goall is to stop the client from trying to get it?
- Maven is trying to retrieve this version because it's in  the maven-metadata.xml file. This file is aggregated by Artifactory from the list of accessible repos given for a virtual repo name.

So what I suggest:
- Make sure the maven-metadata.xml file in myapp-frozen does not contains version that should not be there (who that you don't want).
- By default the "virtual repository" represented by the url http://zoot/artifactory/myapp-frozen is the local repo "myapp-frozen" PLUS all the remote repository. So, to change this behavior you need to create a virtual repository of your own which contains only the local and remote repositories you trust and are frozen. Please check http://www.jfrog.org/confluence/display/RTF/Configuring+Artifactory#ConfiguringArtifactory-VirtualRepositories

Hope this should solve the issue.

On Thu, May 22, 2008 at 4:58 PM, Lalor, Brian <Brian.Lalor@...> wrote:

From: artifactory-users-bounces@... [mailto:artifactory-users-bounces@...] On Behalf Of Frederic Simon
Sent: Thursday, May 22, 2008 8:52 AM
To: artifactory-users@...
Subject: Re: [Artifactory-users] Artifactory retrieving artifacts outside of"walled" repository

This is a known bug of Maven. Please refer to http://www.jfrog.org/confluence/display/RTF/Configuring+Maven
By the way, in maven 2.0.8 and above there is a mirror-any feature.
Did someone used it? Is it solving the bug?
Another issue you have to be careful with is plugin that can be executed without pom.xml file, like archetype. In this case repository settings in profile are not loaded and so maven is going to repo1. The workaround is to have a dummy pom.xml where you are executing the plugin. 
 
Perhaps I wasn't clear.  I'm using Maven v2.0.9 in this particular instance.  Poor POM management is leading to Maven trying to retrieve newer versions of plugins that aren't in my "frozen" repository.

It's also Artifactory (v1.2.5) that is retrieving the artifacts not in the "myapp-frozen" Artifactory repository.  The client is trying to retrieve a non-existant artifact from an Artifactory localRepository, and Artifactory (mistakenly?) goes ahead and retreives it from repo1 instead of returning a 404.  Maven is doing what it's told and only querying Artifactory.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory retrieving artifacts outsideof"walled" repository

by blalor :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

 

From: artifactory-users-bounces@... [mailto:artifactory-users-bounces@...] On Behalf Of Yoav Landman
Sent: Thursday, May 22, 2008 2:57 PM
To: artifactory-users@...
Subject: Re: [Artifactory-users] Artifactory retrieving artifacts outsideof"walled" repository

As, Fred explained this is the behavior of a direct local repo request that is considered as an implicit virtual repo containing the local repo and all other remote caches. This is by design to keep backward compatibility with the behavior before the itroduction of virtual repositories. You can read more on this here: http://www.jfrog.org/jira/browse/RTFACT-306

Though it can be worked around by creating a dedicated virtual repo, with the upcoming release we are thinking towards chaging this behavior to the behavior you expected to see, which seems to be what most people expect anyway. 
Thanks for the explanation, Yoav and Fred.  That does seem to have done the trick.  I do find that behavior in Artifactory to be completely backwards, however.  :-)
 
On a related note, is there a way to rename a repository (local or remote) and keep its contents?  While implementing the above workaround, I changed all of my "real" repository keys to start with an underscore, thinking that I'd create virtual repositories for the ones I wanted exposed. However, when I reloaded the configuration, my existing repositories were gone, until I renamed them back in the config.
 
Thanks again for your help,
Brian

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory retrieving artifacts outsideof"walled" repository

by Yoav Landman :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, May 23, 2008 at 12:08 AM, Lalor, Brian <Brian.Lalor@...> wrote:
 

From: artifactory-users-bounces@... [mailto:artifactory-users-bounces@...] On Behalf Of Yoav Landman
Sent: Thursday, May 22, 2008 2:57 PM

To: artifactory-users@...
Subject: Re: [Artifactory-users] Artifactory retrieving artifacts outsideof"walled" repository

As, Fred explained this is the behavior of a direct local repo request that is considered as an implicit virtual repo containing the local repo and all other remote caches. This is by design to keep backward compatibility with the behavior before the itroduction of virtual repositories. You can read more on this here: http://www.jfrog.org/jira/browse/RTFACT-306

Though it can be worked around by creating a dedicated virtual repo, with the upcoming release we are thinking towards chaging this behavior to the behavior you expected to see, which seems to be what most people expect anyway. 
Thanks for the explanation, Yoav and Fred.  That does seem to have done the trick.  I do find that behavior in Artifactory to be completely backwards, however.  :-)
 
On a related note, is there a way to rename a repository (local or remote) and keep its contents?  While implementing the above workaround, I changed all of my "real" repository keys to start with an underscore, thinking that I'd create virtual repositories for the ones I wanted exposed. However, when I reloaded the configuration, my existing repositories were gone, until I renamed them back in the config.
 

Currently, no, but we are working on that to be supported directly via the UI. The current workaround (which is admittedly not a most intuitive one) is to export the content of the repository you wish to rename, delete all its content, rename the repository and import all the content back into the new one.
 
Thanks again for your help,

Not a problem
 
Brian

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users
LightInTheBox - Buy quality products at wholesale price