You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Kevin Burton <bu...@spinn3r.com> on 2015/07/23 05:13:21 UTC
Fetch all remote dependencies NOT in my pom?
I need a way to fetch all external dependencies that aren’t just a
submodule in my current project.
So things like junit, mockito, log4j, jetty, etc. I want to pull down all
that stuff.
However, I do NOT want it to include dependencies from the project. Those
aren’t built yet so they’re not in any repo.
This is important for our continuous integration system. I want to
pre-cache these so that they’re local. It will shave 2-5 minutes off our
build time. We have about 200 external dependencies that we pull in so
this init process takes a long time (and wastes resources).
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Fetch all remote dependencies NOT in my pom?
Posted by Doug Douglass <do...@gmail.com>.
Kevin,
If the builds are currently downloading the same artifacts only once, then
running any of the suggested mvn commands during your build won't actually
save any time will it?
You mention an "init process"...Are you planning on running the artifact
"pre-caching" outside of your CI build e.g. at container start up? or are
you adding the artifacts to the container definition (e.g. docker layer)?
or is this happening on the host and accessed from the container (e.g.
docker volume)
I'm just curious as one of my teams is considering SAAS CI.
Thanks,
Doug
On Thu, Jul 23, 2015 at 9:22 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> I didn’t try go-offline.. worth a shot!
>
> On Wed, Jul 22, 2015 at 8:57 PM, Curtis Rueden <ct...@wisc.edu> wrote:
>
> > Hi Kevin,
> >
> > > What I want to do is just download all artifacts not in "com.spinn3r.*
> “
> > > group ID.
> >
> > Did you try:
> >
> > mvn dependency:go-offline -DexcludeGroupIds=com.spinn3r ...
> >
> > ?
> >
> > Regards,
> > Curtis
> >
> > On Wed, Jul 22, 2015 at 10:50 PM, Kevin Burton <bu...@spinn3r.com>
> wrote:
> >
> > > What I want to do is just download all artifacts not in "com.spinn3r.*
> “
> > > group ID.
> > >
> > > They aren’t built yet or in any repositories yet. This is for a
> > continuous
> > > integration systems so I want to pre-cache them. After this they’re
> just
> > > restored from cache to build faster.
> > >
> > >
> > > > Is your repo on the same computer as the CI system?
> > > >
> > > >
> > > No.. different.
> > >
> > >
> > > > Maven does cache dependencies locally after they are downloaded from
> > your
> > > > repo.
> > > >
> > >
> > > Yes. But this is a hosted CI build system (CircleCI) so everything is
> a
> > > container and wiped out each time.
> > >
> > >
> > > --
> > >
> > > Founder/CEO Spinn3r.com
> > > Location: *San Francisco, CA*
> > > blog: http://burtonator.wordpress.com
> > > … or check out my Google+ profile
> > > <https://plus.google.com/102718274791889610666/posts>
> > >
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
Re: Fetch all remote dependencies NOT in my pom?
Posted by Kevin Burton <bu...@spinn3r.com>.
I didn’t try go-offline.. worth a shot!
On Wed, Jul 22, 2015 at 8:57 PM, Curtis Rueden <ct...@wisc.edu> wrote:
> Hi Kevin,
>
> > What I want to do is just download all artifacts not in "com.spinn3r.* “
> > group ID.
>
> Did you try:
>
> mvn dependency:go-offline -DexcludeGroupIds=com.spinn3r ...
>
> ?
>
> Regards,
> Curtis
>
> On Wed, Jul 22, 2015 at 10:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> > What I want to do is just download all artifacts not in "com.spinn3r.* “
> > group ID.
> >
> > They aren’t built yet or in any repositories yet. This is for a
> continuous
> > integration systems so I want to pre-cache them. After this they’re just
> > restored from cache to build faster.
> >
> >
> > > Is your repo on the same computer as the CI system?
> > >
> > >
> > No.. different.
> >
> >
> > > Maven does cache dependencies locally after they are downloaded from
> your
> > > repo.
> > >
> >
> > Yes. But this is a hosted CI build system (CircleCI) so everything is a
> > container and wiped out each time.
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> >
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Fetch all remote dependencies NOT in my pom?
Posted by Ron Wheeler <rw...@artifact-software.com>.
Could you not just build it once and copy the contents of the local
cache after it downloads everything?
Ron
On 22/07/2015 11:57 PM, Curtis Rueden wrote:
> Hi Kevin,
>
>> What I want to do is just download all artifacts not in "com.spinn3r.* “
>> group ID.
> Did you try:
>
> mvn dependency:go-offline -DexcludeGroupIds=com.spinn3r ...
>
> ?
>
> Regards,
> Curtis
>
> On Wed, Jul 22, 2015 at 10:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> What I want to do is just download all artifacts not in "com.spinn3r.* “
>> group ID.
>>
>> They aren’t built yet or in any repositories yet. This is for a continuous
>> integration systems so I want to pre-cache them. After this they’re just
>> restored from cache to build faster.
>>
>>
>>> Is your repo on the same computer as the CI system?
>>>
>>>
>> No.. different.
>>
>>
>>> Maven does cache dependencies locally after they are downloaded from your
>>> repo.
>>>
>> Yes. But this is a hosted CI build system (CircleCI) so everything is a
>> container and wiped out each time.
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Fetch all remote dependencies NOT in my pom?
Posted by Curtis Rueden <ct...@wisc.edu>.
Hi Kevin,
> What I want to do is just download all artifacts not in "com.spinn3r.* “
> group ID.
Did you try:
mvn dependency:go-offline -DexcludeGroupIds=com.spinn3r ...
?
Regards,
Curtis
On Wed, Jul 22, 2015 at 10:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> What I want to do is just download all artifacts not in "com.spinn3r.* “
> group ID.
>
> They aren’t built yet or in any repositories yet. This is for a continuous
> integration systems so I want to pre-cache them. After this they’re just
> restored from cache to build faster.
>
>
> > Is your repo on the same computer as the CI system?
> >
> >
> No.. different.
>
>
> > Maven does cache dependencies locally after they are downloaded from your
> > repo.
> >
>
> Yes. But this is a hosted CI build system (CircleCI) so everything is a
> container and wiped out each time.
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
Re: Fetch all remote dependencies NOT in my pom?
Posted by Kevin Burton <bu...@spinn3r.com>.
What I want to do is just download all artifacts not in "com.spinn3r.* “
group ID.
They aren’t built yet or in any repositories yet. This is for a continuous
integration systems so I want to pre-cache them. After this they’re just
restored from cache to build faster.
> Is your repo on the same computer as the CI system?
>
>
No.. different.
> Maven does cache dependencies locally after they are downloaded from your
> repo.
>
Yes. But this is a hosted CI build system (CircleCI) so everything is a
container and wiped out each time.
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Fetch all remote dependencies NOT in my pom?
Posted by Ron Wheeler <rw...@artifact-software.com>.
What repo are you using?
Is your repo on the same computer as the CI system?
Maven does cache dependencies locally after they are downloaded from
your repo.
Ron
On 22/07/2015 11:33 PM, Sander Verhagen wrote:
> Hi Burtonator,
>
>
> The "resolve" goal that you mention "works" for me. I put quotes around "works" because your requirements aren't quite clear to me. But "resolve" gives me a list of dependencies (not transitive ones, since you specified "-DexcludeTransitive=true"). So perhaps you could elaborate on what "doesn't work" means for you.
>
> After your first e-mail I thought you wanted to download the artifacts, rather than list them, and thought you'd be a candidate for "dependency:copy-dependencies" instead.
>
> Not sure what you're trying to do here precisely. But we all have CI environments, so perhaps I can learn from your (attempted) approach!
>
> Best regards,
>
>
> Sander Verhagen
> [ sander@sanderverhagen.net ]
>
> NOTICE: my e-mail address has changed. Please remove Verhagen@Sander.com now and start using Sander@SanderVerhagen.net from now on. Please update your address book. Thank you!
>
>
>> -----Original Message-----
>> From: burtonator2011@gmail.com [mailto:burtonator2011@gmail.com] On
>> Behalf Of Kevin Burton
>> Sent: Wednesday, July 22, 2015 20:27
>> To: Maven Users List
>> Subject: Re: Fetch all remote dependencies NOT in my pom?
>>
>> I tried the following.. but that doesn’t work. hm… stumped on this one.
>>
>> mvn -DexcludeGroupIds=com.spinn3r.artemis -DexcludeTransitive=true
>> dependency:resolve mvn -DexcludeArtifactIds=artemis -
>> DexcludeTransitive=true dependency:resolve
>>
>> On Wed, Jul 22, 2015 at 8:13 PM, Kevin Burton <bu...@spinn3r.com>
>> wrote:
>>
>>> I need a way to fetch all external dependencies that aren’t just a
>>> submodule in my current project.
>>>
>>> So things like junit, mockito, log4j, jetty, etc. I want to pull down
>>> all that stuff.
>>>
>>> However, I do NOT want it to include dependencies from the project.
>>> Those aren’t built yet so they’re not in any repo.
>>>
>>> This is important for our continuous integration system. I want to
>>> pre-cache these so that they’re local. It will shave 2-5 minutes off
>>> our build time. We have about 200 external dependencies that we pull
>>> in so this init process takes a long time (and wastes resources).
>>>
>>> --
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>>
>>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
RE: Fetch all remote dependencies NOT in my pom?
Posted by Sander Verhagen <sa...@sanderverhagen.net>.
Hi Burtonator,
The "resolve" goal that you mention "works" for me. I put quotes around "works" because your requirements aren't quite clear to me. But "resolve" gives me a list of dependencies (not transitive ones, since you specified "-DexcludeTransitive=true"). So perhaps you could elaborate on what "doesn't work" means for you.
After your first e-mail I thought you wanted to download the artifacts, rather than list them, and thought you'd be a candidate for "dependency:copy-dependencies" instead.
Not sure what you're trying to do here precisely. But we all have CI environments, so perhaps I can learn from your (attempted) approach!
Best regards,
Sander Verhagen
[ sander@sanderverhagen.net ]
NOTICE: my e-mail address has changed. Please remove Verhagen@Sander.com now and start using Sander@SanderVerhagen.net from now on. Please update your address book. Thank you!
> -----Original Message-----
> From: burtonator2011@gmail.com [mailto:burtonator2011@gmail.com] On
> Behalf Of Kevin Burton
> Sent: Wednesday, July 22, 2015 20:27
> To: Maven Users List
> Subject: Re: Fetch all remote dependencies NOT in my pom?
>
> I tried the following.. but that doesn’t work. hm… stumped on this one.
>
> mvn -DexcludeGroupIds=com.spinn3r.artemis -DexcludeTransitive=true
> dependency:resolve mvn -DexcludeArtifactIds=artemis -
> DexcludeTransitive=true dependency:resolve
>
> On Wed, Jul 22, 2015 at 8:13 PM, Kevin Burton <bu...@spinn3r.com>
> wrote:
>
> > I need a way to fetch all external dependencies that aren’t just a
> > submodule in my current project.
> >
> > So things like junit, mockito, log4j, jetty, etc. I want to pull down
> > all that stuff.
> >
> > However, I do NOT want it to include dependencies from the project.
> > Those aren’t built yet so they’re not in any repo.
> >
> > This is important for our continuous integration system. I want to
> > pre-cache these so that they’re local. It will shave 2-5 minutes off
> > our build time. We have about 200 external dependencies that we pull
> > in so this init process takes a long time (and wastes resources).
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> >
> >
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Fetch all remote dependencies NOT in my pom?
Posted by Kevin Burton <bu...@spinn3r.com>.
I tried the following.. but that doesn’t work. hm… stumped on this one.
mvn -DexcludeGroupIds=com.spinn3r.artemis -DexcludeTransitive=true
dependency:resolve
mvn -DexcludeArtifactIds=artemis -DexcludeTransitive=true dependency:resolve
On Wed, Jul 22, 2015 at 8:13 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I need a way to fetch all external dependencies that aren’t just a
> submodule in my current project.
>
> So things like junit, mockito, log4j, jetty, etc. I want to pull down all
> that stuff.
>
> However, I do NOT want it to include dependencies from the project. Those
> aren’t built yet so they’re not in any repo.
>
> This is important for our continuous integration system. I want to
> pre-cache these so that they’re local. It will shave 2-5 minutes off our
> build time. We have about 200 external dependencies that we pull in so
> this init process takes a long time (and wastes resources).
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>