You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Curtis Rueden <ct...@wisc.edu> on 2012/05/07 22:31:57 UTC
Reducing SNAPSHOT redundancy
Hi everyone,
I have a question about snapshot deployment.
I have a multi-module project with ~30 modules, all in a Git repository on
GitHub. Whenever someone pushes to the repository, a GitHub notification
hook pings our Jenkins to do a rebuild, which includes a redeploy to our
Nexus. This is all great.
However, there is a lot of redundancy between snapshot JAR files. Often, a
commit will involve only one of the 30 submodules, but all 30 will
ultimately be rebuilt and redeployed, resulting in a plethora of snapshot
versions. At any point in time, there is nearly always a "new" version of
any given submodule of the project.
I was wondering about the best way to reduce this issue. It would be nice
to only redeploy snapshots that have actually changed—or better, for Maven
(client-side) or Nexus (server-side) to detect identical snapshots and not
waste the space creating a superfluous new one. (Of course, for a variety
of reasons, comparing binary hashes between the latest snapshot JAR and the
new snapshot candidate may not be enough—especially if the build process or
CI adds some build-specific information to the JAR. But that is not really
Maven's problem...)
Alternately, we could do something on the CI side to only do the deploy if
the submodule is really known to have changed—probably involving git
reflogs etc. But that road could quickly become fraught with peril...
So my question is: is there a common Maven best practice to mitigate such
redundancy? Or do most people simply live with the proliferation of
snapshots that occurs when using a naive deployment scheme?
Thanks,
Curtis
Re: Reducing SNAPSHOT redundancy
Posted by Marcin Kuthan <ma...@gmail.com>.
How many build plans do you have? If I understood correctly single plan for
all modules (perhaps on aggregate pom).
Even small change in one module rebuilds all modules :-(
Let's consider to define one build plan for each module. In addition define
dependencies between build plans to reflect dependencies between maven
modules (Bamboo discovers dependencies automatically).
For sure, it requires a lot of configuration on continuous server but it
works. Only changed module and all dependents will be built and deployed.
Marcin
On 7 May 2012 23:07, Curtis Rueden <ct...@wisc.edu> wrote:
> Hi Anders,
>
> Thanks for your reply.
>
>
> > I believe the common way to handle this is to schedule purging of old
> > snapshots in the repo manager. At least that's how I handle it in my
> > setups.
> >
>
> Indeed, our Nexus is set to purge old snapshots daily, deleting any that
> are more than 7 days old. This works just fine.
>
> This is very helpful since it effectively bounds the disk space requirement
> based on the number of builds that happen in a 7-day window.
>
> However, my concern is more regarding the fact that the snapshot binaries
> "cycle" all the time. That is, they are being continually replaced with new
> builds, which may in fact be based on identical source code, every time
> someone pushes to the Git repository. The unfortunate side effect is that
> all developers on the project generally have to redownload all submodule
> JARs once every 24 hours (whenever Maven snapshots are refreshed).
>
> Regards,
> Curtis
>
>
> On Mon, May 7, 2012 at 3:53 PM, Anders Hammar <an...@hammar.net> wrote:
>
> > I believe the common way to handle this is to schedule purging of old
> > snapshots in the repo manager. At least that's how I handle it in my
> > setups.
> >
> > /Anders
> >
> > On Mon, May 7, 2012 at 10:31 PM, Curtis Rueden <ct...@wisc.edu>
> wrote:
> > > Hi everyone,
> > >
> > > I have a question about snapshot deployment.
> > >
> > > I have a multi-module project with ~30 modules, all in a Git repository
> > on
> > > GitHub. Whenever someone pushes to the repository, a GitHub
> notification
> > > hook pings our Jenkins to do a rebuild, which includes a redeploy to
> our
> > > Nexus. This is all great.
> > >
> > > However, there is a lot of redundancy between snapshot JAR files.
> Often,
> > a
> > > commit will involve only one of the 30 submodules, but all 30 will
> > > ultimately be rebuilt and redeployed, resulting in a plethora of
> snapshot
> > > versions. At any point in time, there is nearly always a "new" version
> of
> > > any given submodule of the project.
> > >
> > > I was wondering about the best way to reduce this issue. It would be
> nice
> > > to only redeploy snapshots that have actually changed—or better, for
> > Maven
> > > (client-side) or Nexus (server-side) to detect identical snapshots and
> > not
> > > waste the space creating a superfluous new one. (Of course, for a
> variety
> > > of reasons, comparing binary hashes between the latest snapshot JAR and
> > the
> > > new snapshot candidate may not be enough—especially if the build
> process
> > or
> > > CI adds some build-specific information to the JAR. But that is not
> > really
> > > Maven's problem...)
> > >
> > > Alternately, we could do something on the CI side to only do the deploy
> > if
> > > the submodule is really known to have changed—probably involving git
> > > reflogs etc. But that road could quickly become fraught with peril...
> > >
> > > So my question is: is there a common Maven best practice to mitigate
> such
> > > redundancy? Or do most people simply live with the proliferation of
> > > snapshots that occurs when using a naive deployment scheme?
> > >
> > > Thanks,
> > > Curtis
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
Re: Reducing SNAPSHOT redundancy
Posted by Curtis Rueden <ct...@wisc.edu>.
Hi Anders,
Thanks for your reply.
> I believe the common way to handle this is to schedule purging of old
> snapshots in the repo manager. At least that's how I handle it in my
> setups.
>
Indeed, our Nexus is set to purge old snapshots daily, deleting any that
are more than 7 days old. This works just fine.
This is very helpful since it effectively bounds the disk space requirement
based on the number of builds that happen in a 7-day window.
However, my concern is more regarding the fact that the snapshot binaries
"cycle" all the time. That is, they are being continually replaced with new
builds, which may in fact be based on identical source code, every time
someone pushes to the Git repository. The unfortunate side effect is that
all developers on the project generally have to redownload all submodule
JARs once every 24 hours (whenever Maven snapshots are refreshed).
Regards,
Curtis
On Mon, May 7, 2012 at 3:53 PM, Anders Hammar <an...@hammar.net> wrote:
> I believe the common way to handle this is to schedule purging of old
> snapshots in the repo manager. At least that's how I handle it in my
> setups.
>
> /Anders
>
> On Mon, May 7, 2012 at 10:31 PM, Curtis Rueden <ct...@wisc.edu> wrote:
> > Hi everyone,
> >
> > I have a question about snapshot deployment.
> >
> > I have a multi-module project with ~30 modules, all in a Git repository
> on
> > GitHub. Whenever someone pushes to the repository, a GitHub notification
> > hook pings our Jenkins to do a rebuild, which includes a redeploy to our
> > Nexus. This is all great.
> >
> > However, there is a lot of redundancy between snapshot JAR files. Often,
> a
> > commit will involve only one of the 30 submodules, but all 30 will
> > ultimately be rebuilt and redeployed, resulting in a plethora of snapshot
> > versions. At any point in time, there is nearly always a "new" version of
> > any given submodule of the project.
> >
> > I was wondering about the best way to reduce this issue. It would be nice
> > to only redeploy snapshots that have actually changed—or better, for
> Maven
> > (client-side) or Nexus (server-side) to detect identical snapshots and
> not
> > waste the space creating a superfluous new one. (Of course, for a variety
> > of reasons, comparing binary hashes between the latest snapshot JAR and
> the
> > new snapshot candidate may not be enough—especially if the build process
> or
> > CI adds some build-specific information to the JAR. But that is not
> really
> > Maven's problem...)
> >
> > Alternately, we could do something on the CI side to only do the deploy
> if
> > the submodule is really known to have changed—probably involving git
> > reflogs etc. But that road could quickly become fraught with peril...
> >
> > So my question is: is there a common Maven best practice to mitigate such
> > redundancy? Or do most people simply live with the proliferation of
> > snapshots that occurs when using a naive deployment scheme?
> >
> > Thanks,
> > Curtis
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Reducing SNAPSHOT redundancy
Posted by Anders Hammar <an...@hammar.net>.
I believe the common way to handle this is to schedule purging of old
snapshots in the repo manager. At least that's how I handle it in my
setups.
/Anders
On Mon, May 7, 2012 at 10:31 PM, Curtis Rueden <ct...@wisc.edu> wrote:
> Hi everyone,
>
> I have a question about snapshot deployment.
>
> I have a multi-module project with ~30 modules, all in a Git repository on
> GitHub. Whenever someone pushes to the repository, a GitHub notification
> hook pings our Jenkins to do a rebuild, which includes a redeploy to our
> Nexus. This is all great.
>
> However, there is a lot of redundancy between snapshot JAR files. Often, a
> commit will involve only one of the 30 submodules, but all 30 will
> ultimately be rebuilt and redeployed, resulting in a plethora of snapshot
> versions. At any point in time, there is nearly always a "new" version of
> any given submodule of the project.
>
> I was wondering about the best way to reduce this issue. It would be nice
> to only redeploy snapshots that have actually changed—or better, for Maven
> (client-side) or Nexus (server-side) to detect identical snapshots and not
> waste the space creating a superfluous new one. (Of course, for a variety
> of reasons, comparing binary hashes between the latest snapshot JAR and the
> new snapshot candidate may not be enough—especially if the build process or
> CI adds some build-specific information to the JAR. But that is not really
> Maven's problem...)
>
> Alternately, we could do something on the CI side to only do the deploy if
> the submodule is really known to have changed—probably involving git
> reflogs etc. But that road could quickly become fraught with peril...
>
> So my question is: is there a common Maven best practice to mitigate such
> redundancy? Or do most people simply live with the proliferation of
> snapshots that occurs when using a naive deployment scheme?
>
> Thanks,
> Curtis
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Reducing SNAPSHOT redundancy
Posted by Laird Nelson <lj...@gmail.com>.
On Mon, May 7, 2012 at 4:31 PM, Curtis Rueden <ct...@wisc.edu> wrote:
> However, there is a lot of redundancy between snapshot JAR files. Often, a
> commit will involve only one of the 30 submodules, but all 30 will
> ultimately be rebuilt and redeployed, resulting in a plethora of snapshot
> versions.
> I was wondering about the best way to reduce this issue.
Jenkins has an option for incremental builds: you tell it to build from the
root, and it figures out which modules beneath it have actually changed.
Only those modules are rebuilt, and only their snapshots (therefore) are
redeployed.
http://jenkins.361315.n4.nabble.com/Confirming-incremental-build-behavior-td3933958.htmlmight
point you in the right direction.
Best,
Laird
--
http://about.me/lairdnelson
Re: Reducing SNAPSHOT redundancy
Posted by Ron Wheeler <rw...@artifact-software.com>.
I am not a big fan of the automatic triggering of builds.
We have over 60 modules.
The batch building might make sense at the beginning of a product where
the modules are poorly defined and changes affect a lot of modules and
functionality shifts between projects quite often.
As we got into a mature situation where the modules were better divided
by function and were better layered through abstraction and factories,
the scope of changes and the scope of work for individual developers got
smaller.
This made it much more efficient for a development to only check out the
5 or modules that required fixing in a minor upgrade and leave the rest
of the modules alone and get them as dependencies.
We also stopped changing version of modules that did not change so that
application release 1.11 might consist of modules with versions 1.9,
1.10 and perhaps even earlier versions since even major releases did not
affect all modules.
Not a direct answer to your question but something that you might want
to think about as your application matures and you get to a better
module structure where the scope of maintenance fixes and new releases
gets easier to control.
Ron
On 07/05/2012 5:59 PM, Wayne Fay wrote:
>> So my question is: is there a common Maven best practice to mitigate such
>> redundancy? Or do most people simply live with the proliferation of
>> snapshots that occurs when using a naive deployment scheme?
> I can't speak for "most people" but I can say that in my experience,
> people simply live with the proliferation of snapshots and clean
> things up regularly. Jenkins has a "build in clean repo" option and
> the dependency plugin has a purge-local-repository option that may
> also help if you are annoyed with the excessive use of disk space.
>
> You may also consider moving to a nightly build if the raw number of
> builds is too much to handle and you can trust your developers to not
> break the build without the immediate feedback they are getting right
> now from your setup. You can also change the configuration to check
> for new snapshots "always."
>
>> someone pushes to the Git repository. The unfortunate side effect is that
>> all developers on the project generally have to redownload all submodule
>> JARs once every 24 hours (whenever Maven snapshots are refreshed).
> I guess I'm not sure what the problem is here. Are your artifacts
> large such that this is seen as an inconvenience and a waste of time?
> Or is it just annoying? Maybe the first thing your coworkers could do
> when they get in is run a Maven build, then go get their coffee. By
> the time it is done, they have all the latest code and they are ready
> for the day.
>
> Alternatively, you could separate the more stable parts of your
> project so they are built less frequently.
>
> Wayne
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Reducing SNAPSHOT redundancy
Posted by Wayne Fay <wa...@gmail.com>.
> So my question is: is there a common Maven best practice to mitigate such
> redundancy? Or do most people simply live with the proliferation of
> snapshots that occurs when using a naive deployment scheme?
I can't speak for "most people" but I can say that in my experience,
people simply live with the proliferation of snapshots and clean
things up regularly. Jenkins has a "build in clean repo" option and
the dependency plugin has a purge-local-repository option that may
also help if you are annoyed with the excessive use of disk space.
You may also consider moving to a nightly build if the raw number of
builds is too much to handle and you can trust your developers to not
break the build without the immediate feedback they are getting right
now from your setup. You can also change the configuration to check
for new snapshots "always."
> someone pushes to the Git repository. The unfortunate side effect is that
> all developers on the project generally have to redownload all submodule
> JARs once every 24 hours (whenever Maven snapshots are refreshed).
I guess I'm not sure what the problem is here. Are your artifacts
large such that this is seen as an inconvenience and a waste of time?
Or is it just annoying? Maybe the first thing your coworkers could do
when they get in is run a Maven build, then go get their coffee. By
the time it is done, they have all the latest code and they are ready
for the day.
Alternatively, you could separate the more stable parts of your
project so they are built less frequently.
Wayne
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Reducing SNAPSHOT redundancy
Posted by Laird Nelson <lj...@gmail.com>.
On Tue, May 8, 2012 at 12:13 PM, Anders Hammar <an...@hammar.net> wrote:
> Don't add "-o" to this. That's something different not impacting
> artifacts produced in a multi-module build.
>
Right; I know; it sets the offline status. I used it here to prove a
point: that when present, Maven will not go to remote repositories. I
wanted to force readers to think about the fact that no repo other than the
local one had the potential of being used.
> "install" is not required for a multi-module build. (It is sometimes
> when using Maven 2, but everyone should be using Maven 3 so this is
> not a problem.) The so-called reactor of a multi-module build will
> handle the artifact not being present but rather use the content of
> the target folders.
>
Aha; that would explain why my maven-dependency-plugin:unpack goal keeps
failing on trying to unpack a directory. Some days you just can't win.
:-) Thanks for the tip, though!
Best,
Laird
--
http://about.me/lairdnelson
Re: Reducing SNAPSHOT redundancy
Posted by Anders Hammar <an...@hammar.net>.
Don't add "-o" to this. That's something different not impacting
artifacts produced in a multi-module build.
"install" is not required for a multi-module build. (It is sometimes
when using Maven 2, but everyone should be using Maven 3 so this is
not a problem.) The so-called reactor of a multi-module build will
handle the artifact not being present but rather use the content of
the target folders.
/Anders
On Tue, May 8, 2012 at 6:09 PM, Laird Nelson <lj...@gmail.com> wrote:
> On Tue, May 8, 2012 at 12:05 PM, Anders Hammar <an...@hammar.net> wrote:
>
>> Doing "mvn dpeloy" on a larger multi-module build will not do this for
>> the latter part of the build. So, what you want is "mvn test".
>
>
> Dumb, slightly tangential question: will mvn -o clean test on a
> multi-module build work? My assumption (untested, to be sure) is that it
> would not. I would think that "install" is still necessary for tests in a
> given module that depend on artifacts produced by another module, so that
> those artifacts are present in the local repo.
>
> Best,
> Laird
>
> --
> http://about.me/lairdnelson
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Reducing SNAPSHOT redundancy
Posted by Laird Nelson <lj...@gmail.com>.
On Tue, May 8, 2012 at 12:05 PM, Anders Hammar <an...@hammar.net> wrote:
> Doing "mvn dpeloy" on a larger multi-module build will not do this for
> the latter part of the build. So, what you want is "mvn test".
Dumb, slightly tangential question: will mvn -o clean test on a
multi-module build work? My assumption (untested, to be sure) is that it
would not. I would think that "install" is still necessary for tests in a
given module that depend on artifacts produced by another module, so that
those artifacts are present in the local repo.
Best,
Laird
--
http://about.me/lairdnelson
Re: Reducing SNAPSHOT redundancy
Posted by Anders Hammar <an...@hammar.net>.
Yet another way of configuring your multi-module build on CI is to NOT
just do "mvn deploy". The key thing with CI is that you want to verify
the build, right? And get quick feedback on errors.
Doing "mvn dpeloy" on a larger multi-module build will not do this for
the latter part of the build. So, what you want is "mvn test". You
will then not waste time creating jars that you don't really need (for
verifying the build). That build could then trigger a "mvn deploy"
(possibly with -DskipTests) job to push to the repo. Or even better, a
second step of the same job (which will ensure that it's the same code
base being used and not a fresh checkout).
/Anders
On Tue, May 8, 2012 at 3:26 PM, Ron Wheeler
<rw...@artifact-software.com> wrote:
> +1
> Ron
>
> On 08/05/2012 8:39 AM, Thiessen, Todd (Todd) wrote:
>>
>> In my experience, I notice a certain lifecycle with builds. They start
>> off relatively small but as the releases and years pass, the build grows and
>> evolves. More things get tacked on, more modules get added, and so forth.
>>
>> I find that over time, the earlier modules of the build change less and
>> less. There can be any number of reasons for this (feature content for that
>> area is relatively stable and needs fewer and fewer enhancements, etc..).
>> Whatever the reason, what this tells me is that those modules no longer need
>> to be built over and over again. They can be broken out into their own trunk
>> and built and versioned independently. This would shorten your build time,
>> which is vital in a CI kind of environment where you want to have a change
>> tested and get feedback on the build as quickly as possible.
>>
>> Now 30 modules is still relatively small. I am not sure how long your
>> turn around time is. I start to get a bit concerned once a build approaches
>> 10 minutes. Once it does, I start to ask questions like "Are all modules of
>> this build changing constantly? If not, which ones are not and why? Which
>> of these can and should be considered to be pulled out into their own
>> trunk."
>>
>> I find that this line of thinking not only helps to keep all build times
>> fast, but it also helps to keep a healthy high quality code base which
>> promotes re-use.
>>
>> Anyway, that's my 2 cents.
>>
>>> -----Original Message-----
>>> From: ctrueden.wisc@gmail.com [mailto:ctrueden.wisc@gmail.com] On
>>> Behalf Of Curtis Rueden
>>> Sent: Monday, May 07, 2012 4:32 PM
>>> To: Maven Users List
>>> Cc: Johannes Schindelin
>>> Subject: Reducing SNAPSHOT redundancy
>>>
>>> Hi everyone,
>>>
>>> I have a question about snapshot deployment.
>>>
>>> I have a multi-module project with ~30 modules, all in a Git repository
>>> on
>>> GitHub. Whenever someone pushes to the repository, a GitHub
>>> notification
>>> hook pings our Jenkins to do a rebuild, which includes a redeploy to
>>> our
>>> Nexus. This is all great.
>>>
>>> However, there is a lot of redundancy between snapshot JAR files.
>>> Often, a
>>> commit will involve only one of the 30 submodules, but all 30 will
>>> ultimately be rebuilt and redeployed, resulting in a plethora of
>>> snapshot
>>> versions. At any point in time, there is nearly always a "new" version
>>> of
>>> any given submodule of the project.
>>>
>>> I was wondering about the best way to reduce this issue. It would be
>>> nice
>>> to only redeploy snapshots that have actually changed-or better, for
>>> Maven
>>> (client-side) or Nexus (server-side) to detect identical snapshots and
>>> not
>>> waste the space creating a superfluous new one. (Of course, for a
>>> variety
>>> of reasons, comparing binary hashes between the latest snapshot JAR and
>>> the
>>> new snapshot candidate may not be enough-especially if the build
>>> process or
>>> CI adds some build-specific information to the JAR. But that is not
>>> really
>>> Maven's problem...)
>>>
>>> Alternately, we could do something on the CI side to only do the deploy
>>> if
>>> the submodule is really known to have changed-probably involving git
>>> reflogs etc. But that road could quickly become fraught with peril...
>>>
>>> So my question is: is there a common Maven best practice to mitigate
>>> such
>>> redundancy? Or do most people simply live with the proliferation of
>>> snapshots that occurs when using a naive deployment scheme?
>>>
>>> Thanks,
>>> Curtis
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> For additional commands, e-mail: users-help@maven.apache.org
>>
>>
>
>
> --
> Ron Wheeler President Artifact Software Inc email:
> rwheeler@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435,
> ext 102
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Reducing SNAPSHOT redundancy
Posted by Ron Wheeler <rw...@artifact-software.com>.
+1
Ron
On 08/05/2012 8:39 AM, Thiessen, Todd (Todd) wrote:
> In my experience, I notice a certain lifecycle with builds. They start off relatively small but as the releases and years pass, the build grows and evolves. More things get tacked on, more modules get added, and so forth.
>
> I find that over time, the earlier modules of the build change less and less. There can be any number of reasons for this (feature content for that area is relatively stable and needs fewer and fewer enhancements, etc..). Whatever the reason, what this tells me is that those modules no longer need to be built over and over again. They can be broken out into their own trunk and built and versioned independently. This would shorten your build time, which is vital in a CI kind of environment where you want to have a change tested and get feedback on the build as quickly as possible.
>
> Now 30 modules is still relatively small. I am not sure how long your turn around time is. I start to get a bit concerned once a build approaches 10 minutes. Once it does, I start to ask questions like "Are all modules of this build changing constantly? If not, which ones are not and why? Which of these can and should be considered to be pulled out into their own trunk."
>
> I find that this line of thinking not only helps to keep all build times fast, but it also helps to keep a healthy high quality code base which promotes re-use.
>
> Anyway, that's my 2 cents.
>
>> -----Original Message-----
>> From: ctrueden.wisc@gmail.com [mailto:ctrueden.wisc@gmail.com] On
>> Behalf Of Curtis Rueden
>> Sent: Monday, May 07, 2012 4:32 PM
>> To: Maven Users List
>> Cc: Johannes Schindelin
>> Subject: Reducing SNAPSHOT redundancy
>>
>> Hi everyone,
>>
>> I have a question about snapshot deployment.
>>
>> I have a multi-module project with ~30 modules, all in a Git repository
>> on
>> GitHub. Whenever someone pushes to the repository, a GitHub
>> notification
>> hook pings our Jenkins to do a rebuild, which includes a redeploy to
>> our
>> Nexus. This is all great.
>>
>> However, there is a lot of redundancy between snapshot JAR files.
>> Often, a
>> commit will involve only one of the 30 submodules, but all 30 will
>> ultimately be rebuilt and redeployed, resulting in a plethora of
>> snapshot
>> versions. At any point in time, there is nearly always a "new" version
>> of
>> any given submodule of the project.
>>
>> I was wondering about the best way to reduce this issue. It would be
>> nice
>> to only redeploy snapshots that have actually changed-or better, for
>> Maven
>> (client-side) or Nexus (server-side) to detect identical snapshots and
>> not
>> waste the space creating a superfluous new one. (Of course, for a
>> variety
>> of reasons, comparing binary hashes between the latest snapshot JAR and
>> the
>> new snapshot candidate may not be enough-especially if the build
>> process or
>> CI adds some build-specific information to the JAR. But that is not
>> really
>> Maven's problem...)
>>
>> Alternately, we could do something on the CI side to only do the deploy
>> if
>> the submodule is really known to have changed-probably involving git
>> reflogs etc. But that road could quickly become fraught with peril...
>>
>> So my question is: is there a common Maven best practice to mitigate
>> such
>> redundancy? Or do most people simply live with the proliferation of
>> snapshots that occurs when using a naive deployment scheme?
>>
>> Thanks,
>> Curtis
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
--
Ron Wheeler President Artifact Software Inc email:
rwheeler@artifact-software.com skype: ronaldmwheeler phone:
866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
RE: Reducing SNAPSHOT redundancy
Posted by Johannes Schindelin <sc...@wisc.edu>.
Dear Todd,
On Tue, 8 May 2012, Thiessen, Todd (Todd) wrote:
> In my experience, I notice a certain lifecycle with builds. They start
> off relatively small but as the releases and years pass, the build grows
> and evolves. More things get tacked on, more modules get added, and so
> forth.
>
> I find that over time, the earlier modules of the build change less and
> less. There can be any number of reasons for this (feature content for
> that area is relatively stable and needs fewer and fewer enhancements,
> etc..). Whatever the reason, what this tells me is that those modules no
> longer need to be built over and over again. They can be broken out into
> their own trunk and built and versioned independently. This would
> shorten your build time, which is vital in a CI kind of environment
> where you want to have a change tested and get feedback on the build as
> quickly as possible.
I am sorry, I disagree here. Breaking this out into different repositories
-- when the code is logically one single project -- makes an artificial
distinction just for the sake of helping the tools, at the cost of making
it more difficult for new developers and thereby constructing a new
barrier to entry, incurring a substantial social debt.
See, at the moment it is as easy as
git clone git://github.com/imagej/imagej
cd imagej
mvn
Done.
We would like to retain that ease of developing ImageJ, but on the other
hand also reduce the amounts of data downloaded by developers who create
projects on top of ImageJ. They should not need to download new snapshot
versions every day when nothing might have changed.
I appreciate your concern and your help in trying to improve our workflow,
but in my opinion, changing the setup just to accomodate the current
capabilities of the tools would be the wrong thing.
If anybody has a suggestion how to teach Maven to avoid uploading "new"
snapshot versions when nothing changed at all (except for the timestamps
in the .jar file, that's all), I would be very grateful.
Ciao,
Johannes
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
RE: Reducing SNAPSHOT redundancy
Posted by "Thiessen, Todd (Todd)" <tt...@avaya.com>.
In my experience, I notice a certain lifecycle with builds. They start off relatively small but as the releases and years pass, the build grows and evolves. More things get tacked on, more modules get added, and so forth.
I find that over time, the earlier modules of the build change less and less. There can be any number of reasons for this (feature content for that area is relatively stable and needs fewer and fewer enhancements, etc..). Whatever the reason, what this tells me is that those modules no longer need to be built over and over again. They can be broken out into their own trunk and built and versioned independently. This would shorten your build time, which is vital in a CI kind of environment where you want to have a change tested and get feedback on the build as quickly as possible.
Now 30 modules is still relatively small. I am not sure how long your turn around time is. I start to get a bit concerned once a build approaches 10 minutes. Once it does, I start to ask questions like "Are all modules of this build changing constantly? If not, which ones are not and why? Which of these can and should be considered to be pulled out into their own trunk."
I find that this line of thinking not only helps to keep all build times fast, but it also helps to keep a healthy high quality code base which promotes re-use.
Anyway, that's my 2 cents.
> -----Original Message-----
> From: ctrueden.wisc@gmail.com [mailto:ctrueden.wisc@gmail.com] On
> Behalf Of Curtis Rueden
> Sent: Monday, May 07, 2012 4:32 PM
> To: Maven Users List
> Cc: Johannes Schindelin
> Subject: Reducing SNAPSHOT redundancy
>
> Hi everyone,
>
> I have a question about snapshot deployment.
>
> I have a multi-module project with ~30 modules, all in a Git repository
> on
> GitHub. Whenever someone pushes to the repository, a GitHub
> notification
> hook pings our Jenkins to do a rebuild, which includes a redeploy to
> our
> Nexus. This is all great.
>
> However, there is a lot of redundancy between snapshot JAR files.
> Often, a
> commit will involve only one of the 30 submodules, but all 30 will
> ultimately be rebuilt and redeployed, resulting in a plethora of
> snapshot
> versions. At any point in time, there is nearly always a "new" version
> of
> any given submodule of the project.
>
> I was wondering about the best way to reduce this issue. It would be
> nice
> to only redeploy snapshots that have actually changed-or better, for
> Maven
> (client-side) or Nexus (server-side) to detect identical snapshots and
> not
> waste the space creating a superfluous new one. (Of course, for a
> variety
> of reasons, comparing binary hashes between the latest snapshot JAR and
> the
> new snapshot candidate may not be enough-especially if the build
> process or
> CI adds some build-specific information to the JAR. But that is not
> really
> Maven's problem...)
>
> Alternately, we could do something on the CI side to only do the deploy
> if
> the submodule is really known to have changed-probably involving git
> reflogs etc. But that road could quickly become fraught with peril...
>
> So my question is: is there a common Maven best practice to mitigate
> such
> redundancy? Or do most people simply live with the proliferation of
> snapshots that occurs when using a naive deployment scheme?
>
> Thanks,
> Curtis
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org