External vs. Internal dependencies

[wellgoonthen <ri...@googlemail.com>]

Hi,

I'm currently doing some build doctor work on our system build.  I've done
quite a lot of work to implement a java incremental set up using uptodate
etc.  This is great and saves a lot of time/work.  The biggest issue is now
the ivy resolution.  To summarise we have two types dependencies that I
call:

- External - junit, log4j etc.
- Internal - intra build module artifacts

Now I can easily skip the resolution of the external artifacts by checking
the "uptodateness" of the modules ivy.xml file to see if the version of
junit has been changed, or a new 3rd party lib added to that module.  The
issue I have is with the internal dependencies.

For example if my build consists of modules A -> B -> C, built in that
order.  I currently have to rely on ivy resolve to know if A's artifact has
been rebuilt which would therefore trigger a rebuild of B.  

To summarise I'm using ivy:resolve to do this and for one module that takes
a total of 11 seconds, 8 seconds is spent resolving stuff.

Does anyone have any ideas on this?

Many thanks

Richard Cole
-- 
View this message in context: http://old.nabble.com/External-vs.-Internal-dependencies-tp34396743p34396743.html
Sent from the ivy-user mailing list archive at Nabble.com.

Re: External vs. Internal dependencies

[wellgoonthen <ri...@googlemail.com>]

Hi, thanks again for your help.

"If the ivy:resolve is taking so long, it sounds like you're not caching
dependencies when you should be."

All dependencies are cached and picked up from the local machine from my
local cache.  I know this for sure as I can run the build when not connected
to the network.

Is there any hints I could look at as to why resolving something that
already exists on disk may be so slow?

Richard


wellgoonthen wrote:
> 
> Hi,
> 
> I'm currently doing some build doctor work on our system build.  I've done
> quite a lot of work to implement a java incremental set up using uptodate
> etc.  This is great and saves a lot of time/work.  The biggest issue is
> now the ivy resolution.  To summarise we have two types dependencies that
> I call:
> 
> - External - junit, log4j etc.
> - Internal - intra build module artifacts
> 
> Now I can easily skip the resolution of the external artifacts by checking
> the "uptodateness" of the modules ivy.xml file to see if the version of
> junit has been changed, or a new 3rd party lib added to that module.  The
> issue I have is with the internal dependencies.
> 
> For example if my build consists of modules A -> B -> C, built in that
> order.  I currently have to rely on ivy resolve to know if A's artifact
> has been rebuilt which would therefore trigger a rebuild of B.  
> 
> To summarise I'm using ivy:resolve to do this and for one module that
> takes a total of 11 seconds, 8 seconds is spent resolving stuff.
> 
> Does anyone have any ideas on this?
> 
> Many thanks
> 
> Richard Cole
> 

-- 
View this message in context: http://old.nabble.com/External-vs.-Internal-dependencies-tp34396743p34410077.html
Sent from the ivy-user mailing list archive at Nabble.com.

Re: External vs. Internal dependencies

[Mitch Gitman <mg...@gmail.com>]

If A and B and C are all their own Ivy modules with their own ivy.xml
files, then you should look into using the ivy:buildlist Ant task to do the
parent multimodule build. However, even then, you're letting the parent
build build everything unless you state otherwise. Since you're building
from source, you know whether A has changed, so you can tell buildlist to
skip it. See the root and leaf attributes; I always forget which is which.

There are a couple statements of yours that strike me funny.

First: "The reason I'm trying to avoid or at least cut down the situation
where I run an ivy:resolve is due to the amount of time it takes.  If I
have a module with 35 dependencies, it can take over 1 minute in some cases
where the full module build and test time may only be 2 mins."

If the ivy:resolve is taking so long, it sounds like you're not caching
dependencies when you should be.

Second: "As the build process would be run from the top level, as B runs
there very well may have been a rebuild of the A artifact that was pushed
to the local cache on the dev's machine.  So this is where ivy:resolve is
required, it will check if this is the case."

The rebuild of the A artifact should not be pushing to the local cache; it
should be publishing to a local repository (not to be confused with the
cache) on the dev's machine. Because A is something that is constantly
changing, the resolve on B should get A, but it should skip those
transitive dependencies of A that are not changing.

This will have to be it for me on this thread.

On Fri, Sep 7, 2012 at 6:25 AM, wellgoonthen <ri...@googlemail.com>wrote:

>
> Hi Mitch,
>
> Thanks for your response and apologies if my terminology was not accurate.
> Based on your comments I'll try to explain the issue again.  When I talk
> about A, B and C, these are all modules of the same project with a root
> level build.xml calling module level builds + ivy.xml, where A depends on B
> and C can depend on A, B or in fact both.  A, B and C are what I mean by
> internal in the sense I own the code etc.
>
> If I look at the ivy file of B, I may see (pseudo code for sake of the
> email):
>
>  depends on junit
>  depends on log4j
>  depends on A
>
> Based on the simple uptodate logic that can be used from ant I can easily
> remove the need to call ivy:resolve for junit and log4j by checking the
> ivy.xml file for module B.  If it's not newer, I know that someone hasn't
> changed the version of junit etc.
>
> As the build process would be run from the top level, as B runs there very
> well may have been a rebuild of the A artifact that was pushed to the local
> cache on the dev's machine.  So this is where ivy:resolve is required, it
> will check if this is the case.
>
> The reason I'm trying to avoid or at least cut down the situation where I
> run an ivy:resolve is due to the amount of time it takes.  If I have a
> module with 35 dependencies, it can take over 1 minute in some cases where
> the full module build and test time may only be 2 mins.
>
> I hope that's a bit clearer and the purpose of this is really to focus on
> local dev builds speed those up.  I'm not really targeting this from a CI
> build perspective, although I imagine that if I fix the local issue, the CI
> build will also benefit from the same kind of changes.
>
> Richard
>
>

Re: External vs. Internal dependencies

[wellgoonthen <ri...@googlemail.com>]

Hi Mitch,

Thanks for your response and apologies if my terminology was not accurate. 
Based on your comments I'll try to explain the issue again.  When I talk
about A, B and C, these are all modules of the same project with a root
level build.xml calling module level builds + ivy.xml, where A depends on B
and C can depend on A, B or in fact both.  A, B and C are what I mean by
internal in the sense I own the code etc.

If I look at the ivy file of B, I may see (pseudo code for sake of the
email):

 depends on junit
 depends on log4j
 depends on A

Based on the simple uptodate logic that can be used from ant I can easily
remove the need to call ivy:resolve for junit and log4j by checking the
ivy.xml file for module B.  If it's not newer, I know that someone hasn't
changed the version of junit etc.

As the build process would be run from the top level, as B runs there very
well may have been a rebuild of the A artifact that was pushed to the local
cache on the dev's machine.  So this is where ivy:resolve is required, it
will check if this is the case.

The reason I'm trying to avoid or at least cut down the situation where I
run an ivy:resolve is due to the amount of time it takes.  If I have a
module with 35 dependencies, it can take over 1 minute in some cases where
the full module build and test time may only be 2 mins.

I hope that's a bit clearer and the purpose of this is really to focus on
local dev builds speed those up.  I'm not really targeting this from a CI
build perspective, although I imagine that if I fix the local issue, the CI
build will also benefit from the same kind of changes.

Richard






Mitch Gitman wrote:
> 
> Your question is a bit ambiguous, so let me not so much answer it as make
> two points to get the terminology and concepts straight.
> 
> When you talk about skipping the resolution of the external artifacts,
> what
> you really should be saying is that you're still doing an ivy:resolve,
> just
> you're trusting the modules in the Ivy cache are up-to-date and Ivy
> doesn't
> have to check the remote repository again. This is the default behavior,
> and the natural behavior for external libraries and released versions of
> internal libraries.
> 
> For internal libraries still under development, you want to override this
> so that Ivy always checks the repo. Typically, this is done on the
> resolver
> in ivysettings.xml, and I can think of three criteria you can pass to the
> resolver to say that it's an "integration" version:
> 
>    - The publication status in the ivy.xml is "integration" or something
>    comparable.
>    - There's a special suffix in the version, like -SNAPSHOT or a
>    buildnumber or timestamp.
>    - The modules are in a special integration repository directory.
> 
> It's not a bad idea to do all three in combination, although any one will
> do. See:
> http://ant.apache.org/ivy/history/latest-milestone/concept.html#change
> 
> When you talk about using Ivy resolve to tell if A has been rebuilt
> triggering a new build of B, you're applying concepts that apply not to a
> dependency manager like Ivy but rather to a continuous integration server
> like Jenkins. Forgive me if I'm stating something you're already well
> aware
> of, but the way that a change in A triggers a build of B is to have the CI
> server monitor for check-ins to the different projects. If someone commits
> a change to A, it triggers a build of A, and if that build is succeeding,
> a
> new version of the A artifact gets published to the repository. From here,
> the CI server should be configured (whether manually or through an Ivy
> integration) to have a successful build of A trigger a build of B. From
> here, B is assured to get the latest A that was just published.
> 
> Your question implies CI builds as opposed to developer builds, since the
> developer knows what projects they've changed or have incoming changes
> from
> source control.
> 
> On Thu, Sep 6, 2012 at 4:12 AM, wellgoonthen
> <ri...@googlemail.com>wrote:
> 
>>
>> Hi,
>>
>> I'm currently doing some build doctor work on our system build.  I've
>> done
>> quite a lot of work to implement a java incremental set up using uptodate
>> etc.  This is great and saves a lot of time/work.  The biggest issue is
>> now
>> the ivy resolution.  To summarise we have two types dependencies that I
>> call:
>>
>> - External - junit, log4j etc.
>> - Internal - intra build module artifacts
>>
>> Now I can easily skip the resolution of the external artifacts by
>> checking
>> the "uptodateness" of the modules ivy.xml file to see if the version of
>> junit has been changed, or a new 3rd party lib added to that module.  The
>> issue I have is with the internal dependencies.
>>
>> For example if my build consists of modules A -> B -> C, built in that
>> order.  I currently have to rely on ivy resolve to know if A's artifact
>> has
>> been rebuilt which would therefore trigger a rebuild of B.
>>
>> To summarise I'm using ivy:resolve to do this and for one module that
>> takes
>> a total of 11 seconds, 8 seconds is spent resolving stuff.
>>
>> Does anyone have any ideas on this?
>>
>> Many thanks
>>
>> Richard Cole
>> --
>> View this message in context:
>> http://old.nabble.com/External-vs.-Internal-dependencies-tp34396743p34396743.html
>> Sent from the ivy-user mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://old.nabble.com/External-vs.-Internal-dependencies-tp34396743p34402432.html
Sent from the ivy-user mailing list archive at Nabble.com.

Re: External vs. Internal dependencies

[Mitch Gitman <mg...@gmail.com>]

Your question is a bit ambiguous, so let me not so much answer it as make
two points to get the terminology and concepts straight.

When you talk about skipping the resolution of the external artifacts, what
you really should be saying is that you're still doing an ivy:resolve, just
you're trusting the modules in the Ivy cache are up-to-date and Ivy doesn't
have to check the remote repository again. This is the default behavior,
and the natural behavior for external libraries and released versions of
internal libraries.

For internal libraries still under development, you want to override this
so that Ivy always checks the repo. Typically, this is done on the resolver
in ivysettings.xml, and I can think of three criteria you can pass to the
resolver to say that it's an "integration" version:

   - The publication status in the ivy.xml is "integration" or something
   comparable.
   - There's a special suffix in the version, like -SNAPSHOT or a
   buildnumber or timestamp.
   - The modules are in a special integration repository directory.

It's not a bad idea to do all three in combination, although any one will
do. See:
http://ant.apache.org/ivy/history/latest-milestone/concept.html#change

When you talk about using Ivy resolve to tell if A has been rebuilt
triggering a new build of B, you're applying concepts that apply not to a
dependency manager like Ivy but rather to a continuous integration server
like Jenkins. Forgive me if I'm stating something you're already well aware
of, but the way that a change in A triggers a build of B is to have the CI
server monitor for check-ins to the different projects. If someone commits
a change to A, it triggers a build of A, and if that build is succeeding, a
new version of the A artifact gets published to the repository. From here,
the CI server should be configured (whether manually or through an Ivy
integration) to have a successful build of A trigger a build of B. From
here, B is assured to get the latest A that was just published.

Your question implies CI builds as opposed to developer builds, since the
developer knows what projects they've changed or have incoming changes from
source control.

On Thu, Sep 6, 2012 at 4:12 AM, wellgoonthen <ri...@googlemail.com>wrote:

>
> Hi,
>
> I'm currently doing some build doctor work on our system build.  I've done
> quite a lot of work to implement a java incremental set up using uptodate
> etc.  This is great and saves a lot of time/work.  The biggest issue is now
> the ivy resolution.  To summarise we have two types dependencies that I
> call:
>
> - External - junit, log4j etc.
> - Internal - intra build module artifacts
>
> Now I can easily skip the resolution of the external artifacts by checking
> the "uptodateness" of the modules ivy.xml file to see if the version of
> junit has been changed, or a new 3rd party lib added to that module.  The
> issue I have is with the internal dependencies.
>
> For example if my build consists of modules A -> B -> C, built in that
> order.  I currently have to rely on ivy resolve to know if A's artifact has
> been rebuilt which would therefore trigger a rebuild of B.
>
> To summarise I'm using ivy:resolve to do this and for one module that takes
> a total of 11 seconds, 8 seconds is spent resolving stuff.
>
> Does anyone have any ideas on this?
>
> Many thanks
>
> Richard Cole
> --
> View this message in context:
> http://old.nabble.com/External-vs.-Internal-dependencies-tp34396743p34396743.html
> Sent from the ivy-user mailing list archive at Nabble.com.
>
>