Newbie Setup open for review

[ne...@gmail.com]

Hi,

we are using ivy in our project. Now I want to see if I understood the 
concepts correctly and am using it as designed.

First I had it set up so that all libs used were provided by cachepath 
calls. That lead to a lot of ant output and a minimal compile turnaround 
of 11 seconds.

Now I am using a retrieve target that only gets triggered on changes to 
the ivy.xml file. This is way fast, compile turnaround is down to 4 
seconds. The downside is that dynamic revisions will not be picked up 
automatically until you explicitly run the clean target.

In a perfect world I would like a fast build with dynamic revisions. Is 
this possible?

Can anything else be improved with this setup?



Another question: libs that we use for ant targets only are organized in 
confs. Is this the way it's supposed to be? This is an excerpt from our 
ivy.xml:

    <configurations>

        <conf name="ant-checkstyle" visibility="private" 
description="Abhängigkeiten für ant checkstyle task"/>
        <conf name="ant-cobertura" visibility="private" 
description="Abhängigkeiten für ant cobertura tasks"/>
        [....]
    
        <conf name="core" visibility="private" description="Generelle 
Abhängigkeiten der Software"/>
        <conf name="compile" extends="core" visibility="private" 
description="Compilezeit-Abhängigkeiten der Software"/>
        <conf name="runtime" extends="core" 
description="Laufzeit-Abhängigkeiten der Software"/>
        <conf name="test-runtime" extends="runtime" visibility="private" 
description="Abhängigkeiten für Tests der Software"/>
        <conf name="test-compile" extends="compile" visibility="private" 
description="Abhängigkeiten für Tests der Software"/>
    
    </configurations>

    <dependencies>

        <dependency org="com.puppycrawl.tools" name="checkstyle" rev="5.3" 
conf="ant-checkstyle->default"/>
        <dependency org="cobertura" name="cobertura" rev="1.9.4.1" 
conf="ant-cobertura->default"/>

        <!-- core -->
        <dependency org="org.hibernate" name="hibernate-core" 
rev="3.6.0.Final" conf="core->default"/>
        [....]

    </dependencies>

Regards,

Ole

Re: Newbie Setup open for review

[Mitch Gitman <mg...@gmail.com>]

Ole, there was something about the premise to your question that led me into
giving what I see now was a misguided answer. Ivy:cachepath and ivy:retrieve
are both post-resolve tasks. The real heavy lifting is being done by
ivy:resolve. Your performance issues should be the same regardless of which
post-resolve task you use.

Just for sanity's sake, can you construct a target such that only
ivy:resolve is being called? I'd like you to see for yourself that
ivy:resolve is where you're seeing the big performance cost.

Beyond that, I suppose it's possible that your resolve is taking so long
even if everything's already downloaded. This could be a symptom of either
of two things:
* The project itself is really, really big, like containing all the layers
in the application.
* The Ivy confs are too coarse-grained, and you're downloading everything at
once when you only need a subset of dependencies for compiling your main
code.

Even then, there's got to be a bit of a slow-machine factor here, as you
indicate.

On Thu, Nov 18, 2010 at 5:40 AM, Ole Langbehn
<ol...@googlemail.com>wrote:

> Hi Mitch (et al),
>
> thanks for the answer. Let me give you some more info:
>
> Mitch Gitman wrote:
> > I would recommend using ivy:cachepath over ivy:retrieve. With retrieve,
> > you're doing an extra copy of the dependencies, which leaves you
> > vulnerable to the files getting out of sync. Without seeing your Ivy
> > settings, I can't say why things are slower on cachepath or retrieve.
> > If I understand correctly, you're encountering this 11-second cost
> > every time you do a compile. If that's the case, it would mean that Ivy
> > is not trusting your cache at all and forcing a download every time.
>
> No, that's not the case, ivy does not trigger a download from the remote
> repositories, but takes that much extra time simply to resolve the
> dependencies from the cache. A download takes AGES (since the main remote
> repository is on a slow connection).
>
> Maybe this is indeed a problem that only occurs on my machine, since on
> other machines the resolve is quicker, maybe about 2-3 times.
>
>

Re: Newbie Setup open for review

[Tim Brown <tp...@gmail.com>]

On Thu, Nov 18, 2010 at 5:40 AM, Ole Langbehn
<ol...@googlemail.com>wrote:

> No, that's not the case, ivy does not trigger a download from the remote
> repositories, but takes that much extra time simply to resolve the
> dependencies from the cache. A download takes AGES (since the main remote
> repository is on a slow connection).
>
> Maybe this is indeed a problem that only occurs on my machine, since on
> other machines the resolve is quicker, maybe about 2-3 times.
>
>
Check your proxy configuration.  I've seen similar results when we were
inadvertently running through a proxy to get to our repo.

Also try and keep your repo layout as consistent as possible.  The more
artifact patterns you define in your settings, the more potential I/O
operations you'll use.  If you have to use multiple patterns, put the most
frequently used first.

HTH,

~Tim

Re: Newbie Setup open for review

[Ole Langbehn <ol...@googlemail.com>]

Hi Mitch (et al),

thanks for the answer. Let me give you some more info:

Mitch Gitman wrote:
> I would recommend using ivy:cachepath over ivy:retrieve. With retrieve,
> you're doing an extra copy of the dependencies, which leaves you
> vulnerable to the files getting out of sync. Without seeing your Ivy
> settings, I can't say why things are slower on cachepath or retrieve.
> If I understand correctly, you're encountering this 11-second cost
> every time you do a compile. If that's the case, it would mean that Ivy
> is not trusting your cache at all and forcing a download every time.

No, that's not the case, ivy does not trigger a download from the remote 
repositories, but takes that much extra time simply to resolve the 
dependencies from the cache. A download takes AGES (since the main remote 
repository is on a slow connection).

Maybe this is indeed a problem that only occurs on my machine, since on 
other machines the resolve is quicker, maybe about 2-3 times.

> Regarding your other question, I would say it's generally regarded as a
> bad practice to place your confs and dependencies for the build in the
> same ivy.xml as the confs and dependencies for the current project.
> What you'll find is that the confs for the build almost never change
> across projects while the project confs do change. Even if you use
> standard project confs (core, compile, etc.), you'll find that the
> build dependencies never change from project to project while the
> project-proper dependencies do.
>
> The two alternatives I've seen out there to putting build dependencies
> in the ivy.xml are:
> * Give the build its own ivy.xml.

Is there any example for this?

> * Define each build dependency inline in the ivy:resolve as it's
> needed. Do inline="true".

Also not a bad idea.

Regards,

-- 
Ole Langbehn, Breitenfelder Str. 13c, 20251 Hamburg, +4917649802788

Re: Newbie Setup open for review

[Mitch Gitman <mg...@gmail.com>]

I would recommend using ivy:cachepath over ivy:retrieve. With retrieve,
you're doing an extra copy of the dependencies, which leaves you vulnerable
to the files getting out of sync. Without seeing your Ivy settings, I can't
say why things are slower on cachepath or retrieve. If I understand
correctly, you're encountering this 11-second cost every time you do a
compile. If that's the case, it would mean that Ivy is not trusting your
cache at all and forcing a download every time. Just for sanity's sake, you
might want to look in the default cache location, .ivy2/cache under your
user home, and check if the files are indeed being downloaded and if they're
being downloaded every time.

I'm sure someone out there might be able to identify what this is a symptom
of.

Regarding your other question, I would say it's generally regarded as a bad
practice to place your confs and dependencies for the build in the same
ivy.xml as the confs and dependencies for the current project. What you'll
find is that the confs for the build almost never change across projects
while the project confs do change. Even if you use standard project confs
(core, compile, etc.), you'll find that the build dependencies never change
from project to project while the project-proper dependencies do.

The two alternatives I've seen out there to putting build dependencies in
the ivy.xml are:
* Give the build its own ivy.xml.
* Define each build dependency inline in the ivy:resolve as it's needed. Do
inline="true".

On Tue, Nov 16, 2010 at 4:45 AM, <ne...@gmail.com> wrote:

> Hi,
>
> we are using ivy in our project. Now I want to see if I understood the
> concepts correctly and am using it as designed.
>
> First I had it set up so that all libs used were provided by cachepath
> calls. That lead to a lot of ant output and a minimal compile turnaround
> of 11 seconds.
>
> Now I am using a retrieve target that only gets triggered on changes to
> the ivy.xml file. This is way fast, compile turnaround is down to 4
> seconds. The downside is that dynamic revisions will not be picked up
> automatically until you explicitly run the clean target.
>
> In a perfect world I would like a fast build with dynamic revisions. Is
> this possible?
>
> Can anything else be improved with this setup?
>
>
>
> Another question: libs that we use for ant targets only are organized in
> confs. Is this the way it's supposed to be? This is an excerpt from our
> ivy.xml:
>
>    <configurations>
>
>        <conf name="ant-checkstyle" visibility="private"
> description="Abhängigkeiten für ant checkstyle task"/>
>        <conf name="ant-cobertura" visibility="private"
> description="Abhängigkeiten für ant cobertura tasks"/>
>        [....]
>
>        <conf name="core" visibility="private" description="Generelle
> Abhängigkeiten der Software"/>
>        <conf name="compile" extends="core" visibility="private"
> description="Compilezeit-Abhängigkeiten der Software"/>
>        <conf name="runtime" extends="core"
> description="Laufzeit-Abhängigkeiten der Software"/>
>        <conf name="test-runtime" extends="runtime" visibility="private"
> description="Abhängigkeiten für Tests der Software"/>
>        <conf name="test-compile" extends="compile" visibility="private"
> description="Abhängigkeiten für Tests der Software"/>
>
>    </configurations>
>
>    <dependencies>
>
>        <dependency org="com.puppycrawl.tools" name="checkstyle" rev="5.3"
> conf="ant-checkstyle->default"/>
>        <dependency org="cobertura" name="cobertura" rev="1.9.4.1"
> conf="ant-cobertura->default"/>
>
>        <!-- core -->
>        <dependency org="org.hibernate" name="hibernate-core"
> rev="3.6.0.Final" conf="core->default"/>
>        [....]
>
>    </dependencies>
>
> Regards,
>
> Ole
>