You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ant.apache.org by Josh Suereth <jo...@gmail.com> on 2014/04/28 13:59:30 UTC
Ivy slow resolution
On a whim, I implemented parallel download of artifacts by hooking the
downloadArtifacts
method of Resolve engine. While this can potentially speed up download
performance, the resolution times (getDepedencies) still dominates (of
course). Parallel downloads is an often requested feature of sbt, so we
may still release this hook for our users, as they request it, but I think
the *really* just want faster resolution times.
We may be doing some investigation into improving the performance of
getDependencies (and most importantly fetchDependencies). I was curious if
anyone else in the Ivy community has attempted this before and what sorts
of guidance they could offer before we dig in.
Thanks!
- Josh
Re: Ivy slow resolution
Posted by Nicolas Lalevée <ni...@hibnet.org>.
Le 28 avr. 2014 à 16:36, Josh Suereth <jo...@gmail.com> a écrit :
> On Mon, Apr 28, 2014 at 10:13 AM, Nicolas Lalevée <
> nicolas.lalevee@hibnet.org> wrote:
>
>>
>> Le 28 avr. 2014 à 13:59, Josh Suereth <jo...@gmail.com> a écrit :
>>
>>> On a whim, I implemented parallel download of artifacts by hooking the
>>> downloadArtifacts
>>> method of Resolve engine. While this can potentially speed up download
>>> performance, the resolution times (getDepedencies) still dominates (of
>>> course). Parallel downloads is an often requested feature of sbt, so we
>>> may still release this hook for our users, as they request it, but I
>> think
>>> the *really* just want faster resolution times.
>>>
>>> We may be doing some investigation into improving the performance of
>>> getDependencies (and most importantly fetchDependencies). I was curious
>> if
>>> anyone else in the Ivy community has attempted this before and what sorts
>>> of guidance they could offer before we dig in.
>>
>> This has been discussed, but AFAIR nothing has been done other than
>> discussing the potential issues to tackle.
>>
>> Things which would require some attention would be to find a way to
>> properly report the progress of the parallel download: in the console when
>> it's done via Ant or via the listeners for IvyDE.
>> A thing that is worrying me, is that Ivy is quite sensitive to which
>> thread is actually working on the resolve, see IvyContext's thread local.
>> So a parallelization may not be a very upper level than the artifact
>> downloading.
>>
>>
> Totally understand. I haven't had time to check if my resolution on
> different threads has mucked up the ivy context, but I know I can propagate
> context as needed, as long as downloading artifacts is idempotent (i.e. no
> changes to artifacts). The first implementation just defers reporting all
> progress until AFTER the entire download, then notifies, which probably
> breaks IvyDE, except I'm only overriding this in sbt itself, so there is no
> IvyDE there :).
>
> So, if this was something to contribute back to ivy (which I'm not sure it
> is, given the lack of real speed-up we're seeing), I'd have to figure out
> how to maintain the progress reporting notifications with the IvyContext,
> possibly adapting my worker pool such that we register a thread-safe
> IvyContext locally for the reporting events to get fired out of?
>
> Does the event model, as it exist, prevent parallelism?
As far as I can tell, no special care has been taken to ensure that even inner part of Ivy can be run on multiple threads. So this is to be tested and code reviewed.
> In terms of resolution improvements, mostly I was hoping that some
> memoization/cache of immutable graph pieces could help avoid re-resolving
> the same bits of the graph repeatedly. Again, I haven't dug fully into the
> details of IvyNode, ResolveData + VisitNode, but it seems likely to me that
> we should be able to avoid calling "resolve" for an artifact (the same
> ModuleRevisionId) more than once during full graph resolution, which (and I
> may be wrong) appears to be the case when we profile Ivy. I'm mostly
> concerned with this aspect, and was wondering if anyone else has seen this
> or if we're doing something very wrong in sbt.
I don't think Ivy is doing a resolution twice. This part of Ivy is quite complex, so I may be wrong, but we shouldn't forget that a resolve of a module is often contextualized by the "caller", because there are some different versioning constraints, or because the root configuration is not the same.
But I think that it can be parallelized. At least in theory.
Nicolas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Ivy slow resolution
Posted by Josh Suereth <jo...@gmail.com>.
On Mon, Apr 28, 2014 at 10:13 AM, Nicolas Lalevée <
nicolas.lalevee@hibnet.org> wrote:
>
> Le 28 avr. 2014 à 13:59, Josh Suereth <jo...@gmail.com> a écrit :
>
> > On a whim, I implemented parallel download of artifacts by hooking the
> > downloadArtifacts
> > method of Resolve engine. While this can potentially speed up download
> > performance, the resolution times (getDepedencies) still dominates (of
> > course). Parallel downloads is an often requested feature of sbt, so we
> > may still release this hook for our users, as they request it, but I
> think
> > the *really* just want faster resolution times.
> >
> > We may be doing some investigation into improving the performance of
> > getDependencies (and most importantly fetchDependencies). I was curious
> if
> > anyone else in the Ivy community has attempted this before and what sorts
> > of guidance they could offer before we dig in.
>
> This has been discussed, but AFAIR nothing has been done other than
> discussing the potential issues to tackle.
>
> Things which would require some attention would be to find a way to
> properly report the progress of the parallel download: in the console when
> it's done via Ant or via the listeners for IvyDE.
> A thing that is worrying me, is that Ivy is quite sensitive to which
> thread is actually working on the resolve, see IvyContext's thread local.
> So a parallelization may not be a very upper level than the artifact
> downloading.
>
>
Totally understand. I haven't had time to check if my resolution on
different threads has mucked up the ivy context, but I know I can propagate
context as needed, as long as downloading artifacts is idempotent (i.e. no
changes to artifacts). The first implementation just defers reporting all
progress until AFTER the entire download, then notifies, which probably
breaks IvyDE, except I'm only overriding this in sbt itself, so there is no
IvyDE there :).
So, if this was something to contribute back to ivy (which I'm not sure it
is, given the lack of real speed-up we're seeing), I'd have to figure out
how to maintain the progress reporting notifications with the IvyContext,
possibly adapting my worker pool such that we register a thread-safe
IvyContext locally for the reporting events to get fired out of?
Does the event model, as it exist, prevent parallelism?
In terms of resolution improvements, mostly I was hoping that some
memoization/cache of immutable graph pieces could help avoid re-resolving
the same bits of the graph repeatedly. Again, I haven't dug fully into the
details of IvyNode, ResolveData + VisitNode, but it seems likely to me that
we should be able to avoid calling "resolve" for an artifact (the same
ModuleRevisionId) more than once during full graph resolution, which (and I
may be wrong) appears to be the case when we profile Ivy. I'm mostly
concerned with this aspect, and was wondering if anyone else has seen this
or if we're doing something very wrong in sbt.
Re: Ivy slow resolution
Posted by Nicolas Lalevée <ni...@hibnet.org>.
Le 28 avr. 2014 à 13:59, Josh Suereth <jo...@gmail.com> a écrit :
> On a whim, I implemented parallel download of artifacts by hooking the
> downloadArtifacts
> method of Resolve engine. While this can potentially speed up download
> performance, the resolution times (getDepedencies) still dominates (of
> course). Parallel downloads is an often requested feature of sbt, so we
> may still release this hook for our users, as they request it, but I think
> the *really* just want faster resolution times.
>
> We may be doing some investigation into improving the performance of
> getDependencies (and most importantly fetchDependencies). I was curious if
> anyone else in the Ivy community has attempted this before and what sorts
> of guidance they could offer before we dig in.
This has been discussed, but AFAIR nothing has been done other than discussing the potential issues to tackle.
Things which would require some attention would be to find a way to properly report the progress of the parallel download: in the console when it's done via Ant or via the listeners for IvyDE.
A thing that is worrying me, is that Ivy is quite sensitive to which thread is actually working on the resolve, see IvyContext's thread local. So a parallelization may not be a very upper level than the artifact downloading.
Nicolas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org