[PROPOSAL] M2 Dependency Resolution Improvements

[Garrett Conaty <ga...@conaty.net>]

I've been working quite a bit with Maven2 transitive dependencies at  
work, and think that they're one of the best strengths of M2.  In the  
course of building quite a number of artifacts through build,  
deployment, and update I've found that there are number of  
improvements that can be made.  So with the use cases that I've seen  
at work and listening to what other devs in the community have  
experienced, I'm proposing some enhancements.

These points are only an overview, and some are a variation or  
enhancement on features in the Maven 2.1 design docs. I've gone into  
more depth on the rationale and the features themselves on the  
MavenUser Wiki at http://docs.codehaus.org/display/MAVENUSER/Improving 
+Maven2+Dependency+Resolution :

* Virtual Dependencies (also know as spec jars)

* Environmental Context - Use the JDK environment and deploy  
environment as part of the resolution phase.  As an example, you  
don't need to get JAXP API classes from a repository if compiling  
with JDK 1.4

* Pinning Dependencies (Overrides) - If you have a fixed dependency  
tree for a given release, it's important to be able to provided  
patches that can override a node of the tree.  <dependencyManagement>  
only appears to handle this for parent-child projects.

* Filtering Parameters - Let the dependency resolution be able to  
pick artifacts based on things like required JDK version, license,  
signed/unsigned, tags,etc.

* Blacklisting artifacts - Not just repositories.  This is  
appropriate if you don't control the POM that's using the artifact to  
be blacklisted.

* License Acknowledgment - Some commercial vendors (Sun) require a  
clickthrough license before downloading. Also a user getting an  
artifact under an Apache license with an LGPL dependency would need  
to be notified that they are not only accepting the Apache License  
but LGPL.

* Expose dependency resolution to client tools - Provide APIs so that  
clients can pass in different artifacts with different filters to see  
not only the multiple artifacts and paths that can resolve a  
dependency, but also which one would be selected. This would be very  
helpful for debugging and also to let UI tools visualize the  
dependency graphs.

I don't think these features have easy workarounds in Maven 2.0.x,  
but if there are, that would be great.

I've peer reviewed these with the folks at work as well and I'm  
interested in feedback from the community on these improvements as  
I'd like to get started working on them.

Thanks,
Garrett

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: [PROPOSAL] M2 Dependency Resolution Improvements

[Garrett Conaty <ga...@conaty.net>]

John,

Thanks for the feedback.  My responses are now inline:

On Mar 3, 2006, at 7:57 AM, John Casey wrote:

> Comments inline.
>
> Cheers,
>
> John
>
> Garrett Conaty wrote:
>> I've been working quite a bit with Maven2 transitive dependencies  
>> at work, and think that they're one of the best strengths of M2.   
>> In the course of building quite a number of artifacts through  
>> build, deployment, and update I've found that there are number of  
>> improvements that can be made.  So with the use cases that I've  
>> seen at work and listening to what other devs in the community  
>> have experienced, I'm proposing some enhancements.
>> These points are only an overview, and some are a variation or  
>> enhancement on features in the Maven 2.1 design docs. I've gone  
>> into more depth on the rationale and the features themselves on  
>> the MavenUser Wiki at http://docs.codehaus.org/display/MAVENUSER/ 
>> Improving+Maven2+Dependency+Resolution :
>> * Virtual Dependencies (also know as spec jars)
>
> This is something we've been discussing for some time now. We just  
> need to have a concrete discussion about what's involved in  
> implementing it. For instance, we have to provide a portable way to  
> resolve spec -> implementation mappings. This might take a form  
> similar to the plugin-prefix -> implementation metadata file. Once  
> we have that in place, we need some consistent mechanism for  
> choosing one implementation over another. The corresponding  
> mechanisms for plugin selection based on prefix are somewhat crude...

How about putting a new <virtual> in the <dependency> stanzas of the  
POM? Of course, it would still need someway to resolve this map  
(perhaps by putting them in profiles or something).

>
>> * Environmental Context - Use the JDK environment and deploy  
>> environment as part of the resolution phase.  As an example, you  
>> don't need to get JAXP API classes from a repository if compiling  
>> with JDK 1.4
>
> If you can represent pieces like the JDK as a repository with an  
> alternate layout, I don't see why we couldn't tie this into the  
> spec dependencies bit above and make it work.

Yeah, I definitely see these being very related.  So you could define  
a JDK 1.4 environment or Geronimo 1.0 environment (profile?) that  
would be contain the list of the spec JARS.  This would be really  
helpful for building some of the Apache Commons packages that require  
different JARs (currently) depending on what JDK they're using.  I'd  
love it if the build files for these would be able to list all the  
spec JARs that they require and then when you build, Maven could have  
the knowledge of your environment.  Though I'm not sure they'd  
actually be POMs themselves (like for JDK 1.4) that would list  
artifacts.  It might be viewed something more like how Eclipse shows  
you library entries for JDK or app servers (like Tomcat in the  
Eclipse WebToolsProject)

>
>> * Pinning Dependencies (Overrides) - If you have a fixed  
>> dependency tree for a given release, it's important to be able to  
>> provided patches that can override a node of the tree.   
>> <dependencyManagement> only appears to handle this for parent- 
>> child projects.
>
> I suspect an elegant way to handle this might be with an  
> alternative dependency selection implementation that uses some  
> authoritative set of "approved" versions when resolving artifacts...

That makes some sense from an 'approved' perspective, though the use  
case I'm describing is actually a specific patch (e.g. you've built  
your application with a certain release of dependencies, and then  
your vendor tells you you need to slot in this security fix).  This  
is slightly different than having a compliant or standard set of  
dependencies.

>
> Questions I have WRT this are:
>
> 1. will this set of blessed versions be portable (i.e. travel with  
> the POM)?

The simplest scoping would be to have it be an element of your  
application's POM.  I could also see it in /.m2/ specific settings  
for a build machine, though this would be less portable.  A fancy  
solution might be to point to a central server and download updates,  
but this is probably overkill.

> 2. how will developers in a team setting share this info? It seems  
> like implementing this would be an exercise in walking a tightrope  
> between portability and centralized declaration for versions.
(see previous answer)

>
>> * Filtering Parameters - Let the dependency resolution be able to  
>> pick artifacts based on things like required JDK version, license,  
>> signed/unsigned, tags,etc.
>
> Is this a dependency selection technique? It seems like it's just a  
> matter of "approving" a given dependency as a candidate for  
> selection using POM attributes and possibly attributes of the jar  
> itself (which could be a bit more involved), before allowing  
> artifact version selection to proceed. Much of this could probably  
> be factored out of the DefaultArtifactCollector, into some sort of  
> pluggable component.

Exactly, I think these filters are additional parameters to the  
ArtifactCollector.  So just like version ranges let you select which  
artifact to map to a dependency, these would be additional critiera.   
Some of my colleagues have been working on some matching algorithms  
for the collector to let you match the artifacts based on different  
filtering parameters.  Some of these filters can be required (e.g.  
JDK > 5 is required because the library uses annotations) or soft  
(prefer debug versions of the JAR).

A lot of this kind of info should be present in the JAR, though that  
might mean you have to download them as part of the dependency  
resolution/"ArtifactCollector" phase, rather than the current  
artifact resolution phase, whereas if they're in the  
artifactmetadata, this already gets downloaded.

>
>> * Blacklisting artifacts - Not just repositories.  This is  
>> appropriate if you don't control the POM that's using the artifact  
>> to be blacklisted.
>
> ...so you're talking about a global exclusions list? Again, my  
> questions about this center on dissemination of that list.  
> Otherwise, can't you do this with dependency exclusions now?

I'd actually use the same scoping mechanism as per virtual  
dependencies/specJARs.  Unfortunately I don't think you can do this  
with exclusions now, but I could be wrong.  Let's say A depends on B  
depends on C and B specifies a range.  If B is not under my control  
then could I still eliminate the version of C that would match B's  
range in the pom for A?  (I think I might have just figured that this  
would actually work).  If A puts in its own dependency for C as a !C  
(bad version) then I think this would work and is an effective  
blacklist.

>
>> * License Acknowledgment - Some commercial vendors (Sun) require a  
>> clickthrough license before downloading. Also a user getting an  
>> artifact under an Apache license with an LGPL dependency would  
>> need to be notified that they are not only accepting the Apache  
>> License but LGPL.
>
> This might be another thing that gets added to the dependency  
> selection process, when we're gathering candidate artifact/ 
> versions. If you combine it with spec dependencies, the selection  
> algorithm might be able to select an alternate [set of] deps to  
> satisfy the spec dependency you specify (when the user refuses to  
> traverse the click-through)...

Yep, though I'd like to see the APIs on the resolver to expose the  
set of artifacts that could satisfy a dependency, and which would be  
selected by default.  This way a resolver client could present the  
info to the user (like through an IDE tool).

>
> Also, any click-through implementation will have to completely  
> satisfy the legal requirements of people like Sun, which could mean  
> finding a way to embed/interact with a full-featured browser. I  
> don't know of any cross-platform way to do this, without embedding  
> a browser...and I'm not aware of any mature embeddable browser  
> implementations, personally.

I was thinking it would be up to the client (e.g. maven-client) to  
provide the implementation of the plug-ins for this.  The resolver  
would just specify a request/response interface and then the clients  
would need to write their own adapters.

>
>> * Expose dependency resolution to client tools - Provide APIs so  
>> that clients can pass in different artifacts with different  
>> filters to see not only the multiple artifacts and paths that can  
>> resolve a dependency, but also which one would be selected. This  
>> would be very helpful for debugging and also to let UI tools  
>> visualize the dependency graphs.
>
> We can undoubtedly go further in making this accessible, but it  
> should be possible to do what you describe even now. It's just a  
> matter of knowing the APIs...

My understanding of the APIs now are that you can get an  
ArtifactResolutionResult for ArtifactResolver.resolveTransitively 
(...) and then the work is done in the ArtifactCollector.  The thing  
is, these methods return a path for the dependency that's already  
resolved.  Yes, you might be able to plug-in listeners that get  
called at various stages and then build up your own graph of what's  
going on, but I'd like a public API that returns the dependency graph  
for a given artifact that includes all the nodes that 'could' match  
vs just the ones that were picked.  I could be missing something here  
in the APIs, but I'm pretty sure that this functionality isn't  
possible or is unwieldly to do with the current interfaces.

>
>> I don't think these features have easy workarounds in Maven 2.0.x,  
>> but if there are, that would be great.
>> I've peer reviewed these with the folks at work as well and I'm  
>> interested in feedback from the community on these improvements as  
>> I'd like to get started working on them.
>> Thanks,
>> Garrett
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: [PROPOSAL] M2 Dependency Resolution Improvements

[John Casey <jd...@yahoo.com>]

Comments inline.

Cheers,

John

Garrett Conaty wrote:
> I've been working quite a bit with Maven2 transitive dependencies at 
> work, and think that they're one of the best strengths of M2.  In the 
> course of building quite a number of artifacts through build, 
> deployment, and update I've found that there are number of improvements 
> that can be made.  So with the use cases that I've seen at work and 
> listening to what other devs in the community have experienced, I'm 
> proposing some enhancements.
> 
> These points are only an overview, and some are a variation or 
> enhancement on features in the Maven 2.1 design docs. I've gone into 
> more depth on the rationale and the features themselves on the MavenUser 
> Wiki at 
> http://docs.codehaus.org/display/MAVENUSER/Improving+Maven2+Dependency+Resolution 
> :
> 
> * Virtual Dependencies (also know as spec jars)

This is something we've been discussing for some time now. We just need 
to have a concrete discussion about what's involved in implementing it. 
For instance, we have to provide a portable way to resolve spec -> 
implementation mappings. This might take a form similar to the 
plugin-prefix -> implementation metadata file. Once we have that in 
place, we need some consistent mechanism for choosing one implementation 
over another. The corresponding mechanisms for plugin selection based on 
prefix are somewhat crude...

> 
> * Environmental Context - Use the JDK environment and deploy environment 
> as part of the resolution phase.  As an example, you don't need to get 
> JAXP API classes from a repository if compiling with JDK 1.4

If you can represent pieces like the JDK as a repository with an 
alternate layout, I don't see why we couldn't tie this into the spec 
dependencies bit above and make it work.

> 
> * Pinning Dependencies (Overrides) - If you have a fixed dependency tree 
> for a given release, it's important to be able to provided patches that 
> can override a node of the tree.  <dependencyManagement> only appears to 
> handle this for parent-child projects.

I suspect an elegant way to handle this might be with an alternative 
dependency selection implementation that uses some authoritative set of 
"approved" versions when resolving artifacts...

Questions I have WRT this are:

1. will this set of blessed versions be portable (i.e. travel with the POM)?
2. how will developers in a team setting share this info? It seems like 
implementing this would be an exercise in walking a tightrope between 
portability and centralized declaration for versions.

> 
> * Filtering Parameters - Let the dependency resolution be able to pick 
> artifacts based on things like required JDK version, license, 
> signed/unsigned, tags,etc.

Is this a dependency selection technique? It seems like it's just a 
matter of "approving" a given dependency as a candidate for selection 
using POM attributes and possibly attributes of the jar itself (which 
could be a bit more involved), before allowing artifact version 
selection to proceed. Much of this could probably be factored out of the 
DefaultArtifactCollector, into some sort of pluggable component.

> 
> * Blacklisting artifacts - Not just repositories.  This is appropriate 
> if you don't control the POM that's using the artifact to be blacklisted.

...so you're talking about a global exclusions list? Again, my questions 
about this center on dissemination of that list. Otherwise, can't you do 
this with dependency exclusions now?

> 
> * License Acknowledgment - Some commercial vendors (Sun) require a 
> clickthrough license before downloading. Also a user getting an artifact 
> under an Apache license with an LGPL dependency would need to be 
> notified that they are not only accepting the Apache License but LGPL.

This might be another thing that gets added to the dependency selection 
process, when we're gathering candidate artifact/versions. If you 
combine it with spec dependencies, the selection algorithm might be able 
to select an alternate [set of] deps to satisfy the spec dependency you 
specify (when the user refuses to traverse the click-through)...

Also, any click-through implementation will have to completely satisfy 
the legal requirements of people like Sun, which could mean finding a 
way to embed/interact with a full-featured browser. I don't know of any 
cross-platform way to do this, without embedding a browser...and I'm not 
aware of any mature embeddable browser implementations, personally.

> 
> * Expose dependency resolution to client tools - Provide APIs so that 
> clients can pass in different artifacts with different filters to see 
> not only the multiple artifacts and paths that can resolve a dependency, 
> but also which one would be selected. This would be very helpful for 
> debugging and also to let UI tools visualize the dependency graphs.

We can undoubtedly go further in making this accessible, but it should 
be possible to do what you describe even now. It's just a matter of 
knowing the APIs...

> 
> I don't think these features have easy workarounds in Maven 2.0.x, but 
> if there are, that would be great.
> 
> I've peer reviewed these with the folks at work as well and I'm 
> interested in feedback from the community on these improvements as I'd 
> like to get started working on them.
> 
> Thanks,
> Garrett
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org