You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Oleg Gusakov <ol...@gmail.com> on 2008/12/19 19:29:04 UTC
How Mercury resolution works?
a diagram of mercury inner workflow is here:
http://blogs.sonatype.com/people/?p=1016
fyi - I am also working on re-introducing the "old" maven-ant syntax to
mercury-ant-tasks. Herve - as you wished!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
Ralph,
Ralph Goers wrote:
>
> This whole thread makes my head hurt.
>
> Frankly, and at the risk of being repetitious, I think all this work
> into making this resolution based on versioning is pretty much a waste
> of time. Now if your picture incorporated artifact metadata, such as
> "provides" and "requires" attributes I'd sit up and take notice. Until
> it does it really won't much matter how this works since everything is
> going to continue to be locked down with managed dependencies.
>
> All I'm really looking for is a) dependency resolution based on
> artifact attributes, b) notification when an unresolvable conflict
> exists, and c) a way to generate managed dependencies based on a and b
> so that the build can always be repeated.
Good news: I think I have a very good upgrade path towards that. Overall
- I think we should stabilize Maven3, and then enhance Mercury
resolution as an attribute based superset of GAV we have now. The
foundation is already there as I was keeping this path in mind while
designing Mercury.
I will post my proposal and publish it on the list.
Merry XMas!
Oleg
>
> Ralph
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
Ralph Goers wrote:
>
> I understand what you are suggesting, although the example above
> doesn't seem to match since the asm and platform tags are used outside
> the scope.
Maybe because we understand the scope a little differently? For me scope
is just a name - it does not have any attributes. So the example simply
states that in the qa scope I need the binaries, identified by the
specified query, nothing else. So when the client asks to resolve
conflicts for qa - this what Mercury is going to do.
> In any case, what you are discussing here is right on the money with
> what I am looking for.
As far as I understand - what I described is exactly what Jason is
talking about, maybe in different terms.
I guess - this is not the last time we talk about it :)
Thanks,
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Ralph Goers <ra...@dslextreme.com>.
On Dec 25, 2008, at 6:08 PM, Oleg Gusakov wrote:
> Ralph,
>
> If we start walking this road - let's generalize starting from
> existing model, so that it could still be a subset of the the
> solution.
I'm fine with that.
>
>
> more complex (or simple :) spec is:
>
> <dependency>
> <scope>qa</scope>
> <!-- we can use "version range" syntax because asm was declared in
> "version" domain -->
> <asm>[2.0,3.0)</asm>
>
> <!-- this one specifies a list of possible values, including
> negation: !win -->
> <platform>[!win,linux]</platform>
>
> </dependency>
>
>
> Because it's a superset of the current model, we can easily provide
> backward compatibility, as well as introduce all the new features
> discussed here.
>
> Resolution on this model is no different from what Mercury is doing
> now, so implementing it should be straightforward. I need to explore
> the negation option more though ..
I understand what you are suggesting, although the example above
doesn't seem to match since the asm and platform tags are used outside
the scope. In any case, what you are discussing here is right on the
money with what I am looking for. Hopefully, before you (or anyone
else) start implementing it - which I understand is still a ways off -
I would like to see a proposal put forth of what the pom syntax could
be so we can all understand what the end result is we are going for.
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
Ralph,
If we start walking this road - let's generalize starting from existing
model, so that it could still be a subset of the the solution.
So what we have now:
* each artifact "supplies"/"provides"/"has" a set of attributes: groupId
(G), artifactId (A), version (V), classifier (C), type (T)
* each artifact declares a requirement: "dependency", or a wish:
"optional dependency" as an equation on GACT and range expression on V
* scope is really something else - it represents an "enviroment" for
which the resolution is made
Let's just super-size it:
* each artifact has a set of attributes that include at least GAVT,
others are declared in the <attributes> section, similar to
<properties>. If default string comparison rules apply, the declaration
may look like:
<attributes>
<platform>linux</platform>
</attributes>
In order to compare attributes we might need domain-specific
comparators, if they are nor ordinary string equality, for example
<attributes>
<asm domain="version">2.0</asm>
</attributes>
This is required for correct processing of the expressions over this
attribute
* users can define their own scopes and relationships between them.
<scope>
<name>qa</name>
<includes>
<scope>compile</scope>
<scope>dev</scope>
</includes>
</scope>
* each artifact declares it's desired configuration for a particular
scope as logical and over individual attribute expressions; the simplest
for remains the same as now:
<dependency>
<optional>true</optional>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
<version>[3.0,4.0)</version>
<scope>qa</scope>
</dependency>
more complex (or simple :) spec is:
<dependency>
<scope>qa</scope>
<!-- we can use "version range" syntax because asm was declared in
"version" domain -->
<asm>[2.0,3.0)</asm>
<!-- this one specifies a list of possible values, including negation:
!win -->
<platform>[!win,linux]</platform>
</dependency>
Because it's a superset of the current model, we can easily provide
backward compatibility, as well as introduce all the new features
discussed here.
Resolution on this model is no different from what Mercury is doing now,
so implementing it should be straightforward. I need to explore the
negation option more though ..
Thanks,
Oleg
Ralph Goers wrote:
> Jason,
>
> I'm surprised you don't remember this discussion since we've had it so
> many times.
>
> My opinion is that versions are just one piece of metadata, and a not
> very important one at that since they are almost completely arbitrary.
> When I speak of provides and requires I am referring to exactly what
> RPMs do. Yes, every RPM has a version but that is not actually what
> the RPM system uses to determine what can be installed. If you look at
> an RPM you will see provides and requires. A lot of folks think that
> these are used for package version checking, but that is not
> completely true. The general format is
>
> token [operator version]
>
> Usually this might be for the form libstdc++ >= 2.0. But the "tokens"
> are not limited to package names. They can be anything. For example,
> xerces-impl might list as what it provides:
>
> xerces-impl = 2.9.1
> jaxp = 1.2,1.3
> xml = 1.0,1.1
> xml-schema=1.0
> sax = 2.0.2
> xinclude = 1.0
>
> Debian packages handle this a little differently (see section 7.5 at
> http://www.debian.org/doc/debian-policy/ch-relationships.html). There
> the provides is different in that you cannot specify a version so the
> above would have to be specified as
>
> jaxp-1.2
> jaxp-1.3
> xml-1.0
> xml-1.1
> xml-schema-1.0
> sax-2.0.2
> xinclude-1.0
>
> What I envision for Maven would be that the xerces pom might look like:
> <project>
> <groupId>xerces</groupId>
> <artifactId>xercesImpl</artifactId>
> <version>2.9.1</version>
> <provides>
> <feature name="jaxp">1.2,1.3</feature>
> <feature name="xml">1.0,1,1</feature>
> <feature name="xml-schema>1.0</feature>
> <feature name="sax">2.0.2</feature>
> <feature name="xinclude">1.0</feature>
> </provides>
> </project>
>
>
> A component might only be interested in JAXP 1.3 and not really care
> which version of Xerces provides it. It should be able to specify
> something like
>
> <dependency>
> <groupId>xerces</groupId>
> <artifactId>xercesImpl</artifactId>
> <requires>
> <feature name="jaxp">1.2</feature>
> </requires>
> </dependency>
>
> This would allow any version of xerces that provides jaxp 1.2 to be
> selected.
>
> On the other hand, a different component might want both JAXP 1.3 and
> xinclude support. This would limit the number of versions that could
> be selected but would still be resolvable since the latest version
> supports all of the requirements.
>
> While this example for Xerces might seem out of the ordinary, it
> really isn't. If you look at commons-collections history page
> http://commons.apache.org/collections/history.html it should be
> obvious that features could have easily been used to describe the
> various changes made to the project as well as to identify the binary
> incompatibilities that occurred.
>
> Ralph
>
>
> On Dec 25, 2008, at 8:10 AM, Jason van Zyl wrote:
>
>>
>> On 25-Dec-08, at 2:02 AM, Ralph Goers wrote:
>>
>>>
>>> This whole thread makes my head hurt.
>>>
>>> Frankly, and at the risk of being repetitious, I think all this work
>>> into making this resolution based on versioning is pretty much a
>>> waste of time.
>>
>> I don't think anyone is OSGi land would agree with you. This is a
>> critical step along the way.
>>
>>> Now if your picture incorporated artifact metadata, such as
>>> "provides" and "requires" attributes I'd sit up and take notice.
>>> Until it does it really won't much matter how this works since
>>> everything is going to continue to be locked down with managed
>>> dependencies.
>>>
>>
>> The whole point is to get down to a specified list of dependencies
>> that go into the runtime of your application. I don't disagree with
>> you here and Oleg certainly doesn't because we've discussed this at
>> length. What Mercury is provided is a way to arrive at that
>> possibility while along the way using the optimization functions to
>> get there. Otherwise it's a laborious manual process. We want to aid
>> this process during development with resolution and visualization
>> techniques. We don't want a system dynamically resolving everything
>> at runtime. The description of your application may say it can be
>> dynamic, but in practice you limit what is resolved. To manually
>> create your dependencyManagement, or lock down list, is a complete
>> pain in the ass.
>>
>> I'm also not sure what you mean in your provides/requires model. What
>> you provide is something couched in terms your package. In OSGi you
>> generally export, or provide, packages in your library/application
>> you want to expose to others. What you import, or require, are the
>> dependencies your library/application needs. In our model currently
>> what a Maven project provides is itself as an artifact. What it
>> requires are its dependencies. This is how RPM works. If you look in
>> a spec file the list of what's provided is usually itself. I'm not
>> sure were on the same wavelength when talking about
>> requires/provides. A library can't provide anything then the content
>> of what it contains itself. In our course grained model (or requires
>> bundle in OSGi speak) it's the artifact you're making.
>>
>>> All I'm really looking for is a) dependency resolution based on
>>> artifact attributes, b) notification when an unresolvable conflict
>>> exists, and c) a way to generate managed dependencies based on a and
>>> b so that the build can always be repeated.
>>
>> a) is definitely coming and it essentially looks like an LDAP query
>> for lack of anything else to compare it against
>> b) most definitely required and this is handled internally by what we
>> have and is going to be far easier to see given what Oleg has
>> c) exactly
>>
>> We are not talking about different things. To resolve all your
>> dependencies from scratch at runtime would be ill advised. The whole
>> point of the system is so that we can resolve on demand when changes
>> are necessary. Oleg and I have talked for a long time about the
>> dynamic nature of OSGi and balancing that with the fact that once
>> your QA team approves something you better not start throwing new
>> bits into the mix or you'll invalidate what was tested. Ultimately we
>> want a list of pieces for an application that we know don't change
>> when we are at the end of the cycle and are ready to ship. I'm pretty
>> sure what you need is something we have accounted for.
>>
>> When building a standard web application there are really no
>> constraints or metadata establishing what your runtime versioning
>> looks like. So you can happily toss in some new JARs in your
>> WEB-INF/lib directory and if it all links at runtimes everything is
>> good. When you upgrade things maybe you add in a few JARs, or maybe
>> you rebuild the entire application and drop it into your container.
>> In OSGi there are constraints about what pieces can work together at
>> runtime. You do have the option of stating the exact version of your
>> requirements, but this is not generally how people setup their
>> applications. They specify ranges of versions that are acceptable.
>> This is not to say that you want the resolution to go hog wild at
>> runtime. This doesn't happen because people generally manage the
>> bundle repository -- which has no remote capabilities -- and they
>> drop new things in. So the runtime can dynamically accept new
>> versions of dependencies if a range allows it. What is resolved in
>> practice is limited by what's in the bundle repository.
>>
>> So when Oleg is referencing all this resolution and the mechanics of
>> resolving it is toward the end of having a fixed list of something
>> that works and has been tested. We want the mechanics of this to work
>> in Mercury and be available in tooling like m2eclipse and we're also
>> working toward having a system where an application can say "I will
>> take anything you give me to resolve because I can dynamically update
>> myself" but it's going to be talking to Nexus who will say back "yes,
>> little application but I'm going to tell you what you are going to
>> use". To have to redeploy the metadata with the artifacts of an
>> application in order for it to resolve new versions of application on
>> N nodes of a large system is untenable. You need to be able to let
>> the application consume newer versions of artifacts while at the same
>> time limiting what it consumes. We are attempting to shift all this
>> management to Nexus where through a virtual URL an application will
>> resolve against a constrained set of artifacts that might be
>> available. In essence your fix Map of artifacts available from
>> dependencyManagement. So this is our end goal but I don't think
>> Mercury along way is incompatible with what you require.
>>
>>>
>>>
>>> Ralph
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>> Thanks,
>>
>> Jason
>>
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder, Apache Maven
>> jason at sonatype dot com
>> ----------------------------------------------------------
>>
>> A man enjoys his work when he understands the whole and when he
>> is responsible for the quality of the whole
>>
>> -- Christopher Alexander, A Pattern Language
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Jason van Zyl <jv...@sonatype.com>.
On 26-Dec-08, at 2:55 AM, Ralph Goers wrote:
>
> On Dec 25, 2008, at 7:02 PM, Jason van Zyl wrote:
>
>>
>> On 25-Dec-08, at 4:27 PM, Ralph Goers wrote:
>>
>>> Jason,
>>>
>>> I'm surprised you don't remember this discussion since we've had
>>> it so many times.
>>>
>>
>> The only thing I wanted to make sure you weren't doing was passing
>> on requirements as provisions transitively. If you know RPMs then
>> we're on the same wavelength. It was one of your previous emails
>> that made me wonder.
>
> Transitivity would have to come into play in some fashion. If A
> depends on B and C and B depends on an incompatible version of C
> this is a problem. The use of attributes would allow the system to
> be able to determine that. RPMs do the same thing, just differently.
> One of the primary differences is that sometimes it is OK for two
> versions of a component to be installed simultaneously in a Linux
> system, so in some cases this kind of transitivity is not a problem
> on a Linux system - although I'm not sure how the RPM system allows
> it.. In other cases it will be detected and the install will fail.
> With typical class loading this would be a problem in a Maven build
> and would hopefully be detected.
>>
>>
>> Again I don't think we're off to far from what you want Suse and
>> most of the other distributions employ a SAT solver to calculate a
>> set of suitable artifacts. The library that we are using, SAT4J,
>> is capable of doing this and is moreso used like this in P2 which
>> is the update manager technology used in Eclipse. It takes into
>> account attributes like platform and that's taken into account when
>> doing resolution for a particular target platform. But the first
>> thing we need to do with Mercury is make it work for what we do
>> currently.
>
> This is all good to hear and I'm fine with that. I guess I'm just
> trying to emphasize that doing what Maven 2.0.x already does but
> more efficiently really isn't much of a win. I realize that there
> are a few other benefits that go with it, but I'm more interested in
> how this will take us where we want to go, not in the math or logic
> behind version range checking.
I don't think you understand that Maven 2.x doesn't do this at all
correctly. So it's not just a matter of being more efficient, it's a
matter of actually having a system that works. SAT is at all strictly
for anything version specific. We need to replicate the behavior we
have but any attribute is represented as a variable in the system and
when you pile up all the variables in equations we are just solving
for a solution. What is happening in Mercury is absolutely critical.
What Oleg and Daniel are discussing is not limited to how a version is
solved for, but how to solve for anything akin to what you call an
attribute. The SAT solver doesn't care whether you stuff in versions,
platforms, quality markers. It's just how you represent the equations
and optimization functions. So any discussion of how this works is
useful. It buys us a lot understanding how this works, and we don't
have anything currently that would work properly.
>
>>
>>
>>> What I envision for Maven would be that the xerces pom might look
>>> like:
>>> <project>
>>> <groupId>xerces</groupId>
>>> <artifactId>xercesImpl</artifactId>
>>> <version>2.9.1</version>
>>> <provides>
>>> <feature name="jaxp">1.2,1.3</feature>
>>> <feature name="xml">1.0,1,1</feature>
>>> <feature name="xml-schema>1.0</feature>
>>> <feature name="sax">2.0.2</feature>
>>> <feature name="xinclude">1.0</feature>
>>> </provides>
>>> </project>
>>>
>>>
>>> A component might only be interested in JAXP 1.3 and not really
>>> care which version of Xerces provides it. It should be able to
>>> specify something like
>>>
>>> <dependency>
>>> <groupId>xerces</groupId>
>>> <artifactId>xercesImpl</artifactId>
>>> <requires>
>>> <feature name="jaxp">1.2</feature>
>>> </requires>
>>> </dependency>
>>>
>>> This would allow any version of xerces that provides jaxp 1.2 to
>>> be selected.
>>>
>>
>> This is one thing we could certainly do, but honestly something I
>> don't think is useful most of the time. I haven't had many
>> instances myself where I didn't care about the implementation of a
>> particular specification. If you really could guarantee not only
>> API but non-functional requirements like performance then possibly.
>> It won't be hard to do things like this with SAT4J. But I haven't
>> deployed in any sort of standard application server in a long time.
>> So if you're thinking to take everything that a platform might
>> provide in order to mesh that with what requirements your
>> application needs then sure. And something like this is totally
>> possible with Mercury and our use of SAT4J.
>
> I didn't say you didn't care about the implementation. In fact, that
> is why I xercesImpl is named as the artifact. What I don't care
> about is what version gets picked. As long as it meets all the
> features required throughout the build, then Mercury should choose
> the appropriate version - and allow me to create a managed
> dependency file so I can reproduce it. Or, as you had discussed,
> create some sort of label that the repo can use to insure the same
> artifacts are always chosen again.
Yes, this all being predicated on people agreeing on features/
capabilities and how to encode them, and doing the same for non-
functional attributes. I see this taking as long as shifting people
over to a mindset of versioning things. And this is the work we're
signing up for, but I think we're looking at a couple years of work
here at the very least.
>
>>
>>
>>> On the other hand, a different component might want both JAXP 1.3
>>> and xinclude support. This would limit the number of versions that
>>> could be selected but would still be resolvable since the latest
>>> version supports all of the requirements.
>>
>> Yah, I think we're a long way from this being practical and I
>> imagine we would be forging the way here again. OSGi can provide
>> something like the servlet api but I don't think they provide
>> capability maps, or feature sets as you've depicted them above. But
>> OSGi is certainly closer here if you can make the mental jump of
>> mapping a capability to a package like javax.servlet.
>
> No, from what I've seen from OSGi it doesn't solve this problem.
From a complete feature/capability map concept no. For something
simple like "i want the servlet API and I don't care where it comes
from" yes. But also realize there is P2 at eclipse which is slated to
a be a full provisioning solution. So much the same as Mercury where
you'll be able to make queries and get what you require. As far as
runtime partitioning OSGi the best system we have today. That said if
you have a real problem start up another VM.
> From what I understand is what it does do is play games with the
> class loader so that multiple versions of a component can exist
> within a JVM within different bundles. I also read a blog somewhere
> that indicated that this solution has some sort of fatal flaw in it.
It creates a graph of classloaders based on the bundles' specification
of imports and exports. It's in its 5th iteration as a specification
and while not a panacea, could use a lot better tooling, and has a
very lacking component model I don't think anything comes near it as
far as a runtime model. The classloader separation is useful.
>
>>
>>>
>>>
>>> While this example for Xerces might seem out of the ordinary, it
>>> really isn't. If you look at commons-collections history page http://commons.apache.org/collections/history.html
>>> it should be obvious that features could have easily been used to
>>> describe the various changes made to the project as well as to
>>> identify the binary incompatibilities that occurred.
>>>
>>
>> Ralph, I honestly think again on larger scale it is not going to be
>> a technical problem. It's deciding on a global capability map that
>> everyone would use accurate. I mean we still argue with people
>> about versioning their dependencies and "why can't I just use a lib
>> directory of JARs" kind of mentality so while I think this is the
>> direction we should go it's use is going to limited for a while and
>> Mercury is just beginning its life.
>>
> From my experience the people you have those conversations with have
> very little understanding of how class loaders work in Java.
That may be but it's the general population of Java developers, our
users. That's the reality we deal with.
> I have conversations with experienced developers who wonder why they
> are getting class loader exceptions trying to pass data between
> servlets in two different war files.
I assume you mean how something could get constructed that would let
that happen, not why it happens.
> I haven't seen a proposal yet - including OSGi - that really fixes
> this problem. And until it is fixed in Java itself we will need our
> build tools to do as much as they can to help out.
>
OSGi is a runtime model and it goes a long way to solve isolation
problems and reduce conflicts at runtime, the provisioning and
development model is not so strong IMO. There are good things like PDE
tools in the IDE that are great, and things that are generally abysmal
like PDE headless build. If you think anyone else is going to come
with a better runtime model then OSGi I think you're mistaken. Where
we can certainly help is the provisioning, development and build
tooling. As far as the system that you describe in terms of having a
completely attribute-based provisioning model, the runtime most suited
to accept that model is OSGi. How you build and provision things is
inextricably tied to how they run.
I hope to reach some level of interoperability with P2 as it aims to
be complete provisioning tool which is really the same space the Maven
and Mercury play in. We are grabbing a bunch of stuff, given and set
of constraints for an application and we want the application work at
runtime.
At any rate what you need I believe in there in Mercury. I also
believe it is necessary to get Mercury to work for the model we
currently have because what's present in Maven 2.x is not something we
can build on in the future.
> Ralph
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
We know what we are, but know not what we may be.
-- Shakespeare
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Ralph Goers <ra...@dslextreme.com>.
On Dec 25, 2008, at 7:02 PM, Jason van Zyl wrote:
>
> On 25-Dec-08, at 4:27 PM, Ralph Goers wrote:
>
>> Jason,
>>
>> I'm surprised you don't remember this discussion since we've had it
>> so many times.
>>
>
> The only thing I wanted to make sure you weren't doing was passing
> on requirements as provisions transitively. If you know RPMs then
> we're on the same wavelength. It was one of your previous emails
> that made me wonder.
Transitivity would have to come into play in some fashion. If A
depends on B and C and B depends on an incompatible version of C this
is a problem. The use of attributes would allow the system to be able
to determine that. RPMs do the same thing, just differently. One of
the primary differences is that sometimes it is OK for two versions of
a component to be installed simultaneously in a Linux system, so in
some cases this kind of transitivity is not a problem on a Linux
system - although I'm not sure how the RPM system allows it.. In other
cases it will be detected and the install will fail. With typical
class loading this would be a problem in a Maven build and would
hopefully be detected.
>
>
> Again I don't think we're off to far from what you want Suse and
> most of the other distributions employ a SAT solver to calculate a
> set of suitable artifacts. The library that we are using, SAT4J, is
> capable of doing this and is moreso used like this in P2 which is
> the update manager technology used in Eclipse. It takes into account
> attributes like platform and that's taken into account when doing
> resolution for a particular target platform. But the first thing we
> need to do with Mercury is make it work for what we do currently.
This is all good to hear and I'm fine with that. I guess I'm just
trying to emphasize that doing what Maven 2.0.x already does but more
efficiently really isn't much of a win. I realize that there are a few
other benefits that go with it, but I'm more interested in how this
will take us where we want to go, not in the math or logic behind
version range checking.
>
>
>> What I envision for Maven would be that the xerces pom might look
>> like:
>> <project>
>> <groupId>xerces</groupId>
>> <artifactId>xercesImpl</artifactId>
>> <version>2.9.1</version>
>> <provides>
>> <feature name="jaxp">1.2,1.3</feature>
>> <feature name="xml">1.0,1,1</feature>
>> <feature name="xml-schema>1.0</feature>
>> <feature name="sax">2.0.2</feature>
>> <feature name="xinclude">1.0</feature>
>> </provides>
>> </project>
>>
>>
>> A component might only be interested in JAXP 1.3 and not really
>> care which version of Xerces provides it. It should be able to
>> specify something like
>>
>> <dependency>
>> <groupId>xerces</groupId>
>> <artifactId>xercesImpl</artifactId>
>> <requires>
>> <feature name="jaxp">1.2</feature>
>> </requires>
>> </dependency>
>>
>> This would allow any version of xerces that provides jaxp 1.2 to be
>> selected.
>>
>
> This is one thing we could certainly do, but honestly something I
> don't think is useful most of the time. I haven't had many instances
> myself where I didn't care about the implementation of a particular
> specification. If you really could guarantee not only API but non-
> functional requirements like performance then possibly. It won't be
> hard to do things like this with SAT4J. But I haven't deployed in
> any sort of standard application server in a long time. So if you're
> thinking to take everything that a platform might provide in order
> to mesh that with what requirements your application needs then
> sure. And something like this is totally possible with Mercury and
> our use of SAT4J.
I didn't say you didn't care about the implementation. In fact, that
is why I xercesImpl is named as the artifact. What I don't care about
is what version gets picked. As long as it meets all the features
required throughout the build, then Mercury should choose the
appropriate version - and allow me to create a managed dependency file
so I can reproduce it. Or, as you had discussed, create some sort of
label that the repo can use to insure the same artifacts are always
chosen again.
>
>
>> On the other hand, a different component might want both JAXP 1.3
>> and xinclude support. This would limit the number of versions that
>> could be selected but would still be resolvable since the latest
>> version supports all of the requirements.
>
> Yah, I think we're a long way from this being practical and I
> imagine we would be forging the way here again. OSGi can provide
> something like the servlet api but I don't think they provide
> capability maps, or feature sets as you've depicted them above. But
> OSGi is certainly closer here if you can make the mental jump of
> mapping a capability to a package like javax.servlet.
No, from what I've seen from OSGi it doesn't solve this problem. From
what I understand is what it does do is play games with the class
loader so that multiple versions of a component can exist within a JVM
within different bundles. I also read a blog somewhere that indicated
that this solution has some sort of fatal flaw in it.
>
>>
>>
>> While this example for Xerces might seem out of the ordinary, it
>> really isn't. If you look at commons-collections history page http://commons.apache.org/collections/history.html
>> it should be obvious that features could have easily been used to
>> describe the various changes made to the project as well as to
>> identify the binary incompatibilities that occurred.
>>
>
> Ralph, I honestly think again on larger scale it is not going to be
> a technical problem. It's deciding on a global capability map that
> everyone would use accurate. I mean we still argue with people about
> versioning their dependencies and "why can't I just use a lib
> directory of JARs" kind of mentality so while I think this is the
> direction we should go it's use is going to limited for a while and
> Mercury is just beginning its life.
>
From my experience the people you have those conversations with have
very little understanding of how class loaders work in Java. I have
conversations with experienced developers who wonder why they are
getting class loader exceptions trying to pass data between servlets
in two different war files. I haven't seen a proposal yet - including
OSGi - that really fixes this problem. And until it is fixed in Java
itself we will need our build tools to do as much as they can to help
out.
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Jason van Zyl <jv...@sonatype.com>.
On 25-Dec-08, at 4:27 PM, Ralph Goers wrote:
> Jason,
>
> I'm surprised you don't remember this discussion since we've had it
> so many times.
>
The only thing I wanted to make sure you weren't doing was passing on
requirements as provisions transitively. If you know RPMs then we're
on the same wavelength. It was one of your previous emails that made
me wonder.
Again I don't think we're off to far from what you want Suse and most
of the other distributions employ a SAT solver to calculate a set of
suitable artifacts. The library that we are using, SAT4J, is capable
of doing this and is moreso used like this in P2 which is the update
manager technology used in Eclipse. It takes into account attributes
like platform and that's taken into account when doing resolution for
a particular target platform. But the first thing we need to do with
Mercury is make it work for what we do currently.
> My opinion is that versions are just one piece of metadata, and a
> not very important one at that since they are almost completely
> arbitrary. When I speak of provides and requires I am referring to
> exactly what RPMs do. Yes, every RPM has a version but that is not
> actually what the RPM system uses to determine what can be
> installed. If you look at an RPM you will see provides and requires.
> A lot of folks think that these are used for package version
> checking, but that is not completely true. The general format is
>
> token [operator version]
>
> Usually this might be for the form libstdc++ >= 2.0. But the
> "tokens" are not limited to package names. They can be anything. For
> example, xerces-impl might list as what it provides:
>
> xerces-impl = 2.9.1
> jaxp = 1.2,1.3
> xml = 1.0,1.1
> xml-schema=1.0
> sax = 2.0.2
> xinclude = 1.0
>
> Debian packages handle this a little differently (see section 7.5 at http://www.debian.org/doc/debian-policy/ch-relationships.html)
> . There the provides is different in that you cannot specify a
> version so the above would have to be specified as
>
> jaxp-1.2
> jaxp-1.3
> xml-1.0
> xml-1.1
> xml-schema-1.0
> sax-2.0.2
> xinclude-1.0
>
Sure, these are exactly the types of things SAT is good for and is why
most of the package managers employ some form of SAT solver.
> What I envision for Maven would be that the xerces pom might look
> like:
> <project>
> <groupId>xerces</groupId>
> <artifactId>xercesImpl</artifactId>
> <version>2.9.1</version>
> <provides>
> <feature name="jaxp">1.2,1.3</feature>
> <feature name="xml">1.0,1,1</feature>
> <feature name="xml-schema>1.0</feature>
> <feature name="sax">2.0.2</feature>
> <feature name="xinclude">1.0</feature>
> </provides>
> </project>
>
>
> A component might only be interested in JAXP 1.3 and not really care
> which version of Xerces provides it. It should be able to specify
> something like
>
> <dependency>
> <groupId>xerces</groupId>
> <artifactId>xercesImpl</artifactId>
> <requires>
> <feature name="jaxp">1.2</feature>
> </requires>
> </dependency>
>
> This would allow any version of xerces that provides jaxp 1.2 to be
> selected.
>
This is one thing we could certainly do, but honestly something I
don't think is useful most of the time. I haven't had many instances
myself where I didn't care about the implementation of a particular
specification. If you really could guarantee not only API but non-
functional requirements like performance then possibly. It won't be
hard to do things like this with SAT4J. But I haven't deployed in any
sort of standard application server in a long time. So if you're
thinking to take everything that a platform might provide in order to
mesh that with what requirements your application needs then sure. And
something like this is totally possible with Mercury and our use of
SAT4J.
> On the other hand, a different component might want both JAXP 1.3
> and xinclude support. This would limit the number of versions that
> could be selected but would still be resolvable since the latest
> version supports all of the requirements.
Yah, I think we're a long way from this being practical and I imagine
we would be forging the way here again. OSGi can provide something
like the servlet api but I don't think they provide capability maps,
or feature sets as you've depicted them above. But OSGi is certainly
closer here if you can make the mental jump of mapping a capability to
a package like javax.servlet.
>
>
> While this example for Xerces might seem out of the ordinary, it
> really isn't. If you look at commons-collections history page http://commons.apache.org/collections/history.html
> it should be obvious that features could have easily been used to
> describe the various changes made to the project as well as to
> identify the binary incompatibilities that occurred.
>
Ralph, I honestly think again on larger scale it is not going to be a
technical problem. It's deciding on a global capability map that
everyone would use accurate. I mean we still argue with people about
versioning their dependencies and "why can't I just use a lib
directory of JARs" kind of mentality so while I think this is the
direction we should go it's use is going to limited for a while and
Mercury is just beginning its life.
> Ralph
>
>
> On Dec 25, 2008, at 8:10 AM, Jason van Zyl wrote:
>
>>
>> On 25-Dec-08, at 2:02 AM, Ralph Goers wrote:
>>
>>>
>>> This whole thread makes my head hurt.
>>>
>>> Frankly, and at the risk of being repetitious, I think all this
>>> work into making this resolution based on versioning is pretty
>>> much a waste of time.
>>
>> I don't think anyone is OSGi land would agree with you. This is a
>> critical step along the way.
>>
>>> Now if your picture incorporated artifact metadata, such as
>>> "provides" and "requires" attributes I'd sit up and take notice.
>>> Until it does it really won't much matter how this works since
>>> everything is going to continue to be locked down with managed
>>> dependencies.
>>>
>>
>> The whole point is to get down to a specified list of dependencies
>> that go into the runtime of your application. I don't disagree with
>> you here and Oleg certainly doesn't because we've discussed this at
>> length. What Mercury is provided is a way to arrive at that
>> possibility while along the way using the optimization functions to
>> get there. Otherwise it's a laborious manual process. We want to
>> aid this process during development with resolution and
>> visualization techniques. We don't want a system dynamically
>> resolving everything at runtime. The description of your
>> application may say it can be dynamic, but in practice you limit
>> what is resolved. To manually create your dependencyManagement, or
>> lock down list, is a complete pain in the ass.
>>
>> I'm also not sure what you mean in your provides/requires model.
>> What you provide is something couched in terms your package. In
>> OSGi you generally export, or provide, packages in your library/
>> application you want to expose to others. What you import, or
>> require, are the dependencies your library/application needs. In
>> our model currently what a Maven project provides is itself as an
>> artifact. What it requires are its dependencies. This is how RPM
>> works. If you look in a spec file the list of what's provided is
>> usually itself. I'm not sure were on the same wavelength when
>> talking about requires/provides. A library can't provide anything
>> then the content of what it contains itself. In our course grained
>> model (or requires bundle in OSGi speak) it's the artifact you're
>> making.
>>
>>> All I'm really looking for is a) dependency resolution based on
>>> artifact attributes, b) notification when an unresolvable conflict
>>> exists, and c) a way to generate managed dependencies based on a
>>> and b so that the build can always be repeated.
>>
>> a) is definitely coming and it essentially looks like an LDAP query
>> for lack of anything else to compare it against
>> b) most definitely required and this is handled internally by what
>> we have and is going to be far easier to see given what Oleg has
>> c) exactly
>>
>> We are not talking about different things. To resolve all your
>> dependencies from scratch at runtime would be ill advised. The
>> whole point of the system is so that we can resolve on demand when
>> changes are necessary. Oleg and I have talked for a long time about
>> the dynamic nature of OSGi and balancing that with the fact that
>> once your QA team approves something you better not start throwing
>> new bits into the mix or you'll invalidate what was tested.
>> Ultimately we want a list of pieces for an application that we know
>> don't change when we are at the end of the cycle and are ready to
>> ship. I'm pretty sure what you need is something we have accounted
>> for.
>>
>> When building a standard web application there are really no
>> constraints or metadata establishing what your runtime versioning
>> looks like. So you can happily toss in some new JARs in your WEB-
>> INF/lib directory and if it all links at runtimes everything is
>> good. When you upgrade things maybe you add in a few JARs, or maybe
>> you rebuild the entire application and drop it into your container.
>> In OSGi there are constraints about what pieces can work together
>> at runtime. You do have the option of stating the exact version of
>> your requirements, but this is not generally how people setup their
>> applications. They specify ranges of versions that are acceptable.
>> This is not to say that you want the resolution to go hog wild at
>> runtime. This doesn't happen because people generally manage the
>> bundle repository -- which has no remote capabilities -- and they
>> drop new things in. So the runtime can dynamically accept new
>> versions of dependencies if a range allows it. What is resolved in
>> practice is limited by what's in the bundle repository.
>>
>> So when Oleg is referencing all this resolution and the mechanics
>> of resolving it is toward the end of having a fixed list of
>> something that works and has been tested. We want the mechanics of
>> this to work in Mercury and be available in tooling like m2eclipse
>> and we're also working toward having a system where an application
>> can say "I will take anything you give me to resolve because I can
>> dynamically update myself" but it's going to be talking to Nexus
>> who will say back "yes, little application but I'm going to tell
>> you what you are going to use". To have to redeploy the metadata
>> with the artifacts of an application in order for it to resolve new
>> versions of application on N nodes of a large system is untenable.
>> You need to be able to let the application consume newer versions
>> of artifacts while at the same time limiting what it consumes. We
>> are attempting to shift all this management to Nexus where through
>> a virtual URL an application will resolve against a constrained set
>> of artifacts that might be available. In essence your fix Map of
>> artifacts available from dependencyManagement. So this is our end
>> goal but I don't think Mercury along way is incompatible with what
>> you require.
>>
>>>
>>>
>>> Ralph
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>> Thanks,
>>
>> Jason
>>
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder, Apache Maven
>> jason at sonatype dot com
>> ----------------------------------------------------------
>>
>> A man enjoys his work when he understands the whole and when he
>> is responsible for the quality of the whole
>>
>> -- Christopher Alexander, A Pattern Language
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
Our achievements speak for themselves. What we have to keep track
of are our failures, discouragements and doubts. We tend to forget
the past difficulties, the many false starts, and the painful
groping. We see our past achievements as the end result of a
clean forward thrust, and our present difficulties as
signs of decline and decay.
-- Eric Hoffer, Reflections on the Human Condition
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Ralph Goers <ra...@dslextreme.com>.
Jason,
I'm surprised you don't remember this discussion since we've had it so
many times.
My opinion is that versions are just one piece of metadata, and a not
very important one at that since they are almost completely arbitrary.
When I speak of provides and requires I am referring to exactly what
RPMs do. Yes, every RPM has a version but that is not actually what
the RPM system uses to determine what can be installed. If you look at
an RPM you will see provides and requires. A lot of folks think that
these are used for package version checking, but that is not
completely true. The general format is
token [operator version]
Usually this might be for the form libstdc++ >= 2.0. But the "tokens"
are not limited to package names. They can be anything. For example,
xerces-impl might list as what it provides:
xerces-impl = 2.9.1
jaxp = 1.2,1.3
xml = 1.0,1.1
xml-schema=1.0
sax = 2.0.2
xinclude = 1.0
Debian packages handle this a little differently (see section 7.5 at http://www.debian.org/doc/debian-policy/ch-relationships.html)
. There the provides is different in that you cannot specify a version
so the above would have to be specified as
jaxp-1.2
jaxp-1.3
xml-1.0
xml-1.1
xml-schema-1.0
sax-2.0.2
xinclude-1.0
What I envision for Maven would be that the xerces pom might look like:
<project>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.9.1</version>
<provides>
<feature name="jaxp">1.2,1.3</feature>
<feature name="xml">1.0,1,1</feature>
<feature name="xml-schema>1.0</feature>
<feature name="sax">2.0.2</feature>
<feature name="xinclude">1.0</feature>
</provides>
</project>
A component might only be interested in JAXP 1.3 and not really care
which version of Xerces provides it. It should be able to specify
something like
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<requires>
<feature name="jaxp">1.2</feature>
</requires>
</dependency>
This would allow any version of xerces that provides jaxp 1.2 to be
selected.
On the other hand, a different component might want both JAXP 1.3 and
xinclude support. This would limit the number of versions that could
be selected but would still be resolvable since the latest version
supports all of the requirements.
While this example for Xerces might seem out of the ordinary, it
really isn't. If you look at commons-collections history page http://commons.apache.org/collections/history.html
it should be obvious that features could have easily been used to
describe the various changes made to the project as well as to
identify the binary incompatibilities that occurred.
Ralph
On Dec 25, 2008, at 8:10 AM, Jason van Zyl wrote:
>
> On 25-Dec-08, at 2:02 AM, Ralph Goers wrote:
>
>>
>> This whole thread makes my head hurt.
>>
>> Frankly, and at the risk of being repetitious, I think all this
>> work into making this resolution based on versioning is pretty much
>> a waste of time.
>
> I don't think anyone is OSGi land would agree with you. This is a
> critical step along the way.
>
>> Now if your picture incorporated artifact metadata, such as
>> "provides" and "requires" attributes I'd sit up and take notice.
>> Until it does it really won't much matter how this works since
>> everything is going to continue to be locked down with managed
>> dependencies.
>>
>
> The whole point is to get down to a specified list of dependencies
> that go into the runtime of your application. I don't disagree with
> you here and Oleg certainly doesn't because we've discussed this at
> length. What Mercury is provided is a way to arrive at that
> possibility while along the way using the optimization functions to
> get there. Otherwise it's a laborious manual process. We want to aid
> this process during development with resolution and visualization
> techniques. We don't want a system dynamically resolving everything
> at runtime. The description of your application may say it can be
> dynamic, but in practice you limit what is resolved. To manually
> create your dependencyManagement, or lock down list, is a complete
> pain in the ass.
>
> I'm also not sure what you mean in your provides/requires model.
> What you provide is something couched in terms your package. In OSGi
> you generally export, or provide, packages in your library/
> application you want to expose to others. What you import, or
> require, are the dependencies your library/application needs. In our
> model currently what a Maven project provides is itself as an
> artifact. What it requires are its dependencies. This is how RPM
> works. If you look in a spec file the list of what's provided is
> usually itself. I'm not sure were on the same wavelength when
> talking about requires/provides. A library can't provide anything
> then the content of what it contains itself. In our course grained
> model (or requires bundle in OSGi speak) it's the artifact you're
> making.
>
>> All I'm really looking for is a) dependency resolution based on
>> artifact attributes, b) notification when an unresolvable conflict
>> exists, and c) a way to generate managed dependencies based on a
>> and b so that the build can always be repeated.
>
> a) is definitely coming and it essentially looks like an LDAP query
> for lack of anything else to compare it against
> b) most definitely required and this is handled internally by what
> we have and is going to be far easier to see given what Oleg has
> c) exactly
>
> We are not talking about different things. To resolve all your
> dependencies from scratch at runtime would be ill advised. The whole
> point of the system is so that we can resolve on demand when changes
> are necessary. Oleg and I have talked for a long time about the
> dynamic nature of OSGi and balancing that with the fact that once
> your QA team approves something you better not start throwing new
> bits into the mix or you'll invalidate what was tested. Ultimately
> we want a list of pieces for an application that we know don't
> change when we are at the end of the cycle and are ready to ship.
> I'm pretty sure what you need is something we have accounted for.
>
> When building a standard web application there are really no
> constraints or metadata establishing what your runtime versioning
> looks like. So you can happily toss in some new JARs in your WEB-INF/
> lib directory and if it all links at runtimes everything is good.
> When you upgrade things maybe you add in a few JARs, or maybe you
> rebuild the entire application and drop it into your container. In
> OSGi there are constraints about what pieces can work together at
> runtime. You do have the option of stating the exact version of your
> requirements, but this is not generally how people setup their
> applications. They specify ranges of versions that are acceptable.
> This is not to say that you want the resolution to go hog wild at
> runtime. This doesn't happen because people generally manage the
> bundle repository -- which has no remote capabilities -- and they
> drop new things in. So the runtime can dynamically accept new
> versions of dependencies if a range allows it. What is resolved in
> practice is limited by what's in the bundle repository.
>
> So when Oleg is referencing all this resolution and the mechanics of
> resolving it is toward the end of having a fixed list of something
> that works and has been tested. We want the mechanics of this to
> work in Mercury and be available in tooling like m2eclipse and we're
> also working toward having a system where an application can say "I
> will take anything you give me to resolve because I can dynamically
> update myself" but it's going to be talking to Nexus who will say
> back "yes, little application but I'm going to tell you what you are
> going to use". To have to redeploy the metadata with the artifacts
> of an application in order for it to resolve new versions of
> application on N nodes of a large system is untenable. You need to
> be able to let the application consume newer versions of artifacts
> while at the same time limiting what it consumes. We are attempting
> to shift all this management to Nexus where through a virtual URL an
> application will resolve against a constrained set of artifacts that
> might be available. In essence your fix Map of artifacts available
> from dependencyManagement. So this is our end goal but I don't think
> Mercury along way is incompatible with what you require.
>
>>
>>
>> Ralph
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> jason at sonatype dot com
> ----------------------------------------------------------
>
> A man enjoys his work when he understands the whole and when he
> is responsible for the quality of the whole
>
> -- Christopher Alexander, A Pattern Language
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Jason van Zyl <jv...@sonatype.com>.
On 25-Dec-08, at 2:02 AM, Ralph Goers wrote:
>
> This whole thread makes my head hurt.
>
> Frankly, and at the risk of being repetitious, I think all this work
> into making this resolution based on versioning is pretty much a
> waste of time.
I don't think anyone is OSGi land would agree with you. This is a
critical step along the way.
> Now if your picture incorporated artifact metadata, such as
> "provides" and "requires" attributes I'd sit up and take notice.
> Until it does it really won't much matter how this works since
> everything is going to continue to be locked down with managed
> dependencies.
>
The whole point is to get down to a specified list of dependencies
that go into the runtime of your application. I don't disagree with
you here and Oleg certainly doesn't because we've discussed this at
length. What Mercury is provided is a way to arrive at that
possibility while along the way using the optimization functions to
get there. Otherwise it's a laborious manual process. We want to aid
this process during development with resolution and visualization
techniques. We don't want a system dynamically resolving everything at
runtime. The description of your application may say it can be
dynamic, but in practice you limit what is resolved. To manually
create your dependencyManagement, or lock down list, is a complete
pain in the ass.
I'm also not sure what you mean in your provides/requires model. What
you provide is something couched in terms your package. In OSGi you
generally export, or provide, packages in your library/application you
want to expose to others. What you import, or require, are the
dependencies your library/application needs. In our model currently
what a Maven project provides is itself as an artifact. What it
requires are its dependencies. This is how RPM works. If you look in a
spec file the list of what's provided is usually itself. I'm not sure
were on the same wavelength when talking about requires/provides. A
library can't provide anything then the content of what it contains
itself. In our course grained model (or requires bundle in OSGi speak)
it's the artifact you're making.
> All I'm really looking for is a) dependency resolution based on
> artifact attributes, b) notification when an unresolvable conflict
> exists, and c) a way to generate managed dependencies based on a and
> b so that the build can always be repeated.
a) is definitely coming and it essentially looks like an LDAP query
for lack of anything else to compare it against
b) most definitely required and this is handled internally by what we
have and is going to be far easier to see given what Oleg has
c) exactly
We are not talking about different things. To resolve all your
dependencies from scratch at runtime would be ill advised. The whole
point of the system is so that we can resolve on demand when changes
are necessary. Oleg and I have talked for a long time about the
dynamic nature of OSGi and balancing that with the fact that once your
QA team approves something you better not start throwing new bits into
the mix or you'll invalidate what was tested. Ultimately we want a
list of pieces for an application that we know don't change when we
are at the end of the cycle and are ready to ship. I'm pretty sure
what you need is something we have accounted for.
When building a standard web application there are really no
constraints or metadata establishing what your runtime versioning
looks like. So you can happily toss in some new JARs in your WEB-INF/
lib directory and if it all links at runtimes everything is good. When
you upgrade things maybe you add in a few JARs, or maybe you rebuild
the entire application and drop it into your container. In OSGi there
are constraints about what pieces can work together at runtime. You do
have the option of stating the exact version of your requirements, but
this is not generally how people setup their applications. They
specify ranges of versions that are acceptable. This is not to say
that you want the resolution to go hog wild at runtime. This doesn't
happen because people generally manage the bundle repository -- which
has no remote capabilities -- and they drop new things in. So the
runtime can dynamically accept new versions of dependencies if a range
allows it. What is resolved in practice is limited by what's in the
bundle repository.
So when Oleg is referencing all this resolution and the mechanics of
resolving it is toward the end of having a fixed list of something
that works and has been tested. We want the mechanics of this to work
in Mercury and be available in tooling like m2eclipse and we're also
working toward having a system where an application can say "I will
take anything you give me to resolve because I can dynamically update
myself" but it's going to be talking to Nexus who will say back "yes,
little application but I'm going to tell you what you are going to
use". To have to redeploy the metadata with the artifacts of an
application in order for it to resolve new versions of application on
N nodes of a large system is untenable. You need to be able to let the
application consume newer versions of artifacts while at the same time
limiting what it consumes. We are attempting to shift all this
management to Nexus where through a virtual URL an application will
resolve against a constrained set of artifacts that might be
available. In essence your fix Map of artifacts available from
dependencyManagement. So this is our end goal but I don't think
Mercury along way is incompatible with what you require.
>
>
> Ralph
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
A man enjoys his work when he understands the whole and when he
is responsible for the quality of the whole
-- Christopher Alexander, A Pattern Language
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Ralph Goers <ra...@dslextreme.com>.
This whole thread makes my head hurt.
Frankly, and at the risk of being repetitious, I think all this work
into making this resolution based on versioning is pretty much a waste
of time. Now if your picture incorporated artifact metadata, such as
"provides" and "requires" attributes I'd sit up and take notice. Until
it does it really won't much matter how this works since everything is
going to continue to be locked down with managed dependencies.
All I'm really looking for is a) dependency resolution based on
artifact attributes, b) notification when an unresolvable conflict
exists, and c) a way to generate managed dependencies based on a and b
so that the build can always be repeated.
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
Fixed. Thank you!
Tomasz Pik wrote:
> On Fri, Dec 19, 2008 at 7:29 PM, Oleg Gusakov
> <ol...@gmail.com> wrote:
>
>> a diagram of mercury inner workflow is here:
>> http://blogs.sonatype.com/people/?p=1016
>>
>
> One minor change - there should be a1 = 1 instead of a = 1.
>
> Regards,
> Tomek
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
>
Re: How Mercury resolution works?
Posted by Tomasz Pik <to...@gmail.com>.
On Fri, Dec 19, 2008 at 7:29 PM, Oleg Gusakov
<ol...@gmail.com> wrote:
> a diagram of mercury inner workflow is here:
> http://blogs.sonatype.com/people/?p=1016
One minor change - there should be a1 = 1 instead of a = 1.
Regards,
Tomek
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Daniel Le Berre <le...@cril.univ-artois.fr>.
Gilles Scokart a écrit :
> 2008/12/22 Daniel Le Berre <le...@cril.univ-artois.fr>:
>> Gilles,
>>
>> Representation #1 is used to compute which artifacts must be installed to
>> satisfy some artifact dependencies.
>>
>> With representation #2, Oleg only solve the problem of determining which
>> version of the available artifacts should be installed (at least, this is my
>> understanding of its encoding).
>>
>> Daniel
>>
>
> Sorry, but I fail to see the difference.
>
> Gilles
>
in #1, you represent the whole dependencies in a file, and you let the
SAT solver do all the work for you (no need for a dirty tree).
in #2, you build the dirty tree and you feed the SAT solver only with
the information regarding versions of artifacts.
Daniel
--
Daniel Le Berre mailto:leberre@cril.univ-artois.fr
MCF, CRIL-CNRS UMR 8188, Universite d'Artois
http://www.cril.univ-artois.fr/~leberre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Gilles Scokart <gs...@gmail.com>.
2008/12/22 Daniel Le Berre <le...@cril.univ-artois.fr>:
> Gilles,
>
> Representation #1 is used to compute which artifacts must be installed to
> satisfy some artifact dependencies.
>
> With representation #2, Oleg only solve the problem of determining which
> version of the available artifacts should be installed (at least, this is my
> understanding of its encoding).
>
> Daniel
>
Sorry, but I fail to see the difference.
Gilles
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Daniel Le Berre <le...@cril.univ-artois.fr>.
Gilles,
Representation #1 is used to compute which artifacts must be installed
to satisfy some artifact dependencies.
With representation #2, Oleg only solve the problem of determining which
version of the available artifacts should be installed (at least, this
is my understanding of its encoding).
Daniel
Gilles Scokart a écrit :
> 2008/12/20 Daniel Le Berre <le...@cril.univ-artois.fr>:
>> Oleg Gusakov a écrit :
>>> The tree could be walked in two directions, and these approaches are
>>> equivalent, except for b and c optionality:
>>>
>>> Representation #1: P2
>>>
>>> a1 -> b1 or b2 or b3
>>> a1 -> c1 or c2
>>>
>>> b1 + b2 + b3 <= 1 # this one implicates that b is optional
>>> c1 + c2 <= 1 # this one implicates that c is optional
>>>
>>>
>>> Representation #2: Mercury
>>>
>>> b1 -> a1
>>> b2 -> a1
>>> b3 -> a1
>>> c1 -> a1
>>> c2 -> a1
>>> b1 + b2 + b3 = 1
>>> c1 + c2 + c3 = 1
>>>
>
> In representation #1, I think you miss a1 = 1. And I have the feeling
> that representation #1 is easier to generalize than Representation #2.
> What happens if I have one level more. Let say x having a dependency
> on a version [1,2].
> In representation #1, I keep the same equations :
> a1 -> b1 or b2 or b3
> a1 -> c1 or c2
> b1 + b2 + b3 <= 1
> c1 + c2 <= 1
>
> and I add :
> x -> a1 or a2 (dependency expressed)
> a1 + a2 <= 1
>
> Imagine that a2 has no dependencies, or worse that a2 just has a
> dependency on b3. With the representation #1, I can easily represent
> that.
> But with representation #2, I don't see what will be your system of
> constraints ('b3 -> a1' and 'c1+c2=1' is not correct anymore).
>
>
> Gilles
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
Daniel Le Berre mailto:leberre@cril.univ-artois.fr
MCF, CRIL-CNRS UMR 8188, Universite d'Artois
http://www.cril.univ-artois.fr/~leberre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Gilles Scokart <gs...@gmail.com>.
2008/12/20 Daniel Le Berre <le...@cril.univ-artois.fr>:
> Oleg Gusakov a écrit :
>> The tree could be walked in two directions, and these approaches are
>> equivalent, except for b and c optionality:
>>
>> Representation #1: P2
>>
>> a1 -> b1 or b2 or b3
>> a1 -> c1 or c2
>>
>> b1 + b2 + b3 <= 1 # this one implicates that b is optional
>> c1 + c2 <= 1 # this one implicates that c is optional
>>
>>
>> Representation #2: Mercury
>>
>> b1 -> a1
>> b2 -> a1
>> b3 -> a1
>> c1 -> a1
>> c2 -> a1
>> b1 + b2 + b3 = 1
>> c1 + c2 + c3 = 1
>>
In representation #1, I think you miss a1 = 1. And I have the feeling
that representation #1 is easier to generalize than Representation #2.
What happens if I have one level more. Let say x having a dependency
on a version [1,2].
In representation #1, I keep the same equations :
a1 -> b1 or b2 or b3
a1 -> c1 or c2
b1 + b2 + b3 <= 1
c1 + c2 <= 1
and I add :
x -> a1 or a2 (dependency expressed)
a1 + a2 <= 1
Imagine that a2 has no dependencies, or worse that a2 just has a
dependency on b3. With the representation #1, I can easily represent
that.
But with representation #2, I don't see what will be your system of
constraints ('b3 -> a1' and 'c1+c2=1' is not correct anymore).
Gilles
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
Daniel Le Berre wrote:
> Oleg Gusakov a écrit :
>
>> The tree could be walked in two directions, and these approaches are
>> equivalent, except for b and c optionality:
>>
>> Representation #1: P2
>>
>> a1 -> b1 or b2 or b3
>> a1 -> c1 or c2
>>
>> b1 + b2 + b3 <= 1 # this one implicates that b is optional
>> c1 + c2 <= 1 # this one implicates that c is optional
>>
>>
>> Representation #2: Mercury
>>
>> b1 -> a1
>> b2 -> a1
>> b3 -> a1
>> c1 -> a1
>> c2 -> a1
>> b1 + b2 + b3 = 1
>> c1 + c2 + c3 = 1
>>
>> In the second one it's easier for me to keep the tabs on the generated
>> system. Timewise both are solved in 10s of millis.
>>
>
> Oleg,
>
> I agree that you can encode either "provides" or "depends" semantic
> using constraints.
>
> However, you cannot ask the same question:
>
> Representation "depends":
>
> Append the package you want to install (a1=1).
> Ask for a solution consistent with the constraints (or the best one).
>
> (What do I need to install to be able to install a1).
>
> Representation "provides":
>
> Append the packages you have.
> Ask for all the packages that can be safely installed according to the
> constraints.
>
> (what can I install if I have all those packages available in my
> repositories)
>
> If you append a1=1 in your representation, the Xx -> a1 constraints are
> all satisfied, so are useless to discriminate a possible solution.
>
> You find the right solution because of the equalities that force one bi
> and one cj to be satisfied.
>
> Just compare the two approaches on the same example with
>
> additional d1, d2, d3 and e1, e2, e3, e4 in your repository.
>
> I suspect that you will anyway throw them away because they will not be
> part of the dirty tree.
>
> So I have a simpler representation for you:
>
> Representation #3: Mercury new proposal
>
> b1 + b2 + b3 = 1
> c1 + c2 = 1
>
> Together with the objective function, it will find the expected answer :)
>
It would in this particular case. But any of the dependencies could be
declared as optional, then the fun begins ! So Xx -> a1 are essential if
Xx is declared optional. Plus - and it is critical - in the dirty tree a
node is a variable, so optional are not only explicitly declared as
such, but any node can potentially be eliminated.
At first I tried #3 approach and almost got it "working", but luckily,
the first real world try killed it.
Thanks for raising "depends" vs. "provides" semantics question, I have
not seen it that way, will have to add it to my mental model :)
Thanks,
Oleg
> Daniel
>
Re: How Mercury resolution works?
Posted by Daniel Le Berre <le...@cril.univ-artois.fr>.
Oleg Gusakov a écrit :
> The tree could be walked in two directions, and these approaches are
> equivalent, except for b and c optionality:
>
> Representation #1: P2
>
> a1 -> b1 or b2 or b3
> a1 -> c1 or c2
>
> b1 + b2 + b3 <= 1 # this one implicates that b is optional
> c1 + c2 <= 1 # this one implicates that c is optional
>
>
> Representation #2: Mercury
>
> b1 -> a1
> b2 -> a1
> b3 -> a1
> c1 -> a1
> c2 -> a1
> b1 + b2 + b3 = 1
> c1 + c2 + c3 = 1
>
> In the second one it's easier for me to keep the tabs on the generated
> system. Timewise both are solved in 10s of millis.
Oleg,
I agree that you can encode either "provides" or "depends" semantic
using constraints.
However, you cannot ask the same question:
Representation "depends":
Append the package you want to install (a1=1).
Ask for a solution consistent with the constraints (or the best one).
(What do I need to install to be able to install a1).
Representation "provides":
Append the packages you have.
Ask for all the packages that can be safely installed according to the
constraints.
(what can I install if I have all those packages available in my
repositories)
If you append a1=1 in your representation, the Xx -> a1 constraints are
all satisfied, so are useless to discriminate a possible solution.
You find the right solution because of the equalities that force one bi
and one cj to be satisfied.
Just compare the two approaches on the same example with
additional d1, d2, d3 and e1, e2, e3, e4 in your repository.
I suspect that you will anyway throw them away because they will not be
part of the dirty tree.
So I have a simpler representation for you:
Representation #3: Mercury new proposal
b1 + b2 + b3 = 1
c1 + c2 = 1
Together with the objective function, it will find the expected answer :)
Daniel
--
Daniel Le Berre mailto:leberre@cril.univ-artois.fr
MCF, CRIL-CNRS UMR 8188, Universite d'Artois
http://www.cril.univ-artois.fr/~leberre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
Daniel Le Berre wrote:
> Gilles Scokart a �crit :
>
>> An Ivy like function might be nice as well.;-) But there is a piece
>> that I miss. How did you handle conlicts? I means not multiple
>> possible solution, but real conflicts. This seems to be incompatible
>> with a strtict system of constraints.
>>
>> If I take the example :
>> A depends on B:1 and C:1
>> B:1 depends on C:2.
>>
>> I think maven will take C:1 (first levle in the dependency tree), Ivy
>> would by default take C:2 (because it resolve conflicts by taking the
>> 'latest' version). What will mercury do? My understanding is that
>> the SAT resolution will say there is no solution. What practical
>> solution do you propose for Mercury users? Do you ask them to fill a
>> DependencyManagment section for the the module that are in conflicts ?
>>
>
> >From a pure logical point of view, there is no solution to that system
> if you add the constraints that you would like to install A and that C:1
> and C:2 cannot be installed together.
>
Daniel is absolutely right, if the dirty tree looks like that, there is
no solution, Because it represents the dependencies:
a:1 dep.on b:[1,1]
a:1 dep.on c:[1,1]
b:1 dep.on c:[2,2]
No solution, because both b1 and c1 are required for a1, c2 required for
b1 and c1 contradicts c2. This picture would change if some dependencies
are declared "optional", or there are "include/exclude" clauses in any
dependency
But this illustrates an interesting rule we are dealing with:
b:1 as a dependency, could in interpreted in 3 different ways:
1). b:1 = b:[1,1]
2). b:1 = { b:x | x >= 1 } - OSGi standard definition
3). b:1 = { b:x | x exists in the dirty tree } - a.k.a soft range, so
it could be 0.9.7 or 1.3, depending on the tree and "optimization
policy" - see below
Mercury can do all 3, default is #3, #2 could be enabled with an option.
#1 could easily be added, but is too restrictive and will break a lot of
builds right away, so it's an exercise in completeness, not real life.
> Now, since there is an optimization function, it is possible to encode
> conflicts in the constraints:
>
> A depends on (B:1 or ConflictB) and (C:1 or ConflictC)
> B:1 depends on (C:2 or ConflictC)
>
> And minimize the sum of conflicts in the optimization function.
>
> With the optimization function provided earlier by Oleg, the selection
> between C:1 and C:2 would be C:2, because it prefers latest versions,
> like Ivy.
>
As Mercury is a library, the preference ("optimization policy") is
passed in as a list of comparators, and those are used by [mercury]
solver for multi-level sorting of GAVs in order to create the
optimization function. If no comparators are supplied, Mercury uses a
list of two default comparators: "shallowest", then "newest"
Thanks,
Oleg
> Daniel
>
Re: How Mercury resolution works?
Posted by Daniel Le Berre <le...@cril.univ-artois.fr>.
Gilles Scokart a écrit :
> An Ivy like function might be nice as well.;-) But there is a piece
> that I miss. How did you handle conlicts? I means not multiple
> possible solution, but real conflicts. This seems to be incompatible
> with a strtict system of constraints.
>
> If I take the example :
> A depends on B:1 and C:1
> B:1 depends on C:2.
>
> I think maven will take C:1 (first levle in the dependency tree), Ivy
> would by default take C:2 (because it resolve conflicts by taking the
> 'latest' version). What will mercury do? My understanding is that
> the SAT resolution will say there is no solution. What practical
> solution do you propose for Mercury users? Do you ask them to fill a
> DependencyManagment section for the the module that are in conflicts ?
>From a pure logical point of view, there is no solution to that system
if you add the constraints that you would like to install A and that C:1
and C:2 cannot be installed together.
Now, since there is an optimization function, it is possible to encode
conflicts in the constraints:
A depends on (B:1 or ConflictB) and (C:1 or ConflictC)
B:1 depends on (C:2 or ConflictC)
And minimize the sum of conflicts in the optimization function.
With the optimization function provided earlier by Oleg, the selection
between C:1 and C:2 would be C:2, because it prefers latest versions,
like Ivy.
Daniel
--
Daniel Le Berre mailto:leberre@cril.univ-artois.fr
MCF, CRIL-CNRS UMR 8188, Universite d'Artois
http://www.cril.univ-artois.fr/~leberre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
This is an awesome question!
I have tons of ideas in the area, but decided to first make Mercury
compliant with Maven2 scoping and make it works in Maven3, then we can
move forward - see below.
Gilles Scokart wrote:
> Cool,
>
> One more question : how do you handle scopes?
> In ivy, we have something called 'configuration' mapping in which you
> can configure that when you ask for runtime dependency, you have to
> take the compile+runtime transitive dependencies. In maven, this
> mapping is hard-coded.
>
I find an idea of "configuration" very cool, so I tried to make Mercury
as indifferent to the scope impl. as possible and plan to implement
similar concept as soon as Maven3 is stabilized. Developers should be
free to define their scopes and scoping rules, and it's half-way there.
> Also, in maven when you are resolving dependencies, you resolve all
> scopes at once. In ivy, we download each meta-file file once, but we
> make one resolution per configuration (equivalent to scope).
> How is it handled in Mercury?
>
Same idea: the call is made to resolve dependencies in a particular
scope, so what I call "dirty tree" is in essence "scoped dirty tree". As
a result of that - client wind up with a solution per each scope it asks
for.
Each metadata is downloaded only once and cached for subsequent
accesses. Maven is different as it has a concept of a SNAPSHOT, LATEST
and RELEASE versions, these concepts clash with the idea of a cache and
result is so called "repository update policy" which in it's default
implementation, defines cached metadata TTL per remote repository.
Thanks,
Oleg
> Gilles Scokart
>
>
>
> 2008/12/22 Oleg Gusakov <ol...@gmail.com>:
>
>> Gilles Scokart wrote:
>>
>>> I have no doubt that SAT can resolve huge system of constraints, but I
>>> fear that building the dirty tree migh be very heavy. You may have to
>>> download and parse plenty of pom (or other meta-data files).
>>> BTW, when you download a pom, did you also download the jar, or did
>>> you only download the jar whan you know you select the right version.
>>>
>>>
>>>
>>>
>> Mercury dependency resolver operates purely on the "light" metadata. Light
>> in a sense that is does not require full blown POM to do it's job. It only
>> needs:
>>
>> 1). complete content of the <dependency/> element: GAV, classifier, type,
>> scope, optionality, inclusions/exclusions.
>> 2). for each GAV it needs a list of dependencies in the format #1
>>
>> This has been externalized into an interface, and the only implementation
>> (for now :) is Maven3 project builder that reads and interprets POMs,
>> returning back the processed dependencies.
>>
>> This - btw - also opens a road towards separating dependency declarations
>> out of POM to any other storage mechanism - I remember there was a
>> discussion about it. So - this possibility exists.
>>
>> Back to the question: Mercury calculates all the metadata (and also fills in
>> the metadata cache, so that on next invocation all metadata is readily
>> available). And returns back the resolved classpath as a list of metadata
>> objects. There is another call, that takes that list and actually reads the
>> binaries from the repositories.
>>
>> mercury-plexus component combines the two calls into one:
>> resolveConflicts(). It first resolves the conflicts on the metadata level,
>> them reads actual binaries, so that returned metadata always points to local
>> file system for binary.
>>
>>> An Ivy like function might be nice as well.;-)
>>>
>> Several aspects to this remark :)
>>
>> * default Mercury behaves like that - see "optimization policy" in my
>> previous post.
>>
>> * it is possible to implement a different dependency reader, so that we
>> change completely how dependencies are stored
>>
>> * it is possible to access any other repository (GEM, CPAN), as soon as
>> there is a Mercury-compliant implementation. Mercury changed the way
>> repository behaves in order to allow this - it accepts GAV-only calls, makes
>> all the mappings inside, and returns metadata objects. Also see
>> http://docs.codehaus.org/display/MAVEN/Mercury
>>
>> Thanks,
>> Oleg
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
>
Re: How Mercury resolution works?
Posted by Gilles Scokart <gs...@gmail.com>.
Cool,
One more question : how do you handle scopes?
In ivy, we have something called 'configuration' mapping in which you
can configure that when you ask for runtime dependency, you have to
take the compile+runtime transitive dependencies. In maven, this
mapping is hard-coded.
Also, in maven when you are resolving dependencies, you resolve all
scopes at once. In ivy, we download each meta-file file once, but we
make one resolution per configuration (equivalent to scope).
How is it handled in Mercury?
Gilles Scokart
2008/12/22 Oleg Gusakov <ol...@gmail.com>:
>
>
> Gilles Scokart wrote:
>>
>>
>> I have no doubt that SAT can resolve huge system of constraints, but I
>> fear that building the dirty tree migh be very heavy. You may have to
>> download and parse plenty of pom (or other meta-data files).
>> BTW, when you download a pom, did you also download the jar, or did
>> you only download the jar whan you know you select the right version.
>>
>>
>>
>
> Mercury dependency resolver operates purely on the "light" metadata. Light
> in a sense that is does not require full blown POM to do it's job. It only
> needs:
>
> 1). complete content of the <dependency/> element: GAV, classifier, type,
> scope, optionality, inclusions/exclusions.
> 2). for each GAV it needs a list of dependencies in the format #1
>
> This has been externalized into an interface, and the only implementation
> (for now :) is Maven3 project builder that reads and interprets POMs,
> returning back the processed dependencies.
>
> This - btw - also opens a road towards separating dependency declarations
> out of POM to any other storage mechanism - I remember there was a
> discussion about it. So - this possibility exists.
>
> Back to the question: Mercury calculates all the metadata (and also fills in
> the metadata cache, so that on next invocation all metadata is readily
> available). And returns back the resolved classpath as a list of metadata
> objects. There is another call, that takes that list and actually reads the
> binaries from the repositories.
>
> mercury-plexus component combines the two calls into one:
> resolveConflicts(). It first resolves the conflicts on the metadata level,
> them reads actual binaries, so that returned metadata always points to local
> file system for binary.
>>
>> An Ivy like function might be nice as well.;-)
>
> Several aspects to this remark :)
>
> * default Mercury behaves like that - see "optimization policy" in my
> previous post.
>
> * it is possible to implement a different dependency reader, so that we
> change completely how dependencies are stored
>
> * it is possible to access any other repository (GEM, CPAN), as soon as
> there is a Mercury-compliant implementation. Mercury changed the way
> repository behaves in order to allow this - it accepts GAV-only calls, makes
> all the mappings inside, and returns metadata objects. Also see
> http://docs.codehaus.org/display/MAVEN/Mercury
>
> Thanks,
> Oleg
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
Gilles Scokart wrote:
>
>
> I have no doubt that SAT can resolve huge system of constraints, but I
> fear that building the dirty tree migh be very heavy. You may have to
> download and parse plenty of pom (or other meta-data files).
> BTW, when you download a pom, did you also download the jar, or did
> you only download the jar whan you know you select the right version.
>
>
>
Mercury dependency resolver operates purely on the "light" metadata.
Light in a sense that is does not require full blown POM to do it's job.
It only needs:
1). complete content of the <dependency/> element: GAV, classifier,
type, scope, optionality, inclusions/exclusions.
2). for each GAV it needs a list of dependencies in the format #1
This has been externalized into an interface, and the only
implementation (for now :) is Maven3 project builder that reads and
interprets POMs, returning back the processed dependencies.
This - btw - also opens a road towards separating dependency
declarations out of POM to any other storage mechanism - I remember
there was a discussion about it. So - this possibility exists.
Back to the question: Mercury calculates all the metadata (and also
fills in the metadata cache, so that on next invocation all metadata is
readily available). And returns back the resolved classpath as a list of
metadata objects. There is another call, that takes that list and
actually reads the binaries from the repositories.
mercury-plexus component combines the two calls into one:
resolveConflicts(). It first resolves the conflicts on the metadata
level, them reads actual binaries, so that returned metadata always
points to local file system for binary.
>
> An Ivy like function might be nice as well.;-)
Several aspects to this remark :)
* default Mercury behaves like that - see "optimization policy" in my
previous post.
* it is possible to implement a different dependency reader, so that we
change completely how dependencies are stored
* it is possible to access any other repository (GEM, CPAN), as soon as
there is a Mercury-compliant implementation. Mercury changed the way
repository behaves in order to allow this - it accepts GAV-only calls,
makes all the mappings inside, and returns metadata objects. Also see
http://docs.codehaus.org/display/MAVEN/Mercury
Thanks,
Oleg
Re: How Mercury resolution works?
Posted by Gilles Scokart <gs...@gmail.com>.
2008/12/20 Oleg Gusakov <ol...@gmail.com>:
>>>
>>> Also, how do you solve the problem that different version might have
>>> dependency different tree? Did you build the complete "dirty tree"
>>> (meanings that you look up for transitive dependencies of all old
>>> versions that will probably not be selected, an then resolve big the
>>> equations globaly), or did you proceeds somehow iteratively (resolving
>>> some equations to know which part of the tree to "dirty tree" to
>>> extends, update the dirty tree with dependencies of only resolved
>>> revisions, update the equations, solve equations again, ...).
>>>
>
> I create a full "dirty tree" for every version, then use them all in the
> generated system of inequavions. I tried to do some iterative optimizations
> in the process, but SAT does better job and it is also defined to work on
> the huge problems - ours is a toy for it.
>
I have no doubt that SAT can resolve huge system of constraints, but I
fear that building the dirty tree migh be very heavy. You may have to
download and parse plenty of pom (or other meta-data files).
BTW, when you download a pom, did you also download the jar, or did
you only download the jar whan you know you select the right version.
> Manual intervention also proved turned sour because Mercury is flexible to
> allow different selection policies: default is "shallowest", then "newest",
> but it's all configurable with a set of comparators, passed to the solver.
> And if the need comes - we can cover a lot of exotic variations, the first
> one coming to mind - "maven2 compatibility mode" comparator. Maven2 selects
> the first version on the same level, Mercury - the newest.
An Ivy like function might be nice as well.;-) But there is a piece
that I miss. How did you handle conlicts? I means not multiple
possible solution, but real conflicts. This seems to be incompatible
with a strtict system of constraints.
If I take the example :
A depends on B:1 and C:1
B:1 depends on C:2.
I think maven will take C:1 (first levle in the dependency tree), Ivy
would by default take C:2 (because it resolve conflicts by taking the
'latest' version). What will mercury do? My understanding is that
the SAT resolution will say there is no solution. What practical
solution do you propose for Mercury users? Do you ask them to fill a
DependencyManagment section for the the module that are in conflicts ?
Gilles
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Oleg Gusakov <ol...@gmail.com>.
I cannot add much - The Guru has spoken :)
I'll just try to answer the question.
Daniel Le Berre wrote:
> Hi Gilles,
>
> Gilles Scokart a �crit :
>
>> Thanks to remind us the mathematic curses :-)
>>
>> But I have some questions :
>>
>> What does the '>=' constraints represent ?
>>
>> -b1 + a1 >=0
>> -b2 + a1 >=0
>> -b3 + a1 >=0
>> -c1 + a1 >=0
>> -b2 + a1 >=0
>>
>
> the variables are boolean (can only take 0/1 values).
>
> Those constraints are thus called pseudo boolean equations.
> In this case, they simply represent clauses:
> b1 -> a1
> b2 -> a1
>
> etc.
>
>
>
>> If I take -b1 + a1 >=0, I interpret it as "if I take b1, I must take
>> a1". I fail to see why you are adding this contraints to your
>> equation?
>>
>
> I agree.
>
>> I would have expected a system of contraints :
>> a1 = b1 + b2 + b3 (If I take a1, I also take one and exactly one of b1,b2,b3)
>> a1 = c1 + c2 (idem for c)
>> a1 = 1 (it is the root. I take it)
>>
>>
>
> Usually, you need implications, no equivalences:
>
> Here is how it is encoded for Eclipse p2
> (almost, some details are missing)
>
> // dependencies
> a1 -> b1 or b2 or b3
> a1 -> c1 or c2
>
> // just require one version
> b1 + b2 + b3 <= 1
> c1 + c2 <= 1
>
The tree could be walked in two directions, and these approaches are
equivalent, except for b and c optionality:
Representation #1: P2
a1 -> b1 or b2 or b3
a1 -> c1 or c2
b1 + b2 + b3 <= 1 # this one implicates that b is optional
c1 + c2 <= 1 # this one implicates that c is optional
Representation #2: Mercury
b1 -> a1
b2 -> a1
b3 -> a1
c1 -> a1
c2 -> a1
b1 + b2 + b3 = 1
c1 + c2 + c3 = 1
In the second one it's easier for me to keep the tabs on the generated
system. Timewise both are solved in 10s of millis.
>
> // what do I want to install?
> a1 = 1
>
> It is important to use inequalities instead of an equalities for the
> versions
> else any solution would force one version of each package to be installed,
> independently of the packages the user would like to install.
>
>
>> I'm also wondering how to interpret your optimization function. I see
>> that you favor higher versions, but why not Min (4*b1+2*b2+b3 +
>> 2*c2+c1) for example?
>> I fail to understand why it would be much more important to select the
>> latest version of b, than the latest version of c. Is there any
>> reason or is my interpretation of the function incorrect ?
>>
>
> Because then, you are not deterministic: the solver would consider of
> equal importance to install b3 or c2 so if you cannot install both of
> them, on some computers the installation would be b3 c1 and on others c2
> b2. I guess you would like to avoid this :)
>
> The big trouble is to find a metric that allow to sort the packages
> in a meaningful way.
>
This is troublesome, so we had to make special arrangements to make
selection consistent between say, Java5 and Java6. See
http://jira.codehaus.org/browse/MERCURY-40 The pending solutions might
be #1 "the first seen GA", or #2 "sort by GA" - for now it's #1, but -
as indicated in MERCURY-40, it depends on the tree builder doing that.
> Daniel
>
>> Did you have any link where I could find more explanation on the
>> rational of the equations?
>>
Check Wikipedia for "Satisfiability problem", especially - pseudo
boolean approach. Took me several weeks to fully understand the beauty
of this approach, before that I tried to solve the problem myself (your
next question below).
>> Also, how do you solve the problem that different version might have
>> dependency different tree? Did you build the complete "dirty tree"
>> (meanings that you look up for transitive dependencies of all old
>> versions that will probably not be selected, an then resolve big the
>> equations globaly), or did you proceeds somehow iteratively (resolving
>> some equations to know which part of the tree to "dirty tree" to
>> extends, update the dirty tree with dependencies of only resolved
>> revisions, update the equations, solve equations again, ...).
>>
I create a full "dirty tree" for every version, then use them all in the
generated system of inequavions. I tried to do some iterative
optimizations in the process, but SAT does better job and it is also
defined to work on the huge problems - ours is a toy for it.
Manual intervention also proved turned sour because Mercury is flexible
to allow different selection policies: default is "shallowest", then
"newest", but it's all configurable with a set of comparators, passed to
the solver. And if the need comes - we can cover a lot of exotic
variations, the first one coming to mind - "maven2 compatibility mode"
comparator. Maven2 selects the first version on the same level, Mercury
- the newest.
Thanks Daniel for all his help!
Oleg
>
>> Gilles Scokart
>>
>>
>>
>> 2008/12/19 Oleg Gusakov <ol...@gmail.com>:
>>
>>> a diagram of mercury inner workflow is here:
>>> http://blogs.sonatype.com/people/?p=1016
>>>
>>> fyi - I am also working on re-introducing the "old" maven-ant syntax to
>>> mercury-ant-tasks. Herve - as you wished!
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>>
>>
>
>
>
Re: How Mercury resolution works?
Posted by Daniel Le Berre <le...@cril.univ-artois.fr>.
Hi Gilles,
Gilles Scokart a écrit :
> Thanks to remind us the mathematic curses :-)
>
> But I have some questions :
>
> What does the '>=' constraints represent ?
>
> -b1 + a1 >=0
> -b2 + a1 >=0
> -b3 + a1 >=0
> -c1 + a1 >=0
> -b2 + a1 >=0
the variables are boolean (can only take 0/1 values).
Those constraints are thus called pseudo boolean equations.
In this case, they simply represent clauses:
b1 -> a1
b2 -> a1
etc.
> If I take -b1 + a1 >=0, I interpret it as "if I take b1, I must take
> a1". I fail to see why you are adding this contraints to your
> equation?
I agree.
> I would have expected a system of contraints :
> a1 = b1 + b2 + b3 (If I take a1, I also take one and exactly one of b1,b2,b3)
> a1 = c1 + c2 (idem for c)
> a1 = 1 (it is the root. I take it)
>
Usually, you need implications, no equivalences:
Here is how it is encoded for Eclipse p2
(almost, some details are missing)
// dependencies
a1 -> b1 or b2 or b3
a1 -> c1 or c2
// just require one version
b1 + b2 + b3 <= 1
c1 + c2 <= 1
// what do I want to install?
a1 = 1
It is important to use inequalities instead of an equalities for the
versions
else any solution would force one version of each package to be installed,
independently of the packages the user would like to install.
> I'm also wondering how to interpret your optimization function. I see
> that you favor higher versions, but why not Min (4*b1+2*b2+b3 +
> 2*c2+c1) for example?
> I fail to understand why it would be much more important to select the
> latest version of b, than the latest version of c. Is there any
> reason or is my interpretation of the function incorrect ?
Because then, you are not deterministic: the solver would consider of
equal importance to install b3 or c2 so if you cannot install both of
them, on some computers the installation would be b3 c1 and on others c2
b2. I guess you would like to avoid this :)
The big trouble is to find a metric that allow to sort the packages
in a meaningful way.
Daniel
> Did you have any link where I could find more explanation on the
> rational of the equations?
>
> Also, how do you solve the problem that different version might have
> dependency different tree? Did you build the complete "dirty tree"
> (meanings that you look up for transitive dependencies of all old
> versions that will probably not be selected, an then resolve big the
> equations globaly), or did you proceeds somehow iteratively (resolving
> some equations to know which part of the tree to "dirty tree" to
> extends, update the dirty tree with dependencies of only resolved
> revisions, update the equations, solve equations again, ...).
>
>
> Gilles Scokart
>
>
>
> 2008/12/19 Oleg Gusakov <ol...@gmail.com>:
>> a diagram of mercury inner workflow is here:
>> http://blogs.sonatype.com/people/?p=1016
>>
>> fyi - I am also working on re-introducing the "old" maven-ant syntax to
>> mercury-ant-tasks. Herve - as you wished!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
>
--
Daniel Le Berre mailto:leberre@cril.univ-artois.fr
MCF, CRIL-CNRS UMR 8188, Universite d'Artois
http://www.cril.univ-artois.fr/~leberre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: How Mercury resolution works?
Posted by Gilles Scokart <gs...@gmail.com>.
Thanks to remind us the mathematic curses :-)
But I have some questions :
What does the '>=' constraints represent ?
-b1 + a1 >=0
-b2 + a1 >=0
-b3 + a1 >=0
-c1 + a1 >=0
-b2 + a1 >=0
If I take -b1 + a1 >=0, I interpret it as "if I take b1, I must take
a1". I fail to see why you are adding this contraints to your
equation?
I would have expected a system of contraints :
a1 = b1 + b2 + b3 (If I take a1, I also take one and exactly one of b1,b2,b3)
a1 = c1 + c2 (idem for c)
a1 = 1 (it is the root. I take it)
I'm also wondering how to interpret your optimization function. I see
that you favor higher versions, but why not Min (4*b1+2*b2+b3 +
2*c2+c1) for example?
I fail to understand why it would be much more important to select the
latest version of b, than the latest version of c. Is there any
reason or is my interpretation of the function incorrect ?
Did you have any link where I could find more explanation on the
rational of the equations?
Also, how do you solve the problem that different version might have
dependency different tree? Did you build the complete "dirty tree"
(meanings that you look up for transitive dependencies of all old
versions that will probably not be selected, an then resolve big the
equations globaly), or did you proceeds somehow iteratively (resolving
some equations to know which part of the tree to "dirty tree" to
extends, update the dirty tree with dependencies of only resolved
revisions, update the equations, solve equations again, ...).
Gilles Scokart
2008/12/19 Oleg Gusakov <ol...@gmail.com>:
> a diagram of mercury inner workflow is here:
> http://blogs.sonatype.com/people/?p=1016
>
> fyi - I am also working on re-introducing the "old" maven-ant syntax to
> mercury-ant-tasks. Herve - as you wished!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org