You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Jeffrey Sinclair <je...@cooljeff.co.uk> on 2009/08/02 00:14:36 UTC
Solution to native dependencies with Ivy
ivy-user,
The handling of artifacts to populate the java.libary.path has cropped
up a couple of times on the mailgroup and there is an outstanding JIRA
relating to this [IVY-600].
I'm finding that more people are being hit by this, in particular in
Eclipse through IvyDE. I've come up with a solution that I feel will
work well for both Ivy and IvyDE:
http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
As described in the blog entry, I'd like to propose a mixture of using
types as well as a new attribute on the artifact named platform.
I'd like to put a proof of concept together and hence would be grateful
for any feedback on my proposal and evaluation of existing solutions.
Regards,
Jeff
Re: Solution to native dependencies with Ivy
Posted by Niklas Matthies <ml...@nmhq.net>.
Just as an aside: One general problem with native dependencies is that
java.library.path only applies to direct dependencies from Java to
native libraries, but not to transitive dependencies, that is from one
ntaive lirary to another. If a native library is loaded by Java, and
this library has a dependency to another native library, this second
library is not loaded by Java, but natively by the OS, and hence
ignores any java.library.path settings. This is a typical situation
when the first library is a JNI bridge library to an existing native
library. In that case OS specific mechanisms have to be used so that
the dependent native libraries can be correctly located.
-- Niklas Matthies
Re: Solution to native dependencies with Ivy
Posted by Garima Bathla <ga...@gmail.com>.
For the record, the choice of platform/OS is *not *the domain I was trying
to address with the “feature request” thread I started. You might notice
that the example I used in requesting configuration intersections at
the *artifact
*level was copied verbatim—*right down to the artifact names*—from Shawn’s
example in requesting configuration intersections at the *dependency *level.
It just seemed like the coffee drinks domain I started out with was getting
in the way of people looking at the problem, so I wanted to come up with a
domain that would gain a little more acceptance.
I like Mitch’s outline of a proposal, either the “filters” or the “building
blocks,” provided it addresses the fact that my configuration intersections
problem turned out to be more complex than I had originally imagined. Again,
I hate to go back to the coffee drinks domain that people hated, but I need
to be able to specify a mocha-tall combination where I get:
- mocha.jar
- mocha-tall.jar
But I DON’T get:
- tall.jar
To do this strictly by defining special combo Ivy confs would get out of
hand in no time—because, as stated, this example is an oversimplification.
Regards,
Garima.
On Sun, Aug 2, 2009 at 10:32 AM, Mitch Gitman <mg...@gmail.com> wrote:
> Jeffrey:
> I love the way you’ve presented this idea, but this is one idea I would be
> opposed to. The problem is that it confuses Ivy module descriptors with one
> particular application of Ivy module descriptors.
>
> Right now, the ivy.xml XML schema provides a powerful, expressive
> general-purpose language for describing transitive dependencies and
> artifact
> retrieval. You can use it for pulling JARs into a classpath or for
> distributing operating system-specific files or delivering apples and
> oranges for all it cares. It is, and should be, agnostic of its various use
> cases, many of which we probably haven’t even realized yet. The moment we
> promote an application of the language into becoming a first-class part of
> the language itself, we corrupt the language. And worse, we open a
> Pandora’s
> box for inviting more applications of the language into the language,
> further corrupting it and blurring the line between where the language ends
> and its use cases begin.
>
> It seems like with platform or operating system we’re trying to create
> something that is not an Ivy conf itself but is an extra dimension or
> parameter or filter that gets applied to Ivy confs. If we make platform
> part
> of Ivy, what happens when we encounter the next particular
> dimension/parameter/filter that we want to apply to Ivy confs? Do we
> promote
> that into the module descriptor language as well? Every time another
> dimension/parameter/filter arises, we have to rekindle the heated debate
> about which one is worthy of the language and which isn’t. I can see myself
> crying foul, “You promoted platform. Why can’t you promote this?”
>
> And even then, we still don’t have support for arbitrary extra
> dimensions/parameters/filters.
>
> Instead, I suggest we take a step back and try to come up with a
> general-purpose solution to this problem, one that will solve all the use
> cases that come up, not just the choice of operating system. This is where
> I
> go back to an idea Garima broached in the post requesting configuration
> intersections at the artifact level: “And I think the ideal solution would
> be to acknowledge that Ivy confs, as originally conceived, are not
> composite
> constructs; so we need to introduce composite elements that can be used to
> construct confs.”
>
> The way that Ivy confs are now designed, any single Ivy conf can and must
> be
> able to stand on its own. What we need are constructs that don’t stand on
> their own. I can think of two potential high-level ways to accomplish this,
> either of which could address both the operating system problem and the one
> that Garima offered:
>
> - Something like filters that are not themselves Ivy confs but which can
> be applied to Ivy confs. I’m familiar with configuration groupings, and
> they’re not quite the same thing.
> - Building blocks for Ivy confs. So an actual Ivy conf you might specify
> on an artifact is not literally defined as a conf. It’s a composite of
> the
> different building blocks. These building-block elements would coexist
> with
> the first-class Ivy confs. Really, this isn’t such a radical idea. As an
> analogy, consider an email address: mgitman and gmail.com are the
> particular elements that are put together to form mgitman@gmail.com, and
> even the gmail.com Internet domain is formed from gmail and the .com
> top-level domain.
>
> In fact, even the operating system/platform problem seems a lot more
> multi-dimensional than just a platform attribute allows. I mean, ia32.linux
> implies a combination of ia32 and linux.
>
> P.S. I’m writing this without having had a chance to digest Shawn’s
> response.
>
> On Sat, Aug 1, 2009 at 3:14 PM, Jeffrey Sinclair <jeff@cooljeff.co.uk
> >wrote:
>
> > ivy-user,
> >
> > The handling of artifacts to populate the java.libary.path has cropped
> > up a couple of times on the mailgroup and there is an outstanding JIRA
> > relating to this [IVY-600].
> >
> > I'm finding that more people are being hit by this, in particular in
> > Eclipse through IvyDE. I've come up with a solution that I feel will
> > work well for both Ivy and IvyDE:
> >
> >
> >
> http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
> >
> > As described in the blog entry, I'd like to propose a mixture of using
> > types as well as a new attribute on the artifact named platform.
> >
> > I'd like to put a proof of concept together and hence would be grateful
> > for any feedback on my proposal and evaluation of existing solutions.
> >
> > Regards,
> >
> > Jeff
> >
> >
>
RE: Solution to native dependencies with Ivy
Posted by Shawn Castrianni <Sh...@halliburton.com>.
I do agree that the majority of the documentation, if not all, is missing much needed platform specific considerations. A lot of people think that ANT is a java only build tool and they could also think that about IVY. When I first had to tackle this problem of handling platform specific issues, it would have been nice to have more information like best practices in the documentation.
With the feature set that is currently available in IVY, configurations is obviously the way to go. But is it worth it to add direct platform support to IVY instead of using configurations? I am more of a practical person than a purist, so I guess I would say yes. In the past, I even asked for symlink support to be added to IVY but was turned down. This led to my trigger approach, which works fine, but everybody else that needs to solve the same problem would have to duplicate my effort for themselves.
Configurations could be considered a general solution to the platform specific problem. However, this general solution wasn't free. I had to do a lot of coding in ANT to support it and to get the features I wanted. Since this platform specific problem should be a common one that many people would need to solve, there should be a more direct way of solving with less effort on the build engineer. They all shouldn't have to come up with a similar configuration, trigger, and extra attribute solution that I did. Having built-in platform support would save everybody a lot of time and guarantee that everyone is solving it the best way.
---
Shawn Castrianni
-----Original Message-----
From: Jeffrey Sinclair [mailto:jeff@cooljeff.co.uk]
Sent: Monday, August 03, 2009 12:48 AM
To: ivy-user@ant.apache.org
Subject: Re: Solution to native dependencies with Ivy
Mitch, I agree in its current incarnation the ivy schema doesn't have
any special knowledge of JARs. However we also have to admit that most
of the examples given in the documentation use artifacts that are
platform independent.
Shawn, you've got this to work with configurations. Would you have
prefered there to have been a direct way to associate platform specific
information at the artifact level or do you believe configurations are
the way to express this (perhaps with the need for a better way of
performing a projection over the artifacts pulled in).
I'm not saying the platform attribute is the way to go at this stage
(I'll probably update my blog to take into account the helpful feedback
given). My aim was to spark a little debate to see what other ways the
issue could be dealt with :)
Jeff
On Sun, 2009-08-02 at 20:11 -0700, Mitch Gitman wrote:
> I'm not really familiar with this problem space, but adding a peer to source
> and javadoc for IvyDE integration--that sounds sensible to me.
>
> Regarding the additional description that would need to be present in an Ivy
> module descriptor to fully support a native library, again I've got to
> believe that there's a generic way to accomplish that without resorting to
> making the ivy.xml have special knowledge of native libraries. I mean, isn't
> it fair to say that the ivy.xml XML schema in its current incarnation
> doesn't have special knowledge of JARs? That is, JARs are just another
> arbitrary type.
>
> On Sun, Aug 2, 2009 at 3:20 PM, Jeffrey Sinclair <je...@cooljeff.co.uk>wrote:
>
> > Thanks Shawn and Mitch for your responses.
> >
> > It seems as if we are in agreement that the type attribute should
> > identify that the artifact is a native library. In Shawn's case he has
> > opted for type="bin". What there is debate over at the moment is how to
> > declare artifacts being available for different platforms and how to go
> > about resolving them.
> >
> > On the declaration side, I don't agree with Mitch that platform
> > information is an application of the language. The fact is that a native
> > artifact has some additional information that needs to be described in
> > the module descriptor. An end user should be able to look at an ivy file
> > and see what platforms an artifact is available for. This information
> > must also be available in order to segregate artifacts with the same
> > name and extension for the situation where you are building an
> > application on several platforms using a shared cache location (e.g.
> > afs).
> >
> > Shawn has opted to use configurations to identify which platforms an
> > artifact is available for whereas I've opted to use an additional
> > attribute. The main reason why I chose an extra attribute was because I
> > can easily use this as my type token in a file system resolver. I do
> > agree with Mitch that adding an extra attribute is probably not the
> > right way to go but also don't completely like using configurations in
> > the way we have them at the moment. Perhaps configurations are fine or
> > in the future we will have conf-like attributes or functions on confs
> > which will solve this general use-case.
> >
> > >From my perspective what I really want is a solution that will allow
> > IvyDE to integrate with the java.library.path in Eclipse since this
> > appears to be a major gap in the IDE integration. The ultimate solution
> > of how we structure/resolve native libraries is likely to be constrained
> > to the ivy-settings.xml file or in the end user's ivy.xml module
> > descriptor. Hence it seems to me that in order to provide IvyDE
> > integration, it would be sufficient enough to have a native library type
> > filter in the preferences (just like we already have for javadocs and
> > source). This would work regardless of the solution we come up with for
> > how we structure the Ivy file.
> >
> > Would everyone be happy having this added as a feature to IvyDE or see
> > any problems with this?
> >
> > Jeff
> >
> >
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Solution to native dependencies with Ivy
Posted by Jeffrey Sinclair <je...@cooljeff.co.uk>.
Mitch, I agree in its current incarnation the ivy schema doesn't have
any special knowledge of JARs. However we also have to admit that most
of the examples given in the documentation use artifacts that are
platform independent.
Shawn, you've got this to work with configurations. Would you have
prefered there to have been a direct way to associate platform specific
information at the artifact level or do you believe configurations are
the way to express this (perhaps with the need for a better way of
performing a projection over the artifacts pulled in).
I'm not saying the platform attribute is the way to go at this stage
(I'll probably update my blog to take into account the helpful feedback
given). My aim was to spark a little debate to see what other ways the
issue could be dealt with :)
Jeff
On Sun, 2009-08-02 at 20:11 -0700, Mitch Gitman wrote:
> I'm not really familiar with this problem space, but adding a peer to source
> and javadoc for IvyDE integration--that sounds sensible to me.
>
> Regarding the additional description that would need to be present in an Ivy
> module descriptor to fully support a native library, again I've got to
> believe that there's a generic way to accomplish that without resorting to
> making the ivy.xml have special knowledge of native libraries. I mean, isn't
> it fair to say that the ivy.xml XML schema in its current incarnation
> doesn't have special knowledge of JARs? That is, JARs are just another
> arbitrary type.
>
> On Sun, Aug 2, 2009 at 3:20 PM, Jeffrey Sinclair <je...@cooljeff.co.uk>wrote:
>
> > Thanks Shawn and Mitch for your responses.
> >
> > It seems as if we are in agreement that the type attribute should
> > identify that the artifact is a native library. In Shawn's case he has
> > opted for type="bin". What there is debate over at the moment is how to
> > declare artifacts being available for different platforms and how to go
> > about resolving them.
> >
> > On the declaration side, I don't agree with Mitch that platform
> > information is an application of the language. The fact is that a native
> > artifact has some additional information that needs to be described in
> > the module descriptor. An end user should be able to look at an ivy file
> > and see what platforms an artifact is available for. This information
> > must also be available in order to segregate artifacts with the same
> > name and extension for the situation where you are building an
> > application on several platforms using a shared cache location (e.g.
> > afs).
> >
> > Shawn has opted to use configurations to identify which platforms an
> > artifact is available for whereas I've opted to use an additional
> > attribute. The main reason why I chose an extra attribute was because I
> > can easily use this as my type token in a file system resolver. I do
> > agree with Mitch that adding an extra attribute is probably not the
> > right way to go but also don't completely like using configurations in
> > the way we have them at the moment. Perhaps configurations are fine or
> > in the future we will have conf-like attributes or functions on confs
> > which will solve this general use-case.
> >
> > >From my perspective what I really want is a solution that will allow
> > IvyDE to integrate with the java.library.path in Eclipse since this
> > appears to be a major gap in the IDE integration. The ultimate solution
> > of how we structure/resolve native libraries is likely to be constrained
> > to the ivy-settings.xml file or in the end user's ivy.xml module
> > descriptor. Hence it seems to me that in order to provide IvyDE
> > integration, it would be sufficient enough to have a native library type
> > filter in the preferences (just like we already have for javadocs and
> > source). This would work regardless of the solution we come up with for
> > how we structure the Ivy file.
> >
> > Would everyone be happy having this added as a feature to IvyDE or see
> > any problems with this?
> >
> > Jeff
> >
> >
Re: Solution to native dependencies with Ivy
Posted by Mitch Gitman <mg...@gmail.com>.
I'm not really familiar with this problem space, but adding a peer to source
and javadoc for IvyDE integration--that sounds sensible to me.
Regarding the additional description that would need to be present in an Ivy
module descriptor to fully support a native library, again I've got to
believe that there's a generic way to accomplish that without resorting to
making the ivy.xml have special knowledge of native libraries. I mean, isn't
it fair to say that the ivy.xml XML schema in its current incarnation
doesn't have special knowledge of JARs? That is, JARs are just another
arbitrary type.
On Sun, Aug 2, 2009 at 3:20 PM, Jeffrey Sinclair <je...@cooljeff.co.uk>wrote:
> Thanks Shawn and Mitch for your responses.
>
> It seems as if we are in agreement that the type attribute should
> identify that the artifact is a native library. In Shawn's case he has
> opted for type="bin". What there is debate over at the moment is how to
> declare artifacts being available for different platforms and how to go
> about resolving them.
>
> On the declaration side, I don't agree with Mitch that platform
> information is an application of the language. The fact is that a native
> artifact has some additional information that needs to be described in
> the module descriptor. An end user should be able to look at an ivy file
> and see what platforms an artifact is available for. This information
> must also be available in order to segregate artifacts with the same
> name and extension for the situation where you are building an
> application on several platforms using a shared cache location (e.g.
> afs).
>
> Shawn has opted to use configurations to identify which platforms an
> artifact is available for whereas I've opted to use an additional
> attribute. The main reason why I chose an extra attribute was because I
> can easily use this as my type token in a file system resolver. I do
> agree with Mitch that adding an extra attribute is probably not the
> right way to go but also don't completely like using configurations in
> the way we have them at the moment. Perhaps configurations are fine or
> in the future we will have conf-like attributes or functions on confs
> which will solve this general use-case.
>
> >From my perspective what I really want is a solution that will allow
> IvyDE to integrate with the java.library.path in Eclipse since this
> appears to be a major gap in the IDE integration. The ultimate solution
> of how we structure/resolve native libraries is likely to be constrained
> to the ivy-settings.xml file or in the end user's ivy.xml module
> descriptor. Hence it seems to me that in order to provide IvyDE
> integration, it would be sufficient enough to have a native library type
> filter in the preferences (just like we already have for javadocs and
> source). This would work regardless of the solution we come up with for
> how we structure the Ivy file.
>
> Would everyone be happy having this added as a feature to IvyDE or see
> any problems with this?
>
> Jeff
>
>
Re: Solution to native dependencies with Ivy
Posted by Jeffrey Sinclair <je...@cooljeff.co.uk>.
Thanks Shawn and Mitch for your responses.
It seems as if we are in agreement that the type attribute should
identify that the artifact is a native library. In Shawn's case he has
opted for type="bin". What there is debate over at the moment is how to
declare artifacts being available for different platforms and how to go
about resolving them.
On the declaration side, I don't agree with Mitch that platform
information is an application of the language. The fact is that a native
artifact has some additional information that needs to be described in
the module descriptor. An end user should be able to look at an ivy file
and see what platforms an artifact is available for. This information
must also be available in order to segregate artifacts with the same
name and extension for the situation where you are building an
application on several platforms using a shared cache location (e.g.
afs).
Shawn has opted to use configurations to identify which platforms an
artifact is available for whereas I've opted to use an additional
attribute. The main reason why I chose an extra attribute was because I
can easily use this as my type token in a file system resolver. I do
agree with Mitch that adding an extra attribute is probably not the
right way to go but also don't completely like using configurations in
the way we have them at the moment. Perhaps configurations are fine or
in the future we will have conf-like attributes or functions on confs
which will solve this general use-case.
>>From my perspective what I really want is a solution that will allow
IvyDE to integrate with the java.library.path in Eclipse since this
appears to be a major gap in the IDE integration. The ultimate solution
of how we structure/resolve native libraries is likely to be constrained
to the ivy-settings.xml file or in the end user's ivy.xml module
descriptor. Hence it seems to me that in order to provide IvyDE
integration, it would be sufficient enough to have a native library type
filter in the preferences (just like we already have for javadocs and
source). This would work regardless of the solution we come up with for
how we structure the Ivy file.
Would everyone be happy having this added as a feature to IvyDE or see
any problems with this?
Jeff
On Sun, 2009-08-02 at 10:32 -0700, Mitch Gitman wrote:
> Jeffrey:
> I love the way you’ve presented this idea, but this is one idea I would be
> opposed to. The problem is that it confuses Ivy module descriptors with one
> particular application of Ivy module descriptors.
>
> Right now, the ivy.xml XML schema provides a powerful, expressive
> general-purpose language for describing transitive dependencies and artifact
> retrieval. You can use it for pulling JARs into a classpath or for
> distributing operating system-specific files or delivering apples and
> oranges for all it cares. It is, and should be, agnostic of its various use
> cases, many of which we probably haven’t even realized yet. The moment we
> promote an application of the language into becoming a first-class part of
> the language itself, we corrupt the language. And worse, we open a Pandora’s
> box for inviting more applications of the language into the language,
> further corrupting it and blurring the line between where the language ends
> and its use cases begin.
>
> It seems like with platform or operating system we’re trying to create
> something that is not an Ivy conf itself but is an extra dimension or
> parameter or filter that gets applied to Ivy confs. If we make platform part
> of Ivy, what happens when we encounter the next particular
> dimension/parameter/filter that we want to apply to Ivy confs? Do we promote
> that into the module descriptor language as well? Every time another
> dimension/parameter/filter arises, we have to rekindle the heated debate
> about which one is worthy of the language and which isn’t. I can see myself
> crying foul, “You promoted platform. Why can’t you promote this?”
>
> And even then, we still don’t have support for arbitrary extra
> dimensions/parameters/filters.
>
> Instead, I suggest we take a step back and try to come up with a
> general-purpose solution to this problem, one that will solve all the use
> cases that come up, not just the choice of operating system. This is where I
> go back to an idea Garima broached in the post requesting configuration
> intersections at the artifact level: “And I think the ideal solution would
> be to acknowledge that Ivy confs, as originally conceived, are not composite
> constructs; so we need to introduce composite elements that can be used to
> construct confs.”
>
> The way that Ivy confs are now designed, any single Ivy conf can and must be
> able to stand on its own. What we need are constructs that don’t stand on
> their own. I can think of two potential high-level ways to accomplish this,
> either of which could address both the operating system problem and the one
> that Garima offered:
>
> - Something like filters that are not themselves Ivy confs but which can
> be applied to Ivy confs. I’m familiar with configuration groupings, and
> they’re not quite the same thing.
> - Building blocks for Ivy confs. So an actual Ivy conf you might specify
> on an artifact is not literally defined as a conf. It’s a composite of the
> different building blocks. These building-block elements would coexist with
> the first-class Ivy confs. Really, this isn’t such a radical idea. As an
> analogy, consider an email address: mgitman and gmail.com are the
> particular elements that are put together to form mgitman@gmail.com, and
> even the gmail.com Internet domain is formed from gmail and the .com
> top-level domain.
>
> In fact, even the operating system/platform problem seems a lot more
> multi-dimensional than just a platform attribute allows. I mean, ia32.linux
> implies a combination of ia32 and linux.
>
> P.S. I’m writing this without having had a chance to digest Shawn’s
> response.
>
> On Sat, Aug 1, 2009 at 3:14 PM, Jeffrey Sinclair <je...@cooljeff.co.uk>wrote:
>
> > ivy-user,
> >
> > The handling of artifacts to populate the java.libary.path has cropped
> > up a couple of times on the mailgroup and there is an outstanding JIRA
> > relating to this [IVY-600].
> >
> > I'm finding that more people are being hit by this, in particular in
> > Eclipse through IvyDE. I've come up with a solution that I feel will
> > work well for both Ivy and IvyDE:
> >
> >
> > http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
> >
> > As described in the blog entry, I'd like to propose a mixture of using
> > types as well as a new attribute on the artifact named platform.
> >
> > I'd like to put a proof of concept together and hence would be grateful
> > for any feedback on my proposal and evaluation of existing solutions.
> >
> > Regards,
> >
> > Jeff
> >
> >
Re: Solution to native dependencies with Ivy
Posted by Mitch Gitman <mg...@gmail.com>.
Jeffrey:
I love the way you’ve presented this idea, but this is one idea I would be
opposed to. The problem is that it confuses Ivy module descriptors with one
particular application of Ivy module descriptors.
Right now, the ivy.xml XML schema provides a powerful, expressive
general-purpose language for describing transitive dependencies and artifact
retrieval. You can use it for pulling JARs into a classpath or for
distributing operating system-specific files or delivering apples and
oranges for all it cares. It is, and should be, agnostic of its various use
cases, many of which we probably haven’t even realized yet. The moment we
promote an application of the language into becoming a first-class part of
the language itself, we corrupt the language. And worse, we open a Pandora’s
box for inviting more applications of the language into the language,
further corrupting it and blurring the line between where the language ends
and its use cases begin.
It seems like with platform or operating system we’re trying to create
something that is not an Ivy conf itself but is an extra dimension or
parameter or filter that gets applied to Ivy confs. If we make platform part
of Ivy, what happens when we encounter the next particular
dimension/parameter/filter that we want to apply to Ivy confs? Do we promote
that into the module descriptor language as well? Every time another
dimension/parameter/filter arises, we have to rekindle the heated debate
about which one is worthy of the language and which isn’t. I can see myself
crying foul, “You promoted platform. Why can’t you promote this?”
And even then, we still don’t have support for arbitrary extra
dimensions/parameters/filters.
Instead, I suggest we take a step back and try to come up with a
general-purpose solution to this problem, one that will solve all the use
cases that come up, not just the choice of operating system. This is where I
go back to an idea Garima broached in the post requesting configuration
intersections at the artifact level: “And I think the ideal solution would
be to acknowledge that Ivy confs, as originally conceived, are not composite
constructs; so we need to introduce composite elements that can be used to
construct confs.”
The way that Ivy confs are now designed, any single Ivy conf can and must be
able to stand on its own. What we need are constructs that don’t stand on
their own. I can think of two potential high-level ways to accomplish this,
either of which could address both the operating system problem and the one
that Garima offered:
- Something like filters that are not themselves Ivy confs but which can
be applied to Ivy confs. I’m familiar with configuration groupings, and
they’re not quite the same thing.
- Building blocks for Ivy confs. So an actual Ivy conf you might specify
on an artifact is not literally defined as a conf. It’s a composite of the
different building blocks. These building-block elements would coexist with
the first-class Ivy confs. Really, this isn’t such a radical idea. As an
analogy, consider an email address: mgitman and gmail.com are the
particular elements that are put together to form mgitman@gmail.com, and
even the gmail.com Internet domain is formed from gmail and the .com
top-level domain.
In fact, even the operating system/platform problem seems a lot more
multi-dimensional than just a platform attribute allows. I mean, ia32.linux
implies a combination of ia32 and linux.
P.S. I’m writing this without having had a chance to digest Shawn’s
response.
On Sat, Aug 1, 2009 at 3:14 PM, Jeffrey Sinclair <je...@cooljeff.co.uk>wrote:
> ivy-user,
>
> The handling of artifacts to populate the java.libary.path has cropped
> up a couple of times on the mailgroup and there is an outstanding JIRA
> relating to this [IVY-600].
>
> I'm finding that more people are being hit by this, in particular in
> Eclipse through IvyDE. I've come up with a solution that I feel will
> work well for both Ivy and IvyDE:
>
>
> http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
>
> As described in the blog entry, I'd like to propose a mixture of using
> types as well as a new attribute on the artifact named platform.
>
> I'd like to put a proof of concept together and hence would be grateful
> for any feedback on my proposal and evaluation of existing solutions.
>
> Regards,
>
> Jeff
>
>
RE: Solution to native dependencies with Ivy
Posted by Shawn Castrianni <Sh...@halliburton.com>.
I have been using the configuration approach for over a year in my enterprise environment. All of the modules built by my company have all applicable platforms as configurations with each artifact (that is platform specific) showing which conf (platform) it is applicable for. For jars and such, no configuration specification is used to show it is applicable for all configurations defined in this module. All of the dependencies in my modules use the following conf mapping, "*->@" to propagate the platform configuration down to the children for transitive dependencies. In other words, when the end user asks for the win32 conf of module A, then he will get the win32 configuration of all of its transitive dependencies too.
The end user controls which platform configuration he wants. The default is to examine the platform being used to perform the ant build and specify that as the configuration name to be retrieved. If a user wants to get a different platform then what he is running the build on or if he wants to get multiple platforms at once, then he simply sets an ant variable to define which configurations he wants. If a module is an install anywhere module that can produce a cross platform installer, then it might want all configurations (platforms) at once. That is achieved with the wildcard when specifying the configuration (platform) desired.
For thirdparty modules, we have setup our own thirdparty repo internal to our network. We do not use any public IVY or MAVEN repo as our legal department wants to have complete control over the thirdparty dependencies we ship with our products. This means I manually get thirdparty dependencies from their webpage source and add it to our thirdparty repo. This allows me to fully control the ivy.xml file used and add in platform configuration names to match our own modules described above.
On a unix environment, symlinks are typically used for *.so files to have different names for the same thing. For example, libXm.so is a symlink to libXm.so.3 which is a symlink to libXm.so.3.0.2. It can be important to retain this symlinking for modules higher up in the dependencies hierarchy from being able to link and build. Therefore, we have setup a system to maintain these symlinks after a retrieve in IVY. We have added an extra attribute called "aliases" in our ivy.xml files like:
<artifact name="libXm" ext="so.3.0.2" type="bin" ext:aliases="libXm.so.3,libXm.so" conf="linux32"/>
I have created an IVY trigger using "post-retrieve-artifact" (which didn't exist until I requested its creation in IVY 2.1) so that the trigger can check if the extra "aliases" attribute exists and then execute the unix "ln" command to recreate the symlinks on the end users machine.
Finally, by using configurations to handle platforms, it makes it more difficult to use configurations to subset your modules as demonstrated by the IVY documentation where is uses "runtime" and "compile" as example configuration usages. If I have my modules subsetted by platform configurations AND I also want to subset them by "runtime" and "compile" time usages, then I don't want to have to define n*m number of configurations, win32-runtime, win32-compile, win64-runtime, win64-compile, linux32-runtime, linux32-compile, etc. This is why I have also requested configuration intersection. That allows an end user to request only win32 artifacts that are also only applicable for runtime, by using the configuration intersection notation: win32+runtime. This is a controversial subject so please see all of the postings in the last week where I was trying to describe my use case.
In summary, I have been very successful in using configurations to handle all of my platform specific issues. It works very nicely.
---
Shawn Castrianni
-----Original Message-----
From: Jeffrey Sinclair [mailto:jeff@cooljeff.co.uk]
Sent: Saturday, August 01, 2009 5:15 PM
To: ivy-user@ant.apache.org
Subject: Solution to native dependencies with Ivy
ivy-user,
The handling of artifacts to populate the java.libary.path has cropped
up a couple of times on the mailgroup and there is an outstanding JIRA
relating to this [IVY-600].
I'm finding that more people are being hit by this, in particular in
Eclipse through IvyDE. I've come up with a solution that I feel will
work well for both Ivy and IvyDE:
http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
As described in the blog entry, I'd like to propose a mixture of using
types as well as a new attribute on the artifact named platform.
I'd like to put a proof of concept together and hence would be grateful
for any feedback on my proposal and evaluation of existing solutions.
Regards,
Jeff
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Solution to native dependencies with Ivy
Posted by Alexandre Gacon <al...@gmail.com>.
Hi,
We are using IVY to manage dependencies among our C++-based projects, which
are built with several compilers. Thus, most of our artifacts are native
artifacts, except for the documentation and for the header files. Most of
the time, we have a debug and a release version of our artifacts and some of
the external libraries we use (like boost) comes in two versions: a static
one and a dll one.
I don't know if it can help but we handle the different cases like this:
- we handle the debug/release difference by using 2 configurations : a
configuration "run" which match the release libraries and a configuration
"devel", which extends the "run" configuration with the debug version of the
libraries (and some other artifacts). A library is described by two artifact
: the release version is declared in the "run" configuration whereas the
debug version is declared in the "devel" version. The use of the
configurations enables us to transitively retrieve the debug versions of the
dependencies.
- we handle the static/dynamic difference by using 2 configurations : one
for the dynamic libraries and one for the static libraries.
- finally, we handle the different compiler version by a more complex
system. We use an extra attribute to define the platform (for example vc80).
In the unresolved IVY file, we use this attribute to describe the type of
our artifacts : for example a C++ library is types as "lib-${platform}".
When we published a version to the repository for a given compiler, we
published as many IVY files as we have compilers in the same directory (for
example we can have a ivy-vc80.xml and a ivy-gcc.xml). Thus we can share
common artifacts among compilers (for example headers and docs). We
configured the IVY pattern to use
'[organisation]/[module](/[branch])/[revision]/ivy-${platform}.xml' for ivy
descriptor and
'[organisation]/[module](/[branch])/[revision]/[type]s/[artifact].[ext]' for
artifacts.
This works quite well but it didn't answer to your problems because we do
not use IvyDE at all.
Best regards,
Alexandre Gacon
Re: Solution to native dependencies with Ivy
Posted by Maarten Coene <ma...@yahoo.com>.
Jeff,
I think there are different possibilities when working with native libraries. Using extra attributes is one of them, but using configurations is another, and there are probably other solutions too. I think Ivy should give a choice to the user how these native libraries are handled, that's why I don't believe that platform should be a first class citizin in Ivy.
I don't do development on IvyDE, but it seems a usefull enhancement to me.
I suggest you create a JIRA ticket for it...
Maarten
----- Original Message ----
From: Jeffrey Sinclair <je...@cooljeff.co.uk>
To: ivy-user@ant.apache.org
Sent: Tuesday, August 4, 2009 7:11:16 AM
Subject: Re: Solution to native dependencies with Ivy
Thanks Maarten.
The push back that I would have to you guys is that if you were writing
a best practises document for how to use Ivy with JNI related
dependencies (and potentially in a non Java solution), would you be
advocating the use of extra:platform or using configurations?
If the former, then why not have platform as a first class citizen in
Ivy?
Having said this, the solution you've proposed would solve the problem
completely. To remove the complete gap of native dependencies in IvyDE,
would Xavier and yourself be happy with adding a new feature to IvyDE
which allows 'type' to be used in the same way IvyDE currently supports
javadoc and source attachments?
Then, in the future, if arbitrary filters are added to Ivy, we could
look into adding support for using custom filters (in addition to plain
type filters) in IvyDE.
Jeff
On Mon, 2009-08-03 at 14:35 -0700, Maarten Coene wrote:
> Jeff,
>
> excellent blog entry, thanks for sharing this with us.
>
> I suppose you want to make a proof-of-concept of your last proposal (mixture of types and 'platform' attribute).
> I don't like adding such a "platform" attribute to the set of default Ivy artifact-attributes since I don't think Ivy should have knowledge about this (I'm not talking about IvyDE here). But maybe you could make this 'platform' attribute custom/extra attribute and modify the retrieve/cachepath task to make it possible to filter on these extra attributes as well, which is a more general like for instance requested in https://issues.apache.org/jira/browse/IVY-439
>
> regards,
> Maarten
>
>
>
>
>
> ----- Original Message ----
> From: Jeffrey Sinclair <je...@cooljeff.co.uk>
> To: ivy-user@ant.apache.org
> Sent: Sunday, August 2, 2009 12:14:36 AM
> Subject: Solution to native dependencies with Ivy
>
> ivy-user,
>
> The handling of artifacts to populate the java.libary.path has cropped
> up a couple of times on the mailgroup and there is an outstanding JIRA
> relating to this [IVY-600].
>
> I'm finding that more people are being hit by this, in particular in
> Eclipse through IvyDE. I've come up with a solution that I feel will
> work well for both Ivy and IvyDE:
>
> http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
>
> As described in the blog entry, I'd like to propose a mixture of using
> types as well as a new attribute on the artifact named platform.
>
> I'd like to put a proof of concept together and hence would be grateful
> for any feedback on my proposal and evaluation of existing solutions.
>
> Regards,
>
> Jeff
>
>
>
Re: Solution to native dependencies with Ivy
Posted by Jeffrey Sinclair <je...@cooljeff.co.uk>.
Thanks Maarten.
The push back that I would have to you guys is that if you were writing
a best practises document for how to use Ivy with JNI related
dependencies (and potentially in a non Java solution), would you be
advocating the use of extra:platform or using configurations?
If the former, then why not have platform as a first class citizen in
Ivy?
Having said this, the solution you've proposed would solve the problem
completely. To remove the complete gap of native dependencies in IvyDE,
would Xavier and yourself be happy with adding a new feature to IvyDE
which allows 'type' to be used in the same way IvyDE currently supports
javadoc and source attachments?
Then, in the future, if arbitrary filters are added to Ivy, we could
look into adding support for using custom filters (in addition to plain
type filters) in IvyDE.
Jeff
On Mon, 2009-08-03 at 14:35 -0700, Maarten Coene wrote:
> Jeff,
>
> excellent blog entry, thanks for sharing this with us.
>
> I suppose you want to make a proof-of-concept of your last proposal (mixture of types and 'platform' attribute).
> I don't like adding such a "platform" attribute to the set of default Ivy artifact-attributes since I don't think Ivy should have knowledge about this (I'm not talking about IvyDE here). But maybe you could make this 'platform' attribute custom/extra attribute and modify the retrieve/cachepath task to make it possible to filter on these extra attributes as well, which is a more general like for instance requested in https://issues.apache.org/jira/browse/IVY-439
>
> regards,
> Maarten
>
>
>
>
>
> ----- Original Message ----
> From: Jeffrey Sinclair <je...@cooljeff.co.uk>
> To: ivy-user@ant.apache.org
> Sent: Sunday, August 2, 2009 12:14:36 AM
> Subject: Solution to native dependencies with Ivy
>
> ivy-user,
>
> The handling of artifacts to populate the java.libary.path has cropped
> up a couple of times on the mailgroup and there is an outstanding JIRA
> relating to this [IVY-600].
>
> I'm finding that more people are being hit by this, in particular in
> Eclipse through IvyDE. I've come up with a solution that I feel will
> work well for both Ivy and IvyDE:
>
> http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
>
> As described in the blog entry, I'd like to propose a mixture of using
> types as well as a new attribute on the artifact named platform.
>
> I'd like to put a proof of concept together and hence would be grateful
> for any feedback on my proposal and evaluation of existing solutions.
>
> Regards,
>
> Jeff
>
>
>
Re: Solution to native dependencies with Ivy
Posted by Maarten Coene <ma...@yahoo.com>.
Jeff,
excellent blog entry, thanks for sharing this with us.
I suppose you want to make a proof-of-concept of your last proposal (mixture of types and 'platform' attribute).
I don't like adding such a "platform" attribute to the set of default Ivy artifact-attributes since I don't think Ivy should have knowledge about this (I'm not talking about IvyDE here). But maybe you could make this 'platform' attribute custom/extra attribute and modify the retrieve/cachepath task to make it possible to filter on these extra attributes as well, which is a more general like for instance requested in https://issues.apache.org/jira/browse/IVY-439
regards,
Maarten
----- Original Message ----
From: Jeffrey Sinclair <je...@cooljeff.co.uk>
To: ivy-user@ant.apache.org
Sent: Sunday, August 2, 2009 12:14:36 AM
Subject: Solution to native dependencies with Ivy
ivy-user,
The handling of artifacts to populate the java.libary.path has cropped
up a couple of times on the mailgroup and there is an outstanding JIRA
relating to this [IVY-600].
I'm finding that more people are being hit by this, in particular in
Eclipse through IvyDE. I've come up with a solution that I feel will
work well for both Ivy and IvyDE:
http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
As described in the blog entry, I'd like to propose a mixture of using
types as well as a new attribute on the artifact named platform.
I'd like to put a proof of concept together and hence would be grateful
for any feedback on my proposal and evaluation of existing solutions.
Regards,
Jeff