You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by "Bailey, Darragh" <db...@hp.com> on 2010/08/19 21:03:35 UTC
OS specific artifacts
I have a JNI library, dynamic libraries and dynamic & static build libraries for a software component that are all platform specific.
This is causing 2 problems for me:
1) artifact filename clashing. While the dll's and libs used to compile against on windows and linux are all named differently, the jar files are not. Additionally the component is released with the same version, so there's nothing to distinguish with platform the jar files came from. Additionally we've found out that the code in the jar files on the different platforms even with the same release number is not identical, and while we can compile java components using the jar file from one or the other, we have to make sure that the jar file we use at runtime is the one that was shipped with the dll or so file.
Windows:
component.dll <-- dynamic library
component.lib <-- library to compile against to use dll
component_static.lib <-- static library for standalone
component.jar <-- jar file
Linux:
libcomponent.so <-- dynamic library, used to compile against
libcomponent_static.a <-- static library for standalone
component.jar <-- jar file, oh look same name, oh fun!
Right now, I'm not publishing these components to our internal artifact repository, however I plan to in order to shift fully over to using ivy for dependency management. In doing so I'll have to handle the fact that when developers are building for windows or linux, that they get the correct ones, particularly if we want to run any unit tests :/ (developers always run those, right?...).
My only thoughts for a solution at the moment is to modify either the name or path of the jar file when publishing it to include an os specifier. This is the only solution I can really see working, unless of course the team that built the jar file could built it so that it would work wit the native library on either platform, but I doubt that would succeed. Any other suggestions?
2) configurations. I need OS specific configurations to distinguish the artifacts when ivy retrieves them so that the correct set is retrieved for building against whether on linux or windows. Ideally I'll like to be able to specify a 'build' configuration and just have ivy retrieve the correct dependencies depending on the target OS, however, I don't see how that is possible. It looks like to me, that I have to change which configuration I retrieve by relying on some conditionals in the ant scripts.
That's possible for the ant builds, but I'm not sure how to handle this correctly for the eclipse builds and ivyde.
Would it be possible to have a build configuration that extends a variable configuration i.e.
<conf name="build" visibility="public" extends="build-${target.os}" description="--> libraries for build time depending on value of os.target property" />
<conf name="build-linux" visibility="private" description="--> libraries for building for linux targets" />
<conf name="build-windows" visibility="private" description="--> libraries for building for windows targets" />
Then use a property file in ivyde to define target.os? If it's not defined by eclipse by default. I haven't checked.
I haven't tried anything on this just yet, hoping to as soon as I get some time to work on it, but hoping that someone may have already have a solution and will stop me from going down a dead end before I get around to these problems.
--
Regards,
Darragh Bailey
RE: OS specific artifacts
Posted by Shawn Castrianni <Sh...@halliburton.com>.
I use configurations and extra attributes as well to solve this problem. You can probably search this mailing list to find my previous posts where I give more detail as to my solution.
---
Shawn Castrianni
-----Original Message-----
From: Mitch Gitman [mailto:mgitman@gmail.com]
Sent: Friday, August 20, 2010 10:52 AM
To: ivy-user@ant.apache.org
Subject: Re: OS specific artifacts
Nate's got a better answer than I do for the situation where the various
different platforms' artifacts belong in the same Ivy module. So maybe the
question comes down to whether (A) the different platforms represent the
same Ivy module that gets published together and versioned as one and can
coexist on the Ivy repo or (B) each platform is really its own Ivy module
that versions separately.
Nate, it looks like Ivy is just ignoring the type attribute, no?
I recall resisting the idea of making things like OS and platform
first-class citizens in Ivy, but that's a different story from extending Ivy
with these paradigms. And that seems to be a perfect use of the extra
attribute.
On Fri, Aug 20, 2010 at 6:11 AM, Nathan Franzen
<Na...@mmodal.com>wrote:
>
> I think the difficulty is that ivy does not, out of the box, provide enough
> flexibility to distinguish native artifacts. There are lots of axes you
> might want to use in differentiating libraries, for example release vs
> debug, windows vs linux, 32 bit vs 64 bit and so on. And I believe that for
> one codebase it is appropriate to have a one ivy project.
>
> So what to do? "extra" parameters can help some:
>
> ivy.xml:
> <publications>
> <artifact name="libX" type="linux/icc11.32" ext="so"
> extra:os="linux" extra:platform="32" conf="linux32-dynamic"/>
>
> <artifact name="libX" type="linux/icc11.64" ext="so"
> extra:os="linux" extra:platform="64" conf="linux64-dynamic"/>
>
> <artifact name="X" conf="windows32-dynamic"
> extra:os="windows" extra:platform="32"
> type="winnt_vc90/Release"
> ext="dll"
> />
> <artifact name="X" conf="windows32-dynamic"
> extra:os="windows" extra:platform="32"
> type="winnt_vc90/Release"
> ext="lib"
>
>
> And then use those os + platform tags in the ivy resolve task:
>
> <ivy:retrieve conf="${conf.retrieve}"
>
> pattern="${artifact.retrieve.dir}/([os])([platform])/[artifact].[ext]"
>
> ivypattern="${ivyfile.retrieve.dir}/[organization].[module]/ivy.xml"
> file="${basedir}/ivy/ivy.xml"
> revision="${version}" refresh="true" />
>
> * actually, ([os])([platform]/) is probably better in order to avoid cases
> of "//" if those tags are not defined.
>
> I don't have all this working beautifully in our heterogeneous environment
> yet, but there seem to be enough pieces to put it all together. Notice of
> course that the configuration has mutated a little from the ivy traditional
> sense of conf=>"downstream usage" into conf=>"upstream categorization" And
> I've used "type" just to represent part of compilation original path, i.e.
> an alternate (but less meaningful) way of disambiguating the artifacts. I
> know that's a little ugly.
>
> A solution to conflicting names exists, so if you are willing to set the
> configuration outside of ivy somehow, then it can work.
>
> -Nate
>
>
Re: OS specific artifacts
Posted by Mitch Gitman <mg...@gmail.com>.
Nate's got a better answer than I do for the situation where the various
different platforms' artifacts belong in the same Ivy module. So maybe the
question comes down to whether (A) the different platforms represent the
same Ivy module that gets published together and versioned as one and can
coexist on the Ivy repo or (B) each platform is really its own Ivy module
that versions separately.
Nate, it looks like Ivy is just ignoring the type attribute, no?
I recall resisting the idea of making things like OS and platform
first-class citizens in Ivy, but that's a different story from extending Ivy
with these paradigms. And that seems to be a perfect use of the extra
attribute.
On Fri, Aug 20, 2010 at 6:11 AM, Nathan Franzen
<Na...@mmodal.com>wrote:
>
> I think the difficulty is that ivy does not, out of the box, provide enough
> flexibility to distinguish native artifacts. There are lots of axes you
> might want to use in differentiating libraries, for example release vs
> debug, windows vs linux, 32 bit vs 64 bit and so on. And I believe that for
> one codebase it is appropriate to have a one ivy project.
>
> So what to do? "extra" parameters can help some:
>
> ivy.xml:
> <publications>
> <artifact name="libX" type="linux/icc11.32" ext="so"
> extra:os="linux" extra:platform="32" conf="linux32-dynamic"/>
>
> <artifact name="libX" type="linux/icc11.64" ext="so"
> extra:os="linux" extra:platform="64" conf="linux64-dynamic"/>
>
> <artifact name="X" conf="windows32-dynamic"
> extra:os="windows" extra:platform="32"
> type="winnt_vc90/Release"
> ext="dll"
> />
> <artifact name="X" conf="windows32-dynamic"
> extra:os="windows" extra:platform="32"
> type="winnt_vc90/Release"
> ext="lib"
>
>
> And then use those os + platform tags in the ivy resolve task:
>
> <ivy:retrieve conf="${conf.retrieve}"
>
> pattern="${artifact.retrieve.dir}/([os])([platform])/[artifact].[ext]"
>
> ivypattern="${ivyfile.retrieve.dir}/[organization].[module]/ivy.xml"
> file="${basedir}/ivy/ivy.xml"
> revision="${version}" refresh="true" />
>
> * actually, ([os])([platform]/) is probably better in order to avoid cases
> of "//" if those tags are not defined.
>
> I don't have all this working beautifully in our heterogeneous environment
> yet, but there seem to be enough pieces to put it all together. Notice of
> course that the configuration has mutated a little from the ivy traditional
> sense of conf=>"downstream usage" into conf=>"upstream categorization" And
> I've used "type" just to represent part of compilation original path, i.e.
> an alternate (but less meaningful) way of disambiguating the artifacts. I
> know that's a little ugly.
>
> A solution to conflicting names exists, so if you are willing to set the
> configuration outside of ivy somehow, then it can work.
>
> -Nate
>
>
RE: OS specific artifacts
Posted by Nathan Franzen <Na...@mmodal.com>.
I think the difficulty is that ivy does not, out of the box, provide enough flexibility to distinguish native artifacts. There are lots of axes you might want to use in differentiating libraries, for example release vs debug, windows vs linux, 32 bit vs 64 bit and so on. And I believe that for one codebase it is appropriate to have a one ivy project.
So what to do? "extra" parameters can help some:
ivy.xml:
<publications>
<artifact name="libX" type="linux/icc11.32" ext="so" extra:os="linux" extra:platform="32" conf="linux32-dynamic"/>
<artifact name="libX" type="linux/icc11.64" ext="so" extra:os="linux" extra:platform="64" conf="linux64-dynamic"/>
<artifact name="X" conf="windows32-dynamic"
extra:os="windows" extra:platform="32"
type="winnt_vc90/Release"
ext="dll"
/>
<artifact name="X" conf="windows32-dynamic"
extra:os="windows" extra:platform="32" type="winnt_vc90/Release"
ext="lib"
And then use those os + platform tags in the ivy resolve task:
<ivy:retrieve conf="${conf.retrieve}"
pattern="${artifact.retrieve.dir}/([os])([platform])/[artifact].[ext]"
ivypattern="${ivyfile.retrieve.dir}/[organization].[module]/ivy.xml"
file="${basedir}/ivy/ivy.xml" revision="${version}" refresh="true" />
* actually, ([os])([platform]/) is probably better in order to avoid cases of "//" if those tags are not defined.
I don't have all this working beautifully in our heterogeneous environment yet, but there seem to be enough pieces to put it all together. Notice of course that the configuration has mutated a little from the ivy traditional sense of conf=>"downstream usage" into conf=>"upstream categorization" And I've used "type" just to represent part of compilation original path, i.e. an alternate (but less meaningful) way of disambiguating the artifacts. I know that's a little ugly.
A solution to conflicting names exists, so if you are willing to set the configuration outside of ivy somehow, then it can work.
-Nate
> -----Original Message-----
> From: Bailey, Darragh [mailto:dbailey@hp.com]
> Sent: Friday, August 20, 2010 5:49 AM
> To: ivy-user@ant.apache.org
> Subject: RE: OS specific artifacts
>
>
>
>
> > -----Original Message-----
> > From: Mitch Gitman [mailto:mgitman@gmail.com]
> > Sent: 19 August 2010 21:44
> > To: ivy-user@ant.apache.org
> > Subject: Re: OS specific artifacts
> >
> > 1) It sounds like what you "want" to do is create two
> > different files in the
> > same Ivy module and give them the same name, even though they are two
> > different files. This doesn't make sense.
>
> They may be OS specific but they are being delivered to me with the same
> name. I've also no idea how the team providing the jar file managed to
> make them different. One would expect that the jar file should have been
> created to load either the dll or the so based on what platform it's
> being run on. Although I have to admit my understanding of JNI libraries
> is still pretty limited.
>
> > I would suggest, instead of trying to find a way to work
> > around your current
> > state of things, think about what makes the most sense
> > conceptually. From my
> > limited understanding of your situation, this means either of
> > two choices:
> > A. Create one Ivy module for the Windows deployment and
> > another Ivy module
> > for the Linux deployment. The two component.jar files can
> > have the same name
> > since they don't get deployed to the same location.
> > B. Keep everything in one Ivy module and use a Windows Ivy
> > conf and a Linux
> > Ivy conf. In this scenario you would have to name the
> > component.jar files
> > something different, but they are different files. This
> > choice presumes that
> > you're versioning and releasing your Windows and Linux deployments in
> > concert, using a single version.
> >
> > Forgive me for resisting your premise.
>
> You're forgiven, I didn't ask for alternative suggestions any expect to
> get the response that said "yeap your solution is the right one" ;-)
>
> I suspect that there will be no way for IvyDE to automatically select
> the right config using this approach, and not requiring the other
> developers to have to manually change which conf is selected based on
> OS, would be better.
>
>
>
> > 2) Instead of using a ${target.os} placeholder, control the
> > choice between
> > the Linux and Windows Ivy confs outside of ivy.xml. In your
> > build scripts,
> > the ivy:resolve would specify one conf or another. In IvyDE, you would
> > choose one conf or another or both. This brings you back to
> > the problem of
> > conflicting names.
>
> This gives me the result that the developers will have to modify the
> eclipse project settings any time they switch OS.
>
> Ideally this is what I want to avoid. I want that whenever the developer
> opens eclipse, if he's on linux, IvyDE retrieves the linux dependencies,
> and if he's on windows, IvyDE retrieves the windows dependencies. If
> it's being run via an Ant script, I have a lot more flexibility at my
> finger tips in retrieveing configurations.
>
>
> Thinking on it a little more, I suspect that a method for handling the
> jar file will fall out from a solution that allows IvyDE to
> automatically retreive the correct deps for the current platform.
>
>
> --
> Regards,
> Darragh Bailey
RE: OS specific artifacts
Posted by "Bailey, Darragh" <db...@hp.com>.
> -----Original Message-----
> From: Mitch Gitman [mailto:mgitman@gmail.com]
> Sent: 19 August 2010 21:44
> To: ivy-user@ant.apache.org
> Subject: Re: OS specific artifacts
>
> 1) It sounds like what you "want" to do is create two
> different files in the
> same Ivy module and give them the same name, even though they are two
> different files. This doesn't make sense.
They may be OS specific but they are being delivered to me with the same name. I've also no idea how the team providing the jar file managed to make them different. One would expect that the jar file should have been created to load either the dll or the so based on what platform it's being run on. Although I have to admit my understanding of JNI libraries is still pretty limited.
> I would suggest, instead of trying to find a way to work
> around your current
> state of things, think about what makes the most sense
> conceptually. From my
> limited understanding of your situation, this means either of
> two choices:
> A. Create one Ivy module for the Windows deployment and
> another Ivy module
> for the Linux deployment. The two component.jar files can
> have the same name
> since they don't get deployed to the same location.
> B. Keep everything in one Ivy module and use a Windows Ivy
> conf and a Linux
> Ivy conf. In this scenario you would have to name the
> component.jar files
> something different, but they are different files. This
> choice presumes that
> you're versioning and releasing your Windows and Linux deployments in
> concert, using a single version.
>
> Forgive me for resisting your premise.
You're forgiven, I didn't ask for alternative suggestions any expect to get the response that said "yeap your solution is the right one" ;-)
I suspect that there will be no way for IvyDE to automatically select the right config using this approach, and not requiring the other developers to have to manually change which conf is selected based on OS, would be better.
> 2) Instead of using a ${target.os} placeholder, control the
> choice between
> the Linux and Windows Ivy confs outside of ivy.xml. In your
> build scripts,
> the ivy:resolve would specify one conf or another. In IvyDE, you would
> choose one conf or another or both. This brings you back to
> the problem of
> conflicting names.
This gives me the result that the developers will have to modify the eclipse project settings any time they switch OS.
Ideally this is what I want to avoid. I want that whenever the developer opens eclipse, if he's on linux, IvyDE retrieves the linux dependencies, and if he's on windows, IvyDE retrieves the windows dependencies. If it's being run via an Ant script, I have a lot more flexibility at my finger tips in retrieveing configurations.
Thinking on it a little more, I suspect that a method for handling the jar file will fall out from a solution that allows IvyDE to automatically retreive the correct deps for the current platform.
--
Regards,
Darragh Bailey
Re: OS specific artifacts
Posted by Mitch Gitman <mg...@gmail.com>.
1) It sounds like what you "want" to do is create two different files in the
same Ivy module and give them the same name, even though they are two
different files. This doesn't make sense.
I would suggest, instead of trying to find a way to work around your current
state of things, think about what makes the most sense conceptually. From my
limited understanding of your situation, this means either of two choices:
A. Create one Ivy module for the Windows deployment and another Ivy module
for the Linux deployment. The two component.jar files can have the same name
since they don't get deployed to the same location.
B. Keep everything in one Ivy module and use a Windows Ivy conf and a Linux
Ivy conf. In this scenario you would have to name the component.jar files
something different, but they are different files. This choice presumes that
you're versioning and releasing your Windows and Linux deployments in
concert, using a single version.
Forgive me for resisting your premise.
2) Instead of using a ${target.os} placeholder, control the choice between
the Linux and Windows Ivy confs outside of ivy.xml. In your build scripts,
the ivy:resolve would specify one conf or another. In IvyDE, you would
choose one conf or another or both. This brings you back to the problem of
conflicting names.
On Thu, Aug 19, 2010 at 12:03 PM, Bailey, Darragh <db...@hp.com> wrote:
>
>
> I have a JNI library, dynamic libraries and dynamic & static build
> libraries for a software component that are all platform specific.
>
> This is causing 2 problems for me:
>
> 1) artifact filename clashing. While the dll's and libs used to compile
> against on windows and linux are all named differently, the jar files are
> not. Additionally the component is released with the same version, so
> there's nothing to distinguish with platform the jar files came from.
> Additionally we've found out that the code in the jar files on the different
> platforms even with the same release number is not identical, and while we
> can compile java components using the jar file from one or the other, we
> have to make sure that the jar file we use at runtime is the one that was
> shipped with the dll or so file.
>
> Windows:
> component.dll <-- dynamic library
> component.lib <-- library to compile against to use dll
> component_static.lib <-- static library for standalone
> component.jar <-- jar file
>
> Linux:
> libcomponent.so <-- dynamic library, used to compile against
> libcomponent_static.a <-- static library for standalone
> component.jar <-- jar file, oh look same name, oh fun!
>
> Right now, I'm not publishing these components to our internal artifact
> repository, however I plan to in order to shift fully over to using ivy for
> dependency management. In doing so I'll have to handle the fact that when
> developers are building for windows or linux, that they get the correct
> ones, particularly if we want to run any unit tests :/ (developers always
> run those, right?...).
>
> My only thoughts for a solution at the moment is to modify either the name
> or path of the jar file when publishing it to include an os specifier. This
> is the only solution I can really see working, unless of course the team
> that built the jar file could built it so that it would work wit the native
> library on either platform, but I doubt that would succeed. Any other
> suggestions?
>
>
> 2) configurations. I need OS specific configurations to distinguish the
> artifacts when ivy retrieves them so that the correct set is retrieved for
> building against whether on linux or windows. Ideally I'll like to be able
> to specify a 'build' configuration and just have ivy retrieve the correct
> dependencies depending on the target OS, however, I don't see how that is
> possible. It looks like to me, that I have to change which configuration I
> retrieve by relying on some conditionals in the ant scripts.
>
> That's possible for the ant builds, but I'm not sure how to handle this
> correctly for the eclipse builds and ivyde.
>
> Would it be possible to have a build configuration that extends a variable
> configuration i.e.
>
> <conf name="build" visibility="public" extends="build-${target.os}"
> description="--> libraries for build time depending on value of os.target
> property" />
> <conf name="build-linux" visibility="private" description="--> libraries
> for building for linux targets" />
> <conf name="build-windows" visibility="private" description="--> libraries
> for building for windows targets" />
>
> Then use a property file in ivyde to define target.os? If it's not defined
> by eclipse by default. I haven't checked.
>
>
> I haven't tried anything on this just yet, hoping to as soon as I get some
> time to work on it, but hoping that someone may have already have a solution
> and will stop me from going down a dead end before I get around to these
> problems.
>
> --
> Regards,
> Darragh Bailey
>