You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Shawn Castrianni <Sh...@halliburton.com> on 2008/07/08 14:08:27 UTC
better native code support
IVY might be thought of as just a tool for managing dependencies for Java code. Perhaps the same thing is thought of about ANT being just a build tool for Java code. However, I am using both ANT and IVY to build modules with both java and native code on multiple platforms with great success. I use IVY's configuration concept to manage the .so's for the various unix platforms and the .dll's for the various windows platforms. Someone building on a windows machine would get the windows dependencies by default and someone building on a unit platform would get the unix dependencies by default. An installer module could override the default and specify * for the configuration and get all platform's dependencies for the DVD image.
One thing that IVY might be able to improve is symbolic link handling. Typically on the unix platform, .so llibraries have their version encoded into the filename like:
libxerces-c.so.26.0
and then there are symbolic links (sometimes more than one) that point to the real file:
libxerces-c.so.26 -> libxerces-c.so.26.0
libxerces-c.so -> libxerces-c.so.26.0
Currently, I am publishing all three files as artifacts but the symbolic links do not survive the round trip from publish to resolve/retrieve. They get converted to 3 identical files wasting disk space in the repository unnecessarily.
Would it be a good idea to perhaps add another attribute to the artifact xml tag called alias which could be a comma delimited list of alias names? Something like:
....
<artifact name="libxerces-c" ext="so.26.0" type="bin" alias="libxerces-c.so.26,libxerces-c.so"/>
....
Then ivy would only publish one artifact with the real filename of libxerces-c.so.26.0 saving disk space in the ivy repository. Then on a resolve/retrieve, it would download the real file and then on unix, create symbolic links for each alias, and on windows, either ignore the aliases or make real file copies.
I don't know if this has any bad ramifications on the rest of IVY functionality. I am just suggesting it as an idea.
Any comments?
---
Shawn Castrianni
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: better native code support
Posted by Xavier Hanin <xa...@gmail.com>.
On Tue, Jul 8, 2008 at 2:08 PM, Shawn Castrianni <
Shawn.Castrianni@halliburton.com> wrote:
> IVY might be thought of as just a tool for managing dependencies for Java
> code. Perhaps the same thing is thought of about ANT being just a build
> tool for Java code. However, I am using both ANT and IVY to build modules
> with both java and native code on multiple platforms with great success. I
> use IVY's configuration concept to manage the .so's for the various unix
> platforms and the .dll's for the various windows platforms. Someone
> building on a windows machine would get the windows dependencies by default
> and someone building on a unit platform would get the unix dependencies by
> default. An installer module could override the default and specify * for
> the configuration and get all platform's dependencies for the DVD image.
>
> One thing that IVY might be able to improve is symbolic link handling.
> Typically on the unix platform, .so llibraries have their version encoded
> into the filename like:
>
> libxerces-c.so.26.0
>
> and then there are symbolic links (sometimes more than one) that point to
> the real file:
>
> libxerces-c.so.26 -> libxerces-c.so.26.0
> libxerces-c.so -> libxerces-c.so.26.0
>
> Currently, I am publishing all three files as artifacts but the symbolic
> links do not survive the round trip from publish to resolve/retrieve. They
> get converted to 3 identical files wasting disk space in the repository
> unnecessarily.
>
> Would it be a good idea to perhaps add another attribute to the artifact
> xml tag called alias which could be a comma delimited list of alias names?
> Something like:
>
> ....
> <artifact name="libxerces-c" ext="so.26.0" type="bin"
> alias="libxerces-c.so.26,libxerces-c.so"/>
> ....
>
> Then ivy would only publish one artifact with the real filename of
> libxerces-c.so.26.0 saving disk space in the ivy repository. Then on a
> resolve/retrieve, it would download the real file and then on unix, create
> symbolic links for each alias, and on windows, either ignore the aliases or
> make real file copies.
>
> I don't know if this has any bad ramifications on the rest of IVY
> functionality. I am just suggesting it as an idea.
>
> Any comments?
The idea of an alias sounds interesting, even though I think this can be
done using triggers or a task called after retrieve (maybe together with
extra attributes). Another way is to use retrieve with custom patterns, and
use the symlink option supported by the retrieve task.
OTOH, the advantage of supporting this directly in Ivy is that it would
avoid duplicating the same logic for every user needing to use similar
things. The only problem I see is that the role of the alias is still not
absolutely clear to me. Would the alias be used at retrieve time only? And
what would the alias replace? One token, or the full filename as your
example suggests?
Xavier
>
>
> ---
> Shawn Castrianni
>
> ----------------------------------------------------------------------
> This e-mail, including any attached files, may contain confidential and
> privileged information for the sole use of the intended recipient. Any
> review, use, distribution, or disclosure by others is strictly prohibited.
> If you are not the intended recipient (or authorized to receive information
> for the intended recipient), please contact the sender by reply e-mail and
> delete all copies of this message.
--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/
RE: better native code support
Posted by Shawn Castrianni <Sh...@halliburton.com>.
Here is my implementation of Option 1 with Method 1b in case anyone finds it useful. I guess I will go with this unless someone knows of a more elegant solution.
<target name="dependencies.retrieve" depends="dependencies.resolve">
<mkdir dir="${env.DEPENDENCIES_DIR}"/>
<ivy:retrieve pattern="${env.DEPENDENCIES_DIR}/[module]/[type]/[home]/[homeType]/[path]/[artifact](.[ext])" ivypattern="${env.DEPENDENCIES_DIR}/[module]/[type]/[artifact](.[ext])"/>
</target>
<target name="dependencies.process" depends="dependencies.retrieve">
<!--Perform post dependency processing-->
<ac:for param="dependency">
<dirset dir="${env.DEPENDENCIES_DIR}">
<include name="*"/>
</dirset>
<sequential>
<!--Search for any include.zip files and unzip them and delete the zip file-->
<ac:if>
<available file="@{dependency}/include/include.zip"/>
<ac:then>
<unzip src="@{dependency}/include/include.zip" dest="@{dependency}/include"/>
<delete file="@{dependency}/include/include.zip"/>
</ac:then>
</ac:if>
<!--Search for any symbolic links that need to be made-->
<xt:xmltask source="@{dependency}/ivy/ivy.xml" failWithoutMatch="false">
<xt:call path="ivy-module/publications/artifact">
<param name="artifactName" path="@name"/>
<param name="artifactExt" path="@ext"/>
<param name="artifactType" path="@type"/>
<param name="artifactHome" path="@ext:home"/>
<param name="artifactHomeType" path="@ext:homeType"/>
<param name="artifactPath" path="@ext:path"/>
<param name="artifactAliases" path="@ext:aliases" default=""/>
<xt:actions>
<ac:for param="artifactAlias" list="@{artifactAliases}">
<sequential>
<ac:if>
<and>
<available file="@{dependency}/@{artifactType}/@{artifactHome}/@{artifactHomeType}/@{artifactPath}" type="dir"/>
<available file="@{dependency}/@{artifactType}/@{artifactHome}/@{artifactHomeType}/@{artifactPath}/@{artifactName}.@{artifactExt}" type="file"/>
<not>
<available file="@{dependency}/@{artifactType}/@{artifactHome}/@{artifactHomeType}/@{artifactPath}/@{artifactAlias}" type="file"/>
</not>
</and>
<ac:then>
<echo message="Creating symbolic link: @{artifactAlias}->@{artifactName}.@{artifactExt}"/>
<exec executable="ln" dir="${basedir}" failonerror="true">
<arg value="-s"/>
<arg value="@{artifactName}.@{artifactExt}"/>
<arg value="@{dependency}/@{artifactType}/@{artifactHome}/@{artifactHomeType}/@{artifactPath}/@{artifactAlias}"/>
</exec>
</ac:then>
<ac:elseif>
<available file="@{dependency}/@{artifactType}/@{artifactHome}/@{artifactHomeType}/@{artifactPath}/@{artifactAlias}" type="file"/>
<ac:then>
<echo message="Symbolic link already exists: @{artifactAlias}->@{artifactName}.@{artifactExt}"/>
</ac:then>
</ac:elseif>
</ac:if>
</sequential>
</ac:for>
</xt:actions>
</xt:call>
</xt:xmltask>
</sequential>
</ac:for>
</target>
---
Shawn Castrianni
CM Chief Architect
Landmark
Halliburton Drilling, Evaluation and Digital Solutions Building 2
2107 City West Blvd.
Houston, TX 77042
Work: 713-839-3086
Cell: 832-654-0888
Fax: 713-839-2758
From: Shawn Castrianni
Sent: Thursday, October 09, 2008 11:29 PM
To: 'ivy-user@ant.apache.org'
Subject: RE: better native code support
So far only one reply to this post from Xavier. He suggested trying a trigger, or the symlink option on retrieve, or writing an ANT task that is called after retrieve. I thought I would look into trying something but am not sure what would be the most elegant way.
Option 1:
Write some ANT code that gets run right after calling ivy:retrieve that somehow detects which files that need symlinks and running exec to call the ln command. This should work but I need a good way to detect which files need symlinks and what the symlink name should be. I have 2 methods:
Method 1a:
Add published artifacts to the module that will get published with 0 length content. I could name these special artifacts with symlink.<sylmlinkName>.<targetName> such that I could then search for any files with a name that start with symlink and just strip out the name of the symlink from <symlinkName> and point it to the file with <targetName>.
Method 1b:
Add custom extra attributes on the published artifacts for those artifacts that need associated symlinks that provide the name of the symlinks that should point to this artifact. This is what I suggested in this original post. Then the ANT code that would run right after a retrieve would manually parse the XML looking for these special custom extra attributes and perform the link command.
Option 2:
Use triggers combined with custom extra attributes. The custom extra attributes would work just like Method 1b except instead of manually parsing the ivy.xml file, the trigger would be passed the necessary information via ANT properties preventing the need to parse. It would still used the exec task to call the ln command. The problem with this is that I need a post-retrieve-artifact event which doesn't seem to exist. I don't want the post-download-artifact event since I don't want to create the symlink in the cache. I need the symlink down in the retrieved area under ${basedir}.
Anybody have any suggestions for other options?
---
Shawn Castrianni
From: Shawn Castrianni
Sent: Tuesday, July 08, 2008 7:08 AM
To: ivy-user@ant.apache.org
Subject: better native code support
IVY might be thought of as just a tool for managing dependencies for Java code. Perhaps the same thing is thought of about ANT being just a build tool for Java code. However, I am using both ANT and IVY to build modules with both java and native code on multiple platforms with great success. I use IVY's configuration concept to manage the .so's for the various unix platforms and the .dll's for the various windows platforms. Someone building on a windows machine would get the windows dependencies by default and someone building on a unit platform would get the unix dependencies by default. An installer module could override the default and specify * for the configuration and get all platform's dependencies for the DVD image.
One thing that IVY might be able to improve is symbolic link handling. Typically on the unix platform, .so llibraries have their version encoded into the filename like:
libxerces-c.so.26.0
and then there are symbolic links (sometimes more than one) that point to the real file:
libxerces-c.so.26 -> libxerces-c.so.26.0
libxerces-c.so -> libxerces-c.so.26.0
Currently, I am publishing all three files as artifacts but the symbolic links do not survive the round trip from publish to resolve/retrieve. They get converted to 3 identical files wasting disk space in the repository unnecessarily.
Would it be a good idea to perhaps add another attribute to the artifact xml tag called alias which could be a comma delimited list of alias names? Something like:
....
<artifact name="libxerces-c" ext="so.26.0" type="bin" alias="libxerces-c.so.26,libxerces-c.so"/>
....
Then ivy would only publish one artifact with the real filename of libxerces-c.so.26.0 saving disk space in the ivy repository. Then on a resolve/retrieve, it would download the real file and then on unix, create symbolic links for each alias, and on windows, either ignore the aliases or make real file copies.
I don't know if this has any bad ramifications on the rest of IVY functionality. I am just suggesting it as an idea.
Any comments?
---
Shawn Castrianni
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
RE: better native code support
Posted by Shawn Castrianni <Sh...@halliburton.com>.
So far only one reply to this post from Xavier. He suggested trying a trigger, or the symlink option on retrieve, or writing an ANT task that is called after retrieve. I thought I would look into trying something but am not sure what would be the most elegant way.
Option 1:
Write some ANT code that gets run right after calling ivy:retrieve that somehow detects which files that need symlinks and running exec to call the ln command. This should work but I need a good way to detect which files need symlinks and what the symlink name should be. I have 2 methods:
Method 1a:
Add published artifacts to the module that will get published with 0 length content. I could name these special artifacts with symlink.<sylmlinkName>.<targetName> such that I could then search for any files with a name that start with symlink and just strip out the name of the symlink from <symlinkName> and point it to the file with <targetName>.
Method 1b:
Add custom extra attributes on the published artifacts for those artifacts that need associated symlinks that provide the name of the symlinks that should point to this artifact. This is what I suggested in this original post. Then the ANT code that would run right after a retrieve would manually parse the XML looking for these special custom extra attributes and perform the link command.
Option 2:
Use triggers combined with custom extra attributes. The custom extra attributes would work just like Method 1b except instead of manually parsing the ivy.xml file, the trigger would be passed the necessary information via ANT properties preventing the need to parse. It would still used the exec task to call the ln command. The problem with this is that I need a post-retrieve-artifact event which doesn't seem to exist. I don't want the post-download-artifact event since I don't want to create the symlink in the cache. I need the symlink down in the retrieved area under ${basedir}.
Anybody have any suggestions for other options?
---
Shawn Castrianni
From: Shawn Castrianni
Sent: Tuesday, July 08, 2008 7:08 AM
To: ivy-user@ant.apache.org
Subject: better native code support
IVY might be thought of as just a tool for managing dependencies for Java code. Perhaps the same thing is thought of about ANT being just a build tool for Java code. However, I am using both ANT and IVY to build modules with both java and native code on multiple platforms with great success. I use IVY's configuration concept to manage the .so's for the various unix platforms and the .dll's for the various windows platforms. Someone building on a windows machine would get the windows dependencies by default and someone building on a unit platform would get the unix dependencies by default. An installer module could override the default and specify * for the configuration and get all platform's dependencies for the DVD image.
One thing that IVY might be able to improve is symbolic link handling. Typically on the unix platform, .so llibraries have their version encoded into the filename like:
libxerces-c.so.26.0
and then there are symbolic links (sometimes more than one) that point to the real file:
libxerces-c.so.26 -> libxerces-c.so.26.0
libxerces-c.so -> libxerces-c.so.26.0
Currently, I am publishing all three files as artifacts but the symbolic links do not survive the round trip from publish to resolve/retrieve. They get converted to 3 identical files wasting disk space in the repository unnecessarily.
Would it be a good idea to perhaps add another attribute to the artifact xml tag called alias which could be a comma delimited list of alias names? Something like:
....
<artifact name="libxerces-c" ext="so.26.0" type="bin" alias="libxerces-c.so.26,libxerces-c.so"/>
....
Then ivy would only publish one artifact with the real filename of libxerces-c.so.26.0 saving disk space in the ivy repository. Then on a resolve/retrieve, it would download the real file and then on unix, create symbolic links for each alias, and on windows, either ignore the aliases or make real file copies.
I don't know if this has any bad ramifications on the rest of IVY functionality. I am just suggesting it as an idea.
Any comments?
---
Shawn Castrianni
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.