You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Scott Goldstein <Sc...@nextlabs.com> on 2010/03/23 01:47:10 UTC
Dealing with Large/Complex Dependencies (e.g. Windows Platform SDK)
I'm trying to come up with a strategy for dealing with large/complex dependencies. Specifically, in this case, I'm working with the Windows Platform SDK. Although there are a number of environment/configuration issues around using the library, the biggest issue that I see is the mere size. There are almost 3000 artifacts (header files, lib files, etc.). I suppose I could enter all of these into an ivy.xml file, but I'd like to avoid this.
Has anyone had to address a similar problem?
Thanks.
Scott
- --------------------------------------------------------------------
STATEMENT OF CONFIDENTIALITY
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. No representation is made on its accuracy or completeness of the information contained in this electronic message. Certain assumptions may have been made in the preparation of this material as at this date, and are subject to change without notice. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. Please reply to the sender at NextLabs Inc and destroy all copies of this message and any attachments from your system. ======================================================================
RE: Dealing with Large/Complex Dependencies (e.g. Windows Platform
SDK)
Posted by Scott Goldstein <Sc...@nextlabs.com>.
Thanks, Shawn. I'll give this a try.
Scott
-----Original Message-----
From: Shawn Castrianni [mailto:Shawn.Castrianni@halliburton.com]
Sent: Monday, March 22, 2010 6:57 PM
To: 'ivy-user@ant.apache.org'
Subject: RE: Dealing with Large/Complex Dependencies (e.g. Windows Platform SDK)
I solve this with "extra" attributes in the ivy.xml file. Instead of publishing each artifact separately for modules with large number of artifacts, I zip them up into groups and publish the zip files. I add my own extra attribute in the ivy.xml file, ext:archive="true", to these zip files. Then in my ANT script that does the ivy retrieve, I post process the dependencies searching for any that have ext:archive="true" and then automatically unzip them and delete the zip file using ANT tasks.
One caveat with this is if you zip up ALL the artifacts into one huge zip file, you limit yourself in how you can use configurations. For example, I use configurations to split up my modules by platform so that I can download windows artifacts separately from linux artifacts. If you zip up both into one big zip file, you can no longer use the configuration option to only retrieve the artifacts for a single platform since the entire zip file would have to be downloaded. Therefore, choose your groups wisely.
You can make up any "extra" attributes that you want to achieve some nice functionality. For example, I have added ext:executable="true" to tell my retrieve post processing code to do a chmod a+x on the artifact when on unix. I have added, for example, ext:alias="libXXX.so" for an artifact, perhaps called libXXX.so.10.1, so that my retrieve post processing code can do a ln -s libXXX.so.10.1 libXXX.so to maintain unix style symlinks.
It would be nice if IVY had these options built in, but it is not hard at all to write your own retrieve post processing code in ANT that searches for these extra attributes in the ivy.xml and perform the necessary commands. Here is the code using XMLTASK to do the chmod mentioned above. I have modified the code to simplify it so I may have introduced a bug so keep that in mind:
<ac:for param="dependencyDir">
<dirset dir="${env.DEPENDENCIES_DIR}">
<include name="*"/>
</dirset>
<sequential>
<xt:xmltask source="@{dependencyDir}/ivy/ivy.xml" failWithoutMatch="false">
<xt:call path="ivy-module/publications/artifact">
<param name="artifactName" path="@name"/>
<param name="artifactExt" path="@ext"/>
<param name="artifactType" path="@type"/>
<param name="artifactExecutable" path="@ext:executable" default="false"/>
<xt:actions>
<!--Process executables that should be set-->
<ac:if>
<and>
<available file="@{dependencyDir}/@{artifactType}/@{artifactName}.@{artifactExt}"/>
<istrue value="@{artifactExecutable}"/>
</and>
<ac:then>
<echo message="Setting executable bit: @{dependencyDir}/@{artifactType}/@{artifactName}.@{artifactExt}"/>
<exec executable="chmod" dir="${basedir}" failonerror="true">
<arg value="a+x"/>
<arg value="@{dependencyDir}/@{artifactType}/@{artifactName}.@{artifactExt}"/>
</exec>
</ac:then>
</ac:if>
</xt:actions>
</xt:call>
</xt:xmltask>
</sequential>
</ac:for>
---
Shawn Castrianni
-----Original Message-----
From: Scott Goldstein [mailto:Scott.Goldstein@nextlabs.com]
Sent: Monday, March 22, 2010 7:47 PM
To: ivy-user@ant.apache.org
Subject: Dealing with Large/Complex Dependencies (e.g. Windows Platform SDK)
I'm trying to come up with a strategy for dealing with large/complex dependencies. Specifically, in this case, I'm working with the Windows Platform SDK. Although there are a number of environment/configuration issues around using the library, the biggest issue that I see is the mere size. There are almost 3000 artifacts (header files, lib files, etc.). I suppose I could enter all of these into an ivy.xml file, but I'd like to avoid this.
Has anyone had to address a similar problem?
Thanks.
Scott
- --------------------------------------------------------------------
STATEMENT OF CONFIDENTIALITY
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. No representation is made on its accuracy or completeness of the information contained in this electronic message. Certain assumptions may have been made in the preparation of this material as at this date, and are subject to change without notice. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. Please reply to the sender at NextLabs Inc and destroy all copies of this message and any attachments from your system. ======================================================================
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
______________________________________________________________________
- --------------------------------------------------------------------
STATEMENT OF CONFIDENTIALITY
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. No representation is made on its accuracy or completeness of the information contained in this electronic message. Certain assumptions may have been made in the preparation of this material as at this date, and are subject to change without notice. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. Please reply to the sender at NextLabs Inc and destroy all copies of this message and any attachments from your system. ======================================================================
RE: Dealing with Large/Complex Dependencies (e.g. Windows Platform
SDK)
Posted by Shawn Castrianni <Sh...@halliburton.com>.
I solve this with "extra" attributes in the ivy.xml file. Instead of publishing each artifact separately for modules with large number of artifacts, I zip them up into groups and publish the zip files. I add my own extra attribute in the ivy.xml file, ext:archive="true", to these zip files. Then in my ANT script that does the ivy retrieve, I post process the dependencies searching for any that have ext:archive="true" and then automatically unzip them and delete the zip file using ANT tasks.
One caveat with this is if you zip up ALL the artifacts into one huge zip file, you limit yourself in how you can use configurations. For example, I use configurations to split up my modules by platform so that I can download windows artifacts separately from linux artifacts. If you zip up both into one big zip file, you can no longer use the configuration option to only retrieve the artifacts for a single platform since the entire zip file would have to be downloaded. Therefore, choose your groups wisely.
You can make up any "extra" attributes that you want to achieve some nice functionality. For example, I have added ext:executable="true" to tell my retrieve post processing code to do a chmod a+x on the artifact when on unix. I have added, for example, ext:alias="libXXX.so" for an artifact, perhaps called libXXX.so.10.1, so that my retrieve post processing code can do a ln -s libXXX.so.10.1 libXXX.so to maintain unix style symlinks.
It would be nice if IVY had these options built in, but it is not hard at all to write your own retrieve post processing code in ANT that searches for these extra attributes in the ivy.xml and perform the necessary commands. Here is the code using XMLTASK to do the chmod mentioned above. I have modified the code to simplify it so I may have introduced a bug so keep that in mind:
<ac:for param="dependencyDir">
<dirset dir="${env.DEPENDENCIES_DIR}">
<include name="*"/>
</dirset>
<sequential>
<xt:xmltask source="@{dependencyDir}/ivy/ivy.xml" failWithoutMatch="false">
<xt:call path="ivy-module/publications/artifact">
<param name="artifactName" path="@name"/>
<param name="artifactExt" path="@ext"/>
<param name="artifactType" path="@type"/>
<param name="artifactExecutable" path="@ext:executable" default="false"/>
<xt:actions>
<!--Process executables that should be set-->
<ac:if>
<and>
<available file="@{dependencyDir}/@{artifactType}/@{artifactName}.@{artifactExt}"/>
<istrue value="@{artifactExecutable}"/>
</and>
<ac:then>
<echo message="Setting executable bit: @{dependencyDir}/@{artifactType}/@{artifactName}.@{artifactExt}"/>
<exec executable="chmod" dir="${basedir}" failonerror="true">
<arg value="a+x"/>
<arg value="@{dependencyDir}/@{artifactType}/@{artifactName}.@{artifactExt}"/>
</exec>
</ac:then>
</ac:if>
</xt:actions>
</xt:call>
</xt:xmltask>
</sequential>
</ac:for>
---
Shawn Castrianni
-----Original Message-----
From: Scott Goldstein [mailto:Scott.Goldstein@nextlabs.com]
Sent: Monday, March 22, 2010 7:47 PM
To: ivy-user@ant.apache.org
Subject: Dealing with Large/Complex Dependencies (e.g. Windows Platform SDK)
I'm trying to come up with a strategy for dealing with large/complex dependencies. Specifically, in this case, I'm working with the Windows Platform SDK. Although there are a number of environment/configuration issues around using the library, the biggest issue that I see is the mere size. There are almost 3000 artifacts (header files, lib files, etc.). I suppose I could enter all of these into an ivy.xml file, but I'd like to avoid this.
Has anyone had to address a similar problem?
Thanks.
Scott
- --------------------------------------------------------------------
STATEMENT OF CONFIDENTIALITY
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. No representation is made on its accuracy or completeness of the information contained in this electronic message. Certain assumptions may have been made in the preparation of this material as at this date, and are subject to change without notice. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. Please reply to the sender at NextLabs Inc and destroy all copies of this message and any attachments from your system. ======================================================================
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Dealing with Large/Complex Dependencies (e.g. Windows Platform
SDK)
Posted by Archie Cobbs <ar...@gmail.com>.
I don't know from Windows so this may be a dumb question... do you have a
tools issue or a data entry issue?
In other words, if you have to type in 3000 things by hand, it doesn't
really matter what tool you're using, it's going to suck. On the other hand,
if you have the dependency information already electronic, then with some
scripting/programming you should be able to transform that into ivy files
(or whatever else) fairly easily...
-Archie
On Mon, Mar 22, 2010 at 7:47 PM, Scott Goldstein <
Scott.Goldstein@nextlabs.com> wrote:
> I'm trying to come up with a strategy for dealing with large/complex
> dependencies. Specifically, in this case, I'm working with the Windows
> Platform SDK. Although there are a number of environment/configuration
> issues around using the library, the biggest issue that I see is the mere
> size. There are almost 3000 artifacts (header files, lib files, etc.). I
> suppose I could enter all of these into an ivy.xml file, but I'd like to
> avoid this.
>
> Has anyone had to address a similar problem?
>
> Thanks.
>
> Scott
>
> - --------------------------------------------------------------------
> STATEMENT OF CONFIDENTIALITY
>
> The information contained in this electronic message and any attachments to
> this message are intended for the exclusive use of the addressee(s) and may
> contain confidential or privileged information. No representation is made on
> its accuracy or completeness of the information contained in this electronic
> message. Certain assumptions may have been made in the preparation of this
> material as at this date, and are subject to change without notice. If you
> are not the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this e-mail and any attachment(s)
> is strictly prohibited. Please reply to the sender at NextLabs Inc and
> destroy all copies of this message and any attachments from your system.
> ======================================================================
>
--
Archie L. Cobbs