You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Felix Drueke <fd...@orga-systems.com> on 2011/01/12 11:58:30 UTC
Publishing C-module artifacts from different platforms
Hi,
I'd like to manage the dependencies of a set of C-modules with ivy.
Each of these modules needs to be compiled on different platforms (i.e.
Solaris, Linux, HP-UX and others).
This requires that the same sources are compiled in separate workspaces
(on the different hosts).
In a first attempt I setup one ivy.xml with a set of configurations - one for
each platform on that I have to build&publish artifacts.
Being unexperienced with Ivy I didnd't find a way to just publish the
artifact for the one configuration that I built in a workspace.
For example I built module XYZ on Solaris and of course I can only
publish the Solaris-artifact from that workspace. In another worlspace
I got the artifact for Linux and of course I can only publish that.
How do you accomplish to publish just the artifacts for one particular
configuration?
Thanks for any hint,
Felix
The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.
Re: Publishing C-module artifacts from different platforms
Posted by Archie Cobbs <ar...@dellroad.org>.
I don't know of any way to "partially publish". You might need a two-step
process, where the second step gathers the artifacts built on the various
platforms together (and temporarily stored somewhere) and then does the
actual ivy publishing.
-Archie
2011/1/12 Felix Drueke <fd...@orga-systems.com>
> Hi,
>
> I'd like to manage the dependencies of a set of C-modules with ivy.
> Each of these modules needs to be compiled on different platforms (i.e.
> Solaris, Linux, HP-UX and others).
>
> This requires that the same sources are compiled in separate workspaces
> (on the different hosts).
>
> In a first attempt I setup one ivy.xml with a set of configurations - one
> for
> each platform on that I have to build&publish artifacts.
>
> Being unexperienced with Ivy I didnd't find a way to just publish the
> artifact for the one configuration that I built in a workspace.
> For example I built module XYZ on Solaris and of course I can only
> publish the Solaris-artifact from that workspace. In another worlspace
> I got the artifact for Linux and of course I can only publish that.
>
> How do you accomplish to publish just the artifacts for one particular
> configuration?
>
> Thanks for any hint,
> Felix
>
> The information included in this e-mail and any files transmitted with it
> is strictly confidential and may be privileged or otherwise protected from
> disclosure. If you are not the intended recipient, please notify the sender
> immediately by e-mail and delete this e-mail as well as any attachment from
> your system. If you are not the intended recipient you are not authorized to
> use and/or copy this message and/or attachment and/or disclose the contents
> to any other person.
>
--
Archie L. Cobbs
Re: Publishing C-module artifacts from different platforms
Posted by Tim Brown <tp...@gmail.com>.
I'd vote for Go, but I've been an influence in it's development ;-) But
yeah, Hudson and such have greatly improved too.
Currently on CruiseControl handling >30,000 builds a month. 68 pipelines,
with continuous deployment etc etc etc.
We'd looked at QB too (ages ago, 3+ years sounds right), but it didn't suit
our needs.
Anyway, that's off-topic.
For the intermediate artifacts we'd started off down the local publish path,
but have since switched to using an integration versions. The integration
version supersedes the newest "released" version and pops the released off
the stack.
Jacob (cc'ed) can probably give the sorted details if it's of use.
On Thu, Jan 13, 2011 at 8:10 AM, Shawn Castrianni <
Shawn.Castrianni@halliburton.com> wrote:
> I tried Bamboo and Hudson 2 years ago when I was evaluating all of the
> different tools. I found QuickBuild to be, by far, the easiest to setup my
> complicated build pipeline. Some systems, I had to resort to writing shell
> commands to do the actual copy of artifacts from one build machine to
> another. QuickBuild has this concept built in to its default steps.
> Anyway, I strongly urge everyone to take a look at QuickBuild 3.1. It is
> free until you start needing lots of build configurations. Again, sorry for
> the QuickBuild commercial on an IVY mailing list. Perhaps Bamboo and Hudson
> have made this easier in their newer versions, haven't looked at them
> recently.
>
> ---
> Shawn Castrianni
>
>
> -----Original Message-----
> From: Tim Brown [mailto:tpbrown@gmail.com]
> Sent: Thursday, January 13, 2011 9:21 AM
> To: ivy-user@ant.apache.org
> Subject: Re: Publishing C-module artifacts from different platforms
>
> Felix,
>
> But if it's a single source base, for multiple platforms, wouldn't building
> and publishing for a single platform be considered not fully tested?
>
> If the developer wants to share his WIP code with another, why wouldn't he
> just use a patch instead? [or, build it for the other platforms, test it,
> then commit it all so that CI does it's job]
>
> side-note: yeah, you can do the build pipeline similar to Shawn's with
> Hudson [and Go, and Bamboo, etc.]
>
> On Wed, Jan 12, 2011 at 11:23 PM, Felix Drueke <fdrueke@orga-systems.com
> >wrote:
>
> > Shawn, thanks for that valuable input.
> >
> > I assume we're going for a pretty similar procedure.
> > We're using Hudson to verify that the build still works on all platforms
> > and compilers.
> > So I guess it's also possible to have Hudson publish the artifacts right
> > away
> > (didn't try that yet).
> >
> > However there's still one reason why I think that artifacts for just one
> > platform should be publishable (at least for dev-snapshots).
> > That is if a developer wants to publish the results of his latest work
> > in his current workspace. In that case he'll just want to publish for
> > just one platform and that's sufficient in that situation.
> >
> > Felix
> >
> >
> > Shawn Castrianni wrote, on 01/12/2011 05:19 PM:
> >
> > I can't answer your question, but I can tell you how I handle this.
> >>
> >> I felt it was a bad idea to publish a partial module such that one
> >> platform's artifacts are present and other are not. If this happens
> because
> >> of a compiler error on one platform but not others, then you can get
> into a
> >> bad situation. Such as, someone saying that my new feature works on
> linux
> >> but not on windows, not knowing that it works on linux because the new
> >> feature was compiled successfully on that platform and he is using the
> new
> >> published module. But when he tries on windows, he is picking up on old
> >> published module since a new one with windows artifacts is not present
> due
> >> to the compile error. Stuff like that.
> >>
> >> Therefore, what I do is just as you describe, using separate
> configuration
> >> for each platform. But then I use a good automated build system
> (QuickBuild
> >> 3.1) to wrap ivy commands. So my QB build workflow is:
> >>
> >> 0. randomly select a master build machine from the pool, let's call it
> >> MASTER
> >> 1. MASTER: checkout source on any machine
> >> 2. MASTER: tar up source
> >> 3. MASTER: copy source to other platform build machines
> >> 4. EACH PLATFORM: untar onto each of those machines
> >> 5. MASTER: delete tar file
> >> 6. MASTER: clean.all
> >> 7. MASTER: ivy dependencies (specifying proper conf setting so that All
> >> platforms dependency artifacts are downloaded)
> >> 8. MASTER: perform java compile
> >> 9. MASTER: copy dependencies and resulting jars from java compile to
> >> other platform build machines
> >> 10: EACH PLATFORM: perform native code build
> >> 11: EACH PLATFORM: perform unit testing
> >> 12: EACH PLATFORM: copy resulting artifacts from each platform machine
> to
> >> master machine
> >> 13: MASTER: publish all platform artifacts at once
> >>
> >> By using a good automated build system, you can easily handle this multi
> >> platform build workflow and the hand off of control from one node to
> >> another. Then the publishing ONLY happens if ALL platforms are
> successful.
> >> Therefore, I do not need the ability to publish a partial module.
> >>
> >> I am sure there are other ways to do this and there is probably debate
> as
> >> to how efficient this is, but it works very well in a production
> >> environment. If this sounds like an advertisement for QuickBuild, I
> >> apologize. I do not work for them. I am just a passionate happy
> customer.
> >>
> >> ---
> >> Shawn Castrianni
> >>
> >> -----Original Message-----
> >> From: Felix Drueke [mailto:fdrueke@orga-systems.com]
> >> Sent: Wednesday, January 12, 2011 4:59 AM
> >> To: ivy-user@ant.apache.org
> >> Subject: Publishing C-module artifacts from different platforms
> >>
> >> Hi,
> >>
> >> I'd like to manage the dependencies of a set of C-modules with ivy.
> >> Each of these modules needs to be compiled on different platforms (i.e.
> >> Solaris, Linux, HP-UX and others).
> >>
> >> This requires that the same sources are compiled in separate workspaces
> >> (on the different hosts).
> >>
> >> In a first attempt I setup one ivy.xml with a set of configurations -
> one
> >> for
> >> each platform on that I have to build&publish artifacts.
> >>
> >> Being unexperienced with Ivy I didnd't find a way to just publish the
> >> artifact for the one configuration that I built in a workspace.
> >> For example I built module XYZ on Solaris and of course I can only
> >> publish the Solaris-artifact from that workspace. In another worlspace
> >> I got the artifact for Linux and of course I can only publish that.
> >>
> >> How do you accomplish to publish just the artifacts for one particular
> >> configuration?
> >>
> >> Thanks for any hint,
> >> Felix
> >>
> >> The information included in this e-mail and any files transmitted with
> it
> >> is strictly confidential and may be privileged or otherwise protected
> from
> >> disclosure. If you are not the intended recipient, please notify the
> sender
> >> immediately by e-mail and delete this e-mail as well as any attachment
> from
> >> your system. If you are not the intended recipient you are not
> authorized to
> >> use and/or copy this message and/or attachment and/or disclose the
> contents
> >> to any other person.
> >>
> >> ----------------------------------------------------------------------
> >> This e-mail, including any attached files, may contain confidential and
> >> privileged information for the sole use of the intended recipient. Any
> >> review, use, distribution, or disclosure by others is strictly
> prohibited.
> >> If you are not the intended recipient (or authorized to receive
> information
> >> for the intended recipient), please contact the sender by reply e-mail
> and
> >> delete all copies of this message.
> >>
> >>
> > The information included in this e-mail and any files transmitted with it
> > is strictly confidential and may be privileged or otherwise protected
> from
> > disclosure. If you are not the intended recipient, please notify the
> sender
> > immediately by e-mail and delete this e-mail as well as any attachment
> from
> > your system. If you are not the intended recipient you are not authorized
> to
> > use and/or copy this message and/or attachment and/or disclose the
> contents
> > to any other person.
> >
>
RE: Publishing C-module artifacts from different platforms
Posted by Shawn Castrianni <Sh...@halliburton.com>.
I tried Bamboo and Hudson 2 years ago when I was evaluating all of the different tools. I found QuickBuild to be, by far, the easiest to setup my complicated build pipeline. Some systems, I had to resort to writing shell commands to do the actual copy of artifacts from one build machine to another. QuickBuild has this concept built in to its default steps. Anyway, I strongly urge everyone to take a look at QuickBuild 3.1. It is free until you start needing lots of build configurations. Again, sorry for the QuickBuild commercial on an IVY mailing list. Perhaps Bamboo and Hudson have made this easier in their newer versions, haven't looked at them recently.
---
Shawn Castrianni
-----Original Message-----
From: Tim Brown [mailto:tpbrown@gmail.com]
Sent: Thursday, January 13, 2011 9:21 AM
To: ivy-user@ant.apache.org
Subject: Re: Publishing C-module artifacts from different platforms
Felix,
But if it's a single source base, for multiple platforms, wouldn't building
and publishing for a single platform be considered not fully tested?
If the developer wants to share his WIP code with another, why wouldn't he
just use a patch instead? [or, build it for the other platforms, test it,
then commit it all so that CI does it's job]
side-note: yeah, you can do the build pipeline similar to Shawn's with
Hudson [and Go, and Bamboo, etc.]
On Wed, Jan 12, 2011 at 11:23 PM, Felix Drueke <fd...@orga-systems.com>wrote:
> Shawn, thanks for that valuable input.
>
> I assume we're going for a pretty similar procedure.
> We're using Hudson to verify that the build still works on all platforms
> and compilers.
> So I guess it's also possible to have Hudson publish the artifacts right
> away
> (didn't try that yet).
>
> However there's still one reason why I think that artifacts for just one
> platform should be publishable (at least for dev-snapshots).
> That is if a developer wants to publish the results of his latest work
> in his current workspace. In that case he'll just want to publish for
> just one platform and that's sufficient in that situation.
>
> Felix
>
>
> Shawn Castrianni wrote, on 01/12/2011 05:19 PM:
>
> I can't answer your question, but I can tell you how I handle this.
>>
>> I felt it was a bad idea to publish a partial module such that one
>> platform's artifacts are present and other are not. If this happens because
>> of a compiler error on one platform but not others, then you can get into a
>> bad situation. Such as, someone saying that my new feature works on linux
>> but not on windows, not knowing that it works on linux because the new
>> feature was compiled successfully on that platform and he is using the new
>> published module. But when he tries on windows, he is picking up on old
>> published module since a new one with windows artifacts is not present due
>> to the compile error. Stuff like that.
>>
>> Therefore, what I do is just as you describe, using separate configuration
>> for each platform. But then I use a good automated build system (QuickBuild
>> 3.1) to wrap ivy commands. So my QB build workflow is:
>>
>> 0. randomly select a master build machine from the pool, let's call it
>> MASTER
>> 1. MASTER: checkout source on any machine
>> 2. MASTER: tar up source
>> 3. MASTER: copy source to other platform build machines
>> 4. EACH PLATFORM: untar onto each of those machines
>> 5. MASTER: delete tar file
>> 6. MASTER: clean.all
>> 7. MASTER: ivy dependencies (specifying proper conf setting so that All
>> platforms dependency artifacts are downloaded)
>> 8. MASTER: perform java compile
>> 9. MASTER: copy dependencies and resulting jars from java compile to
>> other platform build machines
>> 10: EACH PLATFORM: perform native code build
>> 11: EACH PLATFORM: perform unit testing
>> 12: EACH PLATFORM: copy resulting artifacts from each platform machine to
>> master machine
>> 13: MASTER: publish all platform artifacts at once
>>
>> By using a good automated build system, you can easily handle this multi
>> platform build workflow and the hand off of control from one node to
>> another. Then the publishing ONLY happens if ALL platforms are successful.
>> Therefore, I do not need the ability to publish a partial module.
>>
>> I am sure there are other ways to do this and there is probably debate as
>> to how efficient this is, but it works very well in a production
>> environment. If this sounds like an advertisement for QuickBuild, I
>> apologize. I do not work for them. I am just a passionate happy customer.
>>
>> ---
>> Shawn Castrianni
>>
>> -----Original Message-----
>> From: Felix Drueke [mailto:fdrueke@orga-systems.com]
>> Sent: Wednesday, January 12, 2011 4:59 AM
>> To: ivy-user@ant.apache.org
>> Subject: Publishing C-module artifacts from different platforms
>>
>> Hi,
>>
>> I'd like to manage the dependencies of a set of C-modules with ivy.
>> Each of these modules needs to be compiled on different platforms (i.e.
>> Solaris, Linux, HP-UX and others).
>>
>> This requires that the same sources are compiled in separate workspaces
>> (on the different hosts).
>>
>> In a first attempt I setup one ivy.xml with a set of configurations - one
>> for
>> each platform on that I have to build&publish artifacts.
>>
>> Being unexperienced with Ivy I didnd't find a way to just publish the
>> artifact for the one configuration that I built in a workspace.
>> For example I built module XYZ on Solaris and of course I can only
>> publish the Solaris-artifact from that workspace. In another worlspace
>> I got the artifact for Linux and of course I can only publish that.
>>
>> How do you accomplish to publish just the artifacts for one particular
>> configuration?
>>
>> Thanks for any hint,
>> Felix
>>
>> The information included in this e-mail and any files transmitted with it
>> is strictly confidential and may be privileged or otherwise protected from
>> disclosure. If you are not the intended recipient, please notify the sender
>> immediately by e-mail and delete this e-mail as well as any attachment from
>> your system. If you are not the intended recipient you are not authorized to
>> use and/or copy this message and/or attachment and/or disclose the contents
>> to any other person.
>>
>> ----------------------------------------------------------------------
>> This e-mail, including any attached files, may contain confidential and
>> privileged information for the sole use of the intended recipient. Any
>> review, use, distribution, or disclosure by others is strictly prohibited.
>> If you are not the intended recipient (or authorized to receive information
>> for the intended recipient), please contact the sender by reply e-mail and
>> delete all copies of this message.
>>
>>
> The information included in this e-mail and any files transmitted with it
> is strictly confidential and may be privileged or otherwise protected from
> disclosure. If you are not the intended recipient, please notify the sender
> immediately by e-mail and delete this e-mail as well as any attachment from
> your system. If you are not the intended recipient you are not authorized to
> use and/or copy this message and/or attachment and/or disclose the contents
> to any other person.
>
Re: Publishing C-module artifacts from different platforms
Posted by Tim Brown <tp...@gmail.com>.
Felix,
But if it's a single source base, for multiple platforms, wouldn't building
and publishing for a single platform be considered not fully tested?
If the developer wants to share his WIP code with another, why wouldn't he
just use a patch instead? [or, build it for the other platforms, test it,
then commit it all so that CI does it's job]
side-note: yeah, you can do the build pipeline similar to Shawn's with
Hudson [and Go, and Bamboo, etc.]
On Wed, Jan 12, 2011 at 11:23 PM, Felix Drueke <fd...@orga-systems.com>wrote:
> Shawn, thanks for that valuable input.
>
> I assume we're going for a pretty similar procedure.
> We're using Hudson to verify that the build still works on all platforms
> and compilers.
> So I guess it's also possible to have Hudson publish the artifacts right
> away
> (didn't try that yet).
>
> However there's still one reason why I think that artifacts for just one
> platform should be publishable (at least for dev-snapshots).
> That is if a developer wants to publish the results of his latest work
> in his current workspace. In that case he'll just want to publish for
> just one platform and that's sufficient in that situation.
>
> Felix
>
>
> Shawn Castrianni wrote, on 01/12/2011 05:19 PM:
>
> I can't answer your question, but I can tell you how I handle this.
>>
>> I felt it was a bad idea to publish a partial module such that one
>> platform's artifacts are present and other are not. If this happens because
>> of a compiler error on one platform but not others, then you can get into a
>> bad situation. Such as, someone saying that my new feature works on linux
>> but not on windows, not knowing that it works on linux because the new
>> feature was compiled successfully on that platform and he is using the new
>> published module. But when he tries on windows, he is picking up on old
>> published module since a new one with windows artifacts is not present due
>> to the compile error. Stuff like that.
>>
>> Therefore, what I do is just as you describe, using separate configuration
>> for each platform. But then I use a good automated build system (QuickBuild
>> 3.1) to wrap ivy commands. So my QB build workflow is:
>>
>> 0. randomly select a master build machine from the pool, let's call it
>> MASTER
>> 1. MASTER: checkout source on any machine
>> 2. MASTER: tar up source
>> 3. MASTER: copy source to other platform build machines
>> 4. EACH PLATFORM: untar onto each of those machines
>> 5. MASTER: delete tar file
>> 6. MASTER: clean.all
>> 7. MASTER: ivy dependencies (specifying proper conf setting so that All
>> platforms dependency artifacts are downloaded)
>> 8. MASTER: perform java compile
>> 9. MASTER: copy dependencies and resulting jars from java compile to
>> other platform build machines
>> 10: EACH PLATFORM: perform native code build
>> 11: EACH PLATFORM: perform unit testing
>> 12: EACH PLATFORM: copy resulting artifacts from each platform machine to
>> master machine
>> 13: MASTER: publish all platform artifacts at once
>>
>> By using a good automated build system, you can easily handle this multi
>> platform build workflow and the hand off of control from one node to
>> another. Then the publishing ONLY happens if ALL platforms are successful.
>> Therefore, I do not need the ability to publish a partial module.
>>
>> I am sure there are other ways to do this and there is probably debate as
>> to how efficient this is, but it works very well in a production
>> environment. If this sounds like an advertisement for QuickBuild, I
>> apologize. I do not work for them. I am just a passionate happy customer.
>>
>> ---
>> Shawn Castrianni
>>
>> -----Original Message-----
>> From: Felix Drueke [mailto:fdrueke@orga-systems.com]
>> Sent: Wednesday, January 12, 2011 4:59 AM
>> To: ivy-user@ant.apache.org
>> Subject: Publishing C-module artifacts from different platforms
>>
>> Hi,
>>
>> I'd like to manage the dependencies of a set of C-modules with ivy.
>> Each of these modules needs to be compiled on different platforms (i.e.
>> Solaris, Linux, HP-UX and others).
>>
>> This requires that the same sources are compiled in separate workspaces
>> (on the different hosts).
>>
>> In a first attempt I setup one ivy.xml with a set of configurations - one
>> for
>> each platform on that I have to build&publish artifacts.
>>
>> Being unexperienced with Ivy I didnd't find a way to just publish the
>> artifact for the one configuration that I built in a workspace.
>> For example I built module XYZ on Solaris and of course I can only
>> publish the Solaris-artifact from that workspace. In another worlspace
>> I got the artifact for Linux and of course I can only publish that.
>>
>> How do you accomplish to publish just the artifacts for one particular
>> configuration?
>>
>> Thanks for any hint,
>> Felix
>>
>> The information included in this e-mail and any files transmitted with it
>> is strictly confidential and may be privileged or otherwise protected from
>> disclosure. If you are not the intended recipient, please notify the sender
>> immediately by e-mail and delete this e-mail as well as any attachment from
>> your system. If you are not the intended recipient you are not authorized to
>> use and/or copy this message and/or attachment and/or disclose the contents
>> to any other person.
>>
>> ----------------------------------------------------------------------
>> This e-mail, including any attached files, may contain confidential and
>> privileged information for the sole use of the intended recipient. Any
>> review, use, distribution, or disclosure by others is strictly prohibited.
>> If you are not the intended recipient (or authorized to receive information
>> for the intended recipient), please contact the sender by reply e-mail and
>> delete all copies of this message.
>>
>>
> The information included in this e-mail and any files transmitted with it
> is strictly confidential and may be privileged or otherwise protected from
> disclosure. If you are not the intended recipient, please notify the sender
> immediately by e-mail and delete this e-mail as well as any attachment from
> your system. If you are not the intended recipient you are not authorized to
> use and/or copy this message and/or attachment and/or disclose the contents
> to any other person.
>
RE: Publishing C-module artifacts from different platforms
Posted by Shawn Castrianni <Sh...@halliburton.com>.
For that case, I have used a "local" publish. All of my ivy settings files have a chain of resolvers. The very first one in the chain is always the local resolver which just publishes to a local directory on the developer's computer. When he runs dependencies, it will look there first before trying to find the official one with all platform artifacts in the shared repository for the company. This local publishing specifies only the current platform configuration so only the current platform artifacts that the dev just built are published to this other place on his hard drive. He could then go to another module sandbox that is dependent on the one he just built and locally published, run dependencies, and the locally built artifacts come in as a dependency allowing him to write code against changes he made in that other sandbox. Essentially, what this gives to the dev is the ability to develop in multiple sandbox modules with dependencies between each other and test his changes across modules before committing any code changes to any of their respective source repositories.
Here is the chain I use in production for all 500 of my developers:
<!--Chain resolvers-->
<chain name="published.repository.daily.chain" returnFirst="true">
<resolver ref="published.local"/>
<resolver ref="published.repository.daily"/>
<resolver ref="thirdparty"/>
</chain>
<filesystem name="published.local" force="true" cache="published.local.cache">
<ivy pattern="${env.PUBLISH_LOCAL_ROOT}/${ivy.pattern}"/>
<artifact pattern="${env.PUBLISH_LOCAL_ROOT}/${artifact.pattern}"/>
</filesystem>
<filesystem name="published.repository.daily" force="${published.force}">
<ivy pattern="${ivy.published.root}/daily/${env.RELEASE_NAME}/${ivy.pattern}"/>
<artifact pattern="${ivy.published.root}/daily/${env.RELEASE_NAME}/${artifact.pattern}"/>
</filesystem>
<svn name="thirdparty" repositoryRoot="http://thirdparty.src.lgc.com/anonymous/thirdparty/trunk" userName="renobuild">
<ivy pattern="[organisation]/[module]/[revision]/[type]/[artifact](.[ext])"/>
<artifact pattern="[organisation]/[module]/[revision]/[type]/[artifact](.[ext])"/>
</svn>
You can see that for the local publish resolver, I use ${env.PUBLISH_LOCAL_ROOT} in the pattern. This gives the dev the choice of where he wants his local publish repository to be located on his computer by simply setting the env variable PUBLISH_LOCAL_ROOT. You can then see that the main official repository from the official CM builds come from published.repository.daily. And then if the dependency is not found in either of those, then it looks in the SVN based resolver called thirdparty where we store all of our thirdparty dependencies. Sure, we could have used biblio or ivyrep resolvers to grab thirdparty stuff from the Internet, but our legal team is very strict on the use of thirdparty components. Therefore, each one has to be approved. Once approved, I copy it into our own svn thirdparty repository.
---
Shawn Castrianni
-----Original Message-----
From: Felix Drueke [mailto:fdrueke@orga-systems.com]
Sent: Thursday, January 13, 2011 1:23 AM
To: ivy-user@ant.apache.org
Subject: Re: Publishing C-module artifacts from different platforms
Shawn, thanks for that valuable input.
I assume we're going for a pretty similar procedure.
We're using Hudson to verify that the build still works on all platforms
and compilers.
So I guess it's also possible to have Hudson publish the artifacts right away
(didn't try that yet).
However there's still one reason why I think that artifacts for just one
platform should be publishable (at least for dev-snapshots).
That is if a developer wants to publish the results of his latest work
in his current workspace. In that case he'll just want to publish for
just one platform and that's sufficient in that situation.
Felix
Shawn Castrianni wrote, on 01/12/2011 05:19 PM:
> I can't answer your question, but I can tell you how I handle this.
>
> I felt it was a bad idea to publish a partial module such that one platform's artifacts are present and other are not. If this happens because of a compiler error on one platform but not others, then you can get into a bad situation. Such as, someone saying that my new feature works on linux but not on windows, not knowing that it works on linux because the new feature was compiled successfully on that platform and he is using the new published module. But when he tries on windows, he is picking up on old published module since a new one with windows artifacts is not present due to the compile error. Stuff like that.
>
> Therefore, what I do is just as you describe, using separate configuration for each platform. But then I use a good automated build system (QuickBuild 3.1) to wrap ivy commands. So my QB build workflow is:
>
> 0. randomly select a master build machine from the pool, let's call it MASTER
> 1. MASTER: checkout source on any machine
> 2. MASTER: tar up source
> 3. MASTER: copy source to other platform build machines
> 4. EACH PLATFORM: untar onto each of those machines
> 5. MASTER: delete tar file
> 6. MASTER: clean.all
> 7. MASTER: ivy dependencies (specifying proper conf setting so that All platforms dependency artifacts are downloaded)
> 8. MASTER: perform java compile
> 9. MASTER: copy dependencies and resulting jars from java compile to other platform build machines
> 10: EACH PLATFORM: perform native code build
> 11: EACH PLATFORM: perform unit testing
> 12: EACH PLATFORM: copy resulting artifacts from each platform machine to master machine
> 13: MASTER: publish all platform artifacts at once
>
> By using a good automated build system, you can easily handle this multi platform build workflow and the hand off of control from one node to another. Then the publishing ONLY happens if ALL platforms are successful. Therefore, I do not need the ability to publish a partial module.
>
> I am sure there are other ways to do this and there is probably debate as to how efficient this is, but it works very well in a production environment. If this sounds like an advertisement for QuickBuild, I apologize. I do not work for them. I am just a passionate happy customer.
>
> ---
> Shawn Castrianni
>
> -----Original Message-----
> From: Felix Drueke [mailto:fdrueke@orga-systems.com]
> Sent: Wednesday, January 12, 2011 4:59 AM
> To: ivy-user@ant.apache.org
> Subject: Publishing C-module artifacts from different platforms
>
> Hi,
>
> I'd like to manage the dependencies of a set of C-modules with ivy.
> Each of these modules needs to be compiled on different platforms (i.e.
> Solaris, Linux, HP-UX and others).
>
> This requires that the same sources are compiled in separate workspaces
> (on the different hosts).
>
> In a first attempt I setup one ivy.xml with a set of configurations - one for
> each platform on that I have to build&publish artifacts.
>
> Being unexperienced with Ivy I didnd't find a way to just publish the
> artifact for the one configuration that I built in a workspace.
> For example I built module XYZ on Solaris and of course I can only
> publish the Solaris-artifact from that workspace. In another worlspace
> I got the artifact for Linux and of course I can only publish that.
>
> How do you accomplish to publish just the artifacts for one particular
> configuration?
>
> Thanks for any hint,
> Felix
>
> The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.
>
> ----------------------------------------------------------------------
> This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
>
The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.
Re: Publishing C-module artifacts from different platforms
Posted by Felix Drueke <fd...@orga-systems.com>.
Shawn, thanks for that valuable input.
I assume we're going for a pretty similar procedure.
We're using Hudson to verify that the build still works on all platforms
and compilers.
So I guess it's also possible to have Hudson publish the artifacts right away
(didn't try that yet).
However there's still one reason why I think that artifacts for just one
platform should be publishable (at least for dev-snapshots).
That is if a developer wants to publish the results of his latest work
in his current workspace. In that case he'll just want to publish for
just one platform and that's sufficient in that situation.
Felix
Shawn Castrianni wrote, on 01/12/2011 05:19 PM:
> I can't answer your question, but I can tell you how I handle this.
>
> I felt it was a bad idea to publish a partial module such that one platform's artifacts are present and other are not. If this happens because of a compiler error on one platform but not others, then you can get into a bad situation. Such as, someone saying that my new feature works on linux but not on windows, not knowing that it works on linux because the new feature was compiled successfully on that platform and he is using the new published module. But when he tries on windows, he is picking up on old published module since a new one with windows artifacts is not present due to the compile error. Stuff like that.
>
> Therefore, what I do is just as you describe, using separate configuration for each platform. But then I use a good automated build system (QuickBuild 3.1) to wrap ivy commands. So my QB build workflow is:
>
> 0. randomly select a master build machine from the pool, let's call it MASTER
> 1. MASTER: checkout source on any machine
> 2. MASTER: tar up source
> 3. MASTER: copy source to other platform build machines
> 4. EACH PLATFORM: untar onto each of those machines
> 5. MASTER: delete tar file
> 6. MASTER: clean.all
> 7. MASTER: ivy dependencies (specifying proper conf setting so that All platforms dependency artifacts are downloaded)
> 8. MASTER: perform java compile
> 9. MASTER: copy dependencies and resulting jars from java compile to other platform build machines
> 10: EACH PLATFORM: perform native code build
> 11: EACH PLATFORM: perform unit testing
> 12: EACH PLATFORM: copy resulting artifacts from each platform machine to master machine
> 13: MASTER: publish all platform artifacts at once
>
> By using a good automated build system, you can easily handle this multi platform build workflow and the hand off of control from one node to another. Then the publishing ONLY happens if ALL platforms are successful. Therefore, I do not need the ability to publish a partial module.
>
> I am sure there are other ways to do this and there is probably debate as to how efficient this is, but it works very well in a production environment. If this sounds like an advertisement for QuickBuild, I apologize. I do not work for them. I am just a passionate happy customer.
>
> ---
> Shawn Castrianni
>
> -----Original Message-----
> From: Felix Drueke [mailto:fdrueke@orga-systems.com]
> Sent: Wednesday, January 12, 2011 4:59 AM
> To: ivy-user@ant.apache.org
> Subject: Publishing C-module artifacts from different platforms
>
> Hi,
>
> I'd like to manage the dependencies of a set of C-modules with ivy.
> Each of these modules needs to be compiled on different platforms (i.e.
> Solaris, Linux, HP-UX and others).
>
> This requires that the same sources are compiled in separate workspaces
> (on the different hosts).
>
> In a first attempt I setup one ivy.xml with a set of configurations - one for
> each platform on that I have to build&publish artifacts.
>
> Being unexperienced with Ivy I didnd't find a way to just publish the
> artifact for the one configuration that I built in a workspace.
> For example I built module XYZ on Solaris and of course I can only
> publish the Solaris-artifact from that workspace. In another worlspace
> I got the artifact for Linux and of course I can only publish that.
>
> How do you accomplish to publish just the artifacts for one particular
> configuration?
>
> Thanks for any hint,
> Felix
>
> The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.
>
> ----------------------------------------------------------------------
> This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
>
The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.
RE: Publishing C-module artifacts from different platforms
Posted by Shawn Castrianni <Sh...@halliburton.com>.
I can't answer your question, but I can tell you how I handle this.
I felt it was a bad idea to publish a partial module such that one platform's artifacts are present and other are not. If this happens because of a compiler error on one platform but not others, then you can get into a bad situation. Such as, someone saying that my new feature works on linux but not on windows, not knowing that it works on linux because the new feature was compiled successfully on that platform and he is using the new published module. But when he tries on windows, he is picking up on old published module since a new one with windows artifacts is not present due to the compile error. Stuff like that.
Therefore, what I do is just as you describe, using separate configuration for each platform. But then I use a good automated build system (QuickBuild 3.1) to wrap ivy commands. So my QB build workflow is:
0. randomly select a master build machine from the pool, let's call it MASTER
1. MASTER: checkout source on any machine
2. MASTER: tar up source
3. MASTER: copy source to other platform build machines
4. EACH PLATFORM: untar onto each of those machines
5. MASTER: delete tar file
6. MASTER: clean.all
7. MASTER: ivy dependencies (specifying proper conf setting so that All platforms dependency artifacts are downloaded)
8. MASTER: perform java compile
9. MASTER: copy dependencies and resulting jars from java compile to other platform build machines
10: EACH PLATFORM: perform native code build
11: EACH PLATFORM: perform unit testing
12: EACH PLATFORM: copy resulting artifacts from each platform machine to master machine
13: MASTER: publish all platform artifacts at once
By using a good automated build system, you can easily handle this multi platform build workflow and the hand off of control from one node to another. Then the publishing ONLY happens if ALL platforms are successful. Therefore, I do not need the ability to publish a partial module.
I am sure there are other ways to do this and there is probably debate as to how efficient this is, but it works very well in a production environment. If this sounds like an advertisement for QuickBuild, I apologize. I do not work for them. I am just a passionate happy customer.
---
Shawn Castrianni
-----Original Message-----
From: Felix Drueke [mailto:fdrueke@orga-systems.com]
Sent: Wednesday, January 12, 2011 4:59 AM
To: ivy-user@ant.apache.org
Subject: Publishing C-module artifacts from different platforms
Hi,
I'd like to manage the dependencies of a set of C-modules with ivy.
Each of these modules needs to be compiled on different platforms (i.e.
Solaris, Linux, HP-UX and others).
This requires that the same sources are compiled in separate workspaces
(on the different hosts).
In a first attempt I setup one ivy.xml with a set of configurations - one for
each platform on that I have to build&publish artifacts.
Being unexperienced with Ivy I didnd't find a way to just publish the
artifact for the one configuration that I built in a workspace.
For example I built module XYZ on Solaris and of course I can only
publish the Solaris-artifact from that workspace. In another worlspace
I got the artifact for Linux and of course I can only publish that.
How do you accomplish to publish just the artifacts for one particular
configuration?
Thanks for any hint,
Felix
The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Publishing C-module artifacts from different platforms
Posted by Felix Drueke <fd...@orga-systems.com>.
I already found the cause of my problem.
Apparently it is not a problem to publish just the artifacts for specific
configurations. I just mixed up things when trying to call ivy standalone
(from Makefiles)
Felix Drueke wrote, on 01/12/2011 11:58 AM:
> Hi,
>
> I'd like to manage the dependencies of a set of C-modules with ivy.
> Each of these modules needs to be compiled on different platforms (i.e.
> Solaris, Linux, HP-UX and others).
>
> This requires that the same sources are compiled in separate workspaces
> (on the different hosts).
>
> In a first attempt I setup one ivy.xml with a set of configurations - one for
> each platform on that I have to build&publish artifacts.
>
> Being unexperienced with Ivy I didnd't find a way to just publish the
> artifact for the one configuration that I built in a workspace.
> For example I built module XYZ on Solaris and of course I can only
> publish the Solaris-artifact from that workspace. In another worlspace
> I got the artifact for Linux and of course I can only publish that.
>
> How do you accomplish to publish just the artifacts for one particular
> configuration?
>
> Thanks for any hint,
> Felix
>
> The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.
>
The information included in this e-mail and any files transmitted with it is strictly confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this e-mail as well as any attachment from your system. If you are not the intended recipient you are not authorized to use and/or copy this message and/or attachment and/or disclose the contents to any other person.