You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@continuum.apache.org by Wendy Smoak <ws...@gmail.com> on 2009/01/05 17:48:03 UTC
Re: First Part of Distributed Builds
On Fri, Dec 12, 2008 at 6:46 AM, Marica Tan <ct...@exist.com> wrote:
> I've completed most of the distributed build manager part and there's also
> nap's transport layer using atlassian's xmlrpc. You can checkout the
> continuum-distributed-builds branch to test it.
...
> Comments, Suggestions, and/or Violent Reactions are welcome :D
Any further news on this? I see lots of commits, but not much
discussion... even having a conversation with yourself will help
everyone else keep up with changes and possibly get involved. :)
One thing I'm concerned about is how I'm going to take a dozen
individual Continuum instances and get all that data into the new
master. Has the model changed? First priority is all the
projects/build defs/schedules/etc., and secondly the build results
which are important for statistics.
--
Wendy
Re: First Part of Distributed Builds
Posted by Wendy Smoak <ws...@gmail.com>.
On Wed, Jan 14, 2009 at 12:18 AM, Napoleon Esmundo C. Ramirez
<na...@gmail.com> wrote:
> Having lots of data aggregated from different Continuum instances into one
> will need some kind of an importer tool. I'm thinking this should only
> support data from Continuum instances of the same model version, while data
> from different versions will have to be handled by a separate db migration
> and conversion tool. I'd like to point out that distinction and combining
> the two is an option, but let's take things bit by bit first.
It is the db migration tool that is blocking a release of 1.3 right
now, so the first thing I'm looking for is a way to upgrade a single
instance of 1.2.x to 1.3.1-SNAPSHOT.
The importer tool to consolidate multiple existing instances would be
nice to have, but so far I'm the only one asking. :)
> Briefly:
> 1. The Backup functionality should be available in the Continuum web ui.
> The core functionality is available, making it accessible in the Continuum
> web ui will ease the generation of data to import.
Not sure if you have these in any sort of order, but this is low
priority for me. I'd get it working at the command line before
worrying about adding it to the webapp.
> 2. Create the Import functionality.
> 2.a This accepts the backup files as input.
> 2.b All the data in the backup file will be appended to the existing data
> of the Continuum instance it is imported to.
I think there needs to be an option to import and overwrite -- there
is always some data in a Continuum instance, even a brand new one (the
default project group, default schedule, etc.) If you always append,
there will be no way to restore from a backup and get back to the same
state.
> A basic rule is: no duplicates. In case of duplicates, either a new
Was there more on this thought?
> The complications are:
> 1. The keys of the entities in the imported data may have duplicates in the
> Continuum instance it is imported to. To address this, the keys of the
> imported data will have to be adjusted, shifting the values to next
> available values but giving preference to the existing data (latest key
> value + 1 ??).
> 2. Since the keys will be adjusted, the relationships between the entities
> of the imported data will have adjust as well.
What about the build output, which is stored in numbered directories?
I don't think it is included in the xml backup, you're just expected
to have them in the right place on disk. Those directories would
either have to be re-numbered along with the records in the database,
or we'll have to document that we don't support keeping build output
when you consolidate server config.
Or maybe we just leave out build results altogether for the first pass
at this, and say we only support consolidating the
projects/schedules/installations/etc -- the configuration and not the
historical data.
Thanks,
--
Wendy
Re: First Part of Distributed Builds
Posted by Brett Porter <br...@apache.org>.
Got it - thanks.
On 14/01/2009, at 6:51 PM, Napoleon Esmundo C. Ramirez wrote:
> On Wed, Jan 14, 2009 at 3:36 PM, Brett Porter <br...@apache.org>
> wrote:
>
>>
>> On 14/01/2009, at 6:18 PM, Napoleon Esmundo C. Ramirez wrote:
>>
>>>
>>> Having lots of data aggregated from different Continuum instances
>>> into one
>>> will need some kind of an importer tool.
>>>
>>
>> I'm not sure why this is the case - according to Marica's initial
>> response
>> to this message from Wendy, the build result is always shipped back
>> to the
>> master and not stored on the slave?
>>
>> Why is a data import needed?
>>
>> (FWIW, I think integrating backup/restore into the UI is a good
>> thing, as
>> is improving/solidifying the current upgrade process).
>>
>> - Brett
>>
>
> Wendy mentioned she has a dozen of Continum instances to convert
> into slave
> build agents, and the data from those instances need to be
> transferred to a
> central instance first. I think that's what the importer is trying to
> address.
>
> Thanks,
> Nap
--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/
Re: First Part of Distributed Builds
Posted by "Napoleon Esmundo C. Ramirez" <na...@gmail.com>.
On Wed, Jan 14, 2009 at 3:36 PM, Brett Porter <br...@apache.org> wrote:
>
> On 14/01/2009, at 6:18 PM, Napoleon Esmundo C. Ramirez wrote:
>
>>
>> Having lots of data aggregated from different Continuum instances into one
>> will need some kind of an importer tool.
>>
>
> I'm not sure why this is the case - according to Marica's initial response
> to this message from Wendy, the build result is always shipped back to the
> master and not stored on the slave?
>
> Why is a data import needed?
>
> (FWIW, I think integrating backup/restore into the UI is a good thing, as
> is improving/solidifying the current upgrade process).
>
> - Brett
>
Wendy mentioned she has a dozen of Continum instances to convert into slave
build agents, and the data from those instances need to be transferred to a
central instance first. I think that's what the importer is trying to
address.
Thanks,
Nap
Re: First Part of Distributed Builds
Posted by Brett Porter <br...@apache.org>.
On 14/01/2009, at 6:18 PM, Napoleon Esmundo C. Ramirez wrote:
>
> Having lots of data aggregated from different Continuum instances
> into one
> will need some kind of an importer tool.
I'm not sure why this is the case - according to Marica's initial
response to this message from Wendy, the build result is always
shipped back to the master and not stored on the slave?
Why is a data import needed?
(FWIW, I think integrating backup/restore into the UI is a good thing,
as is improving/solidifying the current upgrade process).
- Brett
--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/
Re: First Part of Distributed Builds
Posted by "Napoleon Esmundo C. Ramirez" <na...@gmail.com>.
On Tue, Jan 6, 2009 at 12:48 AM, Wendy Smoak <ws...@gmail.com> wrote:
> Any further news on this? I see lots of commits, but not much
> discussion... even having a conversation with yourself will help
> everyone else keep up with changes and possibly get involved. :)
>
> One thing I'm concerned about is how I'm going to take a dozen
> individual Continuum instances and get all that data into the new
> master. Has the model changed? First priority is all the
> projects/build defs/schedules/etc., and secondly the build results
> which are important for statistics.
>
> --
> Wendy
>
Having lots of data aggregated from different Continuum instances into one
will need some kind of an importer tool. I'm thinking this should only
support data from Continuum instances of the same model version, while data
from different versions will have to be handled by a separate db migration
and conversion tool. I'd like to point out that distinction and combining
the two is an option, but let's take things bit by bit first.
Briefly:
1. The Backup functionality should be available in the Continuum web ui.
The core functionality is available, making it accessible in the Continuum
web ui will ease the generation of data to import.
2. Create the Import functionality.
2.a This accepts the backup files as input.
2.b All the data in the backup file will be appended to the existing data
of the Continuum instance it is imported to. A basic rule is: no
duplicates. In case of duplicates, either a new
The complications are:
1. The keys of the entities in the imported data may have duplicates in the
Continuum instance it is imported to. To address this, the keys of the
imported data will have to be adjusted, shifting the values to next
available values but giving preference to the existing data (latest key
value + 1 ??).
2. Since the keys will be adjusted, the relationships between the entities
of the imported data will have adjust as well.
What do you think?
Cheers,
Nap
Re: First Part of Distributed Builds
Posted by Wendy Smoak <ws...@gmail.com>.
That will work if all the build servers are the same. (Mine aren't --
they have different OS's and different installed software, so certain
projects have to be built on certain servers.)
Thanks for clarifying!
--
Wendy
On Tue, Jan 6, 2009 at 7:27 PM, Marica Tan <ct...@exist.com> wrote:
> From what I remembered, there were some comments on having to build a
> project to a single slave only. If that slave is unavailable, it will wait
> until the slave becomes available again before the build can be executed.
> Which is why for the first pass, I did not tie up the slave to the build
> environment. So that a build task can be delegated to any slave.
>
> But eventually in the next iteration, there will be some mechanism where the
> slave will be selected based on the required environment.
>
> --
> Marica
>
Re: First Part of Distributed Builds
Posted by Marica Tan <ct...@exist.com>.
>From what I remembered, there were some comments on having to build a
project to a single slave only. If that slave is unavailable, it will wait
until the slave becomes available again before the build can be executed.
Which is why for the first pass, I did not tie up the slave to the build
environment. So that a build task can be delegated to any slave.
But eventually in the next iteration, there will be some mechanism where the
slave will be selected based on the required environment.
--
Marica
Re: First Part of Distributed Builds
Posted by Christian Edward Gruber <cg...@israfil.net>.
I'm liking the vision, that's for sure.
Christian.
On 6-Jan-09, at 20:44 , Wendy Smoak wrote:
> My understanding is that the first pass at this will require you to
> select the slave for each project.
>
> But eventually I envision it eventually being able to balance the load
> across the slaves, selecting one with the right environment and
> preferring the most recently used one (because the local repo there
> will already have needed artifacts).
>
> It should allow slaves to be added and removed on the fly and have a
> notification or monitoring system. I want to hook that up to a
> provisioning system to automatically set up new slaves when the load
> increases, and tear them down if they're not being used "enough" to
> make the cost worth it. Sounds like fun. :)
>
> --
> Wendy
>
> On Tue, Jan 6, 2009 at 6:36 PM, Christian Edward Gruber
> <cg...@israfil.net> wrote:
>> Is there to be a mechanism for allowing distribution to slaves
>> based on
>> available environmental conditions?
>>
>> For example, if I'm CI'ing a Mac OS X app, but I have six machines
>> in my CI
>> cloud, two of which are OS-X, could I tell Continuum to spread out
>> the
>> projects among all slaves whose environments are hosted on OS-X?
>> Or would I
>> need to specify that manually based on my own knowledge of those
>> environments?
>
Re: First Part of Distributed Builds
Posted by Wendy Smoak <ws...@gmail.com>.
My understanding is that the first pass at this will require you to
select the slave for each project.
But eventually I envision it eventually being able to balance the load
across the slaves, selecting one with the right environment and
preferring the most recently used one (because the local repo there
will already have needed artifacts).
It should allow slaves to be added and removed on the fly and have a
notification or monitoring system. I want to hook that up to a
provisioning system to automatically set up new slaves when the load
increases, and tear them down if they're not being used "enough" to
make the cost worth it. Sounds like fun. :)
--
Wendy
On Tue, Jan 6, 2009 at 6:36 PM, Christian Edward Gruber
<cg...@israfil.net> wrote:
> Is there to be a mechanism for allowing distribution to slaves based on
> available environmental conditions?
>
> For example, if I'm CI'ing a Mac OS X app, but I have six machines in my CI
> cloud, two of which are OS-X, could I tell Continuum to spread out the
> projects among all slaves whose environments are hosted on OS-X? Or would I
> need to specify that manually based on my own knowledge of those
> environments?
Re: First Part of Distributed Builds
Posted by Christian Edward Gruber <cg...@israfil.net>.
Is there to be a mechanism for allowing distribution to slaves based
on available environmental conditions?
For example, if I'm CI'ing a Mac OS X app, but I have six machines in
my CI cloud, two of which are OS-X, could I tell Continuum to spread
out the projects among all slaves whose environments are hosted on OS-
X? Or would I need to specify that manually based on my own knowledge
of those environments?
Christian.
On 6-Jan-09, at 18:12 , Wendy Smoak wrote:
> On Tue, Jan 6, 2009 at 4:08 PM, Marica Tan <ct...@exist.com> wrote:
>> On Wed, Jan 7, 2009 at 1:28 AM, Wendy Smoak <ws...@gmail.com> wrote:
>
>>> We currently have the ability to both build and release under a
>>> selected build environment. Continuum itself has to run under JDK
>>> 1.5, so I'm concerned about losing the ability to build and release
>>> with JDK 1.4 or a different Maven version from the one defined in
>>> M2_HOME and PATH.
>>>
>>
>> Ok. Will fix this. Thanks for the inputs.
>
> I thought when we discussed this way back, the idea was to think of a
> Build Environment as everything it is now, plus the "location" the
> build should execute in (which slave).
>
> You mentioned that the only model change is the configuration for
> whether distributed builds are enabled. Then where is the
> url/selected slave for each project (group?) stored? Or did this
> change?
>
> --
> Wendy
>
Re: First Part of Distributed Builds
Posted by Wendy Smoak <ws...@gmail.com>.
On Tue, Jan 6, 2009 at 4:08 PM, Marica Tan <ct...@exist.com> wrote:
> On Wed, Jan 7, 2009 at 1:28 AM, Wendy Smoak <ws...@gmail.com> wrote:
>> We currently have the ability to both build and release under a
>> selected build environment. Continuum itself has to run under JDK
>> 1.5, so I'm concerned about losing the ability to build and release
>> with JDK 1.4 or a different Maven version from the one defined in
>> M2_HOME and PATH.
>>
>
> Ok. Will fix this. Thanks for the inputs.
I thought when we discussed this way back, the idea was to think of a
Build Environment as everything it is now, plus the "location" the
build should execute in (which slave).
You mentioned that the only model change is the configuration for
whether distributed builds are enabled. Then where is the
url/selected slave for each project (group?) stored? Or did this
change?
--
Wendy
Re: First Part of Distributed Builds
Posted by Marica Tan <ct...@exist.com>.
On Wed, Jan 7, 2009 at 1:28 AM, Wendy Smoak <ws...@gmail.com> wrote:
> On Mon, Jan 5, 2009 at 7:53 PM, Marica Tan <ct...@exist.com> wrote:
>
> > Slave agents will use the default installations of maven, java and
> ant
> > when building projects.
>
> Can you clarify this part? Does 'default installations' mean the
> environment for the user running Continuum?
>
It's the environment in the slave machine.
>
> We currently have the ability to both build and release under a
> selected build environment. Continuum itself has to run under JDK
> 1.5, so I'm concerned about losing the ability to build and release
> with JDK 1.4 or a different Maven version from the one defined in
> M2_HOME and PATH.
>
Ok. Will fix this. Thanks for the inputs.
--
Marica
Re: First Part of Distributed Builds
Posted by Wendy Smoak <ws...@gmail.com>.
On Mon, Jan 5, 2009 at 7:53 PM, Marica Tan <ct...@exist.com> wrote:
> Slave agents will use the default installations of maven, java and ant
> when building projects.
Can you clarify this part? Does 'default installations' mean the
environment for the user running Continuum?
We currently have the ability to both build and release under a
selected build environment. Continuum itself has to run under JDK
1.5, so I'm concerned about losing the ability to build and release
with JDK 1.4 or a different Maven version from the one defined in
M2_HOME and PATH.
Thanks,
--
Wendy
Re: First Part of Distributed Builds
Posted by Marica Tan <ct...@exist.com>.
On Tue, Jan 6, 2009 at 12:48 AM, Wendy Smoak <ws...@gmail.com> wrote:
> On Fri, Dec 12, 2008 at 6:46 AM, Marica Tan <ct...@exist.com> wrote:
>
> > I've completed most of the distributed build manager part and there's
> also
> > nap's transport layer using atlassian's xmlrpc. You can checkout the
> > continuum-distributed-builds branch to test it.
> ...
> > Comments, Suggestions, and/or Violent Reactions are welcome :D
>
> Any further news on this? I see lots of commits, but not much
> discussion... even having a conversation with yourself will help
> everyone else keep up with changes and possibly get involved. :)
>
Sorry for the lack of updates :-(
>
> One thing I'm concerned about is how I'm going to take a dozen
> individual Continuum instances and get all that data into the new
> master. Has the model changed? First priority is all the
> projects/build defs/schedules/etc., and secondly the build results
> which are important for statistics.
>
I only added a new field "distributedBuildEnabled" in the
SystemConfiguration table. Build result is returned to the master once the
build is finished.
So here's how the distributed builds work.
*SLAVE*
- slave or the build agent is a separate web application that builds
projects for the master.
- slave can reside in the same machine as the master.
- slave communicates with the master through XML RPC.
- requirements
* slave machine must already have maven2, maven1, ant, java, svn, and
other third-party applications needed to build the projects installed
- setup
* set SLAVE_HOME
* create a continuum-buildagent.xml in SLAVE_HOME/conf/
* edit continuum-buildagent.xml
*Sample:
* <buildOutputDirectory>*/home/slave/buildOutputDirectory*
</buildOutputDirectory>
<workingDirectory>*/home/slave/workingDirectory*</workingDirectory>
<continuumServerUrl>*http://localhost:8080/master-xmlrpc*
</continuumServerUrl>
<installations>
<installation>
<name>*Maven 2 Home*</name>
<type>*maven2*</type>
<varValue>*/opt/apache-maven-2.0.9*</varValue>
</installation>
</installations>
Slave agents will use the default installations of maven, java and ant
when building projects. Installations in the continuum-buildagent.xml will
only be used by the master for viewing.
* deploy continuum-buildagent-webapp.war
* start container
*MASTER*
- master is a continuum instance.
- in the distributed builds mode, the master will still serve all requests
but will delegate all build tasks to the slaves.
- enable/disable distributed builds in the configuration page.
The first thing you have to do is to register the slaves to the master. This
can be done by adding it either in the continuum.xml or using the web UI.
Once you have registered the slaves, the master will start delegating build
tasks in the queue to the slaves. Master will send all information needed by
the slave to build the projects. Once the build is finished, the slave will
send back the build result to the master.
Bugs:
- cancellation of build task
To follow:
- notification
- viewing of slave installations
Question:
1. when there's a connection error in the slave, do we treat it as a
transient error and only log it? ( no build result, set project's state to
previous state )
Will document this later on.
Thanks,
--
Marica