You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@continuum.apache.org by Wendy Smoak <ws...@gmail.com> on 2009/01/17 18:28:30 UTC
Associating projects with distributed build agents
One of the goals at the top of the Distributed Builds wiki page is the
ability to build on multiple platforms.
As we've discussed in some other threads and on irc, we need some way
of telling Continuum which agent(s) to execute builds on.
To me, it seems like this belongs as a field on the build environment,
which is then selected on the build definition.
That would be a good next step IMO -- to pin a particular build def to
a particular agent through the environment.
However, I don't want to lose the effect of distributed builds in
parallel that we're getting from the current 'next available'
selection.
So, how does having groups of agents sound? Then you could associate
a build environment with a pool of agents, and the next available one
of those would be chosen.
You might have a large group of identically configured Linux boxes
that most of your builds can be distributed to. However, maybe there
is one special project group that needs their own private pool to work
on a top secret project. Another team might be targeting Solaris and
want to do their builds on a pool of three Solaris boxes. And of
course the project that only builds on Windows needs to always build
on the one Windows agent you have reluctantly configured for them. ;)
There's an additional requirement that the system admin must be able
to limit which environments a project team is allowed to choose.
While a project developer can modify build definitions and add new
ones, he should not have free choice of *all* the available
environments. This probably means adding an "Allowed Build
Environments" field to the project group.
How does that sound? Does anyone have a different idea of how this should work?
--
Wendy
Re: Associating projects with distributed build agents
Posted by Wendy Smoak <ws...@gmail.com>.
Does anyone know if the necessary functionality is available through
Maven SCM? As I understand it, that's what all the scm access goes
through.
--
Wendy
On Wed, Jan 28, 2009 at 5:15 PM, Christian Edward Gruber
<cg...@israfil.net> wrote:
> I really don't like moving this way. I think that determining whether there
> have been changes isn't so freaky if you use repository commands, not
> workspace commands to do this. This should work with almost every
> client/server SCM system except maybe CVS, and with that you can still
> effectively do it with "cvs log -r X,X" (or diff) commands. What worries me
> is a shift in the approach of continuum away from a really useful system
> based on a limitation of the SCMs. I'd rather have a more functional
> continuum that has fall-back approaches for less feature-rich SCM systems.
Re: Associating projects with distributed build agents
Posted by Christian Edward Gruber <cg...@israfil.net>.
I really don't like moving this way. I think that determining whether
there have been changes isn't so freaky if you use repository
commands, not workspace commands to do this. This should work with
almost every client/server SCM system except maybe CVS, and with that
you can still effectively do it with "cvs log -r X,X" (or diff)
commands. What worries me is a shift in the approach of continuum
away from a really useful system based on a limitation of the SCMs.
I'd rather have a more functional continuum that has fall-back
approaches for less feature-rich SCM systems.
Christian.
On 28-Jan-09, at 19:08 , Wendy Smoak wrote:
> The more I think about this in light of the other discussion about the
> problem of figuring out whether or not there have been scm changes
> (because the last-used scm checkout might be anywhere) I'm leaning
> towards:
>
> 1. associate a build environment with a single build agent [1]
> 2. get parallel builds working on the build agent (CONTINUUM-2045)
>
> It's the effect of concurrent builds that I'm attached to, not
> necessarily selecting an agent at build time. That is, I'd love to
> see permission- and capability-based selection of agents at some
> point, but I can live with something less than that for the moment.
>
> What do you think?
>
> [1] I still need to make sure only an admin controls which projects
> build on which agents...
>
> --
> Wendy
>
> On Mon, Jan 19, 2009 at 3:43 PM, Christian Edward Gruber
> <cg...@israfil.net> wrote:
>> I think the "magic" can be made visible, in that if it is criteria
>> based,
>> the server can display those agents which would apply to the current
>> criteria, so that a user can see the implications of his
>> selection. Since
>> the display will be the same logic as the criteria selection (it's
>> really
>> just straight set theory) then the user is not going to be surprised.
>>
>> Christian
>>
>> On 19-Jan-09, at 02:44 , Brett Porter wrote:
>>
>>> +1
>>>
>>> I think selection needs to be a combination of capability,
>>> permission and
>>> availability. I would mostly prefer the user is in full control of
>>> selecting
>>> these rather than having too much magic (automated selection might
>>> be a nice
>>> feature, but the basic starting point should be a fully configured
>>> system).
> ...
>
Christian E. Gruber - President / Senior Consultant
Isráfíl Consulting Services Corporation
email: cgruber@israfil.net
mobile: +1 (289) 221-9839
web: http://www.israfil.net/
"...keenness of understanding is due to keenness of vision."
Re: Associating projects with distributed build agents
Posted by Wendy Smoak <ws...@gmail.com>.
The more I think about this in light of the other discussion about the
problem of figuring out whether or not there have been scm changes
(because the last-used scm checkout might be anywhere) I'm leaning
towards:
1. associate a build environment with a single build agent [1]
2. get parallel builds working on the build agent (CONTINUUM-2045)
It's the effect of concurrent builds that I'm attached to, not
necessarily selecting an agent at build time. That is, I'd love to
see permission- and capability-based selection of agents at some
point, but I can live with something less than that for the moment.
What do you think?
[1] I still need to make sure only an admin controls which projects
build on which agents...
--
Wendy
On Mon, Jan 19, 2009 at 3:43 PM, Christian Edward Gruber
<cg...@israfil.net> wrote:
> I think the "magic" can be made visible, in that if it is criteria based,
> the server can display those agents which would apply to the current
> criteria, so that a user can see the implications of his selection. Since
> the display will be the same logic as the criteria selection (it's really
> just straight set theory) then the user is not going to be surprised.
>
> Christian
>
> On 19-Jan-09, at 02:44 , Brett Porter wrote:
>
>> +1
>>
>> I think selection needs to be a combination of capability, permission and
>> availability. I would mostly prefer the user is in full control of selecting
>> these rather than having too much magic (automated selection might be a nice
>> feature, but the basic starting point should be a fully configured system).
...
Re: Associating projects with distributed build agents
Posted by Christian Edward Gruber <cg...@israfil.net>.
I think the "magic" can be made visible, in that if it is criteria
based, the server can display those agents which would apply to the
current criteria, so that a user can see the implications of his
selection. Since the display will be the same logic as the criteria
selection (it's really just straight set theory) then the user is not
going to be surprised.
Christian
On 19-Jan-09, at 02:44 , Brett Porter wrote:
> +1
>
> I think selection needs to be a combination of capability,
> permission and availability. I would mostly prefer the user is in
> full control of selecting these rather than having too much magic
> (automated selection might be a nice feature, but the basic starting
> point should be a fully configured system).
>
> - Brett
>
> On 17/01/2009, at 1:36 PM, Christian Edward Gruber wrote:
>
>> I guess then having agent groups as a logical group, but make group
>> be part of the criteria would help with that then, group
>> effectively being a property of the agent.
>>
>> The key for me is to have the whole thing specified as constraints,
>> not explicit solutions, so that new agents can be launched and they
>> will automatically fit into the constraints based on their
>> configuration, rather than having to add them to this or that build
>> definition. Filter/constraint configuration is a great way to
>> simplify config, especially with large banks of agents.
>>
>> Christian
>>
>> On 17-Jan-09, at 16:22 , Wendy Smoak wrote:
>>
>>> I can see that working, if I can define arbitrary criteria like a
>>> property name/value.
>>>
>>> I need a way to make sure that Top Secret Project X builds on agents
>>> 3, 5 and 14 only, and that no other projects ever build on those
>>> agents.
>>>
>>> --
>>> Wendy
>>>
>>> On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
>>> <cg...@israfil.net> wrote:
>>>> Rather than agent groups, could we go with a criteria approach?
>>>> That is to
>>>> say, you can describe and agent's capabilities, then associate
>>>> criteria for
>>>> a project group or project, and then let the system sort out
>>>> which agents
>>>> can fulfill which criteria?
>>>
>>
>> Christian E. Gruber - President / Senior Consultant
>> email: cgruber@israfil.net
>> Isráfíl Consulting Services Corporation
>> mobile: +1 (289) 221-9839
>> "Keenness of understanding is due to keenness of vision..."
>> phone: +1 (905) 640-1119
>>
>>
>>
>>
>>
>
> --
> Brett Porter
> brett@apache.org
> http://blogs.exist.com/bporter/
>
Christian E. Gruber - President / Senior Consultant
email: cgruber@israfil.net
Isráfíl Consulting Services Corporation
mobile: +1 (289) 221-9839
"Keenness of understanding is due to keenness of vision..."
phone: +1 (905) 640-1119
Re: Associating projects with distributed build agents
Posted by Brett Porter <br...@apache.org>.
+1
I think selection needs to be a combination of capability, permission
and availability. I would mostly prefer the user is in full control of
selecting these rather than having too much magic (automated selection
might be a nice feature, but the basic starting point should be a
fully configured system).
- Brett
On 17/01/2009, at 1:36 PM, Christian Edward Gruber wrote:
> I guess then having agent groups as a logical group, but make group
> be part of the criteria would help with that then, group effectively
> being a property of the agent.
>
> The key for me is to have the whole thing specified as constraints,
> not explicit solutions, so that new agents can be launched and they
> will automatically fit into the constraints based on their
> configuration, rather than having to add them to this or that build
> definition. Filter/constraint configuration is a great way to
> simplify config, especially with large banks of agents.
>
> Christian
>
> On 17-Jan-09, at 16:22 , Wendy Smoak wrote:
>
>> I can see that working, if I can define arbitrary criteria like a
>> property name/value.
>>
>> I need a way to make sure that Top Secret Project X builds on agents
>> 3, 5 and 14 only, and that no other projects ever build on those
>> agents.
>>
>> --
>> Wendy
>>
>> On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
>> <cg...@israfil.net> wrote:
>>> Rather than agent groups, could we go with a criteria approach?
>>> That is to
>>> say, you can describe and agent's capabilities, then associate
>>> criteria for
>>> a project group or project, and then let the system sort out which
>>> agents
>>> can fulfill which criteria?
>>
>
> Christian E. Gruber - President / Senior Consultant
> email: cgruber@israfil.net
> Isráfíl Consulting Services Corporation
> mobile: +1 (289) 221-9839
> "Keenness of understanding is due to keenness of vision..."
> phone: +1 (905) 640-1119
>
>
>
>
>
--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/
Re: Associating projects with distributed build agents
Posted by Christian Edward Gruber <cg...@israfil.net>.
I guess then having agent groups as a logical group, but make group be
part of the criteria would help with that then, group effectively
being a property of the agent.
The key for me is to have the whole thing specified as constraints,
not explicit solutions, so that new agents can be launched and they
will automatically fit into the constraints based on their
configuration, rather than having to add them to this or that build
definition. Filter/constraint configuration is a great way to
simplify config, especially with large banks of agents.
Christian
On 17-Jan-09, at 16:22 , Wendy Smoak wrote:
> I can see that working, if I can define arbitrary criteria like a
> property name/value.
>
> I need a way to make sure that Top Secret Project X builds on agents
> 3, 5 and 14 only, and that no other projects ever build on those
> agents.
>
> --
> Wendy
>
> On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
> <cg...@israfil.net> wrote:
>> Rather than agent groups, could we go with a criteria approach?
>> That is to
>> say, you can describe and agent's capabilities, then associate
>> criteria for
>> a project group or project, and then let the system sort out which
>> agents
>> can fulfill which criteria?
>
Christian E. Gruber - President / Senior Consultant
email: cgruber@israfil.net
Isráfíl Consulting Services Corporation
mobile: +1 (289) 221-9839
"Keenness of understanding is due to keenness of vision..."
phone: +1 (905) 640-1119
Re: Associating projects with distributed build agents
Posted by Wendy Smoak <ws...@gmail.com>.
I can see that working, if I can define arbitrary criteria like a
property name/value.
I need a way to make sure that Top Secret Project X builds on agents
3, 5 and 14 only, and that no other projects ever build on those
agents.
--
Wendy
On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
<cg...@israfil.net> wrote:
> Rather than agent groups, could we go with a criteria approach? That is to
> say, you can describe and agent's capabilities, then associate criteria for
> a project group or project, and then let the system sort out which agents
> can fulfill which criteria?
Re: Associating projects with distributed build agents
Posted by Christian Edward Gruber <cg...@israfil.net>.
Rather than agent groups, could we go with a criteria approach? That
is to say, you can describe and agent's capabilities, then associate
criteria for a project group or project, and then let the system sort
out which agents can fulfill which criteria?
Christian.
On 17-Jan-09, at 12:28 , Wendy Smoak wrote:
> One of the goals at the top of the Distributed Builds wiki page is the
> ability to build on multiple platforms.
>
> As we've discussed in some other threads and on irc, we need some way
> of telling Continuum which agent(s) to execute builds on.
>
> To me, it seems like this belongs as a field on the build environment,
> which is then selected on the build definition.
>
> That would be a good next step IMO -- to pin a particular build def to
> a particular agent through the environment.
>
> However, I don't want to lose the effect of distributed builds in
> parallel that we're getting from the current 'next available'
> selection.
>
> So, how does having groups of agents sound? Then you could associate
> a build environment with a pool of agents, and the next available one
> of those would be chosen.
>
> You might have a large group of identically configured Linux boxes
> that most of your builds can be distributed to. However, maybe there
> is one special project group that needs their own private pool to work
> on a top secret project. Another team might be targeting Solaris and
> want to do their builds on a pool of three Solaris boxes. And of
> course the project that only builds on Windows needs to always build
> on the one Windows agent you have reluctantly configured for them. ;)
>
> There's an additional requirement that the system admin must be able
> to limit which environments a project team is allowed to choose.
> While a project developer can modify build definitions and add new
> ones, he should not have free choice of *all* the available
> environments. This probably means adding an "Allowed Build
> Environments" field to the project group.
>
> How does that sound? Does anyone have a different idea of how this
> should work?
>
> --
> Wendy
>
Christian E. Gruber - President / Senior Consultant
email: cgruber@israfil.net
Isráfíl Consulting Services Corporation
mobile: +1 (289) 221-9839
"Keenness of understanding is due to keenness of vision..."
phone: +1 (905) 640-1119