You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@continuum.apache.org by ws...@apache.org on 2009/01/15 01:06:59 UTC
svn commit: r734569 - in /continuum/trunk/continuum-docs/src/site/apt:
administrator_guides/build-agents.apt
administrator_guides/distributed-builds.apt administrator_guides/index.apt
installation/build-agent.apt
Author: wsmoak
Date: Wed Jan 14 16:06:59 2009
New Revision: 734569
URL: http://svn.apache.org/viewvc?rev=734569&view=rev
Log:
[CONTINUUM-1666] Improve documentation on Distributed Builds
Submitted by: Napoleon Esmundo C. Ramirez
Committed with modifications, including changing the term Slave to Build Agent and moving some content to the installation page..
Modified:
continuum/trunk/continuum-docs/src/site/apt/administrator_guides/build-agents.apt
continuum/trunk/continuum-docs/src/site/apt/administrator_guides/distributed-builds.apt
continuum/trunk/continuum-docs/src/site/apt/administrator_guides/index.apt
continuum/trunk/continuum-docs/src/site/apt/installation/build-agent.apt
Modified: continuum/trunk/continuum-docs/src/site/apt/administrator_guides/build-agents.apt
URL: http://svn.apache.org/viewvc/continuum/trunk/continuum-docs/src/site/apt/administrator_guides/build-agents.apt?rev=734569&r1=734568&r2=734569&view=diff
==============================================================================
--- continuum/trunk/continuum-docs/src/site/apt/administrator_guides/build-agents.apt (original)
+++ continuum/trunk/continuum-docs/src/site/apt/administrator_guides/build-agents.apt Wed Jan 14 16:06:59 2009
@@ -5,7 +5,8 @@
Managing Build Agents
Before this, you may want to enable the Distributed Builds option in the {{{configuration.html}General Configuration}}
- and {{{../installation/build-agent.html}install a Build Agent}}.
+ and {{{../installation/build-agent.html}install a Build Agent}}, or read about the {{{distributed-builds.html}Distributed
+ Builds}} feature.
From the menu, choose the 'Build Agents' item
Modified: continuum/trunk/continuum-docs/src/site/apt/administrator_guides/distributed-builds.apt
URL: http://svn.apache.org/viewvc/continuum/trunk/continuum-docs/src/site/apt/administrator_guides/distributed-builds.apt?rev=734569&r1=734568&r2=734569&view=diff
==============================================================================
--- continuum/trunk/continuum-docs/src/site/apt/administrator_guides/distributed-builds.apt (original)
+++ continuum/trunk/continuum-docs/src/site/apt/administrator_guides/distributed-builds.apt Wed Jan 14 16:06:59 2009
@@ -2,24 +2,120 @@
Distributed Builds
------------------
-
Understanding Distributed Builds
- When the 'Distributed Builds' option in General Configuration is enabled, the Continuum instance will become the
- Master and will delegate all builds to configured Build Agents.
+* Introduction
+
+ The Distributed Builds is an extension to the base Continuum
+ functionalities that gives us the ability to process multiple independent
+ builds beyond the capacity of a single server's processing power. It also
+ enables us to execute builds on multiple different platforms while
+ retaining a unified view of all project builds.
+
+* Architecture
+
+ Continuum follows a Client-Server model using XML-RPC as the protocol.
+ However, since it uses a bi-directional XML-RPC implementation, we instead
+ distinguish the components by calling them Master and Build Agent.
+
+ The Master is a Continuum instance that has the ability to
+ delegate the builds to registered Build Agents.
+
+ The Build Agent is a standalone Jetty-bundled webapp that
+ listens for any build requests from the Master it is
+ assigned to.
+
+ There is a one-to-many relationship between the Master and the Slaves.
+ A Master may have many Build Agents, but each Build Agent can only have one Master.
+
+ TODO: insert image here
+
+* Behavior
+
+ Distributed Builds happen at the project group level of Continuum.
+ When the entire project group is built in the Master, independent projects
+ (single project or multi-module project) are distributed to any available
+ registered Slave. A Slave is said to be available when it is currently not
+ building anything as it can only attend to a single build request from the
+ Master.
+
+ In a project group containing a mix of projects, the distribution of work
+ goes through the following steps:
+
+ [[1]] In the Master, a build in the project group is triggered.
+
+ [[2]] Every independent project within the project group is identified,
+ whether as a single project or a multi-module project. Projects with
+ inter-dependencies cannot be distributed separately, so multi-module
+ projects are delegated to a Slave as one build.
+
+ [[3]] For each independent project, the Master iterates over the list of
+ registered Slaves and queries each if available. The query is an XML-RPC
+ <<<ping()>>> followed by an <<<isAvailable()>>> invocation.
+
+ [[4]] If a Build Agent is available, the Master collects the information
+ necessary for the build (SCM url, project id, etc.) and passes it when
+ invoking <<<buildProjects()>>>.
+
+ [[5]] In the Build Agent, the build request is processed: the build is queued and
+ executed. Upon execution, the Build Agent first performs an SCM checkout then
+ the actual build follows. The SCM checkout will result in invoking a
+ <<<returnScmResult()>>> callback method in the Master.
+
+ [[6]] At this point, when the build is running, the Master can invoke
+ <<<cancelBuild()>>> which returns a transient build result, and
+ <<<getBuildResult()>>> that updates the build output viewed in the Master.
+
+ [[7]] After the build, the Build Agent returns the complete build result to the
+ Master by invoking the callback method <<<returnBuildResult()>>>, which the
+ Master aggregates to provide a unified view of projects
+
+ []
+
+ TODO: insert sequence diagram here
+
+* Setup
+
+ * {{{../installation/build-agent.html}Install and Configure}} one or more Build Agents, then
+
+ * {{{configuration.html}Enable}} the Distributed Builds option in the General Configuration, and
+
+ * {{{build-agents.html}Add}} your Build Agents to the Continuum Master.
+
+* Limitations
+
+ * the Build Agent selection policy is currently following a fixed informal next-available policy; a Build Agent cannot be assigned to a specific project build
+
+ * only system administrator can enable/disable distributed builds
+
+ * releases are not supported
+
+ * credentials (s.a. svn credentials) are passed along if specified, but if server cache is used it will need to be done individually on the slaves
+
+ * there is no tracking of scm changes
+
+ * the Build Agent needs a configuration web interface
+
+* Future Enhancements
+
+ * Remote builders
+
+ * Builders can be installed on remote machines, a Continuum manager will send actions to run to builders. An action can be something to run on all builders, on some of them or eventually only to an available builder if we don't want to run more than one build. Actions can be sent with JMS and builders can apply some filters if they don't want to receive all actions. With that, we can do some parallel builds but the dependency tree must be respected for the build order. To work correctly with dependencies, each builders must use a central local repository. Maybe we can use an internal Archiva.
+
+ * With Continuum builders configured to receive all commands, users can run multi-platform build for each build definition execution
+
+ * With Continuum builders configured to receive only some project types, users can use a different builder by project group. In this case, the build of all projects will be done quickly because commands are balanced on few servers
- There is a one to many relationship between the master and the agents. A master may have many agents, but each
- agent only has one master.
+ * With Continuum builders configured to build something when it is available, users can install few builders on several machine to balance the charge. In this case, it will be possible to run some parallel builds.
- Communication happens over XML-RPC, first with the Master contacting an agent to supply the necessary info and invoke
- a build, and then with the agent later contacting the master to return the build results.
+ * When the builder work will be done, a message will be sent to the manager to notify the end of the process.
- An agent may be installed on the same server as the master, or on a separate server.
+ * With JMS used for the communication, we can add some listeners to create reports/statistics, log some informations
- More than one agent may be installed on a server.
+ * Policy-based distribution
- TODO: When a build is enqueued, the choice of which agent to use is defined by ____.
+ * Next available
- TODO: How are multi-module projects handled?
+ * Load balanced
- TODO: How are releases handled?
+ * Targeted environment matching
Modified: continuum/trunk/continuum-docs/src/site/apt/administrator_guides/index.apt
URL: http://svn.apache.org/viewvc/continuum/trunk/continuum-docs/src/site/apt/administrator_guides/index.apt?rev=734569&r1=734568&r2=734569&view=diff
==============================================================================
--- continuum/trunk/continuum-docs/src/site/apt/administrator_guides/index.apt (original)
+++ continuum/trunk/continuum-docs/src/site/apt/administrator_guides/index.apt Wed Jan 14 16:06:59 2009
@@ -40,4 +40,4 @@
* {{{shutdown.html}Shutting Down Continuum}}
- * {{{distributed-builds.html}Distributed Builds}}
\ No newline at end of file
+ * {{{distributed-builds.html}Understanding Distributed Builds}}
\ No newline at end of file
Modified: continuum/trunk/continuum-docs/src/site/apt/installation/build-agent.apt
URL: http://svn.apache.org/viewvc/continuum/trunk/continuum-docs/src/site/apt/installation/build-agent.apt?rev=734569&r1=734568&r2=734569&view=diff
==============================================================================
--- continuum/trunk/continuum-docs/src/site/apt/installation/build-agent.apt (original)
+++ continuum/trunk/continuum-docs/src/site/apt/installation/build-agent.apt Wed Jan 14 16:06:59 2009
@@ -1,29 +1,26 @@
-Installing a Build Agent
+Installing and Configuring a Build Agent
+ First, {{{http://continuum.apache.org/download.html}download}} the Continuum Build Agent standalone
+ distribution or webapp.
- The build agent is available as a web application which you will need to install into a container.
+* Standalone
- First, {{{../../download.html}download}} the webapp.
-
- Next, configure the environment.
-
- The build agent needs two things in order to function properly: A CONTINUUM_BUILDAGENT_HOME environment variable, and a
- continuum-buildagent.xml configuration file in a conf/ directory under $CONTINUUM_BUILDAGENT_HOME.
-
- Set the environment variable:
-
- export CONTINUUM_BUILDAGENT_HOME=/path/to/build/agent/home
-
- or
-
- set CONTINUUM_BUILDAGENT_HOME=c:\path\to\build\agent\home
-
- Create a directory called 'conf' under [CONTINUUM_BUILDAGENT_HOME].
-
- In the 'conf' directory, save the continuum-buildagent.xml file:
+ The Build Agent is available as a separate standalone Jetty-bundled
+ webapp. Installing it is as simple as extracting it to a preferred
+ directory.
+
+ The Build Agent runs on a specific port (defaulting to 8181), so it is possible
+ to install multiple Build Agents in one machine. It is also possible to
+ install Slaves in the same machine as the Master. To configure the Build Agent
+ to run using a different port, edit the
+ <<<$CONTINUUM_BUILDAGENT_HOME/conf/jetty.xml>>> file.
+
+ Another configuration file to look at is the
+ <<<$CONTINUUM_BUILDAGENT_HOME/conf/continuum-buildagent.xml>>>. This file must
+ be manually edited, as there is no web interface for configuration.
+--------------+
<continuum-buildagent-configuration>
@@ -39,14 +36,45 @@
</installations>
</continuum-buildagent-configuration>
+----------------+
+
+ A minimal configuration will have the following fields:
- List all of the installations available on the build agent, including Maven, Ant, JDK, etc.
+ * <<<buildOutputDirectory>>> - similar to the build output directory of Continuum
+ * <<<workingDirectory>>> - similar to the working directory of Continuum
+
+ * <<<continuumServerUrl>>> - the url pointing to the XML-RPC context of the Master Build Agent
+
+ * <<<installations>>> - a list of the installations available on the build agent, including Maven, Ant, JDK, etc.
+
Note that all of your agents need to be identical, as there is no way to control which agent Continuum chooses.
- Finally, drop the webapp into your container, and start it.
+ Now start the Build Agent by executing the startup script available in
+ <<<$CONTINUUM_BUILDAGENT_HOME/bin/>>>. For example:
+
+ <<<continuum-buildagent start>>> (or <<<continuum-buildagent console>>> on Windows).
Next, enable the Distributed Builds option in the {{{../administrator_guides/configuration.html}General Configuration}},
and {{{../administrator_guides/build-agents.html}add the agent}} to the Continuum master server.
- TODO: Provide a Jetty bundle for the agent webapp.
+ TODO: Does the Jetty bundle still want CONTINUUM_BUILDAGENT_HOME set?
+
+* Webapp
+
+ The build agent is also available as a web application which you can install into your own container.
+
+ The build agent needs two things in order to function properly: A CONTINUUM_BUILDAGENT_HOME environment variable,
+ and a continuum-buildagent.xml configuration file in $CONTINUUM_BUILDAGENT_HOME/conf.
+
+ First, set the environment variable:
+
+ export CONTINUUM_BUILDAGENT_HOME=/path/to/build/agent/home
+
+ or
+
+ set CONTINUUM_BUILDAGENT_HOME=c:\path\to\build\agent\home
+
+ Next, create the continuum-buildagent.xml file as shown above, and save it in $CONTINUUM_BUILDAGENT_HOME/conf.
+ You will need to create these directories.
+
+ Finally, drop the webapp into your container, and start it.