You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gump.apache.org by le...@apache.org on 2004/12/27 17:58:38 UTC
svn commit: r123431 - in gump/dynagump/trunk/webapp: . docs docs/howto
Author: leosimons
Date: Mon Dec 27 08:58:37 2004
New Revision: 123431
URL: http://svn.apache.org/viewcvs?view=rev&rev=123431
Log:
Work on new documentation. Still a lot to do...
Added:
gump/dynagump/trunk/webapp/docs/HowGumpWorks.xml
- copied, changed from r123404, gump/dynagump/trunk/webapp/docs/index.xml
gump/dynagump/trunk/webapp/docs/Philosophy.xml
- copied, changed from r123404, gump/dynagump/trunk/webapp/docs/index.xml
gump/dynagump/trunk/webapp/docs/howto/
gump/dynagump/trunk/webapp/docs/howto/index.xml
gump/dynagump/trunk/webapp/docs/process.png (contents, props changed)
Modified:
gump/dynagump/trunk/webapp/docs/index.xml
gump/dynagump/trunk/webapp/docs/sitemap.xmap
gump/dynagump/trunk/webapp/index.xml
Copied: gump/dynagump/trunk/webapp/docs/HowGumpWorks.xml (from r123404, gump/dynagump/trunk/webapp/docs/index.xml)
Url: http://svn.apache.org/viewcvs/gump/dynagump/trunk/webapp/docs/HowGumpWorks.xml?view=diff&rev=123431&p1=gump/dynagump/trunk/webapp/docs/index.xml&r1=123404&p2=gump/dynagump/trunk/webapp/docs/HowGumpWorks.xml&r2=123431
==============================================================================
--- gump/dynagump/trunk/webapp/docs/index.xml (original)
+++ gump/dynagump/trunk/webapp/docs/HowGumpWorks.xml Mon Dec 27 08:58:37 2004
@@ -1,30 +1,99 @@
<page>
- <title>Gump | Documentation</title>
+ <title>Gump | Documentation | How Gump Works</title>
<path>
<item name="Home" url="../"/>
- <current name="Documentation"/>
+ <item name="Documentation" url="./"/>
+ <current name="How Gump Works"/>
</path>
<tabs>
<item name="Results" url="../results/"/>
- <current name="Documentation"/>
+ <current name="Documentation" url="./"/>
+ <item name="Wiki" url="http://wiki.apache.org/gump/"/>
<item name="Home" url="../"/>
</tabs>
- <sidebar>
- <item name="Philosophy" url="">
- <description>Here is where you find out more about the gump philosophy and why it was built.</description>
- </item>
- <item name="Metadata" url="">
- <description>Here is where you find out more about the gump metadata descriptors.</description>
- </item>
- </sidebar>
-
<content>
- <h1>Documentation</h1>
- <p>Just a placeholder for now but hopefully it will grow.</p>
+<h1>How Gump Works</h1>
+<p>Please note that this documentation specifies the plans we have,
+rather than what we actually do right now. The key difference is that
+Gump will be made much more intelligent about whom to send what e-mail
+and what to put in those e-mails. However, the basic ideas for all this
+are quickly falling into place.</p>
+<h2>The actual process</h2>
+<img src="process.png" width="300" align="left"/>
+<p>Lets start with the low-level overview. Gump has two main parts. The
+first part takes care of actually performing the build. Internally, we
+tend to call this part "Gumpy", since its written in python. The second
+part takes care of the reporting and introspection. It is written in
+java using Apache Cocoon.</p>
+<h3>Gumpy, the builder</h3>
+<p>Gumpy is a python script that is fired from cron every few hours. It
+does some self-initialization, looks at it environment, then starts
+figuring out what to do. It updates the gump CVS and SVN repositories.
+The next task is to read in all gump project definitions. For
+maven-based projects, we read in maven's project.xml files and do some
+processing on those. For projects using other build tools (like Ant),
+an XML descriptor describing that project is kept in the Gump
+subversion repository.</p>
+<p>All these project definitions are merged into a database, which we
+call
+the GumpModelDatabase. Once our database is filled, we start walking it
+to figure out the dependencies between all the projects. We then figure
+out a build order. With our model serialized, we know exactly what to
+do.</p>
+<p>In our just-determined order, we run CVS or SVN updates for our
+projects, then try to build them (using ant or maven). The precise
+build commands to use as well as any environment settings (such as
+references to installed packages like javamail) are all figured out
+from the gump project definitions (see HowToGetBuiltByGump for more
+information on how that works).</p>
+<p>Now, the tricky bit is specifying dependencies. Rather than use the
+maven repository or some other place where jars are stored, we set the
+system classpath to include all the dependencies declared for a
+project. And rather than point to a specific versioned jar or library,
+we opt to use the output of the projects that we just built straight
+from bleeding edge SVN.</p>
+<p>Gumpy keeps track of everything it does, and what the results of
+those
+actions are. Success and failure states, build logs, and much more are
+all saved and stored into another database, which we call the
+ResultMetadataDatabase. This database doesn't just store the current
+run, but maintains information for all previous runs as well.</p>
+<p>If a project fails to build, Gumpy will take special actions to try
+and
+figure out what caused that failure. The algorithm used to do this is
+based on a bit of graph theory, which won't be described here.
+Interested people should look at the GumpAlgorithm page. (NOTE: this
+algorithm has been written but not yet implemented!) Suffice saying
+that Gumpy is often able to determine exactly which CVS or SVN commit
+caused the build failure, and in other instances is able to give a good
+hint.</p>
+<h3>DynaGump, the reporter</h3>
+<p>Once Gumpy is done, its time for DynaGump to do its thing. DynaGump
+is
+a web application which hooks into the ResultMetadataDatabase to
+generate reports. These reports allow anyone to find out how stable the
+build for a particular project is, what the main issues are, and the
+like. We produce figures describing the build history of a project.
+This can help developers figuring out which dependencies are causing
+them problems, but also how their changes affect the projects that
+depend on them (their dependees).</p>
+<p>If there's an issue, DynaGump figures out who needs to know about
+that
+and sends them e-mail. For example, if a project introduced a code
+change that results in one or more of their dependees failing to build,
+we'll let that project know, as well as the project that broke. This
+allows all the developers involved to start collaborating immediately
+to resolve the issue.</p>
+<p>Advanced reporting and workflow features (similar to the Jira
+Dashboard) feature will be made available. For example, you will be
+able to create a custom page which lists information about all the
+projects that you are interested in, or get that information as an RSS
+feed.</p>
+
</content>
</page>
Copied: gump/dynagump/trunk/webapp/docs/Philosophy.xml (from r123404, gump/dynagump/trunk/webapp/docs/index.xml)
Url: http://svn.apache.org/viewcvs/gump/dynagump/trunk/webapp/docs/Philosophy.xml?view=diff&rev=123431&p1=gump/dynagump/trunk/webapp/docs/index.xml&r1=123404&p2=gump/dynagump/trunk/webapp/docs/Philosophy.xml&r2=123431
==============================================================================
--- gump/dynagump/trunk/webapp/docs/index.xml (original)
+++ gump/dynagump/trunk/webapp/docs/Philosophy.xml Mon Dec 27 08:58:37 2004
@@ -1,30 +1,29 @@
<page>
- <title>Gump | Documentation</title>
+ <title>Gump | Documentation | Philosophy</title>
<path>
<item name="Home" url="../"/>
- <current name="Documentation"/>
+ <item name="Documentation" url="./"/>
+ <current name="Philosophy"/>
</path>
<tabs>
<item name="Results" url="../results/"/>
- <current name="Documentation"/>
+ <current name="Documentation" url="./"/>
<item name="Home" url="../"/>
</tabs>
- <sidebar>
- <item name="Philosophy" url="">
- <description>Here is where you find out more about the gump philosophy and why it was built.</description>
- </item>
- <item name="Metadata" url="">
- <description>Here is where you find out more about the gump metadata descriptors.</description>
- </item>
- </sidebar>
-
<content>
- <h1>Documentation</h1>
- <p>Just a placeholder for now but hopefully it will grow.</p>
- </content>
+
+<h1>Gump Philosophy</h1>
+<p>
I've read with great amusement the motivations attributed to why Gump
was originally written. The reasons were most definitely not altruistic.
To understand why Gump was written, you need to understand a bit of
the history of the Jakarta and XML projects. But, first, here is a
concise and concrete summary:
</p>
<ul>
<li>It is easier to get a patch accepted against the most current version
of a project than some previous baseline.</li>
<li>It is much more effective to express your opinion on a change that
will affect you <strong>before</strong> that change is released than afterwards.</li>
</ul>
+<h2>1999</h2>
<p>To help set the context, in 1999 the ink was hardly dry on the XML
specifications; and what is now known as XSLT, FOP, and XPath were all
interwoven in a confused jumble. JDK 1.2 was only recently introduced
and wouldn't be widely adopted for many years (to this day, a number
of projects retain JDK 1.1 compatibility).</p>
<p>In this harsh environment, it is amazing that open source software grew
at all. In order to make progress, early versions of Xalan required
specific versions of Xerces. Cocoon required specific versions
of both. Cocoon also depended on other projects which were two years
away from having alpha releases - Turbine and Avalon.</p>
<p>And virtually every project required their own specific version of
Ant.</p>
<p>I found this time very frustrating. As an active developer at the
time on projects such as Ant and Tomcat, I wanted to make improvements
to these projects; but was precluded from using the fruits of my own
labors when working with projects like cocoon. Ultimately, I found
that I could get what I wanted by writing my own scripts to set up
the classpath as I wanted.</p>
+
+<h2>2000</h2>
<p>In 2000, Ant developed the ability to control one's classpath.
This completely defeated my ability to select which versions I wanted
to use. At first, this was done on projects I wasn't active in, but
eventually it encroached everywhere.</p>
<p>Then one day a change was checked in that completely broke me. As
an active committer on the project, I "-1'ed" the change. That -1 was
declared invalid, shouted down, and ignored.</p>
<p>So I did the only thing I could think of. I hacked Ant so that I
could tell it to ignore the classpath elements in build scripts. All of
them. This was then and remains now a dirty, nasty hack. It meant that
test cases could not be run using the recently compiled classes unless I
knew in advance where the classes were to be placed. It means that cactus
can't be built in one run against both Tomcat 3 and Tomcat 4.</p>
<p>But it was the only way I could regain control.</p>
<h2>2001</h2>
<p>Now that I had a technological solution to my problems, the number of
projects I was interested in following grew. This resulted in a number
of problems of scalability: the first being that the scripts were
becoming more difficult to maintain, and second that the long chains of
dependencies were fragile and only worked if the participants were
interested in supporting the combination of projects that I wanted to
run.</p>
<p>The solution to my problems was to automate the generation of these
scripts and to publish the results.</p>
<p>The results exceeded my expectations. People who claimed that version
incompatibility problems were rare and easily addressed could now see
that it was a daily event - in fact to this day, I have only seen a
clean run a mere handful of times. But what I could provide to developers
was timely feedback on the impact of their changes on others, and in turn
help them make their case against changes that would affect them.</p>
<p>Most telling of all were the changes to three projects: Ant, Turbine,
and Avalon. All three were notorious for making changes which broke
their users. In the cases of Turbine and Avalon, these projects were
perennially in alpha, so it was argued that nobody should depend on them.
Now, there are stable releases of each. Deprecation is now actively
used by all three.</p>
+
+<h2>2002</h2>
<p>There is a growing trend for projects to split into multiple cvs trees -
Avalon, Tomcat, and Turbine are examples. The creation of "commons"
areas shared between projects makes it more and more likely that other
developers are interested in multiple code bases and face some of the same
problems that I have faced.</p>
<p>For these reasons, I'd like to see Gump evolve into an developer tool.
The overall concept is as follows: one describes using simple XML
constructs the combination of projects and versions thereof that one
desires in their workspace, and customized scripts are built to order.
The results of one's builds are conveniently published as HTML.</p>
<p>Perhaps even with a GUI.</p>
<p>With Ant becoming stable and backwards compatible, I'd also like to
see a movement towards it becoming something that you install and
occasionally upgrade, as opposed to something that you download with
each and every project.</p>
<p>Now that XML parsers and XSL translators are pluggable and default
versions will be shipping with the JDK, it would also be nice if each
project stopped shipping duplicate copies of these jars too, and instead
used the one of your choosing.</p>
+
+<h2 class="boxed">2003</h2>
<p>As a heavy user of OSS software, combining lots of it into deep stacks,
<a href="http://www.krysalis.org/version/jar-hell.html">JAR Hell</a> became
a pet peave of <a href="http://www.apache.org/~ajack/">Adam Jack</a>.
Supporting heathly re-use became a passion.
</p>
<p>Gump keeps software re-use fresh and healthy and reduces JAR Hell.</p>
</content>
</page>
Added: gump/dynagump/trunk/webapp/docs/howto/index.xml
Url: http://svn.apache.org/viewcvs/gump/dynagump/trunk/webapp/docs/howto/index.xml?view=auto&rev=123431
==============================================================================
--- (empty file)
+++ gump/dynagump/trunk/webapp/docs/howto/index.xml Mon Dec 27 08:58:37 2004
@@ -0,0 +1,31 @@
+<page>
+
+ <title>Gump | Documentation | How-To's</title>
+
+ <path>
+ <item name="Home" url="../../"/>
+ <item name="Documentation" url="../"/>
+ <current name="How-To's"/>
+ </path>
+
+ <tabs>
+ <item name="Results" url="../../results/"/>
+ <current name="Documentation" url="../"/>
+ <item name="Wiki" url="http://wiki.apache.org/gump/"/>
+ <item name="Home" url="../../"/>
+ </tabs>
+
+ <content>
+ <h1>How-To Documentation</h1>
+<p>Concrete guides to working with or on gump:</p>
+<ul>
+ <li><a href="HowToFixGumpBuildProblems.html">How To Fix Gump Build
+Problems</a>: this document explains the steps you should take if gump send you or
+a project you participated in an e-mail.</li>
+ <li><a href="HowToGetBuiltByGump.html">How To Get Built By Gump</a>: this document explains how to add new projects to gump, focussing
+mostly on the gump object model, our xml descriptors that tell gump how
+to "do its thing".</li>
+</ul>
+ </content>
+
+</page>
Modified: gump/dynagump/trunk/webapp/docs/index.xml
Url: http://svn.apache.org/viewcvs/gump/dynagump/trunk/webapp/docs/index.xml?view=diff&rev=123431&p1=gump/dynagump/trunk/webapp/docs/index.xml&r1=123430&p2=gump/dynagump/trunk/webapp/docs/index.xml&r2=123431
==============================================================================
--- gump/dynagump/trunk/webapp/docs/index.xml (original)
+++ gump/dynagump/trunk/webapp/docs/index.xml Mon Dec 27 08:58:37 2004
@@ -10,21 +10,39 @@
<tabs>
<item name="Results" url="../results/"/>
<current name="Documentation"/>
+ <item name="Wiki" url="http://wiki.apache.org/gump/"/>
<item name="Home" url="../"/>
</tabs>
<sidebar>
- <item name="Philosophy" url="">
- <description>Here is where you find out more about the gump philosophy and why it was built.</description>
+ <item name="Philosophy" url="Philosophy.html">
+ <description>Find out why gump was built and what our goals are with the gump project.</description>
</item>
- <item name="Metadata" url="">
- <description>Here is where you find out more about the gump metadata descriptors.</description>
+ <item name="How Gump Works" url="HowGumpWorks.html">
+ <description>Get the 10-mile-high overview of the gump architecture and workflow.</description>
+ </item>
+ <item name="How-To's" url="howto/">
+ <description>Get concrete guides to working with and on gump.</description>
+ </item>
+ <item name="Metadata" url="http://gump.apache.org/metadata/index.html">
+ <description>Look up the details of the gump descriptor syntax.</description>
</item>
</sidebar>
<content>
<h1>Documentation</h1>
- <p>Just a placeholder for now but hopefully it will grow.</p>
+<p>Gump is not a trivial project. We compile thousands of sourcefiles
+every night, and try to send the right reports to hundreds of projects
+when something bad happens. This means that there's a lot to "get"
+about gump. Besides the information on this part of our website, make
+sure to take a look at:</p>
+<ul>
+ <li>the <a href="http://gump.apache.org/">Main gump website</a> -- you're
+looking at our new documentation (ie, a work in progress). The main website
+contains a wealth of other information.</li>
+ <li>the <a href="http://wiki.apache.org/gump/">Gump wiki</a> -- Gump has an
+active wiki with a lot of tidbits of information.</li>
+</ul>
</content>
</page>
Added: gump/dynagump/trunk/webapp/docs/process.png
Url: http://svn.apache.org/viewcvs/gump/dynagump/trunk/webapp/docs/process.png?view=auto&rev=123431
==============================================================================
Binary file. No diff available.
Modified: gump/dynagump/trunk/webapp/docs/sitemap.xmap
Url: http://svn.apache.org/viewcvs/gump/dynagump/trunk/webapp/docs/sitemap.xmap?view=diff&rev=123431&p1=gump/dynagump/trunk/webapp/docs/sitemap.xmap&r1=123430&p2=gump/dynagump/trunk/webapp/docs/sitemap.xmap&r2=123431
==============================================================================
--- gump/dynagump/trunk/webapp/docs/sitemap.xmap (original)
+++ gump/dynagump/trunk/webapp/docs/sitemap.xmap Mon Dec 27 08:58:37 2004
@@ -8,8 +8,36 @@
<map:pipeline>
+ <!-- index -->
<map:match pattern="">
<map:generate src="index.xml"/>
+ <map:transform src="../stylesheets/page2html.xslt"/>
+ <map:serialize type="xhtml"/>
+ </map:match>
+
+ <!-- index -->
+ <map:match pattern="**/">
+ <map:generate src="{1}/index.xml"/>
+ <map:transform src="../stylesheets/page2html.xslt"/>
+ <map:serialize type="xhtml"/>
+ </map:match>
+
+ <!-- images -->
+ <map:match pattern="**.gif">
+ <map:read mime-type="images/gif" src="{1}.gif"/>
+ </map:match>
+
+ <map:match pattern="**.png">
+ <map:read mime-type="images/png" src="{1}.png"/>
+ </map:match>
+
+ <map:match pattern="**.jpg">
+ <map:read mime-type="images/jpeg" src="{1}.jpg"/>
+ </map:match>
+
+ <!-- docs -->
+ <map:match pattern="**.html">
+ <map:generate src="{1}.xml"/>
<map:transform src="../stylesheets/page2html.xslt"/>
<map:serialize type="xhtml"/>
</map:match>
Modified: gump/dynagump/trunk/webapp/index.xml
Url: http://svn.apache.org/viewcvs/gump/dynagump/trunk/webapp/index.xml?view=diff&rev=123431&p1=gump/dynagump/trunk/webapp/index.xml&r1=123430&p2=gump/dynagump/trunk/webapp/index.xml&r2=123431
==============================================================================
--- gump/dynagump/trunk/webapp/index.xml (original)
+++ gump/dynagump/trunk/webapp/index.xml Mon Dec 27 08:58:37 2004
@@ -9,11 +9,39 @@
<tabs>
<item name="Results" url="results/"/>
<item name="Documentation" url="docs/"/>
+ <item name="Wiki" url="http://wiki.apache.org/gump/"/>
<current name="Home"/>
</tabs>
<content>
- <h1>Welcome to Apache Gump</h1>
+ <h1>Welcome to Apache Gump!</h1>
+<p>Gump is Apache's continuous integration tool. It is written in
+python
+and fully supports Apache Ant, Apache Maven and other build tools. Gump
+is unique in that it builds and compiles software against the latest
+development versions of those projects. This allows gump to detect
+potentially incompatible changes to that software just a few hours
+after those changes are checked into the version control system.
+Notifications are sent to the project team as soon as such a change is
+detected, referencing more detailed reports available online.</p>
+<p>You can set up and run Gump on your own machine and run it on your
+own
+projects, however it is currently most famous for building most of
+Apache's java-based projects and their dependencies (which constitutes
+several million lines of code split up into hundreds of projects). For
+this purpose, the gump project maintains its own dedicated server.
+</p>
+
+<h1>Build results</h1>
+<p>Apache dedicates a complete machine to running gump. The
+<a href="results/">Build results</a> page provides a view of the data
+gump generates for apache.</p>
+
+<h1>Documentation</h1>
+<p>We're working on a <a href="docs/">new set of documentation</a>. A
+lot of material is still available on
+<a href="http://gump.apache.org/">our main website</a> only. Also make
+sure to take a look at the <a href="http://wiki.apache.org/gump">wiki</a>.</p>
</content>
</page>