You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@forrest.apache.org by Steven Noels <st...@outerthought.org> on 2002/06/13 19:31:51 UTC
readying the cross-project build environment
All,
Before going further with readying some examples of libre, I was
thinking of progressing in the area of cross-project builds, i.e. having
our forrestbot ready for building other documentation targets than the
Forrest documentation itself.
Marc and I have been discussing this a bit over the day, and we came up
with the following scheme:
1) our main project.xtarget includes a parametrizable, componentized
task that basically grabs a source, builds it (as a static collection
or a deployable webapp) and optionally deploys the result
* there will be three source retrieval mechanisms:
- a simple filecopy (for projects local to the forrestbot host)
- grabbing sources from anoncvs
- fetching sources using scp (public/private key distribution
required)
* two build mechanisms
- Cocoon static file generation
- webapp assembly
* (optional) deploy methods
- filecopy (also for the war?) - readying for rsync operations
- cvs check-in (
- scp (public/private key distribution required)
2) we use an XSLT transformation using the Ant style task transforming
the forrest.xconf to a set of Ant buildfile snippets (one per project) -
it is likely we'll be using the Xalan redirect function here - I want
the output to be granular for resilience reasons - a snippet will
basically consist of a parametrized call of the task mentioned above
3) these generated targets are executed by Ant
What do you think about this? Would this approach work?
Regards,
</Steven>
Re: readying the cross-project build environment
Posted by David Crossley <cr...@indexgeo.com.au>.
Steven Noels wrote:
> Before going further with readying some examples of libre,
> I was thinking of progressing in the area of cross-project builds,
> i.e. having our forrestbot ready for building other documentation
> targets than the Forrest documentation itself.
Yes good idea to get some movement in this area. This will
swell our troops. We could each address our own local needs
for documentation too. This brings more people to work on the
overall issues. It also encourages the other Apache projects
to start experimenting.
> Marc and I have been discussing this a bit over the day, and we came up
> with the following scheme:
>
<snip good stuff, agree/>
Do they also need provision to mount their own sub-sitemap?
I imagine that most stuff will be handled by the main Forrest
src/resources/conf/sitemap.xmap
i.e. matches for todo- faq- changes- book- howto- etc.
Their project.xtarget snippet will define the name of their
skin parameter and the top bar navigation links.
However, i do wonder if they need a way to handle other things.
For example, they might have a collection of task-*.xml and
a task2document.xsl etc.
Perhaps this is a wish for the future. We probably have
enough tools to get started.
--David
Re: readying the cross-project build environment
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Ruedi Anneler - A-R-T GmbH, Switzerland wrote:
> All
> Attached you find a simple graphical representation of my modest
> understanding of the FORREST publishing system under evaluation (logical
> view).
Yeah! :-D
I'm looking at it, and will try to post a more layered view ASAP.
Thanks :-)
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: readying the cross-project build environment
Posted by "Ruedi Anneler - A-R-T GmbH, Switzerland" <ra...@a-r-t.ch>.
All
Attached you find a simple graphical representation of my modest
understanding of the FORREST publishing system under evaluation (logical
view).
My main intention is to have and to share a map supporting the
evaluation process by delivering a "common graphical base". IMHO this
could support the mapping of all aspects discussed to functional nodes
or to interfaces between such nodes. I'm eager to understand all the
things mentioned so far in discussions, and the system as a whole as
well. I've some problems to position the details discussed so far in a
broader context. I wonder what e.g. scp is, anoncvs is and the like (a
glossary/context map would be of great help, especially for newbies).
Your feedback to my map will help me to make it more accurate. After
making the drawing correct, it can/will be expanded by explanatory text.
Ruedi
Steven Noels wrote:
> All,
>
> Before going further with readying some examples of libre, I was
> thinking of progressing in the area of cross-project builds, i.e.
> having our forrestbot ready for building other documentation targets
> than the Forrest documentation itself.
>
> Marc and I have been discussing this a bit over the day, and we came
> up with the following scheme:
>
> 1) our main project.xtarget includes a parametrizable, componentized
> task that basically grabs a source, builds it (as a static collection
> or a deployable webapp) and optionally deploys the result
>
> * there will be three source retrieval mechanisms:
>
> - a simple filecopy (for projects local to the forrestbot host)
> - grabbing sources from anoncvs
> - fetching sources using scp (public/private key distribution
> required)
>
> * two build mechanisms
>
> - Cocoon static file generation
> - webapp assembly
>
> * (optional) deploy methods
>
> - filecopy (also for the war?) - readying for rsync operations
> - cvs check-in (
> - scp (public/private key distribution required)
>
> 2) we use an XSLT transformation using the Ant style task transforming
> the forrest.xconf to a set of Ant buildfile snippets (one per project)
> - it is likely we'll be using the Xalan redirect function here - I
> want the output to be granular for resilience reasons - a snippet will
> basically consist of a parametrized call of the task mentioned above
>
> 3) these generated targets are executed by Ant
>
> What do you think about this? Would this approach work?
>
> Regards,
>
> </Steven>
>
>
RE: readying the cross-project build environment
Posted by David Crossley <cr...@indexgeo.com.au>.
Marc Portier wrote:
<snip/>
> Resilience (Steven's argument) should in fact be achieved by subsequent
> ant-call statements that don't prevent further execution of the next one if
> the current one fails(have to look for the attribute setting that does
> this, quite sure it's there)
<snip/>
Surely a lot could be learned from brother Gump.
There is plenty of good documentation about Gump
that will help. I have started to read, but lots there:
http://jakarta.apache.org/gump/overview.html
--David
RE: readying the cross-project build environment
Posted by Marc Portier <mp...@outerthought.org>.
sorry guys, was intended as a short remark internal message, not a well
thought of post (I shouldn't be working this late) ... sorry for the Dutch
just so you can follow:
I suddenly realized that the generation of more then one sub-ant build file
(xslt-redirect) to call would introduce the dificulty of needing to iterate
through them.
However, the solution should be in letting the XSLT task also provide one
starting point build.xml file that calls upon the others then
And now that I can think of it in English...
Resilience (Steven's argument) should in fact be achieved by subsequent
ant-call statements that don't prevent further execution of the next one if
the current one fails(have to look for the attribute settiing that does
this, quite sure it's there)
In this approach that would mean that we actually can avoid the redirect
trick and just keep all in one build file anyway. (that is if resilience
would be the only argument)
you've see how bad my Dutch is, so how bad are my thoughts this late at
night?
-marc=
> -----Original Message-----
> From: Marc Portier [mailto:mpo@outerthought.org]
> Sent: vrijdag 14 juni 2002 0:43
> To: forrest-dev@xml.apache.org
> Subject: RE: readying the cross-project build environment
>
>
> > 2) we use an XSLT transformation using the Ant style task transforming
> > the forrest.xconf to a set of Ant buildfile snippets (one per project) -
> > it is likely we'll be using the Xalan redirect function here - I want
> > the output to be granular for resilience reasons - a snippet will
>
> resilience is sterk argument, had het na onze discussie nie meer zo in
> koppeke zitten
> enige probleem ermee is dat je moet kunnen itereren door de verschillende
> gegenereerde files... mmm andere kant die xsl zal ook de ene ant kunnen
> maken die dan die andere aanwijst natuurlijk!
>
> > basically consist of a parametrized call of the task mentioned above
> >
> > 3) these generated targets are executed by Ant
> >
> > What do you think about this? Would this approach work?
> >
> > Regards,
> >
> > </Steven>
> >
>
RE: readying the cross-project build environment
Posted by Marc Portier <mp...@outerthought.org>.
> 2) we use an XSLT transformation using the Ant style task transforming
> the forrest.xconf to a set of Ant buildfile snippets (one per project) -
> it is likely we'll be using the Xalan redirect function here - I want
> the output to be granular for resilience reasons - a snippet will
resilience is sterk argument, had het na onze discussie nie meer zo in
koppeke zitten
enige probleem ermee is dat je moet kunnen itereren door de verschillende
gegenereerde files... mmm andere kant die xsl zal ook de ene ant kunnen
maken die dan die andere aanwijst natuurlijk!
> basically consist of a parametrized call of the task mentioned above
>
> 3) these generated targets are executed by Ant
>
> What do you think about this? Would this approach work?
>
> Regards,
>
> </Steven>
>
Re: readying the cross-project build environment
Posted by Stefan Bodewig <bo...@apache.org>.
On 17 Jun 2002, Stefan Bodewig <bo...@apache.org> wrote:
> On Sat, 15 Jun 2002, Stefano Mazzocchi <st...@apache.org> wrote:
>
>> The only difference is that Gump doesn't affect anything directly
>> (we don't build distributions with Gump or things like that),
>
> Nit picking, we build a couple of nightly builds with Gump.
I just realized a bit later that Sam had already said that, sorry.
Sam, you should use a MUA that properly handles the References-Header
so that threading doesn't get broken 8-)
In addition, I've used Gump extensively when working on Ant's 1.5
branch - which shows how important it also would be for Forrest to
support branches. See <http://cvs.apache.org/~bodewig/gump/> where my
Gump results and "nightly" builds of Ant's 1.5 branch can be found.
In the same manner Forrestbot should be able to generate "beta"
documentation for website "refactorings".
As for security considerations. I'm not 100% sure Sam handles it the
same way that I do for above results. The generetaion of content can
be automated (and in Sam's case it is), but the publishing is a manual
step right now - it still requires my passphrase to make the content
travel over to the Apache box.
Some rsync mechanism would be preferable for websites, I guess.
Stefan
Re: readying the cross-project build environment
Posted by Stefan Bodewig <bo...@apache.org>.
On Sat, 15 Jun 2002, Stefano Mazzocchi <st...@apache.org> wrote:
> The only difference is that Gump doesn't affect anything directly
> (we don't build distributions with Gump or things like that),
Nit picking, we build a couple of nightly builds with Gump.
Stefan
Vindico == GumpNG (was Re: readying the cross-project build environment)
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Stefano Mazzocchi wrote:
>
> In my vision, forrestbot is automatic, just like Gump, it runs, build,
> copies and nags. Everyday. Helps you when you do the right thing and
> nags you when you do the bad thing, just like a wise boss :)
>
> It's simple, effective and keeps up the social experiment concept that
> Gump started.
>
> The only difference is that Gump doesn't affect anything directly (we
> don't build distributions with Gump or things like that), while Forrest
> will do both: perform the social test *and* do actual work on the
> project (by publishing their docos)
>
> Is this difference important enough to force us to change the Gump
> model?
>
> What do you think?
I've been discussing this topic for weeks on alexandria-dev (Gump),
krysalis-dev (Centipede), partially here and with many friends :-)
We've come to the unanimous conclusion that there is a Gump-like work
pattern that can be general enough to work with Gump and Forrest, a sort
of big workspace job handler.
This has made me make a proposal of intent on the Gump list that has had
a good reception.
I will refer to this new thing to as "Vindico", the name of one of the
other Alexandria proposals.
There are still conceptual-architectural things to resolve that prevent
me from making a solid-final proposal, and I would like to discuss it
with you guys :-)
-oOo-
(warning: below is my current stream of consciousness;
please read with an open mind)
Attatched is a very crude image of what I currently have in mind, with
all the imperfections and doubts ;-)
Basically, Forrest and Gump could be just modules to plug into a generic
system that runs them on project definitions.
(on the lower part from left to right)
Sources are gotten from a repository, placed locally and "something" is
run on them.
Then results are published somewhere.
As you see, this description can be used to define a build system:
sources are copied in a "build" dir, an ant target is run, then results
are put in a resulting dir under "build".
What makes this different? It works on *workspaces*, ie multiple projects.
Centipede has build components called cents already: junit cent,
forrest.cent...
Yes, forrest.cent (still to upgrade, but the concept remains). I should
be able to run forrest on my local docs during the build... why then
duplicate Forrest as a .cent and as a bot?
Another step: on krysalis-dev, we have defined a buildmap, ie a big
build descriptor that basically assembles macro build components.
Similarly as in Avalon stuff, Centipede becomes a Container of build
Components for a local build.
Vindico would be a Container on a higher level.
So we would have this conceptual build-space:
Workspace (list of projects to build)
|
Projectspace
|
MiniProjectSpace (like Excalibur projects)
Forrest should be able to operate in all spaces... how to do it?
I think that we should keep Forrest, Gump, Centipede, Alexandria (source
xref and Javadocs) in ProjectSpace, and collapse MiniProjectSpace and
Projectspace in a single space; a Projectspace of many mimiprojects is
in reality a Workspace of Projectspaces.
So we will have:
- workspace builders
- project builders
A workspace builder should be able to call project builders on each
project in a workspace or on an aggregation of them, so it needs a
dependency system, both between projectspaces and between build systems.
-oOo-
In essence, Vindico should be able to get the required sources and call
Forrest (for docs), Gump (for fresh builds), Centipede and Maven (for
daily builds), AlexandriaNG (for Javadocs) on each project and publish
the results, all while keeping track of dependencies.
Any ideas?
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: readying the cross-project build environment
Posted by Stefano Mazzocchi <st...@apache.org>.
Your emails often remind me of the wise quote: intelligence is about
posing questions, not answering them. This is for sure one of them.
Diana Shannon wrote:
>
> On Thursday, June 13, 2002, at 01:31 PM, Steven Noels wrote:
>
> Will site updates be based on a manual trigger or time-based trigger
> (once a day, once an hour, etc.)?
This hasn't been defined.
There has been some resistence in the past to automate tasks over the
Apache servers: this is 99% because of security risks they pose since
gaining conctrol of an automated task would lead to asynchronous attacks
which are much harder to track down.
At the same time, since the forrest process will not reside on the same
machine that serves the generated content, this shouldn't be an issue
even for root@apache.org which is normally *very* conservative (and for
a good reason!)
One big not about security of apache.org: imagine a scenario where a
black-hat hacker gains root access on www.apache.org, places a backdoor
in an official apache distribution and removes its traces.
Using proper timing and PR relationships, this person (or group) will be
able to do massive danger to the online economy first, and to the apache
brand second.
By wise forecast of stock market fluctuations, the attackers could
potentially earn millions of dollars.
This is why the apache infrastructure is managed in a very conservative
way: protecting apache means protecting the web and the ecosystem
(social and economical) on top of it, which is worth billion of dollars
and influences millions of individuals.
Since Forrest wants to become a piece of the apache infrastructure, we
must be aware of how important security should be for us.
This said, we should ask how dangerous is to have automatic forrest runs
and what is best from a usability point of view of the apache
infrastructure.
> > * there will be three source retrieval mechanisms:
> >
> > - a simple filecopy (for projects local to the forrestbot host)
> > - grabbing sources from anoncvs
> > - fetching sources using scp (public/private key distribution
> > required)
>
> Sources are retrieved from what cvs branch? For example, with Cocoon,
> would this be the release branch?
Gump is able to specify what branch to checkout. Forrest should do the
same.
> What if someone is in the process of
> commiting a bunch of modified files which, if retrieved by Forrestbot at
> the wrong time, causes a build failure?
Eh, shit happens, one could say :)
No, seriously, this hasn't been a problem on Gump, which is scheduled to
run once a day. For manually-started systems, this could be worse... but
we could use the CVS lock mechanism for that.... even if it's normally
suggested *not* to use locking in a public environment.
> What if I'm updating release and want to turn off any Forrest retrieval
> until I'm finished. Will I be able to do that?
One simple (hacky?) way of doing this is to place a 'retrievable' flag
someplace in the CVS module. It could be a file, an element in the xgump
file and so on.
This is much ligther than a lock since it doesn't require the person
who locked the resource to unlock it (which could potentially create
deadlocks), but also solves the issue.
There is one question though (Sam might want to jump in here): this flag
probably breaks the concept of continous integration and might be abused
by projects that instead of fixing their docs to avoid the nagging
system, they make the project 'unretrievable' for a long period of time,
loosing the benefit of continous integration.
I'm curious to see what others think about this.
> >
> > * two build mechanisms
> >
> > - Cocoon static file generation
> What if a build fails? What happens? Is an email sent to a list?
Yes, this is the behavior I'd like. Of course, it should also provide a
gump-like list of results on a web page.
> Is the
> build tried again later (after another fresh retrieval)?
No, it should not, simply wait for the next round. Of course, if the
build fails, the generated documents should not be uploaded on the
production site.
> > - webapp assembly
> >
> > * (optional) deploy methods
> >
> > - filecopy (also for the war?) - readying for rsync operations
> > - cvs check-in (
>
> Do we need live site cvs equivalent (as is the case today, where the web
> site is a locally checked out copy of the live site repository)?
We have been discussing with root@apache.org about this and it might
well be a possibility. If we add a tag for each update, we are able to
do direct revisioning of the web site.
> Do we
> need version history of what is shown on the web site?
It could be desirable, but I wouldn't think so for our own documents
since:
1) we already revision the original XML sources
2) we include the build documents in every project release
Since the HTML files are simpy the 'result' of the build process, it
would be like versioning both the java and the class files, it doesn't
really make sense since both contain the same exact information.
> What if a new web
> site version, even if built correctly, is problematic and needs to be
> changed immediately. It would be nice to be able to revert to a previous
> version, as is possible right now (through the live site repository).
Then you revert the documents in CVS and the website is rebuilt
correctly. This is how you do patches in java and you recompile the
classes... you don't checkout the previous classes.
> > - scp (public/private key distribution required)
> >
> > 2) we use an XSLT transformation using the Ant style task transforming
> > the forrest.xconf to a set of Ant buildfile snippets (one per
> > project) - it is likely we'll be using the Xalan redirect function
> > here - I want the output to be granular for resilience reasons - a
> > snippet will basically consist of a parametrized call of the task
> > mentioned above
> >
> > 3) these generated targets are executed by Ant
>
> How can Forrestbot be manually stopped, in case of a problem?
I wouldn't implement such a feature. Forrestbot shouldn't be generally
stoppable, just like Gump.
> Will each
> individual project still have that control, or will it be only within
> control of Forrest (committers)?
In my vision, forrestbot is automatic, just like Gump, it runs, build,
copies and nags. Everyday. Helps you when you do the right thing and
nags you when you do the bad thing, just like a wise boss :)
It's simple, effective and keeps up the social experiment concept that
Gump started.
The only difference is that Gump doesn't affect anything directly (we
don't build distributions with Gump or things like that), while Forrest
will do both: perform the social test *and* do actual work on the
project (by publishing their docos)
Is this difference important enough to force us to change the Gump
model?
What do you think?
--
Stefano Mazzocchi One must still have chaos in oneself to be
able to give birth to a dancing star.
<st...@apache.org> Friedrich Nietzsche
--------------------------------------------------------------------
Re: readying the cross-project build environment
Posted by Diana Shannon <sh...@apache.org>.
On Thursday, June 13, 2002, at 01:31 PM, Steven Noels wrote:
Will site updates be based on a manual trigger or time-based trigger
(once a day, once an hour, etc.)?
> * there will be three source retrieval mechanisms:
>
> - a simple filecopy (for projects local to the forrestbot host)
> - grabbing sources from anoncvs
> - fetching sources using scp (public/private key distribution
> required)
Sources are retrieved from what cvs branch? For example, with Cocoon,
would this be the release branch? What if someone is in the process of
commiting a bunch of modified files which, if retrieved by Forrestbot at
the wrong time, causes a build failure?
What if I'm updating release and want to turn off any Forrest retrieval
until I'm finished. Will I be able to do that?
>
> * two build mechanisms
>
> - Cocoon static file generation
What if a build fails? What happens? Is an email sent to a list? Is the
build tried again later (after another fresh retrieval)?
> - webapp assembly
>
> * (optional) deploy methods
>
> - filecopy (also for the war?) - readying for rsync operations
> - cvs check-in (
Do we need live site cvs equivalent (as is the case today, where the web
site is a locally checked out copy of the live site repository)? Do we
need version history of what is shown on the web site? What if a new web
site version, even if built correctly, is problematic and needs to be
changed immediately. It would be nice to be able to revert to a previous
version, as is possible right now (through the live site repository).
> - scp (public/private key distribution required)
>
> 2) we use an XSLT transformation using the Ant style task transforming
> the forrest.xconf to a set of Ant buildfile snippets (one per
> project) - it is likely we'll be using the Xalan redirect function
> here - I want the output to be granular for resilience reasons - a
> snippet will basically consist of a parametrized call of the task
> mentioned above
>
> 3) these generated targets are executed by Ant
How can Forrestbot be manually stopped, in case of a problem? Will each
individual project still have that control, or will it be only within
control of Forrest (committers)?
Diana