You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@forrest.apache.org by David Crossley <cr...@apache.org> on 2005/01/22 07:49:58 UTC
Re: [Proposal] forrestbot at apache.org
Finally we have a demonstration forrestbot running on
an ASF machine. It is on brutus, the untrusted machine
that also runs Apache Gump. That means it can only
ever be a demo, i.e. the sites that it generates
cannot be put into production. Still it is one big step.
A proper machine will be available later.
http://brutus.apache.org:36366/forrestbot-webapp/
It uses the 0.7-dev trunk. The sites are built
automatically every 8 hours.
Anyone can view the built websites and the build logs.
Forrest committers can login and do manual build/deploy.
Find the password in my home directory on minotaur.
Some of the sites have build issues. We should get
those fixed before we announce this more widely.
--David
Re: running multiple forrestbots
Posted by David Crossley <cr...@apache.org>.
Dave Brondsema wrote:
> David Crossley wrote:
[snip]
> >Yes, i had already thought of that too. I was wondering if
> >when forrestbot starts we can create a flag file in the
> >forrestbot-logs directory, which gets removed at the end
> >of the process. I have started a coding experiment.
> >We would probably also need a way to check if there was
> >a stale flag file lying around.
>
> Hrm, that is a tricky issue. The "right way" would be to see if the
> forrest process is still running in the system (But what if you had
> multiple users with write-access to a remotely mounted directory.
> That'd mean different systems with only the log dir in common). But
> doing system-level work would be non-portable anyway. So how to know
> the difference between a long-running forrestbot and one that died
> unexpectedly?
I wondered if adding a parameter to the project configuration file
which provided an indication of how long it should take. Mmm, that
sounds clunky and non-portable too.
> In the forrestbot webapp, it says "running" if the log file is
> incomplete (no BUILD SUCCESFUL). But if the log file's date (last
> updated date, IIRC) is more than a couple minutes old it says "failed"
> instead. Perhaps something like that would work.
In the current code, it is "less than 60 seconds" which is way
too short for something like cocoon-trunk.
A releated issue is that the logfile only gets created when forrest
actually starts. With cocoon-trunk the 'svn update' takes a long
time so the logfile isn't yet updated. That should be easy to fix.
--David
Re: running multiple forrestbots
Posted by Dave Brondsema <da...@brondsema.net>.
David Crossley wrote:
> Dave Brondsema wrote:
>
>>David Crossley wrote:
>>
>>>Dave Brondsema wrote:
>>>
>>>>David Crossley wrote:
>>>>
>>>>
>>>>>I suddenly realised the issue with "cocoon-trunk". It needs
>>>>>to run its 'build docs' before running 'forrest'. It generates
>>>>>some extra source documentation before forrest starts.
>>>>>
>>>>>There is a global parameter "forrest-exec" which can call
>>>>>a shell script to do other things, then call forrest.
>>>>
>>>>A better option would be to have the forrestbot buildfile for
>>>>cocoon-trunk run its 'build docs', then you don't affect any other
>>>>projects.
>>>
>>>That is okay, my forrest-exec shell wrapper has a case statement
>>>to switch based on siteName to handle various special "pre-forrest"
>>>operations. However, the length of time that it takes, causes
>>>the issue that i mention below - forrestbot run via the webapp
>>>will not know if there is a forrestbot already running via
>>>cron and vice versa. I think that can be fixed in my wrapper
>>>by checking/setting the date on the cocoon-trunk forrestbot log.
>>
>>Hmm.. perhaps this should be added to forrest[bot] itself. Not sure how..
>
>
> Yes, i had already thought of that too. I was wondering if
> when forrestbot starts we can create a flag file in the
> forrestbot-logs directory, which gets removed at the end
> of the process. I have started a coding experiment.
> We would probably also need a way to check if there was
> a stale flag file lying around.
>
>
Hrm, that is a tricky issue. The "right way" would be to see if the
forrest process is still running in the system (But what if you had
multiple users with write-access to a remotely mounted directory.
That'd mean different systems with only the log dir in common). But
doing system-level work would be non-portable anyway. So how to know
the difference between a long-running forrestbot and one that died
unexpectedly?
In the forrestbot webapp, it says "running" if the log file is
incomplete (no BUILD SUCCESFUL). But if the log file's date (last
updated date, IIRC) is more than a couple minutes old it says "failed"
instead. Perhaps something like that would work.
--
Dave Brondsema : dave@brondsema.net
http://www.splike.com : programming
http://csx.calvin.edu : student org
http://www.brondsema.net : personal
Re: [Proposal] forrestbot at apache.org
Posted by David Crossley <cr...@apache.org>.
Dave Brondsema wrote:
> David Crossley wrote:
> >Dave Brondsema wrote:
> >>David Crossley wrote:
> >>
> >>>I suddenly realised the issue with "cocoon-trunk". It needs
> >>>to run its 'build docs' before running 'forrest'. It generates
> >>>some extra source documentation before forrest starts.
> >>>
> >>>There is a global parameter "forrest-exec" which can call
> >>>a shell script to do other things, then call forrest.
> >>
> >>A better option would be to have the forrestbot buildfile for
> >>cocoon-trunk run its 'build docs', then you don't affect any other
> >>projects.
> >
> >That is okay, my forrest-exec shell wrapper has a case statement
> >to switch based on siteName to handle various special "pre-forrest"
> >operations. However, the length of time that it takes, causes
> >the issue that i mention below - forrestbot run via the webapp
> >will not know if there is a forrestbot already running via
> >cron and vice versa. I think that can be fixed in my wrapper
> >by checking/setting the date on the cocoon-trunk forrestbot log.
>
> Hmm.. perhaps this should be added to forrest[bot] itself. Not sure how..
Yes, i had already thought of that too. I was wondering if
when forrestbot starts we can create a flag file in the
forrestbot-logs directory, which gets removed at the end
of the process. I have started a coding experiment.
We would probably also need a way to check if there was
a stale flag file lying around.
> >Anyway, thanks, it is good to know that the forrestbot buildfile
> >has that capability - du'oh i should have realised that.
> >Perhaps i can move some of the pre-forrest functionality.
>
> And if you do so then other people can use the same buildfile without
> needing the shell wrapper.
Yes, the more in there the better.
> >Youch, just tried it. I presume that you mean adding an <import>
> >for the cocoon-trunk/build.xml then calling the forrestbot
> >with targets "docs" and then "build". I get errors from cocoon's
> >"docs" target because forrestbot is not running in the top-level
> >of cocoon's source.
>
> Maybe using the 'ant' task would work better; with it you can specify
> the buildfile and directory.
I will try that in my next free time.
> >Also, because forrestbot only does 'svn up' for the exact parts
> >of the sources that it requires, i need to do an 'svn up'
> >for the full cocoon-trunk. This needs to happen over in the
> >work/svn/cocoon-trunk space, then do cocoon's 'build docs' there,
> >then let forrestbot copy the sources to its ${build.work-dir}
>
> Hmm, I don't know enough about cocoon's build procedure. Perhaps the
> cocoon buildfile should provide it's own 'getsrc' implementation (which
> would depend on getsrc.svn and then do more complete updates).
I am afraid to add even more to the cocoon build procedure.
It is already so complex. But yes, that might be a solution.
> >>>I have the forrestbot working now for cocoon-trunk.
> >>>There is potential to have multiple cocoon-trunk builds
> >>>running, so i am currently finding a way to avoid that.
> >>>
> >>>--David
Re: [Proposal] forrestbot at apache.org
Posted by Dave Brondsema <da...@brondsema.net>.
David Crossley wrote:
> Dave Brondsema wrote:
>
>>David Crossley wrote:
>>
>>>I suddenly realised the issue with "cocoon-trunk". It needs
>>>to run its 'build docs' before running 'forrest'. It generates
>>>some extra source documentation before forrest starts.
>>>
>>>There is a global parameter "forrest-exec" which can call
>>>a shell script to do other things, then call forrest.
>>
>>A better option would be to have the forrestbot buildfile for
>>cocoon-trunk run its 'build docs', then you don't affect any other projects.
>
>
> That is okay, my forrest-exec shell wrapper has a case statement
> to switch based on siteName to handle various special "pre-forrest"
> operations. However, the length of time that it takes, causes
> the issue that i mention below - forrestbot run via the webapp
> will not know if there is a forrestbot already running via
> cron and vice versa. I think that can be fixed in my wrapper
> by checking/setting the date on the cocoon-trunk forrestbot log.
>
Hmm.. perhaps this should be added to forrest[bot] itself. Not sure how..
> Anyway, thanks, it is good to know that the forrestbot buildfile
> has that capability - du'oh i should have realised that.
> Perhaps i can move some of the pre-forrest functionality.
>
And if you do so then other people can use the same buildfile without
needing the shell wrapper.
> Youch, just tried it. I presume that you mean adding an <import>
> for the cocoon-trunk/build.xml then calling the forrestbot
> with targets "docs" and then "build". I get errors from cocoon's
> "docs" target because forrestbot is not running in the top-level
> of cocoon's source.
>
Maybe using the 'ant' task would work better; with it you can specify
the buildfile and directory.
> Also, because forrestbot only does 'svn up' for the exact parts
> of the sources that it requires, i need to do an 'svn up'
> for the full cocoon-trunk. This needs to happen over in the
> work/svn/cocoon-trunk space, then do cocoon's 'build docs' there,
> then let forrestbot copy the sources to its ${build.work-dir}
>
Hmm, I don't know enough about cocoon's build procedure. Perhaps the
cocoon buildfile should provide it's own 'getsrc' implementation (which
would depend on getsrc.svn and then do more complete updates).
> --David
>
>
>>>I have the forrestbot working now for cocoon-trunk.
>>>There is potential to have multiple cocoon-trunk builds
>>>running, so i am currently finding a way to avoid that.
>>>
>>>--David
--
Dave Brondsema : dave@brondsema.net
http://www.splike.com : programming
http://csx.calvin.edu : student org
http://www.brondsema.net : personal
Re: [Proposal] forrestbot at apache.org
Posted by David Crossley <cr...@apache.org>.
Dave Brondsema wrote:
> David Crossley wrote:
> >I suddenly realised the issue with "cocoon-trunk". It needs
> >to run its 'build docs' before running 'forrest'. It generates
> >some extra source documentation before forrest starts.
> >
> >There is a global parameter "forrest-exec" which can call
> >a shell script to do other things, then call forrest.
>
> A better option would be to have the forrestbot buildfile for
> cocoon-trunk run its 'build docs', then you don't affect any other projects.
That is okay, my forrest-exec shell wrapper has a case statement
to switch based on siteName to handle various special "pre-forrest"
operations. However, the length of time that it takes, causes
the issue that i mention below - forrestbot run via the webapp
will not know if there is a forrestbot already running via
cron and vice versa. I think that can be fixed in my wrapper
by checking/setting the date on the cocoon-trunk forrestbot log.
Anyway, thanks, it is good to know that the forrestbot buildfile
has that capability - du'oh i should have realised that.
Perhaps i can move some of the pre-forrest functionality.
Youch, just tried it. I presume that you mean adding an <import>
for the cocoon-trunk/build.xml then calling the forrestbot
with targets "docs" and then "build". I get errors from cocoon's
"docs" target because forrestbot is not running in the top-level
of cocoon's source.
Also, because forrestbot only does 'svn up' for the exact parts
of the sources that it requires, i need to do an 'svn up'
for the full cocoon-trunk. This needs to happen over in the
work/svn/cocoon-trunk space, then do cocoon's 'build docs' there,
then let forrestbot copy the sources to its ${build.work-dir}
--David
> >I have the forrestbot working now for cocoon-trunk.
> >There is potential to have multiple cocoon-trunk builds
> >running, so i am currently finding a way to avoid that.
> >
> >--David
Re: [Proposal] forrestbot at apache.org
Posted by Dave Brondsema <da...@brondsema.net>.
David Crossley wrote:
> I suddenly realised the issue with "cocoon-trunk". It needs
> to run its 'build docs' before running 'forrest'. It generates
> some extra source documentation before forrest starts.
>
> There is a global parameter "forrest-exec" which can call
> a shell script to do other things, then call forrest.
A better option would be to have the forrestbot buildfile for
cocoon-trunk run its 'build docs', then you don't affect any other projects.
> I have the forrestbot working now for cocoon-trunk.
> There is potential to have multiple cocoon-trunk builds
> running, so i am currently finding a way to avoid that.
>
> --David
--
Dave Brondsema : dave@brondsema.net
http://www.splike.com : programming
http://csx.calvin.edu : student org
http://www.brondsema.net : personal
Re: [Proposal] forrestbot at apache.org
Posted by David Crossley <cr...@apache.org>.
I suddenly realised the issue with "cocoon-trunk". It needs
to run its 'build docs' before running 'forrest'. It generates
some extra source documentation before forrest starts.
There is a global parameter "forrest-exec" which can call
a shell script to do other things, then call forrest.
I have the forrestbot working now for cocoon-trunk.
There is potential to have multiple cocoon-trunk builds
running, so i am currently finding a way to avoid that.
--David
Re: [Proposal] forrestbot at apache.org
Posted by Antonio Gallardo <ag...@agssa.net>.
On Mie, 23 de Febrero de 2005, 2:23, David Crossley dijo:
> I solved some of the remaining failures with the forrestbot.
> Both "forrest-site" and "forrest-docs-dev" needed to have a full
> 'svn checkout' done before the forrestbot starts. Forrestbot
> only does 'svn co' of part of the source tree and so it misses
> the "conf" directory for "forrest-site". Dunno what the issue
> was with "forrest-docs-dev", but the full checkout fixes it.
> So now we are down to minor build issues and weird error messages
> about logging.
>
> http://brutus.apache.org:36366/forrestbot-webapp/
Wow! That is really wild! It is amazing! :-)
I hope soon we will use this on the ASF.
Best Regards,
Antonio Gallardo
Re: [Proposal] forrestbot at apache.org
Posted by David Crossley <cr...@apache.org>.
I solved some of the remaining failures with the forrestbot.
Both "forrest-site" and "forrest-docs-dev" needed to have a full
'svn checkout' done before the forrestbot starts. Forrestbot
only does 'svn co' of part of the source tree and so it misses
the "conf" directory for "forrest-site". Dunno what the issue
was with "forrest-docs-dev", but the full checkout fixes it.
So now we are down to minor build issues and weird error messages
about logging.
http://brutus.apache.org:36366/forrestbot-webapp/
--David
Re: [Proposal] forrestbot at apache.org
Posted by David Crossley <cr...@apache.org>.
Ross Gardler wrote:
> David Crossley wrote:
> >Ross Gardler wrote:
> >
> >>I note that all the logs show something similar to that copied below at
> >>the end of the build:
> >>
> >>Logging Error: Writing event to closed stream.
> >> [java] Total time: 0 minutes 6 seconds, Site size: 0 Site pages: 0
> >>[trycatch] Caught exception: The following error occurred while
> >>executing this line:
> >>/opt/forrest/forrest-trunk/main/targets/site.xml:40: Java returned: 1
> >>
> >>I've not looked to see if this happens by manually building the site.
> >
> >
> >Yes, each does do the "Logging Error" on local 'forrest'.
> >
> >There is only one of them which has the extra error that you showed.
> >Only "cocoon-trunk" total failure.
>
> Actually, as I type this email it appears in:
>
> cocoon-trunk
>
> forrest-docs-0.6
>
> forrest-docs-dev
>
> forest-site (in addition to Exception in thread "main"
> java.io.FileNotFoundException: The configuration file could not be found.)
>
> incuator
>
> I've not investigated further, no time I'm afraid. just logging the error.
Ah, there are probably various different errors. I note that
"cocoon-doco-global" has the "Logging Error" but not the
trycatch exception.
Cheche is right, someone does need to summarise the errors.
--David
Re: [Proposal] forrestbot at apache.org
Posted by Ross Gardler <rg...@apache.org>.
David Crossley wrote:
> Ross Gardler wrote:
>
>>I note that all the logs show something similar to that copied below at
>>the end of the build:
>>
>>Logging Error: Writing event to closed stream.
>> [java] Total time: 0 minutes 6 seconds, Site size: 0 Site pages: 0
>> [trycatch] Caught exception: The following error occurred while
>>executing this line:
>>/opt/forrest/forrest-trunk/main/targets/site.xml:40: Java returned: 1
>>
>>I've not looked to see if this happens by manually building the site.
>
>
> Yes, each does do the "Logging Error" on local 'forrest'.
>
> There is only one of them which has the extra error that you showed.
> Only "cocoon-trunk" total failure.
Actually, as I type this email it appears in:
cocoon-trunk
forrest-docs-0.6
forrest-docs-dev
forest-site (in addition to Exception in thread "main"
java.io.FileNotFoundException: The configuration file could not be found.)
incuator
I've not investigated further, no time I'm afraid. just logging the error.
Ross
--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.7.2 - Release Date: 21/01/2005
Re: [Proposal] forrestbot at apache.org
Posted by David Crossley <cr...@apache.org>.
Ross Gardler wrote:
>
> I note that all the logs show something similar to that copied below at
> the end of the build:
>
> Logging Error: Writing event to closed stream.
> [java] Total time: 0 minutes 6 seconds, Site size: 0 Site pages: 0
> [trycatch] Caught exception: The following error occurred while
> executing this line:
> /opt/forrest/forrest-trunk/main/targets/site.xml:40: Java returned: 1
>
> I've not looked to see if this happens by manually building the site.
Yes, each does do the "Logging Error" on local 'forrest'.
There is only one of them which has the extra error that you showed.
Only "cocoon-trunk" total failure. Its error might be related to the
obscure error about "tabs". This is a symptom that i have seen before,
but cannot remember the solution.
----
[java] X [0] index.html BROKEN: /opt/forrest/forrestbot-conf/work/cocoon-trunk/build/cocoon-2.2.0-dev/documentation/xdocs/tabs.xml (No such file or directory)
[java] Logging Error: Writing event to closed stream.
[java] Total time: 0 minutes 4 seconds, Site size: 0 Site pages: 0
[trycatch] Caught exception: The following error occurred while executing this line:
/opt/forrest/forrest-trunk/main/targets/site.xml:40: Java returned: 1
----
--David
Re: [Proposal] forrestbot at apache.org
Posted by Ross Gardler <rg...@apache.org>.
David Crossley wrote:
> Finally we have a demonstration forrestbot running on
> an ASF machine. It is on brutus, the untrusted machine
> that also runs Apache Gump. That means it can only
> ever be a demo, i.e. the sites that it generates
> cannot be put into production. Still it is one big step.
> A proper machine will be available later.
>
> http://brutus.apache.org:36366/forrestbot-webapp/
Excellent.
I note that all the logs show something similar to that copied below at
the end of the build:
Logging Error: Writing event to closed stream.
[java] Total time: 0 minutes 6 seconds, Site size: 0 Site pages: 0
[trycatch] Caught exception: The following error occurred while
executing this line:
/opt/forrest/forrest-trunk/main/targets/site.xml:40: Java returned: 1
I've not looked to see if this happens by manually building the site.
Ross
--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.7.2 - Release Date: 21/01/2005
Re: [Proposal] forrestbot at apache.org
Posted by David Crossley <cr...@apache.org>.
Juan Jose Pablos wrote:
> David Crossley wrote:
>
> >Some of the sites have build issues. We should get
> >those fixed before we announce this more widely.
> >
> Could you summarize a bit of those problems? Which one do you want us to
> take care?
The list of problem sites is noted at the top of the page.
Just look at the log for each to see the issues.
--David
Re: [Proposal] forrestbot at apache.org
Posted by Juan Jose Pablos <ch...@apache.org>.
David Crossley wrote:
>Finally we have a demonstration forrestbot running on
>an ASF machine.
>
Nice!
>Some of the sites have build issues. We should get
>those fixed before we announce this more widely.
>
>
>
Could you summarize a bit of those problems? Which one do you want us to
take care?
Cheers,
Cheche