Automatic site generation

[Norman Maurer <nm...@byteaction.de>]

Hi guys,

first to say thanks to Stefano for the nice job. When talkin to him the
idea raise to autmatic build the site everyday.. So it whould be easy to
update the site only by commit changes to svn.

Any thoughts ?

bye
Norman

Re: Automatic site generation

[Danny Angus <da...@gmail.com>]

On 02/08/06, Stefano Bagnara <ap...@bago.org> wrote:

> I could copy and past *all* of your reply back to you, and I think it
> would be valid for me too. So I will not use so much bytes. Please read
> your last message and think like I wrote it to you.

Hey, if you can't take it, don't start it.

d.

Re: Automatic site generation

[Stefano Bagnara <ap...@bago.org>]

Danny Angus wrote:
>> > Agreed. I only said that GUMP is not the answer.
>>
>> Then you were talking to Noel and I thought you were talking to me.
> 
> So why the hell are you continuing to argue about it Stefano???

You're right. I should work on something concrete (as I always try to 
do) instead of trying to answer your answers/critics.

I could copy and past *all* of your reply back to you, and I think it 
would be valid for me too. So I will not use so much bytes. Please read 
your last message and think like I wrote it to you.

Stefano

Re: Automatic site generation

[Danny Angus <da...@gmail.com>]

On 01/08/06, Stefano Bagnara <ap...@bago.org> wrote:
> Danny Angus wrote:
> >> You have to define what is the "World" that you want to integrate.
> >
> > In GUMP's case it is the world of Open Source.
>
> The problem is that you define that CI IS GUMP,

Erm no,  I'm *ONLY* saying that GUMP is run by the GUMP project, that
it isn't ours and that it has very different goals than the things
we're talking about.

If you read my message down to the bottom *before* you started to
reply to it you would have seen that I said "we need a CI build
internal to the project"

> We don't need nightly builds, we need a better CI. I don't care if we
> want to keep or remove the current gump, I'm only saying that we need
> Continous Integration for the James project:

See above, try *reading* what I said before going off on another rant.

> Well it seems to me that this is what the ASF GUMP installation IS, not
> what GUMP is. Btw I don't know GUMP enough to be sure of this.

Yes, and AFAIK that is exactly what we are talking about the ASF GUMP
installation, written maintained and operated by the GUMP project with
its own very particular goals.

I'm trying to tell you that It isn't approprate for what you want to do.

> Just search the internet for some definition of "Continuous Integration"

I'm sure you probably don't mean to be patronising and rude, I hope
not anyway, there's really no need. I'm a professional, I get paid to
do that stuff, I know what I'm talking about.

In fact you were talking about publishing stuff to the web site, that
isn't really a CI task.
check-out, compile, test, and generate reports are though, and I
agreed that we need a James CI cycle for that.

<snip>


> > Agreed. I only said that GUMP is not the answer.
>
> Then you were talking to Noel and I thought you were talking to me.

So why the hell are you continuing to argue about it Stefano???

d.

Re: Automatic site generation

[Stefano Bagnara <ap...@bago.org>]

Danny Angus wrote:
>> You have to define what is the "World" that you want to integrate.
> 
> In GUMP's case it is the world of Open Source.

The problem is that you define that CI IS GUMP, while this is not the 
only truth. I can agree that GUMP is a Continuos Integration tool, but 
configured the way ASF did it is not so useful to James Project as CI 
could be.
We don't need nightly builds, we need a better CI. I don't care if we 
want to keep or remove the current gump, I'm only saying that we need 
Continous Integration for the James project: this mean that it should 
integrate james products and nothings else, and should test things and 
produce results (nightly builds, reports).

>> I personally don't care if commons-collection create a new jar that will
>> not work with James. Instead I would like to know if jspf or postage
>> starts being incompatible with the latest trunk.
> 
> Thats a project level thing though, GUMP is about the bigger picture.

Well it seems to me that this is what the ASF GUMP installation IS, not 
what GUMP is. Btw I don't know GUMP enough to be sure of this. I'm 
really sure that the definition of Continous Integration is not that you 
have to integrate the entire world of dependencies.
Just search the internet for some definition of "Continuous Integration" 
and you will find many references to "integration of many modules of a 
project" or as raw as the Wikipedia definition: "Continuous integration 
is a software engineering term describing a process that completely 
rebuilds and tests an application frequently"

>> Furthermore when (if) we'll move to multi-module for james it will also
>> test the integration between the modules.
>> And I don't really want to have my builds broken by non-working velocity
>> as we don't use it and don't need it.
> 
> I know, and the answer is "thats not what GUMP is for"

That's why I never said that we should use GUMP ;-).
I always thought to run continuum for that... I don't know why the 
thread bring us to GUMP...

>> About our nightly build we currently don't run unit-tests before
>> creating them, we don't create reports, we don't publish other
>> informations. It would really better to have such things done.
> 
> Agree, for which we need to have proper nightly builds.

We probably don't share the definition of nightly builds, because I 
think that we need CI for James products inside the James project.
We need to build, test, produce reports for all of our artifacts. This 
is better defined as CI than by Nighly Builds (IMO).

>> I don't care about defining what CI is or is not,
> 
> No but the GUMP people do!

Ok, I will never say GUMP in the future ;-)

>> but I care about results:
> 
> Fair enough. But that means you want nightly builds and not CI

I don't agree.. but maybe we can talk about "we need what bago define CI 
and danny define nighly builds".. the important thing is that we now 
have a way to understand each other..

>> I don't need to test the latest velocity, but I really need a
>> continous build (maybe done each hour or less) that run a full
>> build/test/report generation/packaging and alert me if something goes 
>> wrong.
> 
> Ok then we need a CI build internal to the project, Maven and
> cruisecontrol work exceedingly well to do this, we have over 20 maven
> build on a continuous loop with cruisecontrol here, and it takes just
> about zero admin.

Exactly what I had in mind. What I want to know is if ASF already 
provide infrastructure for this or if we have to do this ourselves. In 
the latter can I don't care about cruisecontrol vs continuum: the one 
that will setup the things will decide what to use.

>> Currently if I commit something that builds but break tests no one
>> notice this: this is not good.
> 
> Agreed. I only said that GUMP is not the answer.
> 
> d.

Then you were talking to Noel and I thought you were talking to me.
:-)

Stefano

RE: Nightly builds (was automatic site generation)

["Noel J. Bergman" <no...@devtech.com>]

Stefano Bagnara wrote:

> Noel J. Bergman wrote:
>> Danny Angus wrote (on site-dev@):
>>> Stefano wrote:
>>>> About our nightly build we currently don't run unit-tests before
>>>> creating them, we don't create reports, we don't publish other
>>>> informations.
>>> Agree, for which we need to have proper nightly builds.
>> If someone adds a target to run unit-tests to the build script, I would
be
>> happy to include running that during the nightly build process, and I can
>> start posting the session log to the mailing list.
> Already there: "run-unit-tests"
> It needs commons-net-1.4.1.jar in the tools\lib folder

And JUnit 3.8 or later in ${ant}/lib.

Look for a new nightly build report, starting tonight.

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Nightly builds (was automatic site generation)

[Stefano Bagnara <ap...@bago.org>]

Noel J. Bergman wrote:
> Danny Angus wrote (on site-dev@):
> 
>> Stefano wrote:
>>> About our nightly build we currently don't run unit-tests before
>>> creating them, we don't create reports, we don't publish other
>>> informations. It would really better to have such things done.
> 
>> Agree, for which we need to have proper nightly builds.
> 
> If someone adds a target to run unit-tests to the build script, I would be
> happy to include running that during the nightly build process, and I can
> start posting the session log to the mailing list.
> 
> 	--- Noel

Already there: "run-unit-tests"
It needs commons-net-1.4.1.jar in the tools\lib folder

Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Nightly builds (was automatic site generation)

[Norman Maurer <nm...@byteaction.de>]

Am Dienstag, den 01.08.2006, 19:43 +0200 schrieb Stefano Bagnara:
> Noel J. Bergman wrote:
> > Danny Angus wrote (on site-dev@):
> > 
> >> Stefano wrote:
> >>> About our nightly build we currently don't run unit-tests before
> >>> creating them, we don't create reports, we don't publish other
> >>> informations. It would really better to have such things done.
> > 
> >> Agree, for which we need to have proper nightly builds.
> > 
> > If someone adds a target to run unit-tests to the build script, I would be
> > happy to include running that during the nightly build process, and I can
> > start posting the session log to the mailing list.
> > 
> > 	--- Noel
> 
> Already there: "run-unit-tests"
> It needs commons-net-1.4.1.jar in the tools\lib folder
> 
> Stefano

Whould be cool for having such reports.

bye
Norman

Nightly builds (was automatic site generation)

["Noel J. Bergman" <no...@devtech.com>]

Danny Angus wrote (on site-dev@):

> Stefano wrote:
> > About our nightly build we currently don't run unit-tests before
> > creating them, we don't create reports, we don't publish other
> > informations. It would really better to have such things done.

> Agree, for which we need to have proper nightly builds.

If someone adds a target to run unit-tests to the build script, I would be
happy to include running that during the nightly build process, and I can
start posting the session log to the mailing list.

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Automatic site generation

[Danny Angus <da...@gmail.com>]

On 01/08/06, Stefano Bagnara <ap...@bago.org> wrote:
> I don't agree with this.
>
> Elaborating on your point your should do CI under the latest version of
> the kernel on the latest hardware architecture every time ;-)

I know you're just having fun, but the idea *really* is that you
compile against the tip of everything. The kernel and the hardware
don;t impact on that, but the compiler does, and for what its worth,
GUMP bootstraps itself by first building the tip of ant and then using
that for all its builds.


> You have to define what is the "World" that you want to integrate.

In GUMP's case it is the world of Open Source.

> I personally don't care if commons-collection create a new jar that will
> not work with James. Instead I would like to know if jspf or postage
> starts being incompatible with the latest trunk.

Thats a project level thing though, GUMP is about the bigger picture.

> Furthermore when (if) we'll move to multi-module for james it will also
> test the integration between the modules.
> And I don't really want to have my builds broken by non-working velocity
> as we don't use it and don't need it.

I know, and the answer is "thats not what GUMP is for"

> About our nightly build we currently don't run unit-tests before
> creating them, we don't create reports, we don't publish other
> informations. It would really better to have such things done.

Agree, for which we need to have proper nightly builds.

> I don't care about defining what CI is or is not,

No but the GUMP people do!

>but I care about results:

Fair enough. But that means you want nightly builds and not CI

> I don't need to test the latest velocity, but I really need a
> continous build (maybe done each hour or less) that run a full
> build/test/report generation/packaging and alert me if something goes wrong.

Ok then we need a CI build internal to the project, Maven and
cruisecontrol work exceedingly well to do this, we have over 20 maven
build on a continuous loop with cruisecontrol here, and it takes just
about zero admin.

> Currently if I commit something that builds but break tests no one
> notice this: this is not good.

Agreed. I only said that GUMP is not the answer.

d.

Re: Automatic site generation

[Stefano Bagnara <ap...@bago.org>]

Danny Angus wrote:
> On 01/08/06, Stefano Bagnara <ap...@bago.org> wrote:
>> Furthermore I see that gump always run the integration test based on
>> trunk for all the dependencies, I don't know how this is configured but
>> I don't like this for james. I don't want to have a broken build because
>> current velocity trunk is broken and so on (I saw this many times in
>> gump results for james-server)
> 
> Thats the whole point of CI, to give you early warning of changes
> which will break your project.
> When you're at the end of a long food chain like James is it gets a
> bit boring though.
> 
> Setting up nightly builds, not CI ones, is a more appropriate solution
> to all of the things you want to do.

I don't agree with this.

Elaborating on your point your should do CI under the latest version of 
the kernel on the latest hardware architecture every time ;-)

You have to define what is the "World" that you want to integrate.
I personally don't care if commons-collection create a new jar that will 
not work with James. Instead I would like to know if jspf or postage 
starts being incompatible with the latest trunk.
Furthermore when (if) we'll move to multi-module for james it will also 
test the integration between the modules.
And I don't really want to have my builds broken by non-working velocity 
as we don't use it and don't need it.

About our nightly build we currently don't run unit-tests before 
creating them, we don't create reports, we don't publish other 
informations. It would really better to have such things done.

I don't care about defining what CI is or is not, but I care about 
results: I don't need to test the latest velocity, but I really need a 
continous build (maybe done each hour or less) that run a full 
build/test/report generation/packaging and alert me if something goes wrong.

Currently if I commit something that builds but break tests no one 
notice this: this is not good.

Stefano

Re: Automatic site generation

[Danny Angus <da...@gmail.com>]

On 01/08/06, Stefano Bagnara <ap...@bago.org> wrote:
> Furthermore I see that gump always run the integration test based on
> trunk for all the dependencies, I don't know how this is configured but
> I don't like this for james. I don't want to have a broken build because
> current velocity trunk is broken and so on (I saw this many times in
> gump results for james-server)

Thats the whole point of CI, to give you early warning of changes
which will break your project.
When you're at the end of a long food chain like James is it gets a
bit boring though.

Setting up nightly builds, not CI ones, is a more appropriate solution
to all of the things you want to do.

d.

GUMP and CI (was: Automatic site generation)

["Noel J. Bergman" <no...@devtech.com>]

Stefano Bagnara wrote:

> Unfortunately I don't have much knowledge on how to make james and at
> least jspf working under gump and also generating the website.

Generating the web site isn't its GUMP's purpose, but you ought to be able
to get help from the GUMP team for jSPF.

> Furthermore I see that gump always run the integration test based on
> trunk for all the dependencies

Well, yes, and that's the whole point of *CONTINUOUS* Integration from their
perspective.  As opposed to us just doing our own nightly builds and
reporting on their success/failure.  I do have nighly builds running,
although I don't e-mail results to the mailing list at this time.

> I don't want to have a broken build because current velocity
> trunk is broken and so on (I saw this many times in gump
> results for james-server)

Yes, but their point is that if Velocity causes problems, both projects need
to know.  I'm not sure, but there may be a way to change the default
behavior for GUMP.  Again, ask them.

	--- Noel

Re: Automatic site generation

[Stefano Bagnara <ap...@bago.org>]

Noel J. Bergman wrote:
> Stefano Bagnara wrote:
> 
>> Noel J. Bergman wrote:
>>>> Instead I would try to start a continuos integration project for
>>>> server+jspf+site and have a live updated site for our trunks as
>>>> a tool for us.
>>> How does this help?  We're still generating all of the pages.
> 
>> Maybe I don't understand the question, are you asking me why CI is a
>> good thing (helps)?
> 
> No, not at all.  LOL  We already have CI for JAMES.  GUMP builds us every
> night, along with most of the rest of the ASF Java projects.
> 
> I meant, how does CI help us to check-in only the changed files for the
> site?  :-)
> 
>> So maybe it is better if I do all the CI stuff for james in one of my
>> server
> 
> Well, we already have GUMP, but whatever.

Unfortunately I don't have much knowledge on how to make james and at 
least jspf working under gump and also generating the website.

Furthermore I see that gump always run the integration test based on 
trunk for all the dependencies, I don't know how this is configured but 
I don't like this for james. I don't want to have a broken build because 
current velocity trunk is broken and so on (I saw this many times in 
gump results for james-server)

I want to test dependencies like I defined them: if I depend on a 
snapshot than I agree to use newer versions, if I depend on a specific 
version I don't want to build it against newer versions.

Btw the main issue is to be able to automatically run tests and reports 
in the CI environment. I don't know how to do that (if possible) with 
gump so if noone volounteer for this I will try something in september 
configuring a continuum on my server.

Stefano

RE: Automatic site generation

["Noel J. Bergman" <no...@devtech.com>]

Stefano Bagnara wrote:

> Noel J. Bergman wrote:
>>> Instead I would try to start a continuos integration project for
>>> server+jspf+site and have a live updated site for our trunks as
>>> a tool for us.
>> How does this help?  We're still generating all of the pages.

> Maybe I don't understand the question, are you asking me why CI is a
> good thing (helps)?

No, not at all.  LOL  We already have CI for JAMES.  GUMP builds us every
night, along with most of the rest of the ASF Java projects.

I meant, how does CI help us to check-in only the changed files for the
site?  :-)

> So maybe it is better if I do all the CI stuff for james in one of my
> server

Well, we already have GUMP, but whatever.

> and publish it under my personal website (on my server). This
> will make everyone happy.

How so?

> I can't find informations on that stuff and I cannot find
> site-dev@apache.org archives.

For some reason, site-dev archives are private, instead of available for
committers.  I'll look into it.

> I didn't understand if this site-build server is a www.apache.org thing
> or if this is something that I should know about james.apache.org.

The intent is to have an ASF-wide publishing system for sites.

	--- Noel

Re: Automatic site generation

[Stefano Bagnara <ap...@bago.org>]

Noel J. Bergman wrote:
>> Instead I would try to start a continuos integration project for
>> server+jspf+site and have a live updated site for our trunks as
>> a tool for us.
> 
> How does this help?  We're still generating all of the pages.

Maybe I don't understand the question, are you asking me why CI is a 
good thing (helps)?

>> Again I don't know if the ASF policy allow me to do this on my minotaur
>> home.
> 
> In addition to the whole automation issue, you'd be consuming massive
> resources on a shared system.

So maybe it is better if I do all the CI stuff for james in one of my 
server and publish it under my personal website (on my server). This 
will make everyone happy.

> But so you know, the proposed ASF-wide solution is similar: a site-build
> server on which projects could build and maintain their site (a staging copy
> for review, and an approved copy by for pushing to the web servers).  Most
> discussion on this, which has frustratingly yet to jell, takes place on
> site-dev@apache.org.
> 
> 	--- Noel

I can't find informations on that stuff and I cannot find 
site-dev@apache.org archives.
I didn't understand if this site-build server is a www.apache.org thing 
or if this is something that I should know about james.apache.org.

Stefano

RE: Automatic site generation

["Noel J. Bergman" <no...@devtech.com>]

Stefano Bagnara wrote:

> Is there a doc about what I'm allowed to do on minotaur and what not?

Pretty much no.  I asked the same question, myself, years ago.  :-)

The rule of thumb is to remember that these servers are shared resources.
CPU tends to be the most limited resource.  Also never store such things as
private keys on the infrastructure.

> I don't know if I can keep my temporary m2 repository in my public_html

Raise the question on repository@apache.org

> I don't know if I can publish temporary snapshots (or stage) of james
> website for limited time, I don't know if I can use it to make test
> builds and so on.

Sure.  If that were a bad idea, I'd have told you when you started.  But you
can always ask on infrastructure@ if you have a question.

	--- Noel

Re: Automatic site generation

[Stefano Bagnara <ap...@bago.org>]

Noel J. Bergman wrote:
>>> Again I don't know if the ASF policy allow me to do this
>>> on my minotaur home.
> 
> See, for example: http://issues.apache.org/jira/browse/INFRA-900
> 
> 	--- Noel


Is there a doc about what I'm allowed to do on minotaur and what not?
I don't know if I can keep my temporary m2 repository in my public_html, 
I don't know if I can publish temporary snapshots (or stage) of james 
website for limited time, I don't know if I can use it to make test 
builds and so on.

Should I open a JIRA issue to ask, write a message to infrastructure 
list or simply have to dig much more in online documentation?

Stefano

RE: Automatic site generation

["Noel J. Bergman" <no...@devtech.com>]

> > Again I don't know if the ASF policy allow me to do this
> > on my minotaur home.

See, for example: http://issues.apache.org/jira/browse/INFRA-900

	--- Noel

RE: Automatic site generation

["Noel J. Bergman" <no...@devtech.com>]

Stefano Bagnara wrote:

> Otherwise we could at least put a cron for "svn update" of the published
> copy.

> About automatic commit to site/www of the generated site this is more
> delicate as it involve a commit, but we could even do this.

That would be considered a really bad idea.  We're required to review
changes, and automation of that nature would bypass the required human
element.

> if we change things that generate changes in all html files we'll
> change a lot of files every time

> Instead I would try to start a continuos integration project for
> server+jspf+site and have a live updated site for our trunks as
> a tool for us.

How does this help?  We're still generating all of the pages.

> Again I don't know if the ASF policy allow me to do this on my minotaur
> home.

In addition to the whole automation issue, you'd be consuming massive
resources on a shared system.

But so you know, the proposed ASF-wide solution is similar: a site-build
server on which projects could build and maintain their site (a staging copy
for review, and an approved copy by for pushing to the web servers).  Most
discussion on this, which has frustratingly yet to jell, takes place on
site-dev@apache.org.

	--- Noel

Re: Automatic site generation

[Stefano Bagnara <ap...@bago.org>]

I don't know if there is any policy in the use of cron tasks on minotaur.

Otherwise we could at least put a cron for "svn update" of the published 
copy.

About automatic commit to site/www of the generated site this is more 
delicate as it involve a commit, but we could even do this.

The problem is maybe related to space: if we change things that generate 
changes in all html files we'll change a lot of files every time, and 
maybe we can keep this as a manual thing.

Instead I would try to start a continuos integration project for 
server+jspf+site and have a live updated site for our trunks as a tool 
for us.

Again I don't know if the ASF policy allow me to do this on my minotaur 
home.

Any hint?

Stefano

Norman Maurer wrote:
> Hi guys,
> 
> first to say thanks to Stefano for the nice job. When talkin to him the
> idea raise to autmatic build the site everyday.. So it whould be easy to
> update the site only by commit changes to svn.
> 
> Any thoughts ?
> 
> bye
> Norman

RE: Automatic site generation

["Noel J. Bergman" <no...@devtech.com>]

Norman Maurer wrote:

> When talkin to him the idea raise to autmatic build the site everyday.

If that includes automatic publishing to the web server, the answer is
absolutely not.  We are supposed to manually review and approve the site.
The site is part of the project, just like code.  But if this means just
publishing to svn, and we still have to manually svn up the site, that might
be OK.  Except that in the even of a recovery scenariom infrastructure could
potentially recover the wrong revision of the site.

> it whould be easy to update the site only by commit changes to svn.

If we can do that with automation, why can't we do it without?

	--- Noel