[NOTICE] + [PROPOSAL] nightly builds + nightly source checkouts

[Jon Stevens <jo...@latchkey.com>]

Hey all,


[NOTICE]
Brian asked me to remind you to clean up the nightly builds so that we don't
have tons of old stuff eating up disk space.

Example:
/x2/www/jakarta.apache.org/builds/jakarta-tomcat-4.0/nightly

jakarta-tomcat-4.0/nightly > du -sk
424790  .

424790 megs seems like a lot to me.

There are files all the way back to 20010216 which seems a bit excessive.
Also excessive seems to be the .Z and .gz versions. I think it is safe to
assume in this day and age that everyone can install gunzip on their server.


[PROPOSAL]
I would like to finagle this email into another discussion as well which is
coming up with a more centralized system for doing nightly builds as well as
nightly source checkouts.

Right now, we have cronjobs run by a number of different people putting
files in a number of different locations. It would be nice to try to
centralize this down to just one primary script and one configuration file
that handles this the majority of this stuff.

So, in the builds directory, I would like to propose setting something up
that looks like this:

/www/jakarta.apache.org
    builds/
        scripts/
            build-all.sh (or .pl or .py)
            cleanup-all.sh (or .pl or .py)
            build.conf
            projects/
                velocity-source.sh (or .pl or .py)
                velocity-build.sh (or .pl or .py)
                tomcat-4.0-source.sh (or .pl or .py)
                tomcat-4.0-build.sh (or .pl or .py)
                ...
                ...

build.conf:
    baseDirectory=/www/jakarta.apache.org/builds
    projectsDirectory=$baseDirectory/projects
    daysToKeep=7

We could then have the build-all script run through the projects directory
looking for scripts to execute. It would first attempt to execute all of the
"*source*" scripts it finds and then attempt to execute all of the "*build*"
scripts it finds. What this would allow is for projects to have their own
scripts to be run in order to do any source/build specific things for their
projects. It would also allow for individual testing. Adding a new project
is as simple as adding another set of scripts.

The individual scripts could set what their "output" directory is based on
the projectsDirectory + projectName. projectsDirectory would be passed in to
the source/build scripts on ARGV[0].

The cleanup-all.sh script would be a global cleanup script that would run at
the end of the build-all.sh script to prune the files in the directories,
keeping the "daysToKeep" number of files.

I know that I have most of the framework already done for the
"project-build.sh" scripts and the "cleanup-all.sh" scripts, but I don't
have the frameworks for the build scripts, however, I know that some
projects already have that. Maybe we could consolidate those into this
system?

Notes: In order to put a system like this into place, we will also need to
have everyone shut off their existing cronjobs. :-)

Sam, I'm sure that this probably fits into Gump somewhere as well, however,
I'm not sure I want to or feel the need to go that far and come up with the
"perfect" solution at this point. What do you feel about that?

Comments?

-jon


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@jakarta.apache.org
For additional commands, e-mail: general-help@jakarta.apache.org

Re: [NOTICE] + [PROPOSAL] nightly builds + nightly source checkouts

["Craig R. McClanahan" <cr...@apache.org>]

On Tue, 13 Mar 2001, Jon Stevens wrote:

> Hey all,
> 
> 
> [NOTICE]
> Brian asked me to remind you to clean up the nightly builds so that we don't
> have tons of old stuff eating up disk space.
> 
> Example:
> /x2/www/jakarta.apache.org/builds/jakarta-tomcat-4.0/nightly
> 
> jakarta-tomcat-4.0/nightly > du -sk
> 424790  .
> 
> 424790 megs seems like a lot to me.
> 

Will be restarting my "clean out" job shortly.

> There are files all the way back to 20010216 which seems a bit excessive.
> Also excessive seems to be the .Z and .gz versions. I think it is safe to
> assume in this day and age that everyone can install gunzip on their server.
> 
> 
> [PROPOSAL]
> I would like to finagle this email into another discussion as well which is
> coming up with a more centralized system for doing nightly builds as well as
> nightly source checkouts.
> 

+1, but see further discussion below on the details

> Right now, we have cronjobs run by a number of different people putting
> files in a number of different locations. It would be nice to try to
> centralize this down to just one primary script and one configuration file
> that handles this the majority of this stuff.
> 
> So, in the builds directory, I would like to propose setting something up
> that looks like this:
> 
> /www/jakarta.apache.org
>     builds/
>         scripts/
>             build-all.sh (or .pl or .py)
>             cleanup-all.sh (or .pl or .py)
>             build.conf
>             projects/
>                 velocity-source.sh (or .pl or .py)
>                 velocity-build.sh (or .pl or .py)
>                 tomcat-4.0-source.sh (or .pl or .py)
>                 tomcat-4.0-build.sh (or .pl or .py)
>                 ...
>                 ...
> 
> build.conf:
>     baseDirectory=/www/jakarta.apache.org/builds
>     projectsDirectory=$baseDirectory/projects
>     daysToKeep=7
> 
> We could then have the build-all script run through the projects directory
> looking for scripts to execute. It would first attempt to execute all of the
> "*source*" scripts it finds and then attempt to execute all of the "*build*"
> scripts it finds. What this would allow is for projects to have their own
> scripts to be run in order to do any source/build specific things for their
> projects. It would also allow for individual testing. Adding a new project
> is as simple as adding another set of scripts.
> 

We will want each subproject to maintain their own versions
(e.g. tomcat-4.0-source.sh and tomcat-4.0-build.sh) according to the
standard conventions, so that no one person would need to stay up to date
on all the subproject build procedures.

We will also want a few things like stable releases of Ant, XML parsers,
etc. installed in "well known" places for build processes that require
access to them.

> The individual scripts could set what their "output" directory is based on
> the projectsDirectory + projectName. projectsDirectory would be passed in to
> the source/build scripts on ARGV[0].
> 
> The cleanup-all.sh script would be a global cleanup script that would run at
> the end of the build-all.sh script to prune the files in the directories,
> keeping the "daysToKeep" number of files.
> 
> I know that I have most of the framework already done for the
> "project-build.sh" scripts and the "cleanup-all.sh" scripts, but I don't
> have the frameworks for the build scripts, however, I know that some
> projects already have that. Maybe we could consolidate those into this
> system?
> 
> Notes: In order to put a system like this into place, we will also need to
> have everyone shut off their existing cronjobs. :-)
> 

+1 for me (who does Servletapi, Tomcat 4.0, Struts, Watchdog), as soon as
it is ready.

> Sam, I'm sure that this probably fits into Gump somewhere as well, however,
> I'm not sure I want to or feel the need to go that far and come up with the
> "perfect" solution at this point. What do you feel about that?
> 

IMHO Gump serves a different, also valid, purpose as a tinderbox detector
of incompatibilities.

The goal of nightly builds is different -- it is to create something that
ought to be marginally usable.  For that reason, a nightly build would use
a stable Ant (for example), instead of hot-off-the-CVS code that Gump
uses.

That being said, it might well be possible to tweak the Gump scripts so
that they operate in a different mode for nightly builds.  Sam?

> Comments?
> 
> -jon
> 

Craig McClanahan


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@jakarta.apache.org
For additional commands, e-mail: general-help@jakarta.apache.org