You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Kristian Waagan <Kr...@Sun.COM> on 2009/09/03 12:05:47 UTC
Publishing the Derby documentation
Hello,
I have now configured a Hudson job building the Derby documentation.
It looks for changes in the repository once a week (currently late every
Wednesday, can be changed :) ), and if a change has been made it will
build the complete set of manuals.
Further, the output from the last successful build is published with
static URL's:
http://hudson.zones.apache.org/hudson/job/Derby-docs/lastSuccessfulBuild/artifact/docs-out.tar.gz
http://hudson.zones.apache.org/hudson/job/Derby-docs/lastSuccessfulBuild/artifact/docs-out.md5
The tar contains the out directory, and the docs-out.md5 is simply the
MD5 checksum of the tar.
Committers can choose to set up a cron job on their people.apache.org
account. I will provide a sample shell script that should do something
like this:
- At fixed intervals, check for a new version of the docs (for instance
every Wednesday).
This is done by fetching the MD5 checksum from Hudson and comparing
it with the checksum for the docs already published.
I think we can simply save the checksum file in
/www/.../apache/docs/dev/.
- If the docs have changed, fetch the tar file from Hudson.
- Unpack the archive
- Do a few sanity checks.
- Copy the contents of out/ to /www/.../apache/docs/dev/
One must make sure write access is given to the db group here, and
also that the permissions allow people to access the files over HTTP.
If people exercise a little care when deploying their cron jobs, this
process should be pretty low impact, and you can have several committers
ready to publish the docs.
Possible improvements:
- Build only the manuals that have actually changed.
This makes the build date agree with the date of the last change.
- At least one more committer should apply for a Hudson login so that
more people can kick off manual builds or fix problems in Hudson.
- Define the sanity checks.
Simple ones: check archive size, make sure PDFs exist.
- Should the existing published files be deleted?
- Use local rsync instead of normal copy?
I noticed some file permission problems, but I have asked infra to
correct those.
If you have questions, comments or concerns, please share them!
Regards,
--
Kristian
Re: Publishing the Derby documentation
Posted by Kristian Waagan <Kr...@Sun.COM>.
Kim Haase wrote:
> Thanks, Kristian --
>
> Given my elementary knowledge of shell scripts this looks fine ...
>
> One problem is that Myrna's build has already picked up my change --
> COALESCE is in the Ref Manual already -- so probably the script won't
> run Monday after all, if it is checking for changes. We might need to
> wait for the next doc commit to make sure it works.
Actually, the Hudson build hasn't picked it up yet, since it was only
run once a week.
I confirmed that the reference documentation in dev/ has COALESCE, where
as the one in dev_test/ doesn't. The latest build happened early on the
3rd, the commit came late on the 3rd.
Regarding Myrna's build, we can discuss what to do about it.
I'm sure we can find a way to allow both the "build systems" to work
together. If Myrna is okay with continuing to run her build, there's no
point in disabling it just because we now have a different way to do it.
The one thing we should agree on, is how to tell if a build is newer
than the existing one. This can be a checksum of something (this only
work for certain scenarios), a revision number (preferably of the last
change) or something else.
A last alternative is to just keep overwriting the docs, even if the
same build is copied over the existing one (i.e. a newer build date but
no changes to the content).
As I have said earlier, my goal is to put a (easily automated) process
in place that is more transparent to the community. For instance, if
Myrna has to stop her build job for some reason, the community should
still be able to post new docs when changes are made. I think what
remains to achieve this, is to write a wiki page that describes the
process and provides the information needed for a Derby committer to
become a documentation publisher :)
I'll try to get this done, but it may be a while before everything is
ready. Then someone can test out the instructions.
Cheers,
--
Kristian
>
> Kim
>
> On 09/04/09 11:18, Kristian Waagan wrote:
>> Kim Haase wrote:
>>> A nightly build has been lovely (thanks, Myrna!) but realistically,
>>> once a week would be fine for a build of the latest docs.
>>>
>>> I wish I could volunteer for this, since I'm the one who always
>>> complains when the process breaks down, but I can't afford the time
>>> to learn about cron jobs and checksums (about which I am extremely
>>> ignorant). So many thanks to whoever takes this on.
>>
>> Hi Kim,
>>
>> I have written the first version of the script checking for updates
>> and deploying the docs. It must be run from people.a.o. See the
>> script attached, and please comment on it if you something that
>> should be changed.
>> Currently both the deploy directory and the MD5 checksum file must
>> exist (it doesn't have to contain a checksum though).
>>
>> I tested it and hopefully something should appear in
>> http://db.apache.org/derby/docs/dev_test/ in a little while (during
>> tomorrow at latest).
>> To navigate easier, you can use the links from the official
>> documentation page [1], and then just replace dev with dev_test.
>>
>> I have also changed the Hudson job to check for changes every Monday
>> (just after midnight UTC) and every Wednesday (around 20 in the
>> evening/night UTC).
>> My crontab will run the script a little later (to allow the build to
>> finish).
>>
>>
>> Since Kim committed a change yesterday, a new build of the docs
>> should be built and published some time on Monday (under dev_test).
>> Let's see what happens!
>>
>>
Re: Publishing the Derby documentation
Posted by Kim Haase <Ca...@Sun.COM>.
Thanks, Kristian --
Given my elementary knowledge of shell scripts this looks fine ...
One problem is that Myrna's build has already picked up my change --
COALESCE is in the Ref Manual already -- so probably the script won't
run Monday after all, if it is checking for changes. We might need to
wait for the next doc commit to make sure it works.
Kim
On 09/04/09 11:18, Kristian Waagan wrote:
> Kim Haase wrote:
>> A nightly build has been lovely (thanks, Myrna!) but realistically,
>> once a week would be fine for a build of the latest docs.
>>
>> I wish I could volunteer for this, since I'm the one who always
>> complains when the process breaks down, but I can't afford the time to
>> learn about cron jobs and checksums (about which I am extremely
>> ignorant). So many thanks to whoever takes this on.
>
> Hi Kim,
>
> I have written the first version of the script checking for updates and
> deploying the docs. It must be run from people.a.o. See the script
> attached, and please comment on it if you something that should be changed.
> Currently both the deploy directory and the MD5 checksum file must exist
> (it doesn't have to contain a checksum though).
>
> I tested it and hopefully something should appear in
> http://db.apache.org/derby/docs/dev_test/ in a little while (during
> tomorrow at latest).
> To navigate easier, you can use the links from the official
> documentation page [1], and then just replace dev with dev_test.
>
> I have also changed the Hudson job to check for changes every Monday
> (just after midnight UTC) and every Wednesday (around 20 in the
> evening/night UTC).
> My crontab will run the script a little later (to allow the build to
> finish).
>
>
> Since Kim committed a change yesterday, a new build of the docs should
> be built and published some time on Monday (under dev_test). Let's see
> what happens!
>
>
Re: Publishing the Derby documentation
Posted by Kristian Waagan <Kr...@Sun.COM>.
Kim Haase wrote:
> A nightly build has been lovely (thanks, Myrna!) but realistically,
> once a week would be fine for a build of the latest docs.
>
> I wish I could volunteer for this, since I'm the one who always
> complains when the process breaks down, but I can't afford the time to
> learn about cron jobs and checksums (about which I am extremely
> ignorant). So many thanks to whoever takes this on.
Hi Kim,
I have written the first version of the script checking for updates and
deploying the docs. It must be run from people.a.o. See the script
attached, and please comment on it if you something that should be changed.
Currently both the deploy directory and the MD5 checksum file must exist
(it doesn't have to contain a checksum though).
I tested it and hopefully something should appear in
http://db.apache.org/derby/docs/dev_test/ in a little while (during
tomorrow at latest).
To navigate easier, you can use the links from the official
documentation page [1], and then just replace dev with dev_test.
I have also changed the Hudson job to check for changes every Monday
(just after midnight UTC) and every Wednesday (around 20 in the
evening/night UTC).
My crontab will run the script a little later (to allow the build to
finish).
Since Kim committed a change yesterday, a new build of the docs should
be built and published some time on Monday (under dev_test). Let's see
what happens!
--
Kristian
[1] http://db.apache.org/derby/manuals/index.html
>
> Kim
>
> On 09/04/09 07:41, Kristian Waagan wrote:
>> Myrna van Lunteren wrote:
>>> On Thu, Sep 3, 2009 at 3:05 AM, Kristian
>>> Waagan<Kr...@sun.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have now configured a Hudson job building the Derby documentation.
>>>>
>>>
>>> Great!
>>> Thanks for all your work on setting up the Hudson system.
>>>
>>> I have one question...I've a automated process that copies nightly
>>> built docs out to /www/db.apache.org/derby/docs/dev. Can/Should I stop
>>> that job?
>>>
>>
>> Hi Myrna,
>>
>> I think you can stop the job, but I would appreciate if you kept
>> everything around for a while :)
>> Maybe you want to test out the new script when it's ready, or
>> integrate it in such a way that your builds work as a backup system
>> in case Hudson goes down.
>>
>> The docs take a little less than 1 hour to build on the Apache server
>> we're using.
>> How often do we want to check for changes and rebuild the docs?
>>
>> The website also has some stuff that looks like old cruft to me:
>> /www/db.apache.org/derby/docs/10.0-bak
>> /www/db.apache.org/derby/docs/10.1_old
>> /www/db.apache.org/derby/docs/10.4_old
>> docs-10.0.tar
>> docs-10.0.tar.gz
>> docs-10.0.zip
>>
>> Does anyone object to deleting these?
>> There are also some old files lying around in dev/, so I would delete
>> that one too and "repopulate" it.
>>
>>
>> Some of the directories have wrong ownership and/or wrong
>> permissions. I have taken action to fix that by sending mails to
>> Rick, Andrew and the infrastructure team.
>> To allow any committer to fix issues, it is important that the
>> files/dirs have group ownership set to 'db' and that the group is
>> given write access.
>>
>>
Re: Publishing the Derby documentation
Posted by Kim Haase <Ca...@Sun.COM>.
A nightly build has been lovely (thanks, Myrna!) but realistically, once
a week would be fine for a build of the latest docs.
I wish I could volunteer for this, since I'm the one who always
complains when the process breaks down, but I can't afford the time to
learn about cron jobs and checksums (about which I am extremely
ignorant). So many thanks to whoever takes this on.
Kim
On 09/04/09 07:41, Kristian Waagan wrote:
> Myrna van Lunteren wrote:
>> On Thu, Sep 3, 2009 at 3:05 AM, Kristian
>> Waagan<Kr...@sun.com> wrote:
>>
>>> Hello,
>>>
>>> I have now configured a Hudson job building the Derby documentation.
>>>
>>
>> Great!
>> Thanks for all your work on setting up the Hudson system.
>>
>> I have one question...I've a automated process that copies nightly
>> built docs out to /www/db.apache.org/derby/docs/dev. Can/Should I stop
>> that job?
>>
>
> Hi Myrna,
>
> I think you can stop the job, but I would appreciate if you kept
> everything around for a while :)
> Maybe you want to test out the new script when it's ready, or integrate
> it in such a way that your builds work as a backup system in case Hudson
> goes down.
>
> The docs take a little less than 1 hour to build on the Apache server
> we're using.
> How often do we want to check for changes and rebuild the docs?
>
> The website also has some stuff that looks like old cruft to me:
> /www/db.apache.org/derby/docs/10.0-bak
> /www/db.apache.org/derby/docs/10.1_old
> /www/db.apache.org/derby/docs/10.4_old
> docs-10.0.tar
> docs-10.0.tar.gz
> docs-10.0.zip
>
> Does anyone object to deleting these?
> There are also some old files lying around in dev/, so I would delete
> that one too and "repopulate" it.
>
>
> Some of the directories have wrong ownership and/or wrong permissions. I
> have taken action to fix that by sending mails to Rick, Andrew and the
> infrastructure team.
> To allow any committer to fix issues, it is important that the
> files/dirs have group ownership set to 'db' and that the group is given
> write access.
>
>
Re: Publishing the Derby documentation
Posted by Kristian Waagan <Kr...@Sun.COM>.
Myrna van Lunteren wrote:
> On Thu, Sep 3, 2009 at 3:05 AM, Kristian Waagan<Kr...@sun.com> wrote:
>
>> Hello,
>>
>> I have now configured a Hudson job building the Derby documentation.
>>
>
> Great!
> Thanks for all your work on setting up the Hudson system.
>
> I have one question...I've a automated process that copies nightly
> built docs out to /www/db.apache.org/derby/docs/dev. Can/Should I stop
> that job?
>
Hi Myrna,
I think you can stop the job, but I would appreciate if you kept
everything around for a while :)
Maybe you want to test out the new script when it's ready, or integrate
it in such a way that your builds work as a backup system in case Hudson
goes down.
The docs take a little less than 1 hour to build on the Apache server
we're using.
How often do we want to check for changes and rebuild the docs?
The website also has some stuff that looks like old cruft to me:
/www/db.apache.org/derby/docs/10.0-bak
/www/db.apache.org/derby/docs/10.1_old
/www/db.apache.org/derby/docs/10.4_old
docs-10.0.tar
docs-10.0.tar.gz
docs-10.0.zip
Does anyone object to deleting these?
There are also some old files lying around in dev/, so I would delete
that one too and "repopulate" it.
Some of the directories have wrong ownership and/or wrong permissions. I
have taken action to fix that by sending mails to Rick, Andrew and the
infrastructure team.
To allow any committer to fix issues, it is important that the
files/dirs have group ownership set to 'db' and that the group is given
write access.
--
Kristian
> Myrna
>
Re: Publishing the Derby documentation
Posted by Myrna van Lunteren <m....@gmail.com>.
On Thu, Sep 3, 2009 at 3:05 AM, Kristian Waagan<Kr...@sun.com> wrote:
> Hello,
>
> I have now configured a Hudson job building the Derby documentation.
Great!
Thanks for all your work on setting up the Hudson system.
I have one question...I've a automated process that copies nightly
built docs out to /www/db.apache.org/derby/docs/dev. Can/Should I stop
that job?
Myrna