You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Rob Weir <ro...@apache.org> on 2013/10/01 16:17:51 UTC
Automating Error-prone Manual Release-related Files By Using
Centralized Release Artifact Metadata
Every time we release we need to generate quite a few files related to
the release.
Here's the list I know of:
1) The ASC/SHA/MD5 hashes and signature files, one for every released file
2) A CWiki page listing all the built files with hyperlinks to hashes, etc.
https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds#DevelopmentSnapshotBuilds-AOOSnapshot
This is used when voting on a Release Candidate.
3) release_matrix.js, used by the download page's logic to decide what
file to recommend based on user's locale:
http://www.openoffice.org/download/release_matrix.js
4) The other.html page that has a table listing all release files:
http://www.openoffice.org/download/other.html
5) Another download page, on openoffice.apache.org, listing the source
and SDK links, along with hashes:
https://openoffice.apache.org/downloads.html
6) A flat list of all download URLs, used by the download stats script:
http://svn.apache.org/repos/asf/openoffice/devtools/aoo-stats/401.lst
7) The XML files used by the upgrade notification server, one XML file
per release:
https://svn.apache.org/repos/asf/openoffice/ooo-site/trunk/content/projects/update38/ProductUpdateService/check.Update
I imagine this list will grow over time. Certainly new releases and
new translations cause us to update these files. And to the extent
they are manually created they will contain errors. Even if created
by automation, if we don't have a canonical data set that we're all
working from there is the opportunity for error.
So the big question: What can we do to improve on this?
1) Can we have a single, published, canonical, machine-readable
description of what is contained in a release, or even what is
contained in each build:
a) base URL of where the files live
b) based URL to where the hashes and signatures live
c) list of all language codes and platforms included in the build
d) defined rules for creating the full paths and file names given this
information.
2) Have such a data file for every past release, at least back to 3.4.0.
3) Write scripts to generate our other files from these canonical files.
4) Check this all in to a central place so when a new release comes
out we can generate these needed files automatically, with much lower
chance for errors.
5) Spend our extra time drinking beer and imagining how rich we'd all
be if we each received $0.01 for every download of AOO.
How close are we to this? Is any part of this automated already today?
Regards,
-Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Automating Error-prone Manual Release-related Files By Using
Centralized Release Artifact Metadata
Posted by Michal Hriň <mi...@yahoo.com>.
Dňa Tue, 01 Oct 2013 20:38:03 +0200 Marcus (OOo) <ma...@wtnet.de>
napísal:
> Am 10/01/2013 04:17 PM, schrieb Rob Weir:
>> Every time we release we need to generate quite a few files related to
>> the release.
>>
>> Here's the list I know of:
>>
>> 1) The ASC/SHA/MD5 hashes and signature files, one for every released
>> file
>>
>> 2) A CWiki page listing all the built files with hyperlinks to hashes,
>> etc.
>>
>> https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds#DevelopmentSnapshotBuilds-AOOSnapshot
>>
>> This is used when voting on a Release Candidate.
>>
>> 3) release_matrix.js, used by the download page's logic to decide what
>> file to recommend based on user's locale:
>>
>> http://www.openoffice.org/download/release_matrix.js
>
> Plus the file size for the install files. I get it from the Sourceforge
> master mirror via rsync.
>
> A simplification would be to combine "languages.js" and
> "release_matrix.js".
>
>> 4) The other.html page that has a table listing all release files:
>>
>> http://www.openoffice.org/download/other.html
>
> This is already automated via a few variables from "gloablvars.js".
>
>> 5) Another download page, on openoffice.apache.org, listing the source
>> and SDK links, along with hashes:
>>
>> https://openoffice.apache.org/downloads.html
>
> My opinion is that we don't need this. Everything is already available
> on "other.html". We could reduce the webpage to show general information
> but for the specific release things refer to "other.html".
>
>> 6) A flat list of all download URLs, used by the download stats script:
>>
>> http://svn.apache.org/repos/asf/openoffice/devtools/aoo-stats/401.lst
>>
>> 7) The XML files used by the upgrade notification server, one XML file
>> per release:
>>
>> https://svn.apache.org/repos/asf/openoffice/ooo-site/trunk/content/projects/update38/ProductUpdateService/check.Update
>>
>> I imagine this list will grow over time. Certainly new releases and
>> new translations cause us to update these files. And to the extent
>> they are manually created they will contain errors. Even if created
>> by automation, if we don't have a canonical data set that we're all
>> working from there is the opportunity for error.
>>
>> So the big question: What can we do to improve on this?
>>
>> 1) Can we have a single, published, canonical, machine-readable
>> description of what is contained in a release, or even what is
>> contained in each build:
>
> +1
> Yeah, a (so-called) "source of truth" would be great.
>
> For the download website the "globalvars.js" delivers the source data.
> But it needs to be filled with the data.
>
Topic looks interesting,
I have question, what data about released files do you mean ?
It is not enough to have : version+platform+language ?
>> a) base URL of where the files live
>
> http://sourceforge.net/projects/openofficeorg.mirror/files/
>
> Then + VERSION + "/binaries/" + ISO_CODE
>
>> b) based URL to where the hashes and signatures live
>
> http://www.apache.org/dist/openoffice/
>
> Then + VERSION + "/binaries/" + NL_LANGUAGE + "/" + FILENAME
>
> a) and b) can be seen as easy or difficult. Depends on your personal
> point of view.
>
>> c) list of all language codes and platforms included in the build
> >
>> d) defined rules for creating the full paths and file names given this
>> information.
>>
>> 2) Have such a data file for every past release, at least back to 3.4.0.
>
> +1
>
>> 3) Write scripts to generate our other files from these canonical files.
>>
>> 4) Check this all in to a central place so when a new release comes
>> out we can generate these needed files automatically, with much lower
>> chance for errors.
>>
>> 5) Spend our extra time drinking beer and imagining how rich we'd all
>> be if we each received $0.01 for every download of AOO.
>>
>> How close are we to this? Is any part of this automated already today?
>
> The top number #4 is already automated.
>
> But with the bottom #1 - #4 would be an advantage. However, I would be
> satisfied with #1 (source of base data) as everything else is not that
> difficult and time-consuming.
>
> My 2 ct.
>
> Marcus
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
--
Táto správa bola vytvorená poštovým klientom v prehliadači Opera:
http://www.opera.com/mail/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Automating Error-prone Manual Release-related Files By Using
Centralized Release Artifact Metadata
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 10/01/2013 04:17 PM, schrieb Rob Weir:
> Every time we release we need to generate quite a few files related to
> the release.
>
> Here's the list I know of:
>
> 1) The ASC/SHA/MD5 hashes and signature files, one for every released file
>
> 2) A CWiki page listing all the built files with hyperlinks to hashes, etc.
>
> https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds#DevelopmentSnapshotBuilds-AOOSnapshot
>
> This is used when voting on a Release Candidate.
>
> 3) release_matrix.js, used by the download page's logic to decide what
> file to recommend based on user's locale:
>
> http://www.openoffice.org/download/release_matrix.js
Plus the file size for the install files. I get it from the Sourceforge
master mirror via rsync.
A simplification would be to combine "languages.js" and "release_matrix.js".
> 4) The other.html page that has a table listing all release files:
>
> http://www.openoffice.org/download/other.html
This is already automated via a few variables from "gloablvars.js".
> 5) Another download page, on openoffice.apache.org, listing the source
> and SDK links, along with hashes:
>
> https://openoffice.apache.org/downloads.html
My opinion is that we don't need this. Everything is already available
on "other.html". We could reduce the webpage to show general information
but for the specific release things refer to "other.html".
> 6) A flat list of all download URLs, used by the download stats script:
>
> http://svn.apache.org/repos/asf/openoffice/devtools/aoo-stats/401.lst
>
> 7) The XML files used by the upgrade notification server, one XML file
> per release:
>
> https://svn.apache.org/repos/asf/openoffice/ooo-site/trunk/content/projects/update38/ProductUpdateService/check.Update
>
> I imagine this list will grow over time. Certainly new releases and
> new translations cause us to update these files. And to the extent
> they are manually created they will contain errors. Even if created
> by automation, if we don't have a canonical data set that we're all
> working from there is the opportunity for error.
>
> So the big question: What can we do to improve on this?
>
> 1) Can we have a single, published, canonical, machine-readable
> description of what is contained in a release, or even what is
> contained in each build:
+1
Yeah, a (so-called) "source of truth" would be great.
For the download website the "globalvars.js" delivers the source data.
But it needs to be filled with the data.
> a) base URL of where the files live
http://sourceforge.net/projects/openofficeorg.mirror/files/
Then + VERSION + "/binaries/" + ISO_CODE
> b) based URL to where the hashes and signatures live
http://www.apache.org/dist/openoffice/
Then + VERSION + "/binaries/" + NL_LANGUAGE + "/" + FILENAME
a) and b) can be seen as easy or difficult. Depends on your personal
point of view.
> c) list of all language codes and platforms included in the build
>
> d) defined rules for creating the full paths and file names given this
> information.
>
> 2) Have such a data file for every past release, at least back to 3.4.0.
+1
> 3) Write scripts to generate our other files from these canonical files.
>
> 4) Check this all in to a central place so when a new release comes
> out we can generate these needed files automatically, with much lower
> chance for errors.
>
> 5) Spend our extra time drinking beer and imagining how rich we'd all
> be if we each received $0.01 for every download of AOO.
>
> How close are we to this? Is any part of this automated already today?
The top number #4 is already automated.
But with the bottom #1 - #4 would be an advantage. However, I would be
satisfied with #1 (source of base data) as everything else is not that
difficult and time-consuming.
My 2 ct.
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Automating Error-prone Manual Release-related Files By Using
Centralized Release Artifact Metadata
Posted by Kay Schenk <ka...@gmail.com>.
On Tue, Oct 1, 2013 at 10:28 AM, Herbert Duerr <hd...@apache.org> wrote:
> Rob Weir wrote:
> > Every time we release we need to generate quite a few files related to
> > the release.
> >
> > Here's the list I know of:
> >
> > 1) The ASC/SHA/MD5 hashes and signature files, one for every released
> file
>
> We're using scripts for these of course. With the more than 400 packages
> we're providing everything else would be insane. The checksums should
> and are created on the machines they are built on to help protect the
> gigabytes of data as early as possible.
>
> > 2) A CWiki page listing all the built files with hyperlinks to hashes,
> etc.
> >
> >
> https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds#DevelopmentSnapshotBuilds-AOOSnapshot
> >
> > This is used when voting on a Release Candidate.
>
> Jürgen uses a self written script for that. AFAIK it is not publically
> available yet. If needed it can be rewritten with a few lines of a
> script language.
>
> > 3) release_matrix.js, used by the download page's logic to decide what
> > file to recommend based on user's locale:
> > [...]
> > 7) The XML files used by the upgrade notification server, one XML file
> > per release:
>
> Don't know about these. It would be stupid to do all of it manually. I
> suggest the owners of these tasks share their scripts, e.g. in the
> http://svn.apache.org/repos/asf/openoffice/devtools/scripts/ directory.
>
A good idea...even better if there were some Apache service (server) that
would allow access for cron jobs to do maybe a lot of this...we could
pretty creative. Perl and bash scripts are our friends. :)
> > How close are we to this? Is any part of this automated already today?
>
> The parts I'm aware of are automated.
>
> Herbert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
--
-------------------------------------------------------------------------------------------------
MzK
"Truth is stranger than fiction, but it is because Fiction is obliged
to stick to possibilities. Truth isn't."
-- "Following the Equator", Mark Twain
Re: Automating Error-prone Manual Release-related Files By Using
Centralized Release Artifact Metadata
Posted by Herbert Duerr <hd...@apache.org>.
Rob Weir wrote:
> Every time we release we need to generate quite a few files related to
> the release.
>
> Here's the list I know of:
>
> 1) The ASC/SHA/MD5 hashes and signature files, one for every released file
We're using scripts for these of course. With the more than 400 packages
we're providing everything else would be insane. The checksums should
and are created on the machines they are built on to help protect the
gigabytes of data as early as possible.
> 2) A CWiki page listing all the built files with hyperlinks to hashes, etc.
>
> https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds#DevelopmentSnapshotBuilds-AOOSnapshot
>
> This is used when voting on a Release Candidate.
Jürgen uses a self written script for that. AFAIK it is not publically
available yet. If needed it can be rewritten with a few lines of a
script language.
> 3) release_matrix.js, used by the download page's logic to decide what
> file to recommend based on user's locale:
> [...]
> 7) The XML files used by the upgrade notification server, one XML file
> per release:
Don't know about these. It would be stupid to do all of it manually. I
suggest the owners of these tasks share their scripts, e.g. in the
http://svn.apache.org/repos/asf/openoffice/devtools/scripts/ directory.
> How close are we to this? Is any part of this automated already today?
The parts I'm aware of are automated.
Herbert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org