Automating the downloads web page

[Julian Foad <ju...@apache.org>]

I'd like some suggestions or assistance.

Our tool 'release.py' can generate a block of HTML for the downloads 
table for a new release:

[[[
$ release.py write-downloads 1.13.0
<p style=[...]>Apache Subversion 1.13.0</p>
<table class="centered">
[...]
   <td><a 
href="[preferred]subversion/subversion-1.13.0.tar.bz2">subversion-1.13.0.tar.bz2</a></td>
[...]
</table>
]]]

How can we best insert this into 'downloads.html', at the right place?

So far, in my own tool 'svn-roll.sh' I have used some sed scripting to 
search for the "right place" and replace text, but it is fragile and 
incomplete: it works only as far as being provides a starting point for 
manual editing.

I feel like we want either a scripted generation of the whole 
'download.html' from a template, that we can re-run any time anything 
changed and be sure it will DTRT; or else use dynamic inclusion within 
'download.html' to fetch the relevant portions from somewhere.  I feel 
offline generation of a static HTML file is what we should prefer these 
days.  (At least relatively static, regarding inclusion of these tables; 
right now I don't care if it will continue to use some dynamic elements 
as well, such as CGI for mirror selection.)

Thoughts?  Code?

- Julian

Re: Automating the downloads web page

[Nathan Hartman <ha...@gmail.com>]

On Fri, Oct 25, 2019 at 10:04 AM Julian Foad <ju...@apache.org> wrote:

> Of course, there is a bigger picture.  Nearly twenty years on, we're
> still using our home-grown release management (and news feed, etc.)
> infrastructure.  In some sense it "does the job" but in other senses
> surely we might benefit by using some release management (etc.)
> infrastructure that's commonly used?  If we got it right so that it was
> not imposing a huge external system on us (gith... uh... ughb) but
> instead was something that allowed us to adopt small bits of useful
> infrastructure that remain under our control (especially our own domain
> and branding)...
>
> Wishful thinking or are there any real possibilities?


All good points.

I'll do some research in the next few days.

If anyone has some good ideas, please speak up!

Nathan

Re: Automating the downloads web page

[Julian Foad <ju...@apache.org>]

Of course, there is a bigger picture.  Nearly twenty years on, we're 
still using our home-grown release management (and news feed, etc.) 
infrastructure.  In some sense it "does the job" but in other senses 
surely we might benefit by using some release management (etc.) 
infrastructure that's commonly used?  If we got it right so that it was 
not imposing a huge external system on us (gith... uh... ughb) but 
instead was something that allowed us to adopt small bits of useful 
infrastructure that remain under our control (especially our own domain 
and branding)...

Wishful thinking or are there any real possibilities?

- Julian

Re: Automating the downloads web page

[Julian Foad <ju...@apache.org>]

Nathan Hartman wrote:
> Julian Foad  wrote:
>> How can we best insert this into 'downloads.html', at the right place?
> 
> Various [tools...]  insert their generated code between special comments.

I'd like something more standardized.

We already use 'ezt' templating here.  One option would be to generate 
'download.html' from 'download.html.ezt' which would contain

[[[
<h3>Latest regular release ...
[ezt templating here which generates this content]

<h3>Supported LTS releases ...
[ezt templating here which generates this content]

<h3>Pre-releases ...
[ezt templating here which generates this content]
]]]

The generator would have to know how to obtain a list of all the svn 
versions that are currently to be listed for download, along with their 
relevant details (release status, filenames, links, checksums, etc.).

That list should then be defined in some new, machine readable data 
source -- perhaps a JSON or YaML file under version control.  Until now, 
the canonical version of that list has been "whatever is shown in 
download.html".  Changing to use a canonical source data file would be a 
good thing.

That's the sort of thing I'd like to do.

There's a similar but much simpler issue with updating the 'news': we 
copy the same news items into both 'news.html' and 'index.html', and 
manually prune the list in 'index.html' to have approximately 3 entries. 
  That's not a lot of manual work but it's lame.  We should have a 
separate, struxtured, storage location for each news item and then an 
automatic way of pulling and formatting the newest 3 of them into 
'index.html' and all of them into 'news.html'.

- Julian

Re: Automating the downloads web page

[Nathan Hartman <ha...@gmail.com>]

On Fri, Oct 25, 2019 at 7:49 AM Julian Foad <ju...@apache.org> wrote:

>
> How can we best insert this into 'downloads.html', at the right place?


Various code generation tools (e.g. GUI designers) that are designed not to
clobber user edits insert their generated code between special comments.

So in download.html there might be comments like:

<!-- BEGIN AUTOMATICALLY GENERATED CODE -->

...

<!-- END AUTOMATICALLY GENERATED CODE -->

and the script would search for those and replace what's between them.