You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Kristian Waagan (JIRA)" <ji...@apache.org> on 2011/08/22 23:52:29 UTC
[jira] [Created] (DERBY-5388) Write tool to automate archive
process for old release pages
Write tool to automate archive process for old release pages
------------------------------------------------------------
Key: DERBY-5388
URL: https://issues.apache.org/jira/browse/DERBY-5388
Project: Derby
Issue Type: Improvement
Components: Web Site
Affects Versions: 10.9.0.0
Reporter: Kristian Waagan
Although release files themselves are automatically copied to the ASF archive, several further actions most be taken to update the web site when a release is archived.
In short:
a) update the download page itself
b) remove cgi script
c) update links to point to the archive in the HTML file
d) remove CGI-enabling code in the the HTML file
I'm hoping steps b - d can be automated. It's not clear to me how/where to integrate this yet (i.e. in the ant targets in the code repos, or as a script in the web site repos), but I think the process we want to follow may dictate that. We probably don't want to remove existing, working pages/links before we have verified that the new ones are working. Verification takes a while, since we have to wait for the changes to propagate.
The reason for this suggestion is that several issues have been logged by the ASF infra team for problems with the release pages. There has also been errors which are most likely caused by "finger trouble" when manually editing the web pages.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5388) Write tool to automate archive
process for old release pages
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-5388:
-----------------------------------
Issue & fix info: [Patch Available]
Assignee: Kristian Waagan
> Write tool to automate archive process for old release pages
> ------------------------------------------------------------
>
> Key: DERBY-5388
> URL: https://issues.apache.org/jira/browse/DERBY-5388
> Project: Derby
> Issue Type: Improvement
> Components: Web Site
> Affects Versions: 10.9.0.0
> Reporter: Kristian Waagan
> Assignee: Kristian Waagan
> Attachments: archive-release.py
>
>
> Although release files themselves are automatically copied to the ASF archive, several further actions most be taken to update the web site when a release is archived.
> In short:
> a) update the download page itself
> b) remove cgi script
> c) update links to point to the archive in the HTML file
> d) remove CGI-enabling code in the the HTML file
> I'm hoping steps b - d can be automated. It's not clear to me how/where to integrate this yet (i.e. in the ant targets in the code repos, or as a script in the web site repos), but I think the process we want to follow may dictate that. We probably don't want to remove existing, working pages/links before we have verified that the new ones are working. Verification takes a while, since we have to wait for the changes to propagate.
> The reason for this suggestion is that several issues have been logged by the ASF infra team for problems with the release pages. There has also been errors which are most likely caused by "finger trouble" when manually editing the web pages.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5388) Write tool to automate archive
process for old release pages
Posted by "Kristian Waagan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-5388:
-----------------------------------
Issue & fix info: (was: Patch Available)
Clearing patch available flag.
The script has passed manual testing and can be used by the release managers who want to use it.
Based on the communication from ASF infra regarding svnpubsub migration by the end of 2012, it seems reasonable to put this issue to rest for the time being and revisit it once we know which changes (if any) the site will undergo.
> Write tool to automate archive process for old release pages
> ------------------------------------------------------------
>
> Key: DERBY-5388
> URL: https://issues.apache.org/jira/browse/DERBY-5388
> Project: Derby
> Issue Type: Improvement
> Components: Web Site
> Affects Versions: 10.9.0.0
> Reporter: Kristian Waagan
> Assignee: Kristian Waagan
> Attachments: archive-release.py
>
>
> Although release files themselves are automatically copied to the ASF archive, several further actions most be taken to update the web site when a release is archived.
> In short:
> a) update the download page itself
> b) remove cgi script
> c) update links to point to the archive in the HTML file
> d) remove CGI-enabling code in the the HTML file
> I'm hoping steps b - d can be automated. It's not clear to me how/where to integrate this yet (i.e. in the ant targets in the code repos, or as a script in the web site repos), but I think the process we want to follow may dictate that. We probably don't want to remove existing, working pages/links before we have verified that the new ones are working. Verification takes a while, since we have to wait for the changes to propagate.
> The reason for this suggestion is that several issues have been logged by the ASF infra team for problems with the release pages. There has also been errors which are most likely caused by "finger trouble" when manually editing the web pages.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5388) Write tool to automate archive
process for old release pages
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098829#comment-13098829 ]
Kristian Waagan commented on DERBY-5388:
----------------------------------------
Thanks, Rick.
I'm replying to some of your comments/questions below.
RH> Your conversations with the infrastructure group may shed light on
some behaviors of our release management processes which have always
confused me.
KW> Not quite sure what you mean here. If you had anything specific in mind,
let me know.
RH> ... I think the word "archive" is a bit overloaded. Our release
documention uses this term for 3 distinct purposes:
1) Some of our release documentation refers to the Derby
distributions themselves (the tars and zips) as "archives".
KW> To me, these are release artifacts.
RH> 2) The "archives" are the machines/directories where Apache
permanently stores release distributions.
KW> Right - maybe we can use a term like "Apache software archive" to
reduce confusion?
RH> 3) "Archiving" is used to mean updating the download pages so that
they point at release distributions in the Apache archives rather
than on the mirrors.
KW> I agree this is confusing, as it also conflicts with the archiving
that ASF is doing automatically for us - copying deployed release
artifacts to the Apache software archive.
Do we have another word/term we can use for the process of disabling
mirroring and moving a release from one section to another in the
Derby download page?
RH> ... I don't understand what makes old releases disappear from the mirrors.
KW> See the last comment below.
------------------
RH> Currently, our publication instructions tell the release manager to
remove cgi scripts for the old releases at the same time that the new
release is posted. That seems to have worked well in the past but I may
be forgetting some release when this approach caused problems.
KW> In that case, I see no reason why to introduce another step in the
process - we can continue doing what we're doing now.
-------------------
RH> I think we need to switch download pointers before the old releases
disappear from the archives. But I don't know what triggers that
disappearance. I don't understand what harm is caused by continuing
to point old releases at the mirrors.
KW> I suppose you meant "mirrors" in the first sentence above?
The Derby release artifacts are removed from the mirrors shortly after
we explicitly remove them from the main Apache dist directory.
Based on what the ASF infrastructure team has communicated, the harm
caused by not cleaning up the main dist directory is increased load
on parts of the Apache infrastructure and on the infrastructure of
the mirrors. Both ends have to pay in terms of increased network
traffic when the mirrors are syncing up, and both ends have to deal
store old software "nobody" is downloading anymore (at the ASF end,
this probably includes backup too).
> Write tool to automate archive process for old release pages
> ------------------------------------------------------------
>
> Key: DERBY-5388
> URL: https://issues.apache.org/jira/browse/DERBY-5388
> Project: Derby
> Issue Type: Improvement
> Components: Web Site
> Affects Versions: 10.9.0.0
> Reporter: Kristian Waagan
> Assignee: Kristian Waagan
> Attachments: archive-release.py
>
>
> Although release files themselves are automatically copied to the ASF archive, several further actions most be taken to update the web site when a release is archived.
> In short:
> a) update the download page itself
> b) remove cgi script
> c) update links to point to the archive in the HTML file
> d) remove CGI-enabling code in the the HTML file
> I'm hoping steps b - d can be automated. It's not clear to me how/where to integrate this yet (i.e. in the ant targets in the code repos, or as a script in the web site repos), but I think the process we want to follow may dictate that. We probably don't want to remove existing, working pages/links before we have verified that the new ones are working. Verification takes a while, since we have to wait for the changes to propagate.
> The reason for this suggestion is that several issues have been logged by the ASF infra team for problems with the release pages. There has also been errors which are most likely caused by "finger trouble" when manually editing the web pages.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5388) Write tool to automate archive
process for old release pages
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098142#comment-13098142 ]
Rick Hillegas commented on DERBY-5388:
--------------------------------------
Thanks for the patch, Kristian. Your conversations with the infrastructure group may shed light on some behaviors of our release management processes which have always confused me.
I see that the patch is coded in python. Every new tool we require the release manager to understand reduces our pool of potential release managers. This was one of the problems I had with the Eclipse plugins. We can figure out over time whether the python script runs flawlessly every time (like Forrest, another tool I don't understand) or whether it needs tweaking per release (like the Eclipse plugins).
A quick glance at the patch suggests that it may be attempting to automate step (b) in the Description section of this JIRA. However, the header comment of the script suggests that the patch intends to "modify web site to archive old release". Don't know what that means. As I understand it, releases are archived almost immediately after they are posted under /www/www.apache.org/dist/db/derby. Step (b) happens maybe six months later when the next release is posted. I think the word "archive" is a bit overloaded. Our release documention uses this term for 3 distinct purposes:
1) Some of our release documentation refers to the Derby distributions themselves (the tars and zips) as "archives".
2) The "archives" are the machines/directories where Apache permanently stores release distributions.
3) "Archiving" is used to mean updating the download pages so that they point at release distributions in the Apache archives rather than on the mirrors.
It's all a bit muddling to me, I'm afraid. I don't understand what makes old releases disappear from the mirrors.
------------------
Addressing your question about whether this script should be run after the new release has gone live and its downloads appear to be working:
+ An advantage of doing it that way is that it reduces the risk that users will be cut off from older releases in case the new release links are broken and the python script does something goofy.
- A disadvantage of doing it that way is that the release manager has to remember to come back later on to finish up the process. I think that increases the risk that the rele se manager will forget this step.
Currently, our publication instructions tell the release manager to remove cgi scripts for the old releases at the same time that the new release is posted. That seems to have worked well in the past but I may be forgetting some release when this approach caused problems.
-------------------
Addressing your question about when to switch downloads of a release from the mirrors to the archives:
I think we need to switch download pointers before the old releases disappear from the archives. But I don't know what triggers that disappearance. I don't understand what harm is caused by continuing to point old releases at the mirrors.
Thanks,
-Rick
> Write tool to automate archive process for old release pages
> ------------------------------------------------------------
>
> Key: DERBY-5388
> URL: https://issues.apache.org/jira/browse/DERBY-5388
> Project: Derby
> Issue Type: Improvement
> Components: Web Site
> Affects Versions: 10.9.0.0
> Reporter: Kristian Waagan
> Assignee: Kristian Waagan
> Attachments: archive-release.py
>
>
> Although release files themselves are automatically copied to the ASF archive, several further actions most be taken to update the web site when a release is archived.
> In short:
> a) update the download page itself
> b) remove cgi script
> c) update links to point to the archive in the HTML file
> d) remove CGI-enabling code in the the HTML file
> I'm hoping steps b - d can be automated. It's not clear to me how/where to integrate this yet (i.e. in the ant targets in the code repos, or as a script in the web site repos), but I think the process we want to follow may dictate that. We probably don't want to remove existing, working pages/links before we have verified that the new ones are working. Verification takes a while, since we have to wait for the changes to propagate.
> The reason for this suggestion is that several issues have been logged by the ASF infra team for problems with the release pages. There has also been errors which are most likely caused by "finger trouble" when manually editing the web pages.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5388) Write tool to automate archive
process for old release pages
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-5388:
-----------------------------------
Attachment: archive-release.py
First attempt at automating the required tasks to archive a release. We might want to testdrive this tool on 10.8.1.2 when 10.8.2 has been deployed. The approach isn't terribly elegant, but if we keep the HTML/XML as it is today it should work fine (the tool is dependent on a few special strings).
I'm thinking the tool can live in site-trunk/scripts.
To run the script (after you have copied it into the directory mentioned above), type:
python scripts/archive-release.py 10.8.1.2
Patch ready for review.
I don't know if this is automation enough, but it will at least take away the error-prone manual search-and-replace that we currently rely on.
I think it would be best to run this script after the new release has gone live - not at the time of deployment.
Do people agree with this?
We also have to come up with guidelines for how to deal with maintenance releases off older branches. For instance, for how long would we want to mirror a release off the 10.7 branch if we released one in October (after 10.8.2)?
We can also "archive" the release immediately on deployment, but that gives it less visibility on the web site and requires a slightly modified process.
> Write tool to automate archive process for old release pages
> ------------------------------------------------------------
>
> Key: DERBY-5388
> URL: https://issues.apache.org/jira/browse/DERBY-5388
> Project: Derby
> Issue Type: Improvement
> Components: Web Site
> Affects Versions: 10.9.0.0
> Reporter: Kristian Waagan
> Attachments: archive-release.py
>
>
> Although release files themselves are automatically copied to the ASF archive, several further actions most be taken to update the web site when a release is archived.
> In short:
> a) update the download page itself
> b) remove cgi script
> c) update links to point to the archive in the HTML file
> d) remove CGI-enabling code in the the HTML file
> I'm hoping steps b - d can be automated. It's not clear to me how/where to integrate this yet (i.e. in the ant targets in the code repos, or as a script in the web site repos), but I think the process we want to follow may dictate that. We probably don't want to remove existing, working pages/links before we have verified that the new ones are working. Verification takes a while, since we have to wait for the changes to propagate.
> The reason for this suggestion is that several issues have been logged by the ASF infra team for problems with the release pages. There has also been errors which are most likely caused by "finger trouble" when manually editing the web pages.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira