Helping with Releases (Re: TomEE KEYS update)

[David Blevins <da...@gmail.com>]

Digging up this old thread as any offer to help really deserves a response.  I had travelled to help dear old mom these two weeks and didn't get back to you after I had a bit more bandwidth.

> On Sep 18, 2021, at 9:31 AM, Jenkins, Rodney J (Rod) <JE...@nationwide.com> wrote:
> 
> David,
> 
> Thank you for the insights and explanation!
> 
> I completely understand the technical debt and the challenge of making this better during a release.  I would like to jump in and see where I can help.  My problem is I am not a java developer.  What I am good it is automating tasks, if I can be taught to execute them.
> 
> The big ask is:  Would anyone want to take the time with me to educate me on what has to happen for a release (not during a release)?  I am thinking that we could set up a dummy repo that has some simple small java code in it to be a dummy TomEE release candidate.  Create some dummy destinations that mimic where the artifacts must be placed.  Once I understand the process, I can see about making it repeatable.
> 
> Personally, I would like to see it done in a way that someone with lesser skills (like, but not necessarily,  me) does releases.  The way I see it now is the heavy hitters do the releases.  I think their time would be better spent on the technical debt, bugs, etc.  Maybe we could find a small few that would be wiling the be release specialists.  I know some where I work that MAY be interested.  If someone would teach me, I would teach them.

You have the right spirit.  In the early days of Geronimo I did the majority of releases and while I did a good job, wasn't the job I wanted to be doing and could see how me doing all the releases created a knowledge vacuum that actually hurts the community.

What I did was suggest we start a system where each release had a pilot and a copilot.  The pilot would be an experienced person who know what they were doing, the copilot would be someone learning.  Next release the copilot would become the pilot and do the release and a new person would become the copilot.  Each release the documentation got a bit better, the technical debt paid down a bit.  Things improved, releases got more frequent and quality went up.

A significant blocker, however, is that a large number of the steps can't be done unless you're an Apache committer.  Not because of policy, just because they require access to systems only committers can reach.  You won't be able to run a lot of the commands.

That's not a show stopper, it just means it our creativity in how to leverage you will be heavily challenged.  I'm up for it if you are.

I have been trying to revitalize our release tools in java for automating as many release tasks as possible.  I do think that's the right way to go as the majority of people who do releases are java developers.  In the past I've written elaborate scripts in bash and inevitably they decay as I'd be the only one who understood them.  That doesn't necessarily mean there's no role for scripting.

One of the hardest parts of doing releases is remembering all the steps that have to be done and double checking they were done correctly.  This is absolutely something you could script up and run.  I will say in all my attempts to document release processes, I've noticed even in the best circumstances they're always a little out of date.  Turn your back even a little and they become just out of date enough no one even looks at the doc anymore, than it's game over.

If we had a release auditing script that could be pretty amazing.  It could be the checklist people use to "see" the release process as a whole.  Again I'm imagining something like the System V startup output where there's one line per task.  Each line is a step that has to be done a green/red colored status on the right.

Here are some things I frequently see done wrong and not noticed:

 - Forgetting to update the keys file.  Topic of the original thread.  Because it's something you only need to do once in a while, it's easy to forget.  There's a release tools command to help people do it, but you have to remember you rotated your key a few months ago and need to run it.  Perfect thing to audit.

 - Missing signatures once published.  We've had infrastructure ask us to add signatures because we've forgotten a few times.  There's already a script/command to do it, but nothing to check if people use them.  As mentioned, people stop using scripts they find confusing, so they decide to wing it instead.  A script to audit this part would be great.  Apache infra has such a script and does run it every few months, but it's not ideal when they're the ones point it out.

 - Leaving SNAPSHOT or old TomEE dependencies in the examples or build.  I spent three days cleaning out references in the examples to older TomEE 7 and 8 snapshots missed by past releases.  The way you check is delete your ~/.m2/repository directory and build the release tag from git.  If there is anything with a SNAPSHOT dependency in there, that's a fail.  If there are any jars from previous releases in there, that's a fail.  We should not pass these release votes, we should fix the source and roll again.  Again, there is a tool to do this (maven) but it doesn't always catch everything.  It only renames references to the current version to the next version.  If there's a reference to a version say two releases ago, it will not see it.  It will also not rename references for any modules that are commented out, which happens with the examples from time to time.  It really needs auditing.

 - Updating the generated download page HTML.  Frequently people will directly update the download.html page, just hacking in the new release.  The problem of course is all the html pages are generated and it will inevitably get overridden and set back to the previous release the next time the website is generated.  It doesn't seem to matter that repo containing generated HTML files has "do make updates here, this is all generated" written on it.  We still get this mistake from committers and contributors alike.  We in fact just had a PR opened against the generated html repo.  On release, people need to know and remember to update the markdown/asciidoc source which is in a different repo.  There's a specific file we could audit via a simple grep and flag up "tomee-site-generator download.adoc page updated [FAIL]"

 - Forgetting to publish the maven artifacts.  Only happened once, but there was an occasion where we forgot to click the publish button on the nexus repository holding the maven distributions of all the release jars.  They expired and were deleted.  That meant we had a release that was on the website, in the download area for people that wanted to just grab a zip file, but was nowhere maven user could access.  During the release process a special temporary release area is created on repository.apache.org, we need to keep an eye on that and audit that it was in fact released and everything there has reached Maven Central.


I could probably think of a half dozen more checks, but the above is hopefully enough to paint the picture that it would be useful.  There are systems to do everything above, just you know ... darn people being all human :)  None of the above is enough detail to actually implement most of this, so we can have better discussions if we actually want to do any of it.

Another aspect to this idea, Richard Zowalla has been doing a lot of fun experimentations with Github Actions.  There is definitely scripting potential there and some of the above could be triggered when release tags show up in git.  Unlike setting up Jenkins jobs, it's way easier to contribute to Github Actions as it's all git-based so PRs work perfectly.  You could test on a forked repo and then submit a PR.  Some real potential to make outside help possible with this capability.


Some ideas and thoughts.  Hope not an overwhelming amount.  :)

Any thoughts extremely welcome!


-David

Re: Helping with Releases (Re: TomEE KEYS update)

["Zowalla, Richard" <ri...@hs-heilbronn.de>]

I like the idea of pairing people to act as release managers. That is a
great way to transfer knowledge and open the circle of people "who
know". 

By doing so, it might be even possible to have some sort of (mostly) up
2 date release documentation, which could then be integrated in the
release tools. A checklist as a starter (or a script in the next
iteration) is also a good thing + the 4 eye principle. 

If we have some sort of "structured" documentation / approach, we could
try to automate parts of it leveraging CI tools available at ASF (which
might have some restriction due to INFRA restrictions).

> A significant blocker, however, is that a large number of the steps
> can't be done unless you're an Apache committer.  Not because of
> policy, just because they require access to systems only committers
> can reach.  You won't be able to run a lot of the commands.

In addition, it seems that even a "normal" committer cannot act as
release manager unless a special request is sent to INFRA [2] and the
particular access rights are granted. I don't know, if this was already
requested for TomEE ;)

To add some additional things, which we tend to make different from
times to times:

- Creation of Release Notes: We sometimes use the JIRA feature to show
our changes or use the release-tools to create the .adoc file, which is
then put on the website, too. In the past, we didn't create proper
relese notes for our website and peope had to dig into Jira to find
them. I like our current approach (since 9-M7) by adding dedicated
release notes to the website much more. They are even referenced on
Stackoverflow now *lol*.

Gruß
Richard

[1] https://infra.apache.org/release-publishing.html#releasemanager


Am Donnerstag, dem 17.03.2022 um 17:50 -0700 schrieb David Blevins:
> Digging up this old thread as any offer to help really deserves a
> response.  I had travelled to help dear old mom these two weeks and
> didn't get back to you after I had a bit more bandwidth.
> 
> > On Sep 18, 2021, at 9:31 AM, Jenkins, Rodney J (Rod) <
> > JENKIR14@nationwide.com> wrote:
> > 
> > David,
> > 
> > Thank you for the insights and explanation!
> > 
> > I completely understand the technical debt and the challenge of
> > making this better during a release.  I would like to jump in and
> > see where I can help.  My problem is I am not a java
> > developer.  What I am good it is automating tasks, if I can be
> > taught to execute them.
> > 
> > The big ask is:  Would anyone want to take the time with me to
> > educate me on what has to happen for a release (not during a
> > release)?  I am thinking that we could set up a dummy repo that has
> > some simple small java code in it to be a dummy TomEE release
> > candidate.  Create some dummy destinations that mimic where the
> > artifacts must be placed.  Once I understand the process, I can see
> > about making it repeatable.
> > 
> > Personally, I would like to see it done in a way that someone with
> > lesser skills (like, but not necessarily,  me) does releases.  The
> > way I see it now is the heavy hitters do the releases.  I think
> > their time would be better spent on the technical debt, bugs,
> > etc.  Maybe we could find a small few that would be wiling the be
> > release specialists.  I know some where I work that MAY be
> > interested.  If someone would teach me, I would teach them.
> 
> You have the right spirit.  In the early days of Geronimo I did the
> majority of releases and while I did a good job, wasn't the job I
> wanted to be doing and could see how me doing all the releases
> created a knowledge vacuum that actually hurts the community.
> 
> What I did was suggest we start a system where each release had a
> pilot and a copilot.  The pilot would be an experienced person who
> know what they were doing, the copilot would be someone
> learning.  Next release the copilot would become the pilot and do the
> release and a new person would become the copilot.  Each release the
> documentation got a bit better, the technical debt paid down a
> bit.  Things improved, releases got more frequent and quality went
> up.
> 
> A significant blocker, however, is that a large number of the steps
> can't be done unless you're an Apache committer.  Not because of
> policy, just because they require access to systems only committers
> can reach.  You won't be able to run a lot of the commands.
> 
> That's not a show stopper, it just means it our creativity in how to
> leverage you will be heavily challenged.  I'm up for it if you are.
> 
> I have been trying to revitalize our release tools in java for
> automating as many release tasks as possible.  I do think that's the
> right way to go as the majority of people who do releases are java
> developers.  In the past I've written elaborate scripts in bash and
> inevitably they decay as I'd be the only one who understood
> them.  That doesn't necessarily mean there's no role for scripting.
> 
> One of the hardest parts of doing releases is remembering all the
> steps that have to be done and double checking they were done
> correctly.  This is absolutely something you could script up and
> run.  I will say in all my attempts to document release processes,
> I've noticed even in the best circumstances they're always a little
> out of date.  Turn your back even a little and they become just out
> of date enough no one even looks at the doc anymore, than it's game
> over.
> 
> If we had a release auditing script that could be pretty amazing.  It
> could be the checklist people use to "see" the release process as a
> whole.  Again I'm imagining something like the System V startup
> output where there's one line per task.  Each line is a step that has
> to be done a green/red colored status on the right.
> 
> Here are some things I frequently see done wrong and not noticed:
> 
>  - Forgetting to update the keys file.  Topic of the original
> thread.  Because it's something you only need to do once in a while,
> it's easy to forget.  There's a release tools command to help people
> do it, but you have to remember you rotated your key a few months ago
> and need to run it.  Perfect thing to audit.
> 
>  - Missing signatures once published.  We've had infrastructure ask
> us to add signatures because we've forgotten a few times.  There's
> already a script/command to do it, but nothing to check if people use
> them.  As mentioned, people stop using scripts they find confusing,
> so they decide to wing it instead.  A script to audit this part would
> be great.  Apache infra has such a script and does run it every few
> months, but it's not ideal when they're the ones point it out.
> 
>  - Leaving SNAPSHOT or old TomEE dependencies in the examples or
> build.  I spent three days cleaning out references in the examples to
> older TomEE 7 and 8 snapshots missed by past releases.  The way you
> check is delete your ~/.m2/repository directory and build the release
> tag from git.  If there is anything with a SNAPSHOT dependency in
> there, that's a fail.  If there are any jars from previous releases
> in there, that's a fail.  We should not pass these release votes, we
> should fix the source and roll again.  Again, there is a tool to do
> this (maven) but it doesn't always catch everything.  It only renames
> references to the current version to the next version.  If there's a
> reference to a version say two releases ago, it will not see it.  It
> will also not rename references for any modules that are commented
> out, which happens with the examples from time to time.  It really
> needs auditing.
> 
>  - Updating the generated download page HTML.  Frequently people will
> directly update the download.html page, just hacking in the new
> release.  The problem of course is all the html pages are generated
> and it will inevitably get overridden and set back to the previous
> release the next time the website is generated.  It doesn't seem to
> matter that repo containing generated HTML files has "do make updates
> here, this is all generated" written on it.  We still get this
> mistake from committers and contributors alike.  We in fact just had
> a PR opened against the generated html repo.  On release, people need
> to know and remember to update the markdown/asciidoc source which is
> in a different repo.  There's a specific file we could audit via a
> simple grep and flag up "tomee-site-generator download.adoc page
> updated [FAIL]"
> 
>  - Forgetting to publish the maven artifacts.  Only happened once,
> but there was an occasion where we forgot to click the publish button
> on the nexus repository holding the maven distributions of all the
> release jars.  They expired and were deleted.  That meant we had a
> release that was on the website, in the download area for people that
> wanted to just grab a zip file, but was nowhere maven user could
> access.  During the release process a special temporary release area
> is created on repository.apache.org, we need to keep an eye on that
> and audit that it was in fact released and everything there has
> reached Maven Central.
> 
> 
> I could probably think of a half dozen more checks, but the above is
> hopefully enough to paint the picture that it would be useful.  There
> are systems to do everything above, just you know ... darn people
> being all human :)  None of the above is enough detail to actually
> implement most of this, so we can have better discussions if we
> actually want to do any of it.
> 
> Another aspect to this idea, Richard Zowalla has been doing a lot of
> fun experimentations with Github Actions.  There is definitely
> scripting potential there and some of the above could be triggered
> when release tags show up in git.  Unlike setting up Jenkins jobs,
> it's way easier to contribute to Github Actions as it's all git-based 
> so PRs work perfectly.  You could test on a forked repo and then
> submit a PR.  Some real potential to make outside help possible with
> this capability.
> 
> 
> Some ideas and thoughts.  Hope not an overwhelming amount.  :)
> 
> Any thoughts extremely welcome!
> 
> 
> -David
>