[GitHub] [beam] mxm commented on a change in pull request #12455: [BEAM-10630] Include data from load tests in the release process

[GitBox <gi...@apache.org>]

mxm commented on a change in pull request #12455:
URL: https://github.com/apache/beam/pull/12455#discussion_r464971852



##########
File path: website/www/site/content/en/contribute/release-guide.md
##########
@@ -244,7 +247,21 @@ __Attention__: Only PMC has permission to perform this. If you are not a PMC, pl
 **********
 
 
-## 2. Create a release branch in apache/beam repository
+## 3. Investigate performance regressions
+
+Check the Beam load tests for possible performance regressions. Measurements are available on [metrics.beam.apache.org](http://metrics.beam.apache.org).
+
+All Runners which publish data should be checked for the following, in both *batch* and *streaming* mode:
+
+- [ParDo](http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests) and [GBK](http://metrics.beam.apache.org/d/UYZ-oJ3Zk/gbk-load-test): Runtime, latency, checkpoint duration
+- [Nexmark](http://metrics.beam.apache.org/d/ahuaA_zGz/nexmark): Query runtime for all queries
+- [IO](http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow): Runtime
+
+If regressions are found, the release branch can still be created, but the regressions should be investigated and fixed as part of the release process.
+JIRA issues should be created for each regression with the 'Fix Version' set to the to-be-released version.

Review comment:
       I thought the release manager would be responsible for identifying and filing the issues in JIRA. After that, they are part of the open issues for the release. The release manager oversees those, but relies on community members to fix them.
   
   It is reasonable to add a point of contact. Problem with these list of contacts is that they tend to become stale (like some of the OWNERS files we have). Perhaps we could just link to the existing OWNERS files? That way, we wouldn't introduce any additional lists.

##########
File path: website/www/site/content/en/contribute/release-guide.md
##########
@@ -244,7 +247,21 @@ __Attention__: Only PMC has permission to perform this. If you are not a PMC, pl
 **********
 
 
-## 2. Create a release branch in apache/beam repository
+## 3. Investigate performance regressions
+
+Check the Beam load tests for possible performance regressions. Measurements are available on [metrics.beam.apache.org](http://metrics.beam.apache.org).
+
+All Runners which publish data should be checked for the following, in both *batch* and *streaming* mode:
+
+- [ParDo](http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests) and [GBK](http://metrics.beam.apache.org/d/UYZ-oJ3Zk/gbk-load-test): Runtime, latency, checkpoint duration
+- [Nexmark](http://metrics.beam.apache.org/d/ahuaA_zGz/nexmark): Query runtime for all queries
+- [IO](http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow): Runtime
+
+If regressions are found, the release branch can still be created, but the regressions should be investigated and fixed as part of the release process.
+JIRA issues should be created for each regression with the 'Fix Version' set to the to-be-released version.
+Next, the mailing list should be informed to allow fixing the regressions in the course of the release.

Review comment:
       I think we can follow the same process as for other issues marked with the "Fix Version" of the upcoming release.
   
   Generally speaking, it could be good to spell out how the release manager informs the community about the release process. This could include an email with all the pending issues, e.g. bugs, performance regressions, untriaged issues.

##########
File path: website/www/site/content/en/contribute/release-guide.md
##########
@@ -1163,7 +1180,7 @@ Use reporter.apache.org to seed the information about the release into future pr
 **********
 
 
-## 9. Promote the release
+## 11. Promote the release

Review comment:
       It does. We use that feature in the table of contents. (You simply always specify `1.` as the number).
   
   I don't know why we don't use it for the actual sections. I guess to keep it plain-text readable. Either way, it would make sense to stick to just one of the two approaches.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org