You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/08 16:59:41 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
TheNeuralBit opened a new pull request #12196:
URL: https://github.com/apache/beam/pull/12196
Cherry-pick #12067 into 2.23.0.
The bug this fixes is not technically a regression since it existed in previous versions as well, but I think it's serious enough to merit cherry picking into 2.23.0.
R: @tvalentyn
CC: @chamikaramj @robertwb
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-656278107
This change is currently is not in RC1, and we can consider if for RC2 if there has to be an RC2. The release guildeline clearly say that regressions in new functionality should not be blocking the release. I have applied the same criteria to at least 3 or 4 other CP requests, so it would not be fair to have to select which new features are worth delaying the release and which are not. Also, are we confident that the fix would not introduce new issues given that it was merged recently?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-655664851
I agree with the severity, especially as we'll be widely advertising cross-language with the 2.23 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-656906406
Here are some answers to the questions from https://beam.apache.org/contribute/release-guide/#review-cherry-picks:
> Is this a regression from a previous release? (If no, fix could go to a newer version.)
No, it's a bug, but it existed in previous releases.
> Is this a new feature or related to a new feature? (If yes, fix could go to a new version.)
Not a new feature, a bugfix for a >1 year old feature.
> Would this impact production workloads for users? (E.g. if this is a direct runner only fix it may not need to be a cherry pick.)
Yes, python users may want to use external transforms in production, and they could be blocked by this bug.
> What percentage of users would be impacted by this issue if it is not fixed? (E.g. If this is predicted to be a small number it may not need to be a cherry pick.)
Hard to say. I'm not sure how many users are trying out external transforms, and of those that are it's possible they won't ever encounter this bug.
> Would it be possible for the impacted users to skip this version? (If users could skip this version, fix could go to a newer version.)
Not sure this applies since there isn't a "working" version for users to stay on and skip 2.23.0.
To be honest I think this situation is somewhat of a gap in the guidance there. There may be cases where its worth delaying a release for a severe bugfix even if it's a longstanding issue and not a regression, and that's not addressed in our release guide. We could draw a line somewhere: e.g. maybe data loss/incorrect result bugs can delay, crashes can not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-656278107
This change is currently is not in RC1, and we can consider if for RC2 if there has to be an RC2. The release guildeline clearly say that bugs in new functionality should not be blocking the release. I have applied the same criteria to at least 3 or 4 other CP requests, so it would not be fair to have to select which new features are worth delaying the release and which are not. Also, are we confident that the fix would not introduce new issues given that it was merged recently?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-655684620
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-656866592
xlang is new on Dataflow, but has been supported in Flink, Spark, etc. for over a year now, so I wouldn't call it a "new" feature. I have high confidence that this would not introduce new issues. That being said, this is your call as the release manager.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-658441886
> To be honest I think this situation is somewhat of a gap in the guidance there. There may be cases where its worth delaying a release for a severe bugfix even if it's a longstanding issue and not a regression, and that's not addressed in our release guide. We could draw a line somewhere: e.g. maybe data loss/incorrect result bugs can delay, crashes can not?
In my opinion for these situations releasing a patch version (2.23.1) following a release (2.23.0), would be most preferable to users: new features/bugfixes already in RC1 will become available earlier, and cherry-pick worthy long-standing issues that were recently fixed and easy to back-port won't have to wait 6 weeks to be released. If anything goes wrong (cherry-pick conflict, new bug), 2.23.0 will still stay around.
For example, for this particular change I think we would only need to update Python SDK artifacts. We can propose to make 2.23.1 with this fix right after 2.23.0, without waiting for 2.24.0, if you think it's urgent. Thoughts @TheNeuralBit @robertwb ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-656201084
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn merged pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
tvalentyn merged pull request #12196:
URL: https://github.com/apache/beam/pull/12196
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12196: [release-2.23.0][BEAM-10308] Make component ID assignments consistent across Pipeline…
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12196:
URL: https://github.com/apache/beam/pull/12196#issuecomment-658446507
I think that's fair. If we happen to cut a RC2 we can consider including this. Otherwise we can wait for next release. If we determine that some of our users are hitting this after the release, we can consider cutting a 2.23.1.
I verified that Kafka transforms work with Beam 2.23.0 and updated the release spreadsheet. Thanks Valentyn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org