You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/03/23 22:15:34 UTC
[GitHub] [beam] kennknowles opened a new pull request #14317: Roll forward Dataflow V2 portable job submission changes, but without making it the default
kennknowles opened a new pull request #14317:
URL: https://github.com/apache/beam/pull/14317
This should make it easy to test whether making it the default is safe, by rolling forward all the sickbay lists and fixes that were necessary to get it green.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806325641
Ah! Comparing the log view that pops open when I hit the button, I see the "Go To Logs Explorer" button is context-sensitive so I was clicking it in a context that filtered for only job messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806223228
My prefernce is to make steady progress that will not need to be rolled back. Multiple commits in the same PR are fine as long as whoever is rolling back knows how to bisect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806317105
I pulled job ids from https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2_PR/18/ and quickly browsed the logs.
If this is not an environmental problem, then there is probably something obviously wrong with the change somehow. Maybe it breaks non-portable job submission?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles edited a comment on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles edited a comment on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806225989
Well, nvm of course this still does not turn on portable job submission so we expect everything to be unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #14317:
URL: https://github.com/apache/beam/pull/14317#discussion_r601011497
##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -1177,7 +1232,7 @@ public DataflowPipelineJob run(Pipeline pipeline) {
jobResult.getId(),
options,
jobSpecification.getStepNames(),
- pipelineProto);
+ dataflowV1PipelineProto);
Review comment:
This one is supposed to be a V2 proto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806331908
Run Java Examples on Dataflow Runner V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-807231266
The alternative is to adjust this PR to stage the V1 proto, and the follow-up that turns it on will both set the flag and change with proto is uploaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806224779
If Pubsub works and examples work, I am confident in our understanding of ValidatesRunner results. No need to waste resources running that eagerly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806053023
Hi Kenn,
Any reason that we want to separate PR14297 and this one? It would be easy to rollback or check the history if all changes are in the same PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806224506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806222284
Only because this seemed unhealthy that that change seemed the minimum to make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806244501
run java postcommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-810480571
Merged #14339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806244358
Merging in changes to apply portable replacements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806316604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806318182
Actualy no.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-805313168
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806321611
Checking job: 2021-03-24_17_34_31-11200910305787742169
It seems like certain step is missing environment id: https://screenshot.googleplex.com/6VDZsbds8DWfgS2.png
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806244544
Run Java Examples on Dataflow Runner V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-807229666
Oh I think I figured it out and it is a bit obvious: the staged pipeline is portable so the steps in the v1beta3 graph do not exist. So it just makes sense to roll it all forward together. I opened #14339 and will test it for the regressions that happened last time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806225989
Well, nvm of course this still should be turning on portable job submission so we expect everything to be unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806317262
If course, it would break Pubsub.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806224318
More specifically
- Ideally the sickbay change is kept as that is the true sickbay for non-deprecated runner V2 paths.
- The portable pipeline being uploaded today is somewhat wrong for runner v2. Making that right is somewhat separable from the remaining changes.
But if this PR can go in and be green that is even better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-806334601
Run Java Examples on Dataflow Runner V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles closed pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles closed pull request #14317:
URL: https://github.com/apache/beam/pull/14317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #14317: Roll forward Dataflow V2 portable job submission changes from #14074 but without making it the default
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #14317:
URL: https://github.com/apache/beam/pull/14317#issuecomment-805313265
Run Java Examples on Dataflow Runner V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org