You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/13 00:32:13 UTC
[GitHub] [beam] boyuanzz opened a new pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
boyuanzz opened a new pull request #12561:
URL: https://github.com/apache/beam/pull/12561
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz merged pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
boyuanzz merged pull request #12561:
URL: https://github.com/apache/beam/pull/12561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673585531
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673188099
Apparently we were not setting is early enough and Create transform ended up picking up pubsub instead of inmemory source resulting in invalid timstamps in the UW pipeline. @boyuanzz probably we should document the issue here or in the JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673538098
We already publish that use_runner_v2 enables streaming engine: https://screenshot.googleplex.com/dekvY5yf6eN
Pipeline differences come from here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L760
So seems like this will not change user expectations ? Note that this issue affects all SDFs including x-lang Kafka.
cc: @robertwb @lukecwik
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ananvay commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
ananvay commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673186087
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-674184094
PythonDocs PreCommit is currently disabled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673182421
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
youngoli commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673771916
cc: @youngoli
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673181439
r: @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673854629
Run PythonDocs PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj edited a comment on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
chamikaramj edited a comment on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673538098
We already publish that use_runner_v2 enables streaming engine: https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-runner-v2
Pipeline differences come from here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L760
So seems like this will not change user expectations ? Note that this issue affects all SDFs including x-lang Kafka.
cc: @robertwb @lukecwik
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] angoenka commented on pull request #12561: [BEAM-10698] Enable dataflow streaming engine when running with runner_v2 and streaming.
Posted by GitBox <gi...@apache.org>.
angoenka commented on pull request #12561:
URL: https://github.com/apache/beam/pull/12561#issuecomment-673350523
I see, My understanding is that streaming appliance and streaming engine should not have any pipeline graph difference.
The idea was to to make streaming engine vs streaming appliance choice at the service so that we can control it.
Secondly, as streaming engine has a billing implication, we don't want to change the default behavior from streaming appliance to streaming engine without announcement.
Was their a specific test which was failing. In that case, we should set the enable_streaming_engine flag in the test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org