You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/12/03 18:32:14 UTC
[GitHub] [beam] tvalentyn opened a new pull request #13475: [DO NOT MERGE] Do not add unnecessary experiment use_multiple_sdk_containers.
tvalentyn opened a new pull request #13475:
URL: https://github.com/apache/beam/pull/13475
When Dataflow runs portable python pipelines, such as all Python streaming pipelines and all pipelines on Runner V2, Dataflow service currently starts as many SDK containers as there are CPU cores on the VMs.
This behavior was originally initiated by the SDK setting an experiment use_multiple_sdk_containers, and it was possible to disable this behavior by supplying `--experiment=no_use_multiple_sdk_containers`.
Dataflow service no longer needs SDK to supply these experiments, so we clean up this logic.
Dataflow service may still recognize `--experiment=no_use_multiple_sdk_containers` for some time but it is NOT RECOMMENDED to use this knob: in the future Dataflow may have better algorithms for deciding how many SDK containers to start, and specifying this knob may interfere with these algorithms.
Users can control the number of cores on the VMs by setting an appropriate `--machine_type`. Note that there are custom machine types, where users can select number of cores and number of memory GBs, such as `--machine_type=custom-1-13312-ext` which will have 1 core and 13GB memory.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #13475: [DO NOT MERGE] Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-745455276
LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #13475: Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-763231069
@psobot Thanks for your feedback. It's a reasonable use-case scenario. `--experiment=no_use_multiple_sdk_containers` should be working again now on 2.25.0+ SDKs. We'll look into adding proper documentation for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #13475: Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-750719561
Hi, @psobot, thanks for reaching out! I would be curious to learn more about your observations of performance difference between
- setting a `--experiment=no_use_multiple_sdk_containers` and using a multi-core VM
vs
- using a singlecore VM
on the use-cases that you mention.
There is no exact alternative to `--experiment=no_use_multiple_sdk_containers` right now, but this was never documented as a supported customization. We don't have tests targeting this functionality and noticed that it stopped work as of 2.25.0 on Dataflow Runner V2, due to a combination of SDK + Dataflow service changes, but may start working again after another series of service changes rolls out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] psobot commented on pull request #13475: Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
psobot commented on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-748317475
Hi @tvalentyn and @chamikaramj!
> Dataflow service may still recognize --experiment=no_use_multiple_sdk_containers for some time but it is NOT RECOMMENDED to use this knob: in the future Dataflow may have better algorithms for deciding how many SDK containers to start, and specifying this knob may interfere with these algorithms.
>
> Users can control the number of cores on the VMs by setting an appropriate --machine_type. Note that there are custom machine types, where users can select number of cores and number of memory GBs, such as --machine_type=custom-1-13312-ext which will have 1 core and 13GB memory.
While this is true (that it is possible to control the number of cores with `machine-type`), there are many situations in which it's desirable for a job to use multiple cores while processing one element for performance reasons. (e.g.: running ML inference within Dataflow.) Is there a proposed alternative for workloads that benefit from multi-core parallelism without multiple SDK workers?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #13475: [DO NOT MERGE] Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-745450306
R: @chamikaramj @angoenka - with new default XLANG codepath, can we safely remove this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #13475: Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-750719561
Hi, @psobot, thanks for reaching out! I would be curious to learn more about your observations of performance difference between
- setting a `--experiment=no_use_multiple_sdk_containers` and using a multi-core VM
vs
- using a singlecore VM
on the use-cases that you mention.
There is no exact alternative to `--experiment=no_use_multiple_sdk_containers` right now, but this was never documented as a supported customization. We don't have tests targeting this functionality and noticed that it stopped working on 2.25.0 and newer SDKs on Dataflow Runner V2, due to a combination of SDK + Dataflow service changes, but may start working again after another series of service changes rolls out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #13475: Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-745483283
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] psobot commented on pull request #13475: Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
psobot commented on pull request #13475:
URL: https://github.com/apache/beam/pull/13475#issuecomment-754235246
Happy to include more details! One example use case: running TensorFlow inference in a Beam job, where TF can use multiple cores simultaneously when processing one element. Using a single-core VM restricts us to, well, one core - which means our processing happens slowly. Using a multi-core VM with only one SDK container allows us to use all of those cores from a single SDK container, by virtue of the fact that TensorFlow breaks out of the GIL and uses all available cores when processing an element.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn merged pull request #13475: Do not add unnecessary experiment use_multiple_sdk_containers.
Posted by GitBox <gi...@apache.org>.
tvalentyn merged pull request #13475:
URL: https://github.com/apache/beam/pull/13475
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org