You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/28 17:26:11 UTC
[GitHub] [beam] steveniemitz opened a new pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
steveniemitz opened a new pull request #11849:
URL: https://github.com/apache/beam/pull/11849
#11710 added the plumbing to use `workerCacheMb` parameter to size the streaming dataflow worker state cache. However, the parameter itself is inaccessible from user jobs because it's in `DataflowWorkerHarnessOptions`, which is only exposed in the worker itself.
Trying to set it produces:
```
$ java -jar myjob.jar ... --runner=DataflowRunner --workerCacheMb=400
Exception in thread "main" java.lang.IllegalArgumentException: Class interface ...MyOptions missing a property named 'workerCacheMb'.
at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1625)
```
This simply moves it to a user-accessible location.
R: @omarismail94 @pabloem
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11849:
URL: https://github.com/apache/beam/pull/11849
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11849:
URL: https://github.com/apache/beam/pull/11849#issuecomment-636084490
Just a question - I am a little confused. How come the `DataflowPipelineDebugOptions` class is visible, but `DataflowWorkerHarnessOptions` isn't? If you inherit from it, shouldn't you be able to use it?
It may be that we generally encourage users to rely on DataflowPipelineDebugOptions for their Dataflow pipeline needs? If so, it makes sense to move the option... I'm just confused about what is affecting the visibility of the classes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11849:
URL: https://github.com/apache/beam/pull/11849#issuecomment-636086329
gotcha. Makes sense. LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #11849:
URL: https://github.com/apache/beam/pull/11849#issuecomment-635696194
> gahh so sorry that I missed this. I guess you did have to end up contributing this : )
heh no problem, teamwork! :highfive:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #11849:
URL: https://github.com/apache/beam/pull/11849#issuecomment-635697264
=/ looks like the dataflow precommit succeeded but the API call to update it here failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz edited a comment on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on pull request #11849:
URL: https://github.com/apache/beam/pull/11849#issuecomment-636086069
If you look at [DataflowPipelineOptions](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java) it doesn't include `DataflowWorkerHarnessOptions`. In fact, `DataflowWorkerHarnessOptions` implement `DataflowPipelineOptions` instead. The harness options are used in the harness itself, while the `DataflowPipelineOptions` are what are validated against in the dataflow runner.
edit: Also to clarify, user's don't (in general) directly implement `DataflowPipelineOptions`, they're included implicitly when the dataflow runner is used. One could specifically implement `DataflowWorkerHarnessOptions` (or even just define the property in any options they have, we actually used to just do that) if they wanted to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11849:
URL: https://github.com/apache/beam/pull/11849#issuecomment-635663865
gahh so sorry that I missed this. I guess you did have to end up contributing this : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #11849: [BEAM-9964] Move workerCacheMb to a user-visible place
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on pull request #11849:
URL: https://github.com/apache/beam/pull/11849#issuecomment-636086069
If you look at [DataflowPipelineOptions](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java) it doesn't include `DataflowWorkerHarnessOptions`. In fact, `DataflowWorkerHarnessOptions` implement `DataflowPipelineOptions` instead. The harness options are used in the harness itself, while the `DataflowPipelineOptions` are what are validated against in the dataflow runner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org