You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/10/03 18:22:00 UTC
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom
options for Runner
[ https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=150880&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150880 ]
ASF GitHub Bot logged work on BEAM-5442:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Oct/18 18:21
Start Date: 03/Oct/18 18:21
Worklog Time Spent: 10m
Work Description: mxm opened a new pull request #6557: [BEAM-5442] Pass unknown SDK pipeline options to Runner
URL: https://github.com/apache/beam/pull/6557
Unknown Python SDK pipelin options were previously ignored. Pipeline options had
to be replicated in the Python SDK to be passed on to the Runner.
Validating known options in the SDK is a good way to report failures back to the
user early. However, it also requires to maintain options in multiple places.
This change passes on unknown options as long as they are in the form of
'--option value' or '--option=value'. It logs whenever that happens. All other
options are discarded, as previously the case. Discarding option is now logged
with a warn. For example:
```
INFO:root:Parsing unknown args: ['-invalid=value', '--valid=value']
WARNING:root:Discarding unparseable args: ['-invalid=value']
```
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 150880)
Time Spent: 3h (was: 2h 50m)
> PortableRunner swallows custom options for Runner
> -------------------------------------------------
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Labels: portability, portability-flink
> Fix For: 2.8.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)