[jira] [Commented] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written

["Suzen Fylke (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191064#comment-17191064 ] 

Suzen Fylke commented on BEAM-9093:
-----------------------------------

I'm a bit confused about how this is supposed to work with the DataflowRunner, since {{no_use_public_ips}} isn't listed in the docs: [https://cloud.google.com/dataflow/docs/guides/specifying-exec-params#setting-other-cloud-dataflow-pipeline-options]

I'm using the DataflowRunner with TFX ([https://github.com/tensorflow/tfx]) and have to pass the pipeline options as a list of args. Currently, directly passing a PipelineOptions object is not supported, and I've filed a ticket for that in the TFX repo ([https://github.com/tensorflow/tfx/issues/2332]). 

What's the recommended way to convert \{'use_public_ips': False} to a flag?
 * "--use_public_ips False" causes
 WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['False']"
 * "--use_public_ips=False" causes "argparse.ArgumentError: argument --use_public_ips: ignored explicit argument 'False'"
 * Setting \{'no_use_public_ips': True} in my PipelineOptions causes "WARNING:apache_beam.options.pipeline_options:Discarding invalid overrides: \{'no_use_public_ips': True}"

What I'm doing right now is a little hacky:

    pipeline_options = PipelineOptions(**\{'use_public_ips': False, etc.})
     all_options = pipeline_options.get_all_options(drop_default=True)
     dataflow_args = []
     all_options.pop('use_public_ips')
     dataflow_args.append('--no_use_public_ips')

> Pipeline options which with different underlying store variable does not get over written
> -----------------------------------------------------------------------------------------
>
>                 Key: BEAM-9093
>                 URL: https://issues.apache.org/jira/browse/BEAM-9093
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Ankur Goenka
>            Priority: P3
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)