Re: Problems with conflicting option names when running tests on 2.28.0

[Ahmet Altay <al...@google.com>]

/cc +Valentyn Tymofieiev <va...@google.com> - might have an idea.

On Sat, Feb 27, 2021 at 2:42 AM adam moore <ne...@gmail.com> wrote:

> Hi there beam-team.
>
> I've recently upgraded to 2.28.0 but I'm having problems running tests
> which were previously working on 2.27.0.
>
> Here's a description of the situation, and I'm wondering if we might be
> able to suggest a way to move forward.
>
> * The project is using python
> * The project contains multiple pipelines, which run independent of each
> other, but share some transforms.
> * Each pipeline has its own set of options, which subclass
> `PipelineOptions`, and use the .view_as pattern as described in the
> PipelineOptions docs [1]
> * Recently, the test runner changed to include a call to
> options.get_all_options [2]
> * The pipelines individually contain conflicting option names (eg.
> log_level)
> * When running tests, this now throws an error:
>
>     E       argparse.ArgumentError: argument --log_level: conflicting
> option string: --log_level
>
> When running the individual pipelines, only one subclass is ever imported,
> so the conflicting options problem doesn't present itself. I'm wondering if
> this is an unexpected side effect of the fix for BEAM-11736 [3][4], or if I
> should maybe:
>
> * Create a "CommonOptions" class which is imported independently by both
> pipelines
> * Use independent names eg. --pipeline1_log_level, --pipeline2_log_level
>
> Looking forward to people's input!
>
> Regards,
>
> Adam
>
>
> [1]
> https://beam.apache.org/releases/pydoc/2.17.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions
> [2]
> https://github.com/apache/beam/blame/master/sdks/python/apache_beam/runners/direct/direct_runner.py#L124
> [3] https://issues.apache.org/jira/browse/BEAM-11736
> [4] https://github.com/apache/beam/pull/13723
>

Re: Problems with conflicting option names when running tests on 2.28.0

[Valentyn Tymofieiev <va...@google.com>]

Hi Adam,

The code changes you mention are likely the causing the change in behavior.
You can confirm this by using `git bisect` if you want, but looking in the
PR I am seeing that a few changes had to be made to our test code as well.

Could you try to add something like this in your tests and see if it helps?

https://github.com/dandy10/beam/blob/f3b15fe4ce78ee915b8ca24a27290bf9efb045eb/sdks/python/apache_beam/examples/snippets/snippets_test.py#L521-L522

Thanks!

On Mon, Mar 8, 2021 at 5:02 PM Ahmet Altay <al...@google.com> wrote:

> /cc +Valentyn Tymofieiev <va...@google.com> - might have an idea.
>
> On Sat, Feb 27, 2021 at 2:42 AM adam moore <ne...@gmail.com> wrote:
>
>> Hi there beam-team.
>>
>> I've recently upgraded to 2.28.0 but I'm having problems running tests
>> which were previously working on 2.27.0.
>>
>> Here's a description of the situation, and I'm wondering if we might be
>> able to suggest a way to move forward.
>>
>> * The project is using python
>> * The project contains multiple pipelines, which run independent of each
>> other, but share some transforms.
>> * Each pipeline has its own set of options, which subclass
>> `PipelineOptions`, and use the .view_as pattern as described in the
>> PipelineOptions docs [1]
>> * Recently, the test runner changed to include a call to
>> options.get_all_options [2]
>> * The pipelines individually contain conflicting option names (eg.
>> log_level)
>> * When running tests, this now throws an error:
>>
>>     E       argparse.ArgumentError: argument --log_level: conflicting
>> option string: --log_level
>>
>> When running the individual pipelines, only one subclass is ever
>> imported, so the conflicting options problem doesn't present itself. I'm
>> wondering if this is an unexpected side effect of the fix for BEAM-11736
>> [3][4], or if I should maybe:
>>
>> * Create a "CommonOptions" class which is imported independently by both
>> pipelines
>> * Use independent names eg. --pipeline1_log_level, --pipeline2_log_level
>>
>> Looking forward to people's input!
>>
>> Regards,
>>
>> Adam
>>
>>
>> [1]
>> https://beam.apache.org/releases/pydoc/2.17.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions
>> [2]
>> https://github.com/apache/beam/blame/master/sdks/python/apache_beam/runners/direct/direct_runner.py#L124
>> [3] https://issues.apache.org/jira/browse/BEAM-11736
>> [4] https://github.com/apache/beam/pull/13723
>>
>