You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Dennis Yung <de...@gmail.com> on 2020/11/24 02:26:14 UTC
Best way to resolve conflict between PipelineOption classes
Hi all,
I discovered that the io module aws2 (sdk.io.aws2) has a PipelineOption
class AwsOptions that is incompatible with the DataflowPipelineOptions
class, from the Dataflow runner module. (I have created a JIRA issue
BEAM-11331 <https://issues.apache.org/jira/projects/BEAM/issues/BEAM-11331>
).
I'd like to work on this issue, but I am not sure the proper way to address
this.
One way would be to rename the conflicting property. Specifically, I can
rename the property "region" to "awsRegion", which is the same with the
ver1 aws module (sdk.io.aws). However, this may break existing pipelines
that is using the "region field"
Is there a better way? Thanks!
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
不含病毒。www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Best way to resolve conflict between PipelineOption classes
Posted by Kenneth Knowles <ke...@apache.org>.
Yes I think this renaming is OK. If you make awsRegion required, then
existing pipelines will fail because it is missing. That is a feature - you
can make the error message explain the change.
In a perfect world we would also fix the mistake in the Dataflow options
and have it be `gcpRegion` (or `dataflowRegion`). That one is
not @Experimental and widely used, though, so we need to start with an
alias and deprecation and resolve it later.
Kenn
On Mon, Nov 23, 2020 at 8:33 PM Kyle Weaver <kc...@google.com> wrote:
> > One way would be to rename the conflicting property. Specifically, I can
> rename the property "region" to "awsRegion", which is the same with the
> ver1 aws module (sdk.io.aws). However, this may break existing pipelines
> that is using the "region field"
>
> Since aws2 is still marked as @Experimental, I think renaming would be
> fine. Though you should mention it in the release notes [1] and get a
> review from one of the maintainers of the aws2 module (which I am not).
>
> [1] https://github.com/apache/beam/blob/master/CHANGES.md#breaking-changes
>
> On Mon, Nov 23, 2020 at 6:26 PM Dennis Yung <de...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I discovered that the io module aws2 (sdk.io.aws2) has a PipelineOption
>> class AwsOptions that is incompatible with the DataflowPipelineOptions
>> class, from the Dataflow runner module. (I have created a JIRA issue
>> BEAM-11331
>> <https://issues.apache.org/jira/projects/BEAM/issues/BEAM-11331>).
>>
>> I'd like to work on this issue, but I am not sure the proper way to
>> address this.
>> One way would be to rename the conflicting property. Specifically, I can
>> rename the property "region" to "awsRegion", which is the same with the
>> ver1 aws module (sdk.io.aws). However, this may break existing pipelines
>> that is using the "region field"
>>
>> Is there a better way? Thanks!
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>> 不含病毒。www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>> <#m_-3065701516022339487_m_6546976657711853841_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>
Re: Best way to resolve conflict between PipelineOption classes
Posted by Kyle Weaver <kc...@google.com>.
> One way would be to rename the conflicting property. Specifically, I can
rename the property "region" to "awsRegion", which is the same with the
ver1 aws module (sdk.io.aws). However, this may break existing pipelines
that is using the "region field"
Since aws2 is still marked as @Experimental, I think renaming would be
fine. Though you should mention it in the release notes [1] and get a
review from one of the maintainers of the aws2 module (which I am not).
[1] https://github.com/apache/beam/blob/master/CHANGES.md#breaking-changes
On Mon, Nov 23, 2020 at 6:26 PM Dennis Yung <de...@gmail.com> wrote:
> Hi all,
>
> I discovered that the io module aws2 (sdk.io.aws2) has a PipelineOption
> class AwsOptions that is incompatible with the DataflowPipelineOptions
> class, from the Dataflow runner module. (I have created a JIRA issue
> BEAM-11331
> <https://issues.apache.org/jira/projects/BEAM/issues/BEAM-11331>).
>
> I'd like to work on this issue, but I am not sure the proper way to
> address this.
> One way would be to rename the conflicting property. Specifically, I can
> rename the property "region" to "awsRegion", which is the same with the
> ver1 aws module (sdk.io.aws). However, this may break existing pipelines
> that is using the "region field"
>
> Is there a better way? Thanks!
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> 不含病毒。www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_6546976657711853841_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>