You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Chamikara Jayalath <ch...@google.com> on 2021/09/14 22:30:39 UTC
URN convention for cross-language transforms
Hi All,
As cross-language transforms become widely used I think there's a greater
chance of two or more cross-language transforms defined by different
transform authors ending up using the same URN. This will be
problematic for anyone who wishes to use cross-language transforms with
conflicting URNs in the same expansion service. To prevent this, I propose
adapting a recommended URN convention for cross-language transforms. I
defined the following short doc that proposes such a convention.
https://docs.google.com/document/d/1JOHPBNv6x6ziMdwr_96EPSP-Bx7C4IrLU1j267MSCts/edit?usp=sharing
Please take a look and let me know what you think.
Thanks,
Cham
Re: URN convention for cross-language transforms
Posted by Chamikara Jayalath <ch...@google.com>.
https://github.com/apache/beam/pull/15748 was submitted.
As a result of this, Java cross-language transforms released with Beam will
start using new URNs starting Beam 2.35.0.
This should not be an issue for most users since corresponding Python
cross-language wrappers automatically download the correct jar for that
particular Beam version.
But, theoretically, it can be an issue for someone using Java transforms
without using the corresponding Python wrappers. They'll have to change
code to use the new URN when upgrading Beam. So I'll mark this as a
breaking change for Beam 2.35.0.
Also, +user <us...@beam.apache.org> in case some existing users think they
might run into issues due to this.
Thanks,
Cham
On Mon, Oct 18, 2021 at 6:35 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Hi Folks,
>
> There were no additional comments in the doc. So I think we can adopt it
> and use Option 2 for cross-language transform URNs.
> I sent https://github.com/apache/beam/pull/15748 to update existing URNs
> in the Beam repo.
>
> Thanks,
> Cham
>
> On Mon, Sep 20, 2021 at 5:15 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Hi Folks,
>>
>> Thanks for the comments in the doc so far.
>>
>> In the document I've defined two possible options for the URN convention.
>>
>> * Option 1: URN convention exactly identifies the SDK language where the
>> transform is implemented and the transform implementation
>> * Option 2: URN convention abstracts out the SDK language and the
>> transform implementation
>>
>> Option 2 has the benefit of preventing most of the conflicts in practice
>> while not leaking implementation details of the transforms hence we are
>> currently leaning towards that.
>>
>> I'll keep the document open for a few more days for additional comments.
>> Once the document is finalized I hope to update the Beam documentation [1]
>> to mention the finalized convention and update existing cross-language
>> transform URNs in the Beam repo to match the convention.
>>
>> Thanks,
>> Cham
>>
>> [1]
>> http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines
>> <http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines>
>>
>>
>> On Tue, Sep 14, 2021 at 3:30 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> As cross-language transforms become widely used I think there's a
>>> greater chance of two or more cross-language transforms defined by
>>> different transform authors ending up using the same URN. This will be
>>> problematic for anyone who wishes to use cross-language transforms with
>>> conflicting URNs in the same expansion service. To prevent this, I propose
>>> adapting a recommended URN convention for cross-language transforms. I
>>> defined the following short doc that proposes such a convention.
>>>
>>>
>>> https://docs.google.com/document/d/1JOHPBNv6x6ziMdwr_96EPSP-Bx7C4IrLU1j267MSCts/edit?usp=sharing
>>>
>>> Please take a look and let me know what you think.
>>>
>>> Thanks,
>>> Cham
>>>
>>
Re: URN convention for cross-language transforms
Posted by Chamikara Jayalath <ch...@google.com>.
https://github.com/apache/beam/pull/15748 was submitted.
As a result of this, Java cross-language transforms released with Beam will
start using new URNs starting Beam 2.35.0.
This should not be an issue for most users since corresponding Python
cross-language wrappers automatically download the correct jar for that
particular Beam version.
But, theoretically, it can be an issue for someone using Java transforms
without using the corresponding Python wrappers. They'll have to change
code to use the new URN when upgrading Beam. So I'll mark this as a
breaking change for Beam 2.35.0.
Also, +user <us...@beam.apache.org> in case some existing users think they
might run into issues due to this.
Thanks,
Cham
On Mon, Oct 18, 2021 at 6:35 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Hi Folks,
>
> There were no additional comments in the doc. So I think we can adopt it
> and use Option 2 for cross-language transform URNs.
> I sent https://github.com/apache/beam/pull/15748 to update existing URNs
> in the Beam repo.
>
> Thanks,
> Cham
>
> On Mon, Sep 20, 2021 at 5:15 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Hi Folks,
>>
>> Thanks for the comments in the doc so far.
>>
>> In the document I've defined two possible options for the URN convention.
>>
>> * Option 1: URN convention exactly identifies the SDK language where the
>> transform is implemented and the transform implementation
>> * Option 2: URN convention abstracts out the SDK language and the
>> transform implementation
>>
>> Option 2 has the benefit of preventing most of the conflicts in practice
>> while not leaking implementation details of the transforms hence we are
>> currently leaning towards that.
>>
>> I'll keep the document open for a few more days for additional comments.
>> Once the document is finalized I hope to update the Beam documentation [1]
>> to mention the finalized convention and update existing cross-language
>> transform URNs in the Beam repo to match the convention.
>>
>> Thanks,
>> Cham
>>
>> [1]
>> http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines
>> <http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines>
>>
>>
>> On Tue, Sep 14, 2021 at 3:30 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> As cross-language transforms become widely used I think there's a
>>> greater chance of two or more cross-language transforms defined by
>>> different transform authors ending up using the same URN. This will be
>>> problematic for anyone who wishes to use cross-language transforms with
>>> conflicting URNs in the same expansion service. To prevent this, I propose
>>> adapting a recommended URN convention for cross-language transforms. I
>>> defined the following short doc that proposes such a convention.
>>>
>>>
>>> https://docs.google.com/document/d/1JOHPBNv6x6ziMdwr_96EPSP-Bx7C4IrLU1j267MSCts/edit?usp=sharing
>>>
>>> Please take a look and let me know what you think.
>>>
>>> Thanks,
>>> Cham
>>>
>>
Re: URN convention for cross-language transforms
Posted by Chamikara Jayalath <ch...@google.com>.
Hi Folks,
There were no additional comments in the doc. So I think we can adopt it
and use Option 2 for cross-language transform URNs.
I sent https://github.com/apache/beam/pull/15748 to update existing URNs in
the Beam repo.
Thanks,
Cham
On Mon, Sep 20, 2021 at 5:15 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Hi Folks,
>
> Thanks for the comments in the doc so far.
>
> In the document I've defined two possible options for the URN convention.
>
> * Option 1: URN convention exactly identifies the SDK language where the
> transform is implemented and the transform implementation
> * Option 2: URN convention abstracts out the SDK language and the
> transform implementation
>
> Option 2 has the benefit of preventing most of the conflicts in practice
> while not leaking implementation details of the transforms hence we are
> currently leaning towards that.
>
> I'll keep the document open for a few more days for additional comments.
> Once the document is finalized I hope to update the Beam documentation [1]
> to mention the finalized convention and update existing cross-language
> transform URNs in the Beam repo to match the convention.
>
> Thanks,
> Cham
>
> [1]
> http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines
> <http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines>
>
>
> On Tue, Sep 14, 2021 at 3:30 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Hi All,
>>
>> As cross-language transforms become widely used I think there's a greater
>> chance of two or more cross-language transforms defined by different
>> transform authors ending up using the same URN. This will be
>> problematic for anyone who wishes to use cross-language transforms with
>> conflicting URNs in the same expansion service. To prevent this, I propose
>> adapting a recommended URN convention for cross-language transforms. I
>> defined the following short doc that proposes such a convention.
>>
>>
>> https://docs.google.com/document/d/1JOHPBNv6x6ziMdwr_96EPSP-Bx7C4IrLU1j267MSCts/edit?usp=sharing
>>
>> Please take a look and let me know what you think.
>>
>> Thanks,
>> Cham
>>
>
Re: URN convention for cross-language transforms
Posted by Chamikara Jayalath <ch...@google.com>.
Hi Folks,
Thanks for the comments in the doc so far.
In the document I've defined two possible options for the URN convention.
* Option 1: URN convention exactly identifies the SDK language where the
transform is implemented and the transform implementation
* Option 2: URN convention abstracts out the SDK language and the transform
implementation
Option 2 has the benefit of preventing most of the conflicts in practice
while not leaking implementation details of the transforms hence we are
currently leaning towards that.
I'll keep the document open for a few more days for additional comments.
Once the document is finalized I hope to update the Beam documentation [1]
to mention the finalized convention and update existing cross-language
transform URNs in the Beam repo to match the convention.
Thanks,
Cham
[1]
http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines
<http://beam.apache.org/documentation/programming-guide/#multi-language-pipelines>
On Tue, Sep 14, 2021 at 3:30 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Hi All,
>
> As cross-language transforms become widely used I think there's a greater
> chance of two or more cross-language transforms defined by different
> transform authors ending up using the same URN. This will be
> problematic for anyone who wishes to use cross-language transforms with
> conflicting URNs in the same expansion service. To prevent this, I propose
> adapting a recommended URN convention for cross-language transforms. I
> defined the following short doc that proposes such a convention.
>
>
> https://docs.google.com/document/d/1JOHPBNv6x6ziMdwr_96EPSP-Bx7C4IrLU1j267MSCts/edit?usp=sharing
>
> Please take a look and let me know what you think.
>
> Thanks,
> Cham
>