You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Heejong Lee <he...@google.com> on 2022/02/11 23:43:57 UTC
[PROPOSAL] Python External Transform Registry
Hi,
I wanted to share the design doc for Python External Transform Registry.
This could improve Python expansion service usability before we officially
support a multi-language Java pipeline using Python external transforms.
https://s.apache.org/python-external-transform-registry
Thanks!
Re: [PROPOSAL] Python External Transform Registry
Posted by Brian Hulette <bh...@google.com>.
I'm also +1 for adding some kind of discovery API to the expansion
service that could be used in conjunction with John's proposal.
On Mon, Feb 14, 2022 at 12:01 PM Chamikara Jayalath <ch...@google.com>
wrote:
> +1. I think part of John's proposal is defining a standard way for the
> structure of a PTransform so that they can be identified and instantiated
> in a language-neutral way. This is potentially something that can be
> integrated to the expansion service so that such transforms can be
> discovered dynamically (and hopefully something that can be integrated to a
> future expansion services UI :) ).
>
> Thanks,
> Cham
>
> On Mon, Feb 14, 2022 at 11:47 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> I was thinking that if an execution service had a way to vend the
>> transforms that it offers, with their config parameters, this could be
>> used to automatically generate transform wrappers (either ahead of
>> time or on the fly) in a language-agnostic way. This is, however,
>> orthogonal to Heejong's proposal.
>>
>> On Mon, Feb 14, 2022 at 10:47 AM Chamikara Jayalath
>> <ch...@google.com> wrote:
>> >
>> > Yeah, this proposal defines the underlying API for defining
>> cross-language transforms in Python. Hopefully John's work can be extended
>> to generate cross-language wrappers for Java/Go for Python cross-language
>> transforms defined using this API.
>> >
>> > Thanks,
>> > Cham
>> >
>> > On Mon, Feb 14, 2022 at 9:51 AM Brian Hulette <bh...@google.com>
>> wrote:
>> >>
>> >> I'm not sure it is solving the same problem. This is a design for
>> registering Python SDK transforms to expose in an expansion service -
>> similar to the function performed by ExternalTransformRegistrar [1] and
>> friends in the Java SDK. Fundamentally John's proposal is focused on the
>> calling SDK, and this is focused on Python as the expanding SDK.
>> >>
>> >> Brian
>> >>
>> >> [1]
>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
>> >>
>> >> On Mon, Feb 14, 2022 at 8:33 AM John Casey <jo...@google.com>
>> wrote:
>> >>>
>> >>> This looks like it is working towards solving the same problem, which
>> is cross language usability.
>> >>> These approaches are complementary, though we may not need both if
>> either approach simplifies the usability problem enough.
>> >>>
>> >>> John
>> >>>
>> >>> On Mon, Feb 14, 2022 at 9:44 AM Kerry Donny-Clark <ke...@google.com>
>> wrote:
>> >>>>
>> >>>> Is this work complementary to the approach John proposes here:
>> >>>>
>> https://docs.google.com/document/d/1SAe9tG3JeKbyWVlJf-oGROHsu_fHk02X_I1j6kfvClM/edit#heading=h.cfh6hf1hyp3a
>> >>>> Or do they both achieve the same objective?
>> >>>> Kerry
>> >>>>
>> >>>> On Fri, Feb 11, 2022 at 6:44 PM Heejong Lee <he...@google.com>
>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I wanted to share the design doc for Python External Transform
>> Registry. This could improve Python expansion service usability before we
>> officially support a multi-language Java pipeline using Python external
>> transforms.
>> >>>>>
>> >>>>> https://s.apache.org/python-external-transform-registry
>> >>>>>
>> >>>>> Thanks!
>>
>
Re: [PROPOSAL] Python External Transform Registry
Posted by Chamikara Jayalath <ch...@google.com>.
+1. I think part of John's proposal is defining a standard way for the
structure of a PTransform so that they can be identified and instantiated
in a language-neutral way. This is potentially something that can be
integrated to the expansion service so that such transforms can be
discovered dynamically (and hopefully something that can be integrated to a
future expansion services UI :) ).
Thanks,
Cham
On Mon, Feb 14, 2022 at 11:47 AM Robert Bradshaw <ro...@google.com>
wrote:
> I was thinking that if an execution service had a way to vend the
> transforms that it offers, with their config parameters, this could be
> used to automatically generate transform wrappers (either ahead of
> time or on the fly) in a language-agnostic way. This is, however,
> orthogonal to Heejong's proposal.
>
> On Mon, Feb 14, 2022 at 10:47 AM Chamikara Jayalath
> <ch...@google.com> wrote:
> >
> > Yeah, this proposal defines the underlying API for defining
> cross-language transforms in Python. Hopefully John's work can be extended
> to generate cross-language wrappers for Java/Go for Python cross-language
> transforms defined using this API.
> >
> > Thanks,
> > Cham
> >
> > On Mon, Feb 14, 2022 at 9:51 AM Brian Hulette <bh...@google.com>
> wrote:
> >>
> >> I'm not sure it is solving the same problem. This is a design for
> registering Python SDK transforms to expose in an expansion service -
> similar to the function performed by ExternalTransformRegistrar [1] and
> friends in the Java SDK. Fundamentally John's proposal is focused on the
> calling SDK, and this is focused on Python as the expanding SDK.
> >>
> >> Brian
> >>
> >> [1]
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
> >>
> >> On Mon, Feb 14, 2022 at 8:33 AM John Casey <jo...@google.com>
> wrote:
> >>>
> >>> This looks like it is working towards solving the same problem, which
> is cross language usability.
> >>> These approaches are complementary, though we may not need both if
> either approach simplifies the usability problem enough.
> >>>
> >>> John
> >>>
> >>> On Mon, Feb 14, 2022 at 9:44 AM Kerry Donny-Clark <ke...@google.com>
> wrote:
> >>>>
> >>>> Is this work complementary to the approach John proposes here:
> >>>>
> https://docs.google.com/document/d/1SAe9tG3JeKbyWVlJf-oGROHsu_fHk02X_I1j6kfvClM/edit#heading=h.cfh6hf1hyp3a
> >>>> Or do they both achieve the same objective?
> >>>> Kerry
> >>>>
> >>>> On Fri, Feb 11, 2022 at 6:44 PM Heejong Lee <he...@google.com>
> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I wanted to share the design doc for Python External Transform
> Registry. This could improve Python expansion service usability before we
> officially support a multi-language Java pipeline using Python external
> transforms.
> >>>>>
> >>>>> https://s.apache.org/python-external-transform-registry
> >>>>>
> >>>>> Thanks!
>
Re: [PROPOSAL] Python External Transform Registry
Posted by Robert Bradshaw <ro...@google.com>.
I was thinking that if an execution service had a way to vend the
transforms that it offers, with their config parameters, this could be
used to automatically generate transform wrappers (either ahead of
time or on the fly) in a language-agnostic way. This is, however,
orthogonal to Heejong's proposal.
On Mon, Feb 14, 2022 at 10:47 AM Chamikara Jayalath
<ch...@google.com> wrote:
>
> Yeah, this proposal defines the underlying API for defining cross-language transforms in Python. Hopefully John's work can be extended to generate cross-language wrappers for Java/Go for Python cross-language transforms defined using this API.
>
> Thanks,
> Cham
>
> On Mon, Feb 14, 2022 at 9:51 AM Brian Hulette <bh...@google.com> wrote:
>>
>> I'm not sure it is solving the same problem. This is a design for registering Python SDK transforms to expose in an expansion service - similar to the function performed by ExternalTransformRegistrar [1] and friends in the Java SDK. Fundamentally John's proposal is focused on the calling SDK, and this is focused on Python as the expanding SDK.
>>
>> Brian
>>
>> [1] https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
>>
>> On Mon, Feb 14, 2022 at 8:33 AM John Casey <jo...@google.com> wrote:
>>>
>>> This looks like it is working towards solving the same problem, which is cross language usability.
>>> These approaches are complementary, though we may not need both if either approach simplifies the usability problem enough.
>>>
>>> John
>>>
>>> On Mon, Feb 14, 2022 at 9:44 AM Kerry Donny-Clark <ke...@google.com> wrote:
>>>>
>>>> Is this work complementary to the approach John proposes here:
>>>> https://docs.google.com/document/d/1SAe9tG3JeKbyWVlJf-oGROHsu_fHk02X_I1j6kfvClM/edit#heading=h.cfh6hf1hyp3a
>>>> Or do they both achieve the same objective?
>>>> Kerry
>>>>
>>>> On Fri, Feb 11, 2022 at 6:44 PM Heejong Lee <he...@google.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I wanted to share the design doc for Python External Transform Registry. This could improve Python expansion service usability before we officially support a multi-language Java pipeline using Python external transforms.
>>>>>
>>>>> https://s.apache.org/python-external-transform-registry
>>>>>
>>>>> Thanks!
Re: [PROPOSAL] Python External Transform Registry
Posted by Chamikara Jayalath <ch...@google.com>.
Yeah, this proposal defines the underlying API for defining cross-language
transforms in Python. Hopefully John's work can be extended to generate
cross-language wrappers for Java/Go for Python cross-language transforms
defined using this API.
Thanks,
Cham
On Mon, Feb 14, 2022 at 9:51 AM Brian Hulette <bh...@google.com> wrote:
> I'm not sure it is solving the same problem. This is a design for
> registering Python SDK transforms to expose in an expansion service -
> similar to the function performed by ExternalTransformRegistrar [1] and
> friends in the Java SDK. Fundamentally John's proposal is focused on the
> calling SDK, and this is focused on Python as the expanding SDK.
>
> Brian
>
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
>
> On Mon, Feb 14, 2022 at 8:33 AM John Casey <jo...@google.com> wrote:
>
>> This looks like it is working towards solving the same problem, which is
>> cross language usability.
>> These approaches are complementary, though we may not need both if either
>> approach simplifies the usability problem enough.
>>
>> John
>>
>> On Mon, Feb 14, 2022 at 9:44 AM Kerry Donny-Clark <ke...@google.com>
>> wrote:
>>
>>> Is this work complementary to the approach John proposes here:
>>>
>>> https://docs.google.com/document/d/1SAe9tG3JeKbyWVlJf-oGROHsu_fHk02X_I1j6kfvClM/edit#heading=h.cfh6hf1hyp3a
>>> Or do they both achieve the same objective?
>>> Kerry
>>>
>>> On Fri, Feb 11, 2022 at 6:44 PM Heejong Lee <he...@google.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I wanted to share the design doc for Python External Transform
>>>> Registry. This could improve Python expansion service usability before we
>>>> officially support a multi-language Java pipeline using Python external
>>>> transforms.
>>>>
>>>> https://s.apache.org/python-external-transform-registry
>>>>
>>>> Thanks!
>>>>
>>>
Re: [PROPOSAL] Python External Transform Registry
Posted by Brian Hulette <bh...@google.com>.
I'm not sure it is solving the same problem. This is a design for
registering Python SDK transforms to expose in an expansion service -
similar to the function performed by ExternalTransformRegistrar [1] and
friends in the Java SDK. Fundamentally John's proposal is focused on the
calling SDK, and this is focused on Python as the expanding SDK.
Brian
[1]
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
On Mon, Feb 14, 2022 at 8:33 AM John Casey <jo...@google.com> wrote:
> This looks like it is working towards solving the same problem, which is
> cross language usability.
> These approaches are complementary, though we may not need both if either
> approach simplifies the usability problem enough.
>
> John
>
> On Mon, Feb 14, 2022 at 9:44 AM Kerry Donny-Clark <ke...@google.com>
> wrote:
>
>> Is this work complementary to the approach John proposes here:
>>
>> https://docs.google.com/document/d/1SAe9tG3JeKbyWVlJf-oGROHsu_fHk02X_I1j6kfvClM/edit#heading=h.cfh6hf1hyp3a
>> Or do they both achieve the same objective?
>> Kerry
>>
>> On Fri, Feb 11, 2022 at 6:44 PM Heejong Lee <he...@google.com> wrote:
>>
>>> Hi,
>>>
>>> I wanted to share the design doc for Python External Transform Registry.
>>> This could improve Python expansion service usability before we officially
>>> support a multi-language Java pipeline using Python external transforms.
>>>
>>> https://s.apache.org/python-external-transform-registry
>>>
>>> Thanks!
>>>
>>
Re: [PROPOSAL] Python External Transform Registry
Posted by John Casey <jo...@google.com>.
This looks like it is working towards solving the same problem, which is
cross language usability.
These approaches are complementary, though we may not need both if either
approach simplifies the usability problem enough.
John
On Mon, Feb 14, 2022 at 9:44 AM Kerry Donny-Clark <ke...@google.com>
wrote:
> Is this work complementary to the approach John proposes here:
>
> https://docs.google.com/document/d/1SAe9tG3JeKbyWVlJf-oGROHsu_fHk02X_I1j6kfvClM/edit#heading=h.cfh6hf1hyp3a
> Or do they both achieve the same objective?
> Kerry
>
> On Fri, Feb 11, 2022 at 6:44 PM Heejong Lee <he...@google.com> wrote:
>
>> Hi,
>>
>> I wanted to share the design doc for Python External Transform Registry.
>> This could improve Python expansion service usability before we officially
>> support a multi-language Java pipeline using Python external transforms.
>>
>> https://s.apache.org/python-external-transform-registry
>>
>> Thanks!
>>
>
Re: [PROPOSAL] Python External Transform Registry
Posted by Kerry Donny-Clark <ke...@google.com>.
Is this work complementary to the approach John proposes here:
https://docs.google.com/document/d/1SAe9tG3JeKbyWVlJf-oGROHsu_fHk02X_I1j6kfvClM/edit#heading=h.cfh6hf1hyp3a
Or do they both achieve the same objective?
Kerry
On Fri, Feb 11, 2022 at 6:44 PM Heejong Lee <he...@google.com> wrote:
> Hi,
>
> I wanted to share the design doc for Python External Transform Registry.
> This could improve Python expansion service usability before we officially
> support a multi-language Java pipeline using Python external transforms.
>
> https://s.apache.org/python-external-transform-registry
>
> Thanks!
>