You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Brian Hulette <bh...@google.com> on 2020/01/13 22:25:10 UTC

[PROPOSAL] Leveraging SQL TableProviders for Cross-Language IOs

Hi everyone,
I have a proposal that I think can unify two problem sets:
  1) adding more IOs for Beam SQL, and
  2) making more (Row-based) Java IOs available in Python as cross-language
transforms

The basic idea is to create a single cross-language transform that exposes
all Beam SQL IOs via the TableProvider interface. A design document is
available here: https://s.apache.org/xlang-table-provider

Please take a look and let me know what you think. Thanks!
Brian

[PROPOSAL] Abstractions for Schema-Aware IOs (Was: Leveraging SQL TableProviders for Cross-Language IOs)

Posted by Brian Hulette <bh...@google.com>.

Hi everyone,
You may remember the proposal [1] I sent out early this year aiming to
unify Beam SQL Tables and Row-oriented cross-language IOs. I received a lot
of feedback that it was a good idea, but the specific implementation was a
little odd since it relied on existing Beam SQL concepts.

I've re-written the proposal with a different approach: adding a new set of
interfaces to the core Beam Java SDK [2]. These interfaces can be used for
both Beam SQL Tables, as well as for cross-language IOs. Please take a look
and let me know what you think!

Full disclosure: there is a Google intern starting this week that I would
like to have help out with this work once they are ramped up.

Thanks,
Brian

[1] https://s.apache.org/xlang-table-provider
[2] https://s.apache.org/beam-schema-io

On Mon, Mar 30, 2020 at 3:38 PM Robert Bradshaw <ro...@google.com> wrote:

> A belated but very enthusiastic +1 to this proposal. Added some
> comments to the doc.
>
> On Thu, Jan 16, 2020 at 9:05 AM Kenneth Knowles <ke...@apache.org> wrote:
> >
> > Nice! This is quite clever.
> >
> > Kenn
> >
> > On Mon, Jan 13, 2020 at 5:08 PM Chamikara Jayalath <ch...@google.com>
> wrote:
> >>
> >> Thanks Brian. Added some comments.
> >>
> >> On Mon, Jan 13, 2020 at 2:25 PM Brian Hulette <bh...@google.com>
> wrote:
> >>>
> >>> Hi everyone,
> >>> I have a proposal that I think can unify two problem sets:
> >>>   1) adding more IOs for Beam SQL, and
> >>>   2) making more (Row-based) Java IOs available in Python as
> cross-language transforms
> >>>
> >>> The basic idea is to create a single cross-language transform that
> exposes all Beam SQL IOs via the TableProvider interface. A design document
> is available here: https://s.apache.org/xlang-table-provider
> >>>
> >>> Please take a look and let me know what you think. Thanks!
> >>> Brian
>

Re: [PROPOSAL] Leveraging SQL TableProviders for Cross-Language IOs

Posted by Robert Bradshaw <ro...@google.com>.

A belated but very enthusiastic +1 to this proposal. Added some
comments to the doc.

On Thu, Jan 16, 2020 at 9:05 AM Kenneth Knowles <ke...@apache.org> wrote:
>
> Nice! This is quite clever.
>
> Kenn
>
> On Mon, Jan 13, 2020 at 5:08 PM Chamikara Jayalath <ch...@google.com> wrote:
>>
>> Thanks Brian. Added some comments.
>>
>> On Mon, Jan 13, 2020 at 2:25 PM Brian Hulette <bh...@google.com> wrote:
>>>
>>> Hi everyone,
>>> I have a proposal that I think can unify two problem sets:
>>>   1) adding more IOs for Beam SQL, and
>>>   2) making more (Row-based) Java IOs available in Python as cross-language transforms
>>>
>>> The basic idea is to create a single cross-language transform that exposes all Beam SQL IOs via the TableProvider interface. A design document is available here: https://s.apache.org/xlang-table-provider
>>>
>>> Please take a look and let me know what you think. Thanks!
>>> Brian

Re: [PROPOSAL] Leveraging SQL TableProviders for Cross-Language IOs

Posted by Kenneth Knowles <ke...@apache.org>.

Nice! This is quite clever.

Kenn

On Mon, Jan 13, 2020 at 5:08 PM Chamikara Jayalath <ch...@google.com>
wrote:

> Thanks Brian. Added some comments.
>
> On Mon, Jan 13, 2020 at 2:25 PM Brian Hulette <bh...@google.com> wrote:
>
>> Hi everyone,
>> I have a proposal that I think can unify two problem sets:
>>   1) adding more IOs for Beam SQL, and
>>   2) making more (Row-based) Java IOs available in Python as
>> cross-language transforms
>>
>> The basic idea is to create a single cross-language transform that
>> exposes all Beam SQL IOs via the TableProvider interface. A design document
>> is available here: https://s.apache.org/xlang-table-provider
>>
>> Please take a look and let me know what you think. Thanks!
>> Brian
>>
>

Re: [PROPOSAL] Leveraging SQL TableProviders for Cross-Language IOs

Posted by Chamikara Jayalath <ch...@google.com>.

Thanks Brian. Added some comments.

On Mon, Jan 13, 2020 at 2:25 PM Brian Hulette <bh...@google.com> wrote:

> Hi everyone,
> I have a proposal that I think can unify two problem sets:
>   1) adding more IOs for Beam SQL, and
>   2) making more (Row-based) Java IOs available in Python as
> cross-language transforms
>
> The basic idea is to create a single cross-language transform that exposes
> all Beam SQL IOs via the TableProvider interface. A design document is
> available here: https://s.apache.org/xlang-table-provider
>
> Please take a look and let me know what you think. Thanks!
> Brian
>