You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Eleanore Jin <el...@gmail.com> on 2021/01/26 21:06:46 UTC
Beam support Flink Async I/O operator
Hi community,
Does Beam support Flink Async I/O operator? if so, can you please share the
doc, and if not, is there any workaround to achieve the same in Beam
semantics?
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
Thanks a lot!
Eleanore
Re: Beam support Flink Async I/O operator
Posted by Eleanore Jin <el...@gmail.com>.
ha, yes, sorry I forgot about that
On Wed, Jan 27, 2021 at 4:31 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> I guess it was already discussed a while ago [1] and the conclusion was
> that Flink Async I/O is a specific Flink operator and it can’t be used
> directly from Beam since Beam should provide a unified model and its
> implementation for all supported runners.
>
> Did a proposed workaround, based on GroupIntoBatches [2], work for you?
>
> [1]
> https://lists.apache.org/thread.html/rc1e087a15036c18564d3147c3570c7d80c4963ff7d48cf8eaf180758%40%3Cuser.beam.apache.org%3E
> [2]
> https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/
>
> On 26 Jan 2021, at 22:09, Boyuan Zhang <bo...@google.com> wrote:
>
> +dev <de...@beam.apache.org>
>
> On Tue, Jan 26, 2021 at 1:07 PM Eleanore Jin <el...@gmail.com>
> wrote:
>
>> Hi community,
>>
>> Does Beam support Flink Async I/O operator? if so, can you please share
>> the doc, and if not, is there any workaround to achieve the same in Beam
>> semantics?
>>
>>
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
>>
>> Thanks a lot!
>> Eleanore
>>
>
>
Re: Beam support Flink Async I/O operator
Posted by Eleanore Jin <el...@gmail.com>.
ha, yes, sorry I forgot about that
On Wed, Jan 27, 2021 at 4:31 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> I guess it was already discussed a while ago [1] and the conclusion was
> that Flink Async I/O is a specific Flink operator and it can’t be used
> directly from Beam since Beam should provide a unified model and its
> implementation for all supported runners.
>
> Did a proposed workaround, based on GroupIntoBatches [2], work for you?
>
> [1]
> https://lists.apache.org/thread.html/rc1e087a15036c18564d3147c3570c7d80c4963ff7d48cf8eaf180758%40%3Cuser.beam.apache.org%3E
> [2]
> https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/
>
> On 26 Jan 2021, at 22:09, Boyuan Zhang <bo...@google.com> wrote:
>
> +dev <de...@beam.apache.org>
>
> On Tue, Jan 26, 2021 at 1:07 PM Eleanore Jin <el...@gmail.com>
> wrote:
>
>> Hi community,
>>
>> Does Beam support Flink Async I/O operator? if so, can you please share
>> the doc, and if not, is there any workaround to achieve the same in Beam
>> semantics?
>>
>>
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
>>
>> Thanks a lot!
>> Eleanore
>>
>
>
Re: Beam support Flink Async I/O operator
Posted by Alexey Romanenko <ar...@gmail.com>.
I guess it was already discussed a while ago [1] and the conclusion was that Flink Async I/O is a specific Flink operator and it can’t be used directly from Beam since Beam should provide a unified model and its implementation for all supported runners.
Did a proposed workaround, based on GroupIntoBatches [2], work for you?
[1] https://lists.apache.org/thread.html/rc1e087a15036c18564d3147c3570c7d80c4963ff7d48cf8eaf180758%40%3Cuser.beam.apache.org%3E
[2] https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/
> On 26 Jan 2021, at 22:09, Boyuan Zhang <bo...@google.com> wrote:
>
> +dev <ma...@beam.apache.org>
>
> On Tue, Jan 26, 2021 at 1:07 PM Eleanore Jin <eleanore.jin@gmail.com <ma...@gmail.com>> wrote:
> Hi community,
>
> Does Beam support Flink Async I/O operator? if so, can you please share the doc, and if not, is there any workaround to achieve the same in Beam semantics?
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html>
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673>
>
> Thanks a lot!
> Eleanore
Re: Beam support Flink Async I/O operator
Posted by Alexey Romanenko <ar...@gmail.com>.
I guess it was already discussed a while ago [1] and the conclusion was that Flink Async I/O is a specific Flink operator and it can’t be used directly from Beam since Beam should provide a unified model and its implementation for all supported runners.
Did a proposed workaround, based on GroupIntoBatches [2], work for you?
[1] https://lists.apache.org/thread.html/rc1e087a15036c18564d3147c3570c7d80c4963ff7d48cf8eaf180758%40%3Cuser.beam.apache.org%3E
[2] https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/
> On 26 Jan 2021, at 22:09, Boyuan Zhang <bo...@google.com> wrote:
>
> +dev <ma...@beam.apache.org>
>
> On Tue, Jan 26, 2021 at 1:07 PM Eleanore Jin <eleanore.jin@gmail.com <ma...@gmail.com>> wrote:
> Hi community,
>
> Does Beam support Flink Async I/O operator? if so, can you please share the doc, and if not, is there any workaround to achieve the same in Beam semantics?
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html>
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673>
>
> Thanks a lot!
> Eleanore
Re: Beam support Flink Async I/O operator
Posted by Boyuan Zhang <bo...@google.com>.
+dev <de...@beam.apache.org>
On Tue, Jan 26, 2021 at 1:07 PM Eleanore Jin <el...@gmail.com> wrote:
> Hi community,
>
> Does Beam support Flink Async I/O operator? if so, can you please share
> the doc, and if not, is there any workaround to achieve the same in Beam
> semantics?
>
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
>
> Thanks a lot!
> Eleanore
>
Re: Beam support Flink Async I/O operator
Posted by Boyuan Zhang <bo...@google.com>.
+dev <de...@beam.apache.org>
On Tue, Jan 26, 2021 at 1:07 PM Eleanore Jin <el...@gmail.com> wrote:
> Hi community,
>
> Does Beam support Flink Async I/O operator? if so, can you please share
> the doc, and if not, is there any workaround to achieve the same in Beam
> semantics?
>
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
>
> Thanks a lot!
> Eleanore
>