You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Eleanore Jin <el...@gmail.com> on 2020/06/15 02:21:26 UTC

Beam supports Flink RichAsyncFunction

Hi Community,

I am trying to convert an existing Flink job into Beam pipeline. In the
current Flink job, we have async I/O operator (
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html)
which extends RichAsyncFunction
<https://ci.apache.org/projects/flink/flink-docs-master/api/java/index.html?org/apache/flink/streaming/api/functions/async/RichAsyncFunction.html>
.

I did not find any document online for Beam to support this, if it is
documented somewhere, can you please point to me?

In case Beam does not support it, is there any suggested 'work around' for
it?

Thanks a lot!
Eleanore

Re: Beam supports Flink RichAsyncFunction

Posted by Eleanore Jin <el...@gmail.com>.

Thanks Luke for the info. I will take a look.

Eleanore

On Mon, Jun 15, 2020 at 12:48 PM Luke Cwik <lc...@google.com> wrote:

> The intent is that users shouldn't have to use async I/O since the idea is
> that the runner should increase the number of workers/threads being
> processed automatically so that you never need to special case this.
> Unfortunately Dataflow is the only one who does this today so you are
> forced to use something like GroupIntoBatches[1] to gather input elements
> that you convert into requests you want to send and manage your own threads
> / completion.
>
> 1:
> https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/
>
> On Sun, Jun 14, 2020 at 7:21 PM Eleanore Jin <el...@gmail.com>
> wrote:
>
>> Hi Community,
>>
>> I am trying to convert an existing Flink job into Beam pipeline. In the
>> current Flink job, we have async I/O operator (
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html)
>> which extends RichAsyncFunction
>> <https://ci.apache.org/projects/flink/flink-docs-master/api/java/index.html?org/apache/flink/streaming/api/functions/async/RichAsyncFunction.html>
>> .
>>
>> I did not find any document online for Beam to support this, if it is
>> documented somewhere, can you please point to me?
>>
>> In case Beam does not support it, is there any suggested 'work around'
>> for it?
>>
>> Thanks a lot!
>> Eleanore
>>
>

Re: Beam supports Flink RichAsyncFunction

Posted by Luke Cwik <lc...@google.com>.

The intent is that users shouldn't have to use async I/O since the idea is
that the runner should increase the number of workers/threads being
processed automatically so that you never need to special case this.
Unfortunately Dataflow is the only one who does this today so you are
forced to use something like GroupIntoBatches[1] to gather input elements
that you convert into requests you want to send and manage your own threads
/ completion.

1:
https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/

On Sun, Jun 14, 2020 at 7:21 PM Eleanore Jin <el...@gmail.com> wrote:

> Hi Community,
>
> I am trying to convert an existing Flink job into Beam pipeline. In the
> current Flink job, we have async I/O operator (
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html)
> which extends RichAsyncFunction
> <https://ci.apache.org/projects/flink/flink-docs-master/api/java/index.html?org/apache/flink/streaming/api/functions/async/RichAsyncFunction.html>
> .
>
> I did not find any document online for Beam to support this, if it is
> documented somewhere, can you please point to me?
>
> In case Beam does not support it, is there any suggested 'work around' for
> it?
>
> Thanks a lot!
> Eleanore
>