Async operator with a KeyedStream

[bastien dine <ba...@gmail.com>]

Hello,

I would like to know if you can use a KeyedStream with the Async operator :
I want to use the async operator to insert some stuff in my database but I
want to limit 1 request per element (with key=id) at a time
With a regular keyBy / map, it's working, but it's too slow (i don't have
enough ressources to increase my parallelism),

As far as I have seen, this is not possible
When I write something like
Async.orderedWait(myStream.keyBy(myKeyselector)), the keyBy is totally
ignored

Have you a solution for this?

Best Regards,
Bastien

------------------

Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io

Re: Async operator with a KeyedStream

[vino yang <ya...@gmail.com>]

Hi Bastien,

Your analysis of using KeyedStream in Async I/O is correct. It will not
figure out the key.

In your scene, the good practice about interacting with DB is async I/O +
thread pool[1] + connection Pool.

You can use a connection pool to reuse and limit the mysql connection.

Best,
Vino

[1]:
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html#implementation-tips

bastien dine <ba...@gmail.com> 于2019年10月31日周四 下午4:36写道：

> Hello,
>
> I would like to know if you can use a KeyedStream with the Async operator :
> I want to use the async operator to insert some stuff in my database but I
> want to limit 1 request per element (with key=id) at a time
> With a regular keyBy / map, it's working, but it's too slow (i don't have
> enough ressources to increase my parallelism),
>
> As far as I have seen, this is not possible
> When I write something like
> Async.orderedWait(myStream.keyBy(myKeyselector)), the keyBy is totally
> ignored
>
> Have you a solution for this?
>
> Best Regards,
> Bastien
>
> ------------------
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io
>