You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by patricia lee <pl...@gmail.com> on 2023/09/01 08:26:39 UTC
Send data asynchronously to a 3rd party via SinkFunction
I'd like to ask if there is a way to send data to a vendor (SDK plugin,
which is also an HTTP request) asynchronously in flink 1.17?
After transformation on the data, I usually collate them as a List to my
custom SinkFunction. I initialized a CompleteableFuture inside the invoke()
method. However I read about the Async I/O from the documentation but I
couldn't figure out how to use it in my use case.
How can I close the resources initialized in SinkFunction properly? e.g.
the job failed.
Is using completableFuture inside SinkFunction a good approach?
Regards,
Pat
Re: Send data asynchronously to a 3rd party via SinkFunction
Posted by liu ron <ro...@gmail.com>.
Hi, patricia
If you want to use SinkFunction, maybe you should use
`RichSinkFunction`[1], you can close the resources in close method.
[1]
https://github.com/apache/flink/blob/7d8f9821d2b3ed9876eae4ffe2e3c8b86af2d88a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/RichSinkFunction.java#L25
Best,
Ron
Feng Jin <ji...@gmail.com> 于2023年9月1日周五 22:18写道:
> hi, patricia
>
> I suggest using the generic asynchronous base sink.
>
> https://flink.apache.org/2022/03/16/the-generic-asynchronous-base-sink/
>
>
> Best,
> Feng
>
> On Fri, Sep 1, 2023 at 6:07 PM patricia lee <pl...@gmail.com> wrote:
>
>>
>> I'd like to ask if there is a way to send data to a vendor (SDK plugin,
>> which is also an HTTP request) asynchronously in flink 1.17?
>>
>> After transformation on the data, I usually collate them as a List to my
>> custom SinkFunction. I initialized a CompleteableFuture inside the invoke()
>> method. However I read about the Async I/O from the documentation but I
>> couldn't figure out how to use it in my use case.
>>
>>
>> How can I close the resources initialized in SinkFunction properly? e.g.
>> the job failed.
>> Is using completableFuture inside SinkFunction a good approach?
>>
>>
>> Regards,
>> Pat
>>
>>
Re: Send data asynchronously to a 3rd party via SinkFunction
Posted by Feng Jin <ji...@gmail.com>.
hi, patricia
I suggest using the generic asynchronous base sink.
https://flink.apache.org/2022/03/16/the-generic-asynchronous-base-sink/
Best,
Feng
On Fri, Sep 1, 2023 at 6:07 PM patricia lee <pl...@gmail.com> wrote:
>
> I'd like to ask if there is a way to send data to a vendor (SDK plugin,
> which is also an HTTP request) asynchronously in flink 1.17?
>
> After transformation on the data, I usually collate them as a List to my
> custom SinkFunction. I initialized a CompleteableFuture inside the invoke()
> method. However I read about the Async I/O from the documentation but I
> couldn't figure out how to use it in my use case.
>
>
> How can I close the resources initialized in SinkFunction properly? e.g.
> the job failed.
> Is using completableFuture inside SinkFunction a good approach?
>
>
> Regards,
> Pat
>
>