Re: Stateful AsyncIO operator

[Taher Koitawala <ta...@gmail.com>]

Thank you, this was helpful

On Fri, 9 Aug 2024 at 8:17 PM, Eric Xiao <er...@decodable.co.invalid>
wrote:

> Hey Taher,
>
> To answer question 2 and 3 - per the flink documentation[1] in-flight
> requests are stored in checkpoints and only the unsent records will be sent
> when recovering from a failure.
>
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/#fault-tolerance-guarantees
>
> Best,
>
> Eric
>
> On Tue, Aug 6, 2024 at 6:47 AM Taher Koitawala <ta...@gmail.com> wrote:
>
> > Hi All,
> >          I have a use case where I read from Kafkf and have some records
> > that I want to submit from Flink to a rest service with throttling, I
> > thought that the AsyncIO operator would be the best for such use cases,
> > however I wanted to know if AsyncIO is a stateful operator?
> >
> > What I want to know is:
> >
> >    1. Backpressure Management:
> >       1. I expect that the operator will not cause a backpressure Flink,
> >       TPM of the rest service is approx 10000, so I assume AsyncIO will
> > buffer
> >       records to state
> >    2. In case of failures and restart will the AsyncIO operator send the
> >    last lost set of records only?
> >    3. If let's say 200 records are to be sent and only 100 are sent and
> >    flink fails on restart does it send only 100 more?
> >
> > If AsyncIO is not the right operator please can you tell me what we can
> use
> > instead to achieve all this.
> >
> > Regards,
> > Taher Koitawala
> >
>