Blocking in an Async function

[Olivier Nouguier <ol...@teads.com>]

Hi,
  We have a rather classical situation where we use an AsyncRich function
to read from Cassandra (C*) and enrich some domain elements.

It works great, but some time, C* is not reachable, how could we handle
this in async function ?

 0/ Let it crash && restart.
 1/ Thread.sleep() and retry seem a terrible idea to me.
 2/ handle timeout / error and schedule a retry: is it possible in
asyncInvoke?
 3/ Queueing inflight (incomplete) élement and retry later in a
sync/blocking function.

Thanks.


-- 

Olivier Nouguier

Senior Software Engineer

e | olivier.nouguier@teads.com m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
Montpellier, France
[image: image] <https://www.teads.com>

The information in this email is confidential and intended only for the
addressee(s) named above. If you are not the intended recipient any
disclosure, copying, distribution or any action taken or omitted to be
taken in reliance on it is prohibited and may be unlawful. Teads does not
warrant that any attachment(s) are free from viruses or other defects and
accept no liability for any losses resulting from infected email
transmission. Please note that any views expressed in this email may be
those of the originator and do not necessarily reflect those of the
organization.

Re: Blocking in an Async function

[Arvid Heise <ar...@apache.org>]

Hi Oliver,

if you are truly running async in asyncIO (remember you need an external
thread pool provided by an async lib or by yourself), then option 1) sounds
best to me. Could you elaborate why you'd think it's terrible? The only
downside is that you'd need to disable timeout in asyncIO and handle
timeouts yourself.

There is no support for option 2). If you are using ordered asyncIO, I'm
also not sure if it makes much sense or what the benefit is over option 1.

Option 3 is certainly possible but I'm not seeing the benefit immediately -
if C is down, you cannot progress anyways. Same is true for Option 0.

On Thu, Apr 29, 2021 at 11:54 AM Arian Rohani <ar...@gmail.com> wrote:

> Is this your own implementation of the AsyncRich function?
>
> There is already an API available for Flink for async I/O for external
> data access that solves some of the challenges that you seem to be
> encountering.
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
>
> Best,
> Arian Rohani
>
>
> Den tors 29 apr. 2021 kl 11:52 skrev Olivier Nouguier <
> olivier.nouguier@teads.com>:
>
>> Hi,
>>   We have a rather classical situation where we use an AsyncRich function
>> to read from Cassandra (C*) and enrich some domain elements.
>>
>> It works great, but some time, C* is not reachable, how could we handle
>> this in async function ?
>>
>>  0/ Let it crash && restart.
>>  1/ Thread.sleep() and retry seem a terrible idea to me.
>>  2/ handle timeout / error and schedule a retry: is it possible in
>> asyncInvoke?
>>  3/ Queueing inflight (incomplete) élement and retry later in a
>> sync/blocking function.
>>
>> Thanks.
>>
>>
>> --
>>
>> Olivier Nouguier
>>
>> Senior Software Engineer
>>
>> e | olivier.nouguier@teads.com m | 0651383971
>>
>> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
>> Montpellier, France
>> [image: image] <https://www.teads.com>
>>
>> The information in this email is confidential and intended only for the
>> addressee(s) named above. If you are not the intended recipient any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it is prohibited and may be unlawful. Teads does
>> not warrant that any attachment(s) are free from viruses or other defects
>> and accept no liability for any losses resulting from infected email
>> transmission. Please note that any views expressed in this email may be
>> those of the originator and do not necessarily reflect those of the
>> organization.
>>
>

Re: Blocking in an Async function

[Arian Rohani <ar...@gmail.com>]

Is this your own implementation of the AsyncRich function?

There is already an API available for Flink for async I/O for external data
access that solves some of the challenges that you seem to be encountering.
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html

Best,
Arian Rohani


Den tors 29 apr. 2021 kl 11:52 skrev Olivier Nouguier <
olivier.nouguier@teads.com>:

> Hi,
>   We have a rather classical situation where we use an AsyncRich function
> to read from Cassandra (C*) and enrich some domain elements.
>
> It works great, but some time, C* is not reachable, how could we handle
> this in async function ?
>
>  0/ Let it crash && restart.
>  1/ Thread.sleep() and retry seem a terrible idea to me.
>  2/ handle timeout / error and schedule a retry: is it possible in
> asyncInvoke?
>  3/ Queueing inflight (incomplete) élement and retry later in a
> sync/blocking function.
>
> Thanks.
>
>
> --
>
> Olivier Nouguier
>
> Senior Software Engineer
>
> e | olivier.nouguier@teads.com m | 0651383971
>
> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
> Montpellier, France
> [image: image] <https://www.teads.com>
>
> The information in this email is confidential and intended only for the
> addressee(s) named above. If you are not the intended recipient any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. Teads does not
> warrant that any attachment(s) are free from viruses or other defects and
> accept no liability for any losses resulting from infected email
> transmission. Please note that any views expressed in this email may be
> those of the originator and do not necessarily reflect those of the
> organization.
>