Async and checkpointing

[anurag <an...@gmail.com>]

Hi All,
Thanks for your help in advance. I am using async I/O with  flink 1.5.5, I
am using

AsyncDataStream.unorderedWait method  also my capacity is set to
100.My question is since my capacity is 100, each thread will be
processing one record.Say sequence number of my records is
S1,S2....S100 and say at a particular point in time , thread T1 is
processing record S1, T2 is processing record S2  and T100 is
processing record S100 and say all the threads T1..T100 except T50
failed when sending the record to the Sink. In this will the
checkpoint pointer move at all or not.

Apologies in advance if my question is not clear.

Thanks,
Anurag

Re: Async and checkpointing

[anurag <an...@gmail.com>]

Hi Yun,
Thanks for your reply. I am sorry if I was not clear . What I meant was
that as records are processed, the checkpoint and the corresponding stream
position will move.
But in case of async with unordered, will the checkpoint and the
corresponding stream position move in the above scenario where Tasks
T1....T100 are currently running in parallel and all tasks T1...T100 pass
except for T50.
Many thanks.
Anurag

On Thu, Oct 10, 2019 at 11:13 AM Yun Tang <my...@live.com> wrote:

> Hi  Anurag
>
> What do you mean "will the checkpoint pointer move at all or not"?
>
> If one of your thread failed to send record, and if it would cause that
> sub-task to fail, it would lead to the job failover. When job failover, any
> on-going checkpoint would be aborted and job would then just restore from
> last latest checkpoint.
> If failing to send record would not cause that sub-task to fail, nothing
> would happen and job continues to run but this might be not what you want.
>
> Best
> Yun Tang
> ------------------------------
> *From:* anurag <an...@gmail.com>
> *Sent:* Friday, October 11, 2019 2:03
> *To:* user@flink.apache.org <us...@flink.apache.org>
> *Subject:* Async and checkpointing
>
> Hi All,
> Thanks for your help in advance. I am using async I/O with  flink 1.5.5, I
> am using
>
> AsyncDataStream.unorderedWait method  also my capacity is set to 100.My question is since my capacity is 100, each thread will be processing one record.Say sequence number of my records is S1,S2....S100 and say at a particular point in time , thread T1 is processing record S1, T2 is processing record S2  and T100 is processing record S100 and say all the threads T1..T100 except T50 failed when sending the record to the Sink. In this will the checkpoint pointer move at all or not.
>
> Apologies in advance if my question is not clear.
>
> Thanks,
> Anurag
>
>

Re: Async and checkpointing

[Yun Tang <my...@live.com>]

Hi  Anurag

What do you mean "will the checkpoint pointer move at all or not"?

If one of your thread failed to send record, and if it would cause that sub-task to fail, it would lead to the job failover. When job failover, any on-going checkpoint would be aborted and job would then just restore from last latest checkpoint.
If failing to send record would not cause that sub-task to fail, nothing would happen and job continues to run but this might be not what you want.

Best
Yun Tang
________________________________
From: anurag <an...@gmail.com>
Sent: Friday, October 11, 2019 2:03
To: user@flink.apache.org <us...@flink.apache.org>
Subject: Async and checkpointing

Hi All,
Thanks for your help in advance. I am using async I/O with  flink 1.5.5, I am using

AsyncDataStream.unorderedWait method  also my capacity is set to 100.My question is since my capacity is 100, each thread will be processing one record.Say sequence number of my records is S1,S2....S100 and say at a particular point in time , thread T1 is processing record S1, T2 is processing record S2  and T100 is processing record S100 and say all the threads T1..T100 except T50 failed when sending the record to the Sink. In this will the checkpoint pointer move at all or not.


Apologies in advance if my question is not clear.

Thanks,
Anurag