You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Niels van Kaam <ni...@vankaam.net> on 2018/10/17 08:58:30 UTC
Checkpointing when one of the sources has completed
Hi All,
I am debugging an issue where the periodic checkpointing has halted. I
noticed that one of the sources of my job has completed (finished). The
other sources and operators would however still be able to produce output.
Does anyone know if Flink's periodic checkpoints are supposed to continue
when one or more sources of a job are in the "FINISHED" state?
Cheers,
Niels
Re: Checkpointing when one of the sources has completed
Posted by Paul Lam <pa...@gmail.com>.
Hi Niels,
The link was broken, it should be https://issues.apache.org/jira/browse/FLINK-2491 <https://issues.apache.org/jira/browse/FLINK-2491>.
A similar question was asked a few days ago.
Best,
Paul Lam
> 在 2018年10月17日，19:56，Niels van Kaam <ni...@vankaam.net> 写道：
>
> Hi All,
>
> Thanks for the responses, the finished source explains my issue then. I can work around the problem by letting my sources negotiate a "final" checkpoint via zookeeper.
>
> @Paul, I think your answer was meant for the earlier question asked by Joshua?
>
> Cheers,
> Niels
>
> On Wed, Oct 17, 2018 at 11:15 AM Joshua Fan <joshuafatfan@gmail.com <ma...@gmail.com>> wrote:
> Hi Niels,
>
> Probably not, an operator begins to do checkpoint until it gets all the barriers from all the upstream sources, if one source can not send a barrier, the downstream operator can not do checkpoint, FYI.
>
> Yours sincerely
> Joshua
>
> On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam <niels@vankaam.net <ma...@vankaam.net>> wrote:
> Hi All,
>
> I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.
>
> Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?
>
> Cheers,
> Niels
>
Re: Checkpointing when one of the sources has completed
Posted by Niels van Kaam <ni...@vankaam.net>.
Hi All,
Thanks for the responses, the finished source explains my issue then. I can
work around the problem by letting my sources negotiate a "final"
checkpoint via zookeeper.
@Paul, I think your answer was meant for the earlier question asked by
Joshua?
Cheers,
Niels
On Wed, Oct 17, 2018 at 11:15 AM Joshua Fan <jo...@gmail.com> wrote:
> Hi Niels,
>
> Probably not, an operator begins to do checkpoint until it gets all the
> barriers from all the upstream sources, if one source can not send a
> barrier, the downstream operator can not do checkpoint, FYI.
>
> Yours sincerely
> Joshua
>
> On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam <ni...@vankaam.net> wrote:
>
>> Hi All,
>>
>> I am debugging an issue where the periodic checkpointing has halted. I
>> noticed that one of the sources of my job has completed (finished). The
>> other sources and operators would however still be able to produce output.
>>
>> Does anyone know if Flink's periodic checkpoints are supposed to continue
>> when one or more sources of a job are in the "FINISHED" state?
>>
>> Cheers,
>> Niels
>>
>>
Re: Checkpointing when one of the sources has completed
Posted by Joshua Fan <jo...@gmail.com>.
Hi Niels,
Probably not, an operator begins to do checkpoint until it gets all the
barriers from all the upstream sources, if one source can not send a
barrier, the downstream operator can not do checkpoint, FYI.
Yours sincerely
Joshua
On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam <ni...@vankaam.net> wrote:
> Hi All,
>
> I am debugging an issue where the periodic checkpointing has halted. I
> noticed that one of the sources of my job has completed (finished). The
> other sources and operators would however still be able to produce output.
>
> Does anyone know if Flink's periodic checkpoints are supposed to continue
> when one or more sources of a job are in the "FINISHED" state?
>
> Cheers,
> Niels
>
>
Re: Checkpointing when one of the sources has completed
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Niels,
Checkpoints can only complete if all sources are running.
That's because the checkpoint mechanism relies on injecting checkpoint
barriers into the stream at the sources.
Best, Fabian
Am Mi., 17. Okt. 2018 um 11:11 Uhr schrieb Paul Lam <pa...@gmail.com>:
> Hi Niels,
>
> Please see https://issues.apache.org/jira/browse/FLINK-249.
>
> Best,
> Paul Lam
>
> 在 2018年10月17日，16:58，Niels van Kaam <ni...@vankaam.net> 写道：
>
> Hi All,
>
> I am debugging an issue where the periodic checkpointing has halted. I
> noticed that one of the sources of my job has completed (finished). The
> other sources and operators would however still be able to produce output.
>
> Does anyone know if Flink's periodic checkpoints are supposed to continue
> when one or more sources of a job are in the "FINISHED" state?
>
> Cheers,
> Niels
>
>
>
Re: Checkpointing when one of the sources has completed
Posted by Paul Lam <pa...@gmail.com>.
Hi Niels,
Please see https://issues.apache.org/jira/browse/FLINK-249 <https://issues.apache.org/jira/browse/FLINK-249>.
Best,
Paul Lam
> 在 2018年10月17日，16:58，Niels van Kaam <ni...@vankaam.net> 写道：
>
> Hi All,
>
> I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.
>
> Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?
>
> Cheers,
> Niels
>