You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ma GuoWei <ma...@outlook.com> on 2016/03/10 06:58:56 UTC
a question about operatorChain.broadcastCheckpointBarrier
hi,all
Why does the broadcastCheckpointBarrier need in the lock scope?
thanks a lot.
[cid:3612d450-1757-4dcb-9bb5-0d0cd8b89b1a]
Re: a question about operatorChain.broadcastCheckpointBarrier
Posted by Aljoscha Krettek <al...@apache.org>.
Yes, you are right. In other tasks the lock is not strictly necessary but we have one method in the base class that covers all tasks.
> On 11 Mar 2016, at 05:04, Ma GuoWei <ma...@outlook.com> wrote:
>
> it is very helpful。 thanks a lot
>
> now i know the souce task needs the lock。
>
> Other tasks like the StreamTask will handle broadcast and checkpoint then process element
>
> so lock is “no use” in those tasks。
>
> am i right？
>
> thanks your help very much！
>
>
> 发自我的 iPhone
>
>> 在 2016年3月10日，下午7:45，Aljoscha Krettek <al...@apache.org> 写道：
>>
>> Hi,
>> that call needs to be in the lock scope because we need to ensure that no element processing/element emission happens while we forward the checkpoint barrier and perform the checkpoint.
>>
>> Let me illustrate with an example. Say we have a source. What can happen in the source if the call is not in the checkpoint lock scope is the following scenario:
>>
>> Source Forwards Checkpoint Barrier
>> Source Emits Element 13
>> Source Emits Element 14
>> Source Performs Checkpoint
>>
>> Now, the operations downstream from the source also performs a checkpoint. In the state of this checkpoint we don’t have the changes caused by Element 13 and Element 14 reflected. The source checkpoint, however, says that we correctly forwarded Elements 13 and 14. Therefore the checkpoints of the source and the downstream operations are not consistent.
>>
>> Basically the lock ensures that nothing can happen between “Forward Checkpoint Barrier” and “Perform Checkpoint”. So the scenarios could be this, for example:
>>
>> Source Emits Element 13
>> Source Emits Element 14
>> Source Forwards Checkpoint Barrier
>> Source Performs Checkpoint
>>
>> Now, both the checkpoint in the source and the checkpoints of downstream operations correctly say that elements 13 and 14 are reflected in state changes.
>>
>> I hope this helps but let me know if you need to know more.
>>
>> Cheers,
>> Aljoscha
>>
>>
>>
>>> On 10 Mar 2016, at 06:58, Ma GuoWei <ma...@outlook.com> wrote:
>>>
>>> hi,all
>>> Why does the broadcastCheckpointBarrier need in the lock scope?
>>>
>>> thanks a lot.
>>
Re: a question about operatorChain.broadcastCheckpointBarrier
Posted by Ma GuoWei <ma...@outlook.com>.
it is very helpful。 thanks a lot
now i know the souce task needs the lock。
Other tasks like the StreamTask will handle broadcast and checkpoint then process element
so lock is “no use” in those tasks。
am i right？
thanks your help very much！
发自我的 iPhone
> 在 2016年3月10日，下午7:45，Aljoscha Krettek <al...@apache.org> 写道：
>
> Hi,
> that call needs to be in the lock scope because we need to ensure that no element processing/element emission happens while we forward the checkpoint barrier and perform the checkpoint.
>
> Let me illustrate with an example. Say we have a source. What can happen in the source if the call is not in the checkpoint lock scope is the following scenario:
>
> Source Forwards Checkpoint Barrier
> Source Emits Element 13
> Source Emits Element 14
> Source Performs Checkpoint
>
> Now, the operations downstream from the source also performs a checkpoint. In the state of this checkpoint we don’t have the changes caused by Element 13 and Element 14 reflected. The source checkpoint, however, says that we correctly forwarded Elements 13 and 14. Therefore the checkpoints of the source and the downstream operations are not consistent.
>
> Basically the lock ensures that nothing can happen between “Forward Checkpoint Barrier” and “Perform Checkpoint”. So the scenarios could be this, for example:
>
> Source Emits Element 13
> Source Emits Element 14
> Source Forwards Checkpoint Barrier
> Source Performs Checkpoint
>
> Now, both the checkpoint in the source and the checkpoints of downstream operations correctly say that elements 13 and 14 are reflected in state changes.
>
> I hope this helps but let me know if you need to know more.
>
> Cheers,
> Aljoscha
>
>
>
>> On 10 Mar 2016, at 06:58, Ma GuoWei <ma...@outlook.com> wrote:
>>
>> hi,all
>> Why does the broadcastCheckpointBarrier need in the lock scope?
>>
>> thanks a lot.
>
Re: a question about operatorChain.broadcastCheckpointBarrier
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
that call needs to be in the lock scope because we need to ensure that no element processing/element emission happens while we forward the checkpoint barrier and perform the checkpoint.
Let me illustrate with an example. Say we have a source. What can happen in the source if the call is not in the checkpoint lock scope is the following scenario:
Source Forwards Checkpoint Barrier
Source Emits Element 13
Source Emits Element 14
Source Performs Checkpoint
Now, the operations downstream from the source also performs a checkpoint. In the state of this checkpoint we don’t have the changes caused by Element 13 and Element 14 reflected. The source checkpoint, however, says that we correctly forwarded Elements 13 and 14. Therefore the checkpoints of the source and the downstream operations are not consistent.
Basically the lock ensures that nothing can happen between “Forward Checkpoint Barrier” and “Perform Checkpoint”. So the scenarios could be this, for example:
Source Emits Element 13
Source Emits Element 14
Source Forwards Checkpoint Barrier
Source Performs Checkpoint
Now, both the checkpoint in the source and the checkpoints of downstream operations correctly say that elements 13 and 14 are reflected in state changes.
I hope this helps but let me know if you need to know more.
Cheers,
Aljoscha
> On 10 Mar 2016, at 06:58, Ma GuoWei <ma...@outlook.com> wrote:
>
> hi,all
> Why does the broadcastCheckpointBarrier need in the lock scope?
>
> thanks a lot.
>
>