You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Yun Gao <yu...@aliyun.com.INVALID> on 2021/06/28 08:45:20 UTC
[VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi all,
For FLIP-147[1] which targets at supports checkpoints after tasks finished and modify operator
API and implementation to ensures the commit of last piece of data, since after the last vote
we have more discussions[2][3] and a few updates, including changes to PublicEvolving API,
I'd like to have another VOTE on the current state of the FLIP.
The vote will last at least 72 hours (Jul 1st), following the consensus
voting process.
thanks,
Yun
[1] https://cwiki.apache.org/confluence/x/mw-ZCQ
[2] https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
[3] https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi all,
I must admit I think I agree with Till that the relation between
life-cycles of the (Stream)Task and the Operator is missing in the FLIP.
Moreover I am now a bit concerned that we missed the relation with
stopping with savepoint. I am still positive we can incorporate that in
the current efforts. At the same time I think the proposed Operator APIs
still make sense.
First I'd like to start from the Operator's point of view. *
*
*endOfInput vs finish*
The way I see the relation between endOfInput/finish/MAX_WATERMARK/close
is that basically endOfInput() and finish() are the same for one-input
operators and the difference for multi input operators is that
endOfInput is called for each input separately and then once all inputs
finish the finish() is called. For simplicity I will from now on use
just the finish method and leave the endOfInput out of the discussion.
*w drain vs w/o drain*
I agree we have two cases of finishing based on the reason:
1. we consumed all data (bounded source or stop-with-savepoint w drain)
2. stop-with-savepoint w/o drain
I think we're at the same page that finish() should be called only in
the first case. I suggest we keep the current behaviour that the we emit
MAX_WATERMARK separately and then call finish(). For even-time based
scenarios this might be duplicated information, but it does not harm.
The order would be that we first emit MAX_WATERMARK and then call
finish(). Of course we would emit MAX_WATERMARK only in the first case.
Lastly the assumption should be that if there is finish() there will be
one extra snapshotState/notifyCheckpointComplete. For the w/o drain the
operator should not care when it is closed() or which savepoint was the
last one.
That's from the operator point of view.
Now I'd like to move on to the (Stream)Task point of view. I think here
lay the problems that we missed and Till rightly points those out.
*When should we call finish()*
In the current implementation we call finish() once we receive the
EndOfPartitionEvent which, as Till said, makes it impossible to bring
down the entire topology in a single savepoint. We can finish all
chained operators in a single Task, but not all Tasks in a DAG. It makes
it also a bit harder to distinguish if we should call the finish() then
or not. Point taken and it is an important omission from our side, in my
opinion.
However, we introduced EndOfUserRecordsEvent which says there will be no
more records, but the channel is not closed and you can expect other
events such as e.g. CheckpointBarriers. The original idea behind this
event was to make unaligned checkpoints work well with checkpoints after
tasks finished. I think we could and should use that event to call the
finish() method, though. That way we could call finish() before the
producer task finishes. This would also solve the problem of savepoint
w/o drain, as this event would not be emitted in that case and we would
not call the finish().
Having that a StreamTask could finish if all following conditions are met:
* in case of all data consumed
o received EndOfUserRecordsEvent from all channels
o called finish() method
o checkpoint was triggered and its completion was acknowledged
(notifyCheckpointComplete)
o received EndOfPartitionEvent from all channels
* stop-with-savepoint w/o drain
o savepoint acknowledged
o received EndOfPartitionEvent from all channels
We can distinguish that from the previous case, because we would receive
EndOfPartitionEvent from all channels without receiving
EndOfUserRecordsEvent before that.
Lets then see how this could look for end of data for two Tasks:
Task1 (Source) -> Task2
Task1:
(consume all data) -> call finish() -> send EndOfUserRecordsEvent ->
wait for checkpoint x trigger -> wait for checkpoint x complete + wait
for EndOfUserRecordsEvent acknowledged -> close / send EndOfPartitionEvent
Task2:
receive EndOfUserRecordsEvent -> send EndOfUserRecordsEvent -> wait for
checkpoint y barrier (it could happen that y=x) -> wait for checkpoint y
complete + wait for EndOfUserRecordsEvent acknowledged -> close
The savepoint w drain is very similar, it just replaces consume all
data, and in this case y will always be x, because all sources will
finish with the same savepoint.
For savepoint w/o drain it's slightly different:
Task1:
savepoint w/o drain -> wait for savepoint complete -> close / send
EndOfPartitionEvent
Task 2:
EndOfPartition received + wait for savepoint complete -> close
Does that make sense?
Best,
Dawid
On 16/07/2021 11:26, Till Rohrmann wrote:
> Ok, so the plan is that finish() will flush all pending events and then
> send the MAX_WATERMARK.
>
> What I am missing is the connection between the lifecycle of the operator
> and signals the StreamTask and Task might receive (and also their life
> cycles). At the moment we do have the EndOfPartitionEvent that we send once
> the Task reaches its end. Based on what signals will
> StreamOperator.finish() be called? What kind of other signals for the
> StreamTask do we have to introduce in order to support stop-with-savepoint
> w/o --drain if we don't want to call finish()?
>
> If finish() sends MAX_WATERMARK after flushing all pending events and
> finish() itself is being triggered by some endOfInputEvent from the
> StreamTask, then we might actually send MAX_WATERMARK multiple times.
>
> I think what I am lacking a bit is how the StreamOperator lifecycle will
> fit together with the lifecycle of its owner and the corresponding signals
> we have to send.
>
> Cheers,
> Till
>
> Cheers,
> Till
>
> On Fri, Jul 16, 2021 at 10:40 AM Yun Gao <yu...@aliyun.com> wrote:
>
>> Hi Till,
>>
>> Sorry that I do not see the reply when sending the last mail, I think as a
>> whole we are on the same page for the
>> 1. For normal stop-with-savepoint, we do not call finish() and do not emit
>> MAX_WATERMARK
>> 2. For normal finish / stop-with-savepoint --drain, we would call finish()
>> and emit MAX_WATERMARK
>>
>>> But then there is the question, how do we signal the operator that the
>> next checkpoint is supposed to stop the operator
>>> (how will the operator's lifecycle look in this case)? Maybe we simply
>> don't tell the operator and handle this situation on the
>>> StreamTask level.
>> Logically I think in this case UDF seems do not need to know the next
>> checkpoint is supposed to stop the operator since the final
>> checkpoint in this case have no difference with the ordinary checkpoints.
>>
>>> So I guess the question is will finish() advance the time to the end or
>> is this a separate mechanism (e.g. explicit watermarks).
>>
>> I tend to have an explicit MAX_WATERMARK since it makes watermark
>> processing to be unified with normal cases and make the meanings of
>> each event explicit. But this might be a private preference and both
>> methods would work.
>>
>> Very sorry for not making the whole thing clear in the FLIP again, if
>> there are no other concerns I'll update the FLIP with the above conclusions
>> to make it precise in this part.
>>
>>
>> Best,
>> Yun
>>
>> ------------------------------------------------------------------
>> From:Till Rohrmann <tr...@apache.org>
>> Send Time:2021 Jul. 16 (Fri.) 16:00
>> To:dev <de...@flink.apache.org>
>> Cc:Yun Gao <yu...@aliyun.com>
>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>> Finished
>>
>> I think we should try to sort this out because it might affect how and
>> when finish() will be called (or in general how the operator lifecycle
>> looks like).
>>
>> To give an example let's take a look at the stop-with-savepoint w/ and w/o
>> --drain:
>>
>> 1) stop-with-savepoint w/o --drain: Conceptually what we would like to do
>> is to stop processing w/o completing any windows or waiting for the
>> AsyncOperator to finish its operations. All these unfinished operations
>> should become part of the final checkpoint so that we can resume from it at
>> a later point. Depending on what finish() does (flush unfinished windows or
>> not), this method must or must not be called. Assuming that finish()
>> flushes unfinished windows/waits for uncompleted async operations, we
>> clearly shouldn't call it. But then there is the question, how do we
>> signal the operator that the next checkpoint is supposed to stop the
>> operator (how will the operator's lifecycle look in this case)? Maybe we
>> simply don't tell the operator and handle this situation on the StreamTask
>> level. If finish() does not flush unfinished windows, then it shouldn't be
>> a problem.
>>
>> 2) stop-with-savepoint w/ --drain: Here we want to complete all pending
>> operations and flush out all results because we don't intend to resume the
>> job. Conceptually, we tell the system that we have reached MAX_WATERMARK.
>> If finish() is defined so that it implicitly advances the watermark to
>> MAX_WATERMARK, then there is no problem. If finish() does not have this
>> semantic, then we need to send the MAX_WATERMARK before sending the
>> endOfInput event to a downstream task. In fact, stop-with-savepoint /w
>> --drain shouldn't be a lot different from a bounded source that reaches its
>> end. It would also send MAX_WATERMARK and then signal the endOfInput event
>> (note that endOfInput is decoupled from the event time here).
>>
>> So I guess the question is will finish() advance the time to the end or is
>> this a separate mechanism (e.g. explicit watermarks).
>>
>> Concerning how to handle processing time, I am a bit unsure tbh. I can see
>> arguments for completing processing time windows/firing processing time
>> timers when calling stop-with-savepoint w/ --drain. On the other hand, I
>> could also see that people want to define actions based on the wall clock
>> time that are independent of the stream state and, thus, would want to
>> ignore them if the Flink application is stopped before reaching this time.
>>
>> Cheers,
>> Till
>>
>> On Fri, Jul 16, 2021 at 7:48 AM Piotr Nowojski <pn...@apache.org>
>> wrote:
>> Hi Till,
>>
>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>
>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>> WindowOperator close all windows and emit the results upon calling
>>> finish, for example?
>> 1) currently they are independent but parallel mechanisms. With event time,
>> they are basically the same.
>> 2) it probably should for the sake of processing time windows.
>>
>> Here you are touching the bit of the current design that I like the least.
>> We basically have now three different ways of conveying very similar
>> things:
>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
>> processing time?)
>> b) endInput(), used for example by AsyncWaitOperator to flush it's internal
>> state
>> c) finish(), used for example by ContinuousFileReaderOperator
>>
>> It's a bit messy and I'm not sure if this should be strengthened out? Each
>> one of those has a little bit different semantic/meaning, but at the same
>> time they are very similar. For single input operators `endInput()` and
>> `finish()` are actually the very same thing.
>>
>> Piotrek
>>
>> czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
>>
>>> Thanks for updating the FLIP. Based on the new section about
>>> stop-with-savepoint [--drain] I got two other questions:
>>>
>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>
>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>> WindowOperator close all windows and emit the results upon calling
>>> finish, for example?
>>>
>>> Cheers,
>>> Till
>>>
>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Thanks a lot for your answers and clarifications Yun.
>>>>
>>>> 1+2) Agreed, this can be a future improvement if this becomes a
>> problem.
>>>> 3) Great, this will help a lot with understanding the FLIP.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
>>>> wrote:
>>>>
>>>>> Hi Till,
>>>>>
>>>>> Very thanks for the review and comments!
>>>>>
>>>>> 1) First I think in fact we could be able to do the computation
>> outside
>>>>> of the main thread,
>>>>> and the current implementation mainly due to the computation is in
>>>>> general fast and we
>>>>> initially want to have a simplified first version.
>>>>>
>>>>> The main requirement here is to have a constant view of the state of
>> the
>>>>> tasks, otherwise
>>>>> for example if we have A -> B, if A is running when we check if we
>> need
>>>>> to trigger A, we will
>>>>> mark A as have to trigger, but if A gets to finished when we check B,
>> we
>>>>> will also mark B as
>>>>> have to trigger, then B will receive both rpc trigger and checkpoint
>>>>> barrier, which would break
>>>>> some assumption on the task side and complicate the implementation.
>>>>>
>>>>> But to cope this issue, we in fact could first have a snapshot of the
>>>>> tasks' state and then do the
>>>>> computation, both the two step do not need to be in the main thread.
>>>>>
>>>>> 2) For the computation logic, in fact currently we benefit a lot from
>>>>> some shortcuts on all-to-all
>>>>> edges and job vertex with all tasks running, these shortcuts could do
>>>>> checks on the job vertex level
>>>>> first and skip some job vertices as a whole. With this optimization we
>>>>> have a O(V) algorithm, and the
>>>>> current running time of the worst case for a job with 320,000 tasks is
>>>>> less than 100ms. For
>>>>> daily graph sizes the time would be further reduced linearly.
>>>>>
>>>>> If we do the computation based on the last triggered tasks, we may not
>>>>> easily encode this information
>>>>> into the shortcuts on the job vertex level. And since the time seems
>> to
>>>>> be short, perhaps it is enough
>>>>> to do re-computation from the scratch in consideration of the tradeoff
>>>>> between the performance and
>>>>> the complexity ?
>>>>>
>>>>> 3) We are going to emit the EndOfInput event exactly after the
>> finish()
>>>>> method and before the last
>>>>> snapshotState() method so that we could shut down the whole topology
>>> with
>>>>> a single final checkpoint.
>>>>> Very sorry for not include enough details for this part and I'll
>>>>> complement the FLIP with the details on
>>>>> the process of the final checkpoint / savepoint.
>>>>>
>>>>> Best,
>>>>> Yun
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> From:Till Rohrmann <tr...@apache.org>
>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
>>>>> To:dev <de...@flink.apache.org>
>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>>>> Finished
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I am a bit late to the voting party but let me ask three questions:
>>>>>
>>>>> 1) Why do we execute the trigger plan computation in the main thread
>> if
>>> we
>>>>> cannot guarantee that all tasks are still running when triggering the
>>>>> checkpoint? Couldn't we do the computation in a different thread in
>>> order
>>>>> to relieve the main thread a bit.
>>>>>
>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator seems to
>> go
>>>>> over the whole topology for every calculation. Wouldn't it be more
>>>>> efficient to maintain the set of current tasks to trigger and check
>>>>> whether
>>>>> anything has changed and if so check the succeeding tasks until we
>> have
>>>>> found the current checkpoint trigger frontier?
>>>>>
>>>>> 3) When are we going to send the endOfInput events to a downstream
>> task?
>>>>> If
>>>>> this happens after we call finish on the upstream operator but before
>>>>> snapshotState then it would be possible to shut down the whole
>> topology
>>>>> with a single final checkpoint. I think this part could benefit from a
>>> bit
>>>>> more detailed description in the FLIP.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
>> vote
>>>>> now.
>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>>>>>>
>>>>>> - Dawid Wysakowicz (binding)
>>>>>> - Piotr Nowojski(binding)
>>>>>> - Jiangang Liu (binding)
>>>>>> - Arvid Heise (binding)
>>>>>> - Jing Zhang (binding)
>>>>>> - Leonard Xu (non-binding)
>>>>>> - Guowei Ma (binding)
>>>>>>
>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
>>> accepted.
>>>>>> Very thanks everyone!
>>>>>>
>>>>>> Best,
>>>>>> Yun
>>>>>>
>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>>>>
>>
>>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Till, Dawid
Very thanks for all the thoughts!
1) Regarding the event used to notify finish()
Logically there indeed two purposes to introduce the new event, one is for working
around the limitations fo the unaligned checkpoint and one for notifying the finish(),
and as Dawid has pointed out for implementation we want to unify them into one event and
currently it is named as EndOfUserRecordsEvent in realistic. Sorry for the inconsistency and
I'll unify the terminology in the FLIP.
2) Regarding the lifecycle of the tasks & operators
Glad that we should be in general on the same page, and the FSM description indeed simplify the
understanding of the process a lot.
For the order of emit MAX_WATERMARK and call finish(), I should have made a mistake, and as
both of you have pointed out finish() should be called after emitting MAX_WATERMARK, this indeed
should be two atomic subprocess in order: handing MAX_WATERMARK -> get to FINISHED, and
we in fact could not call finish() between aligned on MAX_WATERMARK and emit MAX_WATERMARK.
A small detail for the relationship between emit MAX_WATERMARK and finish() is that although MAX_WATERMARK
would fire all the event time triggers and finalize structures like windows, perhaps sometimes users may still need
to do some flush in the finish() method for non-event-time based data, like when a file sink uses a rolling policy
based on size, then on finish() it needs to flush and close the current opened files.
We also mentioned different methods to distinguish whether we need to emit MAX_WATERMARK and call finish(),
like not emit such events or emit events with special flags, I think all these methods would be able to achieve the
target and perhaps we could leave the choices of the detailed methods to when we are implementing the code.
For waiting for the next checkpoint / savepoint after FINISHED, I also agree with that currently we could have the
same process for both the final checkpoint and stop-with-savepoint cases, and do not need to distinguish if we are
waiting for a checkpoint or a savepoint.
Best,
Yun
------------------------------------------------------------------
From:Till Rohrmann <tr...@apache.org>
Send Time:2021 Jul. 16 (Fri.) 23:05
To:Yun Gao <yu...@aliyun.com>
Cc:dev <de...@flink.apache.org>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Thanks for your responses Dawid and Yun,
I agree with you that the whole lifecycle section of the
StreamOperator/StreamTask/Task in the FLIP needs to be better fleshed out.
I have to admit that I am a bit confused by the new events you are
mentioning because I couldn't find them in the FLIP. I assume that
EndOfUserRecordsEvent has been introduced to work around the limitations of
unaligned checkpoints and that EndofInputEvent is something that does not
exist yet. Maybe properly writing things down and agreeing on the same
terminology could help our discussion.
I also agree with you that we need some kind of new event (one that is
conceptually bound to the StreamTask) that says that we have reached the
end of input. Let's name this event for the further discussion
EndOfInputEvent.
The way I've thought about the problem is the following: First let us
define two simple state machines for the StreamOperator and the StreamTask:
StreamOperator FSM: RUNNING -> FINISHED -> CLOSED
RUNNING: We process input records and produce output records.
RUNNING -> FINISHED: Happens upon calling StreamOperator.finish() or if the
source reaches the end of data (does not mean that we flush anything). This
also emits EndOfInputEvent to downstream operators
FINISHED: The operator must not output events.
FINISHED -> CLOSED: Happens upon calling StreamOperator.close() (close all
resources)
StreamTask FSM: RUNNING -> WAITING_FOR_FINAL_CHECKPOINT -> CLOSED
RUNNING: The StreamOperator is running.
RUNNING -> WAITING_FOR_FINAL_CHECKPOINT: Upon receiving EndOfInputEvent on
all inputs. This will also call StreamOperator.finish() and send
EndOfInputEvent to its downstream tasks. (Alternatively if the
EndOfInputEvent processing happens in the StreamOperator, then we can
listen to the state transition RUNNING -> FINISHED of the StreamOperator).
WAITING_FOR_FINAL_CHECKPOINT: As the name says we wait for the next
successful checkpoint to complete.
WAITING_FOR_FINAL_CHECKPOINT -> CLOSED: Once the next checkpoint completes
successfully (e.g. StreamTask.notifyCheckpointComplete is called).
So the first thing to note is that StreamOperator.finish() does not flush
anything but simply tells the StreamOperator that it should no longer
produce output. The basic idea would be that whenever we need to flush
our buffered data, then we send a MAX_WATERMARK before sending an
EndOfInputEvent. Of course, we could also integrate this property in the
EndOfInputEvent(boolean flush_buffered_data) but we already have the
watermark mechanism. Of course, we could also change
StreamOperator.finish() into .finish(boolean flush) and translate a
MAX_WATERMARK into .finish(true).
What would it look like if the source reaches its end?
Upon reaching the end, the source would send a MAX_WATERMARK followed by an
EndOfInputEvent and then go into the FINISHED state. This will cause the
StreamTask to go into the WAITING_FOR_FINAL_CHECKPOINT state. The
downstream tasks upon receiving MAX_WATERMARK and EndOfInputEvent on all
channels, would first flush all their data (because of MAX_WATERMARK), then
forward MAX_WATERMARK to its downstream operator, go into the FINISHED
state (because of EndOfInputEvent) and finally also sent an EndOfInputEvent
to its downstream operators. The owning StreamTask would also go into the
WAITING_FOR_FINAL_CHECKPOINT state. And all StreamOperators and StreamTasks
would terminate after the next successful checkpoint. That would then
trigger the EndOfPartitionEvent that will terminate the TCP connections.
What would it look like if we trigger stop-with-savepoint w/ drain?
We would send our sources a signal that they should stop processing and
advance the time. This means that they emit a MAX_WATERMARK and then go
into the FINISHED state (meaning that they send an EndOfInputEvent). Then
everything is the same as with the first case only that we will trigger a
savepoint just after sending the stop processing signal.
What would it look like if we trigger stop-with-savepoint w/o drain?
We would send our sources a signal that they should stop processing. This
means that the sources go into FINISHED which means that we are sending an
EndOfInputEvent. Then everything is the same as with the first case only
that we will trigger a savepoint just after sending the stop processing
signal.
One thing I would like to emphasize here is that the
StreamOperators/StreamTasks don't have to know anything about a specific
savepoint/checkpoint. They simply follow their state machine that says that
you have to wait for a checkpoint if you want to stop. It just happens that
in the case of stop-with-savepoint the next checkpoint will be a savepoint.
That way we decouple the lifecycle management and the stop-with-savepoint
operations a bit better.
Cheers,
Till
On Fri, Jul 16, 2021 at 2:22 PM Yun Gao <yu...@aliyun.com> wrote:
> Hi Till,
>
> > Ok, so the plan is that finish() will flush all pending events and then
> send the MAX_WATERMARK.
>
> > What I am missing is the connection between the lifecycle of the
> operator and signals the StreamTask and Task might receive (and also their
> life cycles).
> > At the moment we do have the EndOfPartitionEvent that we send once the
> Task reaches its end. Based on what signals will
> > StreamOperator.finish() be called? What kind of other signals for the
> StreamTask do we have to introduce in order to support stop-with-savepoint
> w/o --drain if we don't want to call finish()?
>
> Currently the plan is to introduce a new event (like EndofInputEvent) to
> first notify all the tasks from sources to
> sink to first call finish() (or skip finish() if stop-with-savepoint),
> then the tasks would wait for the last checkpoint and the finally emit
> EndOfPartition. We could
> add a flag to `EndofInputEvent` to indicating if the finish() method
> would be called.
>
> From the view of a single task, the life cycle of a source task would
> looks like
>
> 1. Open & Initialize
> 2. Emits all the records
> 3. Call finish() for all the operators
> 4. Emit MAX_WATERMARK
> 5. Emit EndofInputEvent {finish = true}
> 6. Wait for the checkpoint trigger, emit CheckpointBarrier and wait for
> the notification of the final checkpoint if needed.
> 7. Call close() for all the operators
> 8. Emit EndOfPartitionEvent
>
>
> for normal finish. For stop-with-savepoint [--drain], it would looks like
>
> 1. Open & Initialize
> 2. Received stop-with-savepoint (--drain) request
> 3. Finish the task (e.g. interrupt the legacy source thread or suspend the
> mailbox for the new source).
> 4. Call finish() for all the operators.
> 5. Emit MAX_WATERMARK if --drain
> 6. Emit EndofInputEvent {finish = is drain}
> 7. Emit CheckpointBarrier for this savepoint and wait for the
> notification of the final savepoint if needed.
> 8. Call close() for all the operators
> 9. Emit EndOfPartitionEvent
>
> Then the lifecycle of a non-source task would be
>
> 1. Open & Initialize
> 2. runMailboxLoop()
> 3. Received MAX_WATERMARK from all the channels. --> trigger all the
> timers.
> 4. Received EndofInputEvent from all the input channels --> Call
> finish() for all the operators.
> 5. Emit MAX_WATERMARK if --drain (this is nature with the watermark
> alignment mechanism)
> 6. Emit the received EndofInputEvent.
> 7. Wait for the checkpoint barrier aligned, emit CheckpointBarrier and
> wait for the notification of the final checkpoint / savepoint if needed.
> 8. Call close() for all the operators
> 9. Wait for received all EndOfPartitionEvent, release network resources
> and emit EndOfPartitionEvent
>
> > If finish() sends MAX_WATERMARK after flushing all pending events and
> finish() itself is being triggered by some endOfInputEvent from the
> StreamTask, then we might actually send MAX_WATERMARK multiple times.
>
> I think we might not send MAX_WATERMARK multiple times in that for
> non-source tasks, both MAX_WATERMARK and EndOfInputEvent would be
> processed only after they are received from event input channels.
> Then after process the current task would emit MAX_WATERMARK and EndOfInputEvent
> again to the following tasks, thus both the two events would only be sent
> once for each task ?
>
> Best,
> Yun
>
>
>
> ------------------------------------------------------------------
> From:Till Rohrmann <tr...@apache.org>
> Send Time:2021 Jul. 16 (Fri.) 17:26
> To:Yun Gao <yu...@aliyun.com>
> Cc:dev <de...@flink.apache.org>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> Ok, so the plan is that finish() will flush all pending events and then
> send the MAX_WATERMARK.
>
> What I am missing is the connection between the lifecycle of the operator
> and signals the StreamTask and Task might receive (and also their life
> cycles). At the moment we do have the EndOfPartitionEvent that we send once
> the Task reaches its end. Based on what signals will
> StreamOperator.finish() be called? What kind of other signals for the
> StreamTask do we have to introduce in order to support stop-with-savepoint
> w/o --drain if we don't want to call finish()?
>
> If finish() sends MAX_WATERMARK after flushing all pending events and
> finish() itself is being triggered by some endOfInputEvent from the
> StreamTask, then we might actually send MAX_WATERMARK multiple times.
>
> I think what I am lacking a bit is how the StreamOperator lifecycle will
> fit together with the lifecycle of its owner and the corresponding signals
> we have to send.
>
> Cheers,
> Till
>
> Cheers,
> Till
>
> On Fri, Jul 16, 2021 at 10:40 AM Yun Gao <yu...@aliyun.com> wrote:
> Hi Till,
>
> Sorry that I do not see the reply when sending the last mail, I think as a
> whole we are on the same page for the
> 1. For normal stop-with-savepoint, we do not call finish() and do not emit
> MAX_WATERMARK
> 2. For normal finish / stop-with-savepoint --drain, we would call finish()
> and emit MAX_WATERMARK
>
> > But then there is the question, how do we signal the operator that the
> next checkpoint is supposed to stop the operator
> > (how will the operator's lifecycle look in this case)? Maybe we simply
> don't tell the operator and handle this situation on the
> > StreamTask level.
>
> Logically I think in this case UDF seems do not need to know the next
> checkpoint is supposed to stop the operator since the final
> checkpoint in this case have no difference with the ordinary checkpoints.
>
> > So I guess the question is will finish() advance the time to the end or
> is this a separate mechanism (e.g. explicit watermarks).
>
> I tend to have an explicit MAX_WATERMARK since it makes watermark
> processing to be unified with normal cases and make the meanings of
> each event explicit. But this might be a private preference and both
> methods would work.
>
> Very sorry for not making the whole thing clear in the FLIP again, if
> there are no other concerns I'll update the FLIP with the above conclusions
> to make it precise in this part.
>
>
> Best,
> Yun
>
> ------------------------------------------------------------------
> From:Till Rohrmann <tr...@apache.org>
> Send Time:2021 Jul. 16 (Fri.) 16:00
> To:dev <de...@flink.apache.org>
> Cc:Yun Gao <yu...@aliyun.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> I think we should try to sort this out because it might affect how and
> when finish() will be called (or in general how the operator lifecycle
> looks like).
>
> To give an example let's take a look at the stop-with-savepoint w/ and w/o
> --drain:
>
> 1) stop-with-savepoint w/o --drain: Conceptually what we would like to do
> is to stop processing w/o completing any windows or waiting for the
> AsyncOperator to finish its operations. All these unfinished operations
> should become part of the final checkpoint so that we can resume from it at
> a later point. Depending on what finish() does (flush unfinished windows or
> not), this method must or must not be called. Assuming that finish()
> flushes unfinished windows/waits for uncompleted async operations, we
> clearly shouldn't call it. But then there is the question, how do we
> signal the operator that the next checkpoint is supposed to stop the
> operator (how will the operator's lifecycle look in this case)? Maybe we
> simply don't tell the operator and handle this situation on the StreamTask
> level. If finish() does not flush unfinished windows, then it shouldn't be
> a problem.
>
> 2) stop-with-savepoint w/ --drain: Here we want to complete all pending
> operations and flush out all results because we don't intend to resume the
> job. Conceptually, we tell the system that we have reached MAX_WATERMARK.
> If finish() is defined so that it implicitly advances the watermark to
> MAX_WATERMARK, then there is no problem. If finish() does not have this
> semantic, then we need to send the MAX_WATERMARK before sending the
> endOfInput event to a downstream task. In fact, stop-with-savepoint /w
> --drain shouldn't be a lot different from a bounded source that reaches its
> end. It would also send MAX_WATERMARK and then signal the endOfInput event
> (note that endOfInput is decoupled from the event time here).
>
> So I guess the question is will finish() advance the time to the end or is
> this a separate mechanism (e.g. explicit watermarks).
>
> Concerning how to handle processing time, I am a bit unsure tbh. I can see
> arguments for completing processing time windows/firing processing time
> timers when calling stop-with-savepoint w/ --drain. On the other hand, I
> could also see that people want to define actions based on the wall clock
> time that are independent of the stream state and, thus, would want to
> ignore them if the Flink application is stopped before reaching this time.
>
> Cheers,
> Till
>
> On Fri, Jul 16, 2021 at 7:48 AM Piotr Nowojski <pn...@apache.org>
> wrote:
> Hi Till,
>
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
>
> 1) currently they are independent but parallel mechanisms. With event time,
> they are basically the same.
> 2) it probably should for the sake of processing time windows.
>
> Here you are touching the bit of the current design that I like the least.
> We basically have now three different ways of conveying very similar
> things:
> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
> processing time?)
> b) endInput(), used for example by AsyncWaitOperator to flush it's internal
> state
> c) finish(), used for example by ContinuousFileReaderOperator
>
> It's a bit messy and I'm not sure if this should be strengthened out? Each
> one of those has a little bit different semantic/meaning, but at the same
> time they are very similar. For single input operators `endInput()` and
> `finish()` are actually the very same thing.
>
> Piotrek
>
> czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
>
> > Thanks for updating the FLIP. Based on the new section about
> > stop-with-savepoint [--drain] I got two other questions:
> >
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
> >
> > Cheers,
> > Till
> >
> > On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Thanks a lot for your answers and clarifications Yun.
> > >
> > > 1+2) Agreed, this can be a future improvement if this becomes a
> problem.
> > >
> > > 3) Great, this will help a lot with understanding the FLIP.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > > wrote:
> > >
> > >> Hi Till,
> > >>
> > >> Very thanks for the review and comments!
> > >>
> > >> 1) First I think in fact we could be able to do the computation
> outside
> > >> of the main thread,
> > >> and the current implementation mainly due to the computation is in
> > >> general fast and we
> > >> initially want to have a simplified first version.
> > >>
> > >> The main requirement here is to have a constant view of the state of
> the
> > >> tasks, otherwise
> > >> for example if we have A -> B, if A is running when we check if we
> need
> > >> to trigger A, we will
> > >> mark A as have to trigger, but if A gets to finished when we check B,
> we
> > >> will also mark B as
> > >> have to trigger, then B will receive both rpc trigger and checkpoint
> > >> barrier, which would break
> > >> some assumption on the task side and complicate the implementation.
> > >>
> > >> But to cope this issue, we in fact could first have a snapshot of the
> > >> tasks' state and then do the
> > >> computation, both the two step do not need to be in the main thread.
> > >>
> > >> 2) For the computation logic, in fact currently we benefit a lot from
> > >> some shortcuts on all-to-all
> > >> edges and job vertex with all tasks running, these shortcuts could do
> > >> checks on the job vertex level
> > >> first and skip some job vertices as a whole. With this optimization we
> > >> have a O(V) algorithm, and the
> > >> current running time of the worst case for a job with 320,000 tasks is
> > >> less than 100ms. For
> > >> daily graph sizes the time would be further reduced linearly.
> > >>
> > >> If we do the computation based on the last triggered tasks, we may not
> > >> easily encode this information
> > >> into the shortcuts on the job vertex level. And since the time seems
> to
> > >> be short, perhaps it is enough
> > >> to do re-computation from the scratch in consideration of the tradeoff
> > >> between the performance and
> > >> the complexity ?
> > >>
> > >> 3) We are going to emit the EndOfInput event exactly after the
> finish()
> > >> method and before the last
> > >> snapshotState() method so that we could shut down the whole topology
> > with
> > >> a single final checkpoint.
> > >> Very sorry for not include enough details for this part and I'll
> > >> complement the FLIP with the details on
> > >> the process of the final checkpoint / savepoint.
> > >>
> > >> Best,
> > >> Yun
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------------
> > >> From:Till Rohrmann <tr...@apache.org>
> > >> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >> To:dev <de...@flink.apache.org>
> > >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >> Finished
> > >>
> > >> Hi everyone,
> > >>
> > >> I am a bit late to the voting party but let me ask three questions:
> > >>
> > >> 1) Why do we execute the trigger plan computation in the main thread
> if
> > we
> > >> cannot guarantee that all tasks are still running when triggering the
> > >> checkpoint? Couldn't we do the computation in a different thread in
> > order
> > >> to relieve the main thread a bit.
> > >>
> > >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to
> go
> > >> over the whole topology for every calculation. Wouldn't it be more
> > >> efficient to maintain the set of current tasks to trigger and check
> > >> whether
> > >> anything has changed and if so check the succeeding tasks until we
> have
> > >> found the current checkpoint trigger frontier?
> > >>
> > >> 3) When are we going to send the endOfInput events to a downstream
> task?
> > >> If
> > >> this happens after we call finish on the upstream operator but before
> > >> snapshotState then it would be possible to shut down the whole
> topology
> > >> with a single final checkpoint. I think this part could benefit from a
> > bit
> > >> more detailed description in the FLIP.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> > >> wrote:
> > >>
> > >> > Hi there,
> > >> >
> > >> > Since the voting time of FLIP-147[1] has passed, I'm closing the
> vote
> > >> now.
> > >> >
> > >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >> >
> > >> > - Dawid Wysakowicz (binding)
> > >> > - Piotr Nowojski(binding)
> > >> > - Jiangang Liu (binding)
> > >> > - Arvid Heise (binding)
> > >> > - Jing Zhang (binding)
> > >> > - Leonard Xu (non-binding)
> > >> > - Guowei Ma (binding)
> > >> >
> > >> > Thus I'm happy to announce that the update to the FLIP-147 is
> > accepted.
> > >> >
> > >> > Very thanks everyone!
> > >> >
> > >> > Best,
> > >> > Yun
> > >> >
> > >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>
> > >>
> >
>
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
Thanks for your responses Dawid and Yun,
I agree with you that the whole lifecycle section of the
StreamOperator/StreamTask/Task in the FLIP needs to be better fleshed out.
I have to admit that I am a bit confused by the new events you are
mentioning because I couldn't find them in the FLIP. I assume that
EndOfUserRecordsEvent has been introduced to work around the limitations of
unaligned checkpoints and that EndofInputEvent is something that does not
exist yet. Maybe properly writing things down and agreeing on the same
terminology could help our discussion.
I also agree with you that we need some kind of new event (one that is
conceptually bound to the StreamTask) that says that we have reached the
end of input. Let's name this event for the further discussion
EndOfInputEvent.
The way I've thought about the problem is the following: First let us
define two simple state machines for the StreamOperator and the StreamTask:
StreamOperator FSM: RUNNING -> FINISHED -> CLOSED
RUNNING: We process input records and produce output records.
RUNNING -> FINISHED: Happens upon calling StreamOperator.finish() or if the
source reaches the end of data (does not mean that we flush anything). This
also emits EndOfInputEvent to downstream operators
FINISHED: The operator must not output events.
FINISHED -> CLOSED: Happens upon calling StreamOperator.close() (close all
resources)
StreamTask FSM: RUNNING -> WAITING_FOR_FINAL_CHECKPOINT -> CLOSED
RUNNING: The StreamOperator is running.
RUNNING -> WAITING_FOR_FINAL_CHECKPOINT: Upon receiving EndOfInputEvent on
all inputs. This will also call StreamOperator.finish() and send
EndOfInputEvent to its downstream tasks. (Alternatively if the
EndOfInputEvent processing happens in the StreamOperator, then we can
listen to the state transition RUNNING -> FINISHED of the StreamOperator).
WAITING_FOR_FINAL_CHECKPOINT: As the name says we wait for the next
successful checkpoint to complete.
WAITING_FOR_FINAL_CHECKPOINT -> CLOSED: Once the next checkpoint completes
successfully (e.g. StreamTask.notifyCheckpointComplete is called).
So the first thing to note is that StreamOperator.finish() does not flush
anything but simply tells the StreamOperator that it should no longer
produce output. The basic idea would be that whenever we need to flush
our buffered data, then we send a MAX_WATERMARK before sending an
EndOfInputEvent. Of course, we could also integrate this property in the
EndOfInputEvent(boolean flush_buffered_data) but we already have the
watermark mechanism. Of course, we could also change
StreamOperator.finish() into .finish(boolean flush) and translate a
MAX_WATERMARK into .finish(true).
What would it look like if the source reaches its end?
Upon reaching the end, the source would send a MAX_WATERMARK followed by an
EndOfInputEvent and then go into the FINISHED state. This will cause the
StreamTask to go into the WAITING_FOR_FINAL_CHECKPOINT state. The
downstream tasks upon receiving MAX_WATERMARK and EndOfInputEvent on all
channels, would first flush all their data (because of MAX_WATERMARK), then
forward MAX_WATERMARK to its downstream operator, go into the FINISHED
state (because of EndOfInputEvent) and finally also sent an EndOfInputEvent
to its downstream operators. The owning StreamTask would also go into the
WAITING_FOR_FINAL_CHECKPOINT state. And all StreamOperators and StreamTasks
would terminate after the next successful checkpoint. That would then
trigger the EndOfPartitionEvent that will terminate the TCP connections.
What would it look like if we trigger stop-with-savepoint w/ drain?
We would send our sources a signal that they should stop processing and
advance the time. This means that they emit a MAX_WATERMARK and then go
into the FINISHED state (meaning that they send an EndOfInputEvent). Then
everything is the same as with the first case only that we will trigger a
savepoint just after sending the stop processing signal.
What would it look like if we trigger stop-with-savepoint w/o drain?
We would send our sources a signal that they should stop processing. This
means that the sources go into FINISHED which means that we are sending an
EndOfInputEvent. Then everything is the same as with the first case only
that we will trigger a savepoint just after sending the stop processing
signal.
One thing I would like to emphasize here is that the
StreamOperators/StreamTasks don't have to know anything about a specific
savepoint/checkpoint. They simply follow their state machine that says that
you have to wait for a checkpoint if you want to stop. It just happens that
in the case of stop-with-savepoint the next checkpoint will be a savepoint.
That way we decouple the lifecycle management and the stop-with-savepoint
operations a bit better.
Cheers,
Till
On Fri, Jul 16, 2021 at 2:22 PM Yun Gao <yu...@aliyun.com> wrote:
> Hi Till,
>
> > Ok, so the plan is that finish() will flush all pending events and then
> send the MAX_WATERMARK.
>
> > What I am missing is the connection between the lifecycle of the
> operator and signals the StreamTask and Task might receive (and also their
> life cycles).
> > At the moment we do have the EndOfPartitionEvent that we send once the
> Task reaches its end. Based on what signals will
> > StreamOperator.finish() be called? What kind of other signals for the
> StreamTask do we have to introduce in order to support stop-with-savepoint
> w/o --drain if we don't want to call finish()?
>
> Currently the plan is to introduce a new event (like EndofInputEvent) to
> first notify all the tasks from sources to
> sink to first call finish() (or skip finish() if stop-with-savepoint),
> then the tasks would wait for the last checkpoint and the finally emit
> EndOfPartition. We could
> add a flag to `EndofInputEvent` to indicating if the finish() method
> would be called.
>
> From the view of a single task, the life cycle of a source task would
> looks like
>
> 1. Open & Initialize
> 2. Emits all the records
> 3. Call finish() for all the operators
> 4. Emit MAX_WATERMARK
> 5. Emit EndofInputEvent {finish = true}
> 6. Wait for the checkpoint trigger, emit CheckpointBarrier and wait for
> the notification of the final checkpoint if needed.
> 7. Call close() for all the operators
> 8. Emit EndOfPartitionEvent
>
>
> for normal finish. For stop-with-savepoint [--drain], it would looks like
>
> 1. Open & Initialize
> 2. Received stop-with-savepoint (--drain) request
> 3. Finish the task (e.g. interrupt the legacy source thread or suspend the
> mailbox for the new source).
> 4. Call finish() for all the operators.
> 5. Emit MAX_WATERMARK if --drain
> 6. Emit EndofInputEvent {finish = is drain}
> 7. Emit CheckpointBarrier for this savepoint and wait for the
> notification of the final savepoint if needed.
> 8. Call close() for all the operators
> 9. Emit EndOfPartitionEvent
>
> Then the lifecycle of a non-source task would be
>
> 1. Open & Initialize
> 2. runMailboxLoop()
> 3. Received MAX_WATERMARK from all the channels. --> trigger all the
> timers.
> 4. Received EndofInputEvent from all the input channels --> Call
> finish() for all the operators.
> 5. Emit MAX_WATERMARK if --drain (this is nature with the watermark
> alignment mechanism)
> 6. Emit the received EndofInputEvent.
> 7. Wait for the checkpoint barrier aligned, emit CheckpointBarrier and
> wait for the notification of the final checkpoint / savepoint if needed.
> 8. Call close() for all the operators
> 9. Wait for received all EndOfPartitionEvent, release network resources
> and emit EndOfPartitionEvent
>
> > If finish() sends MAX_WATERMARK after flushing all pending events and
> finish() itself is being triggered by some endOfInputEvent from the
> StreamTask, then we might actually send MAX_WATERMARK multiple times.
>
> I think we might not send MAX_WATERMARK multiple times in that for
> non-source tasks, both MAX_WATERMARK and EndOfInputEvent would be
> processed only after they are received from event input channels.
> Then after process the current task would emit MAX_WATERMARK and EndOfInputEvent
> again to the following tasks, thus both the two events would only be sent
> once for each task ?
>
> Best,
> Yun
>
>
>
> ------------------------------------------------------------------
> From:Till Rohrmann <tr...@apache.org>
> Send Time:2021 Jul. 16 (Fri.) 17:26
> To:Yun Gao <yu...@aliyun.com>
> Cc:dev <de...@flink.apache.org>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> Ok, so the plan is that finish() will flush all pending events and then
> send the MAX_WATERMARK.
>
> What I am missing is the connection between the lifecycle of the operator
> and signals the StreamTask and Task might receive (and also their life
> cycles). At the moment we do have the EndOfPartitionEvent that we send once
> the Task reaches its end. Based on what signals will
> StreamOperator.finish() be called? What kind of other signals for the
> StreamTask do we have to introduce in order to support stop-with-savepoint
> w/o --drain if we don't want to call finish()?
>
> If finish() sends MAX_WATERMARK after flushing all pending events and
> finish() itself is being triggered by some endOfInputEvent from the
> StreamTask, then we might actually send MAX_WATERMARK multiple times.
>
> I think what I am lacking a bit is how the StreamOperator lifecycle will
> fit together with the lifecycle of its owner and the corresponding signals
> we have to send.
>
> Cheers,
> Till
>
> Cheers,
> Till
>
> On Fri, Jul 16, 2021 at 10:40 AM Yun Gao <yu...@aliyun.com> wrote:
> Hi Till,
>
> Sorry that I do not see the reply when sending the last mail, I think as a
> whole we are on the same page for the
> 1. For normal stop-with-savepoint, we do not call finish() and do not emit
> MAX_WATERMARK
> 2. For normal finish / stop-with-savepoint --drain, we would call finish()
> and emit MAX_WATERMARK
>
> > But then there is the question, how do we signal the operator that the
> next checkpoint is supposed to stop the operator
> > (how will the operator's lifecycle look in this case)? Maybe we simply
> don't tell the operator and handle this situation on the
> > StreamTask level.
>
> Logically I think in this case UDF seems do not need to know the next
> checkpoint is supposed to stop the operator since the final
> checkpoint in this case have no difference with the ordinary checkpoints.
>
> > So I guess the question is will finish() advance the time to the end or
> is this a separate mechanism (e.g. explicit watermarks).
>
> I tend to have an explicit MAX_WATERMARK since it makes watermark
> processing to be unified with normal cases and make the meanings of
> each event explicit. But this might be a private preference and both
> methods would work.
>
> Very sorry for not making the whole thing clear in the FLIP again, if
> there are no other concerns I'll update the FLIP with the above conclusions
> to make it precise in this part.
>
>
> Best,
> Yun
>
> ------------------------------------------------------------------
> From:Till Rohrmann <tr...@apache.org>
> Send Time:2021 Jul. 16 (Fri.) 16:00
> To:dev <de...@flink.apache.org>
> Cc:Yun Gao <yu...@aliyun.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> I think we should try to sort this out because it might affect how and
> when finish() will be called (or in general how the operator lifecycle
> looks like).
>
> To give an example let's take a look at the stop-with-savepoint w/ and w/o
> --drain:
>
> 1) stop-with-savepoint w/o --drain: Conceptually what we would like to do
> is to stop processing w/o completing any windows or waiting for the
> AsyncOperator to finish its operations. All these unfinished operations
> should become part of the final checkpoint so that we can resume from it at
> a later point. Depending on what finish() does (flush unfinished windows or
> not), this method must or must not be called. Assuming that finish()
> flushes unfinished windows/waits for uncompleted async operations, we
> clearly shouldn't call it. But then there is the question, how do we
> signal the operator that the next checkpoint is supposed to stop the
> operator (how will the operator's lifecycle look in this case)? Maybe we
> simply don't tell the operator and handle this situation on the StreamTask
> level. If finish() does not flush unfinished windows, then it shouldn't be
> a problem.
>
> 2) stop-with-savepoint w/ --drain: Here we want to complete all pending
> operations and flush out all results because we don't intend to resume the
> job. Conceptually, we tell the system that we have reached MAX_WATERMARK.
> If finish() is defined so that it implicitly advances the watermark to
> MAX_WATERMARK, then there is no problem. If finish() does not have this
> semantic, then we need to send the MAX_WATERMARK before sending the
> endOfInput event to a downstream task. In fact, stop-with-savepoint /w
> --drain shouldn't be a lot different from a bounded source that reaches its
> end. It would also send MAX_WATERMARK and then signal the endOfInput event
> (note that endOfInput is decoupled from the event time here).
>
> So I guess the question is will finish() advance the time to the end or is
> this a separate mechanism (e.g. explicit watermarks).
>
> Concerning how to handle processing time, I am a bit unsure tbh. I can see
> arguments for completing processing time windows/firing processing time
> timers when calling stop-with-savepoint w/ --drain. On the other hand, I
> could also see that people want to define actions based on the wall clock
> time that are independent of the stream state and, thus, would want to
> ignore them if the Flink application is stopped before reaching this time.
>
> Cheers,
> Till
>
> On Fri, Jul 16, 2021 at 7:48 AM Piotr Nowojski <pn...@apache.org>
> wrote:
> Hi Till,
>
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
>
> 1) currently they are independent but parallel mechanisms. With event time,
> they are basically the same.
> 2) it probably should for the sake of processing time windows.
>
> Here you are touching the bit of the current design that I like the least.
> We basically have now three different ways of conveying very similar
> things:
> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
> processing time?)
> b) endInput(), used for example by AsyncWaitOperator to flush it's internal
> state
> c) finish(), used for example by ContinuousFileReaderOperator
>
> It's a bit messy and I'm not sure if this should be strengthened out? Each
> one of those has a little bit different semantic/meaning, but at the same
> time they are very similar. For single input operators `endInput()` and
> `finish()` are actually the very same thing.
>
> Piotrek
>
> czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
>
> > Thanks for updating the FLIP. Based on the new section about
> > stop-with-savepoint [--drain] I got two other questions:
> >
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
> >
> > Cheers,
> > Till
> >
> > On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Thanks a lot for your answers and clarifications Yun.
> > >
> > > 1+2) Agreed, this can be a future improvement if this becomes a
> problem.
> > >
> > > 3) Great, this will help a lot with understanding the FLIP.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > > wrote:
> > >
> > >> Hi Till,
> > >>
> > >> Very thanks for the review and comments!
> > >>
> > >> 1) First I think in fact we could be able to do the computation
> outside
> > >> of the main thread,
> > >> and the current implementation mainly due to the computation is in
> > >> general fast and we
> > >> initially want to have a simplified first version.
> > >>
> > >> The main requirement here is to have a constant view of the state of
> the
> > >> tasks, otherwise
> > >> for example if we have A -> B, if A is running when we check if we
> need
> > >> to trigger A, we will
> > >> mark A as have to trigger, but if A gets to finished when we check B,
> we
> > >> will also mark B as
> > >> have to trigger, then B will receive both rpc trigger and checkpoint
> > >> barrier, which would break
> > >> some assumption on the task side and complicate the implementation.
> > >>
> > >> But to cope this issue, we in fact could first have a snapshot of the
> > >> tasks' state and then do the
> > >> computation, both the two step do not need to be in the main thread.
> > >>
> > >> 2) For the computation logic, in fact currently we benefit a lot from
> > >> some shortcuts on all-to-all
> > >> edges and job vertex with all tasks running, these shortcuts could do
> > >> checks on the job vertex level
> > >> first and skip some job vertices as a whole. With this optimization we
> > >> have a O(V) algorithm, and the
> > >> current running time of the worst case for a job with 320,000 tasks is
> > >> less than 100ms. For
> > >> daily graph sizes the time would be further reduced linearly.
> > >>
> > >> If we do the computation based on the last triggered tasks, we may not
> > >> easily encode this information
> > >> into the shortcuts on the job vertex level. And since the time seems
> to
> > >> be short, perhaps it is enough
> > >> to do re-computation from the scratch in consideration of the tradeoff
> > >> between the performance and
> > >> the complexity ?
> > >>
> > >> 3) We are going to emit the EndOfInput event exactly after the
> finish()
> > >> method and before the last
> > >> snapshotState() method so that we could shut down the whole topology
> > with
> > >> a single final checkpoint.
> > >> Very sorry for not include enough details for this part and I'll
> > >> complement the FLIP with the details on
> > >> the process of the final checkpoint / savepoint.
> > >>
> > >> Best,
> > >> Yun
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------------
> > >> From:Till Rohrmann <tr...@apache.org>
> > >> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >> To:dev <de...@flink.apache.org>
> > >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >> Finished
> > >>
> > >> Hi everyone,
> > >>
> > >> I am a bit late to the voting party but let me ask three questions:
> > >>
> > >> 1) Why do we execute the trigger plan computation in the main thread
> if
> > we
> > >> cannot guarantee that all tasks are still running when triggering the
> > >> checkpoint? Couldn't we do the computation in a different thread in
> > order
> > >> to relieve the main thread a bit.
> > >>
> > >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to
> go
> > >> over the whole topology for every calculation. Wouldn't it be more
> > >> efficient to maintain the set of current tasks to trigger and check
> > >> whether
> > >> anything has changed and if so check the succeeding tasks until we
> have
> > >> found the current checkpoint trigger frontier?
> > >>
> > >> 3) When are we going to send the endOfInput events to a downstream
> task?
> > >> If
> > >> this happens after we call finish on the upstream operator but before
> > >> snapshotState then it would be possible to shut down the whole
> topology
> > >> with a single final checkpoint. I think this part could benefit from a
> > bit
> > >> more detailed description in the FLIP.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> > >> wrote:
> > >>
> > >> > Hi there,
> > >> >
> > >> > Since the voting time of FLIP-147[1] has passed, I'm closing the
> vote
> > >> now.
> > >> >
> > >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >> >
> > >> > - Dawid Wysakowicz (binding)
> > >> > - Piotr Nowojski(binding)
> > >> > - Jiangang Liu (binding)
> > >> > - Arvid Heise (binding)
> > >> > - Jing Zhang (binding)
> > >> > - Leonard Xu (non-binding)
> > >> > - Guowei Ma (binding)
> > >> >
> > >> > Thus I'm happy to announce that the update to the FLIP-147 is
> > accepted.
> > >> >
> > >> > Very thanks everyone!
> > >> >
> > >> > Best,
> > >> > Yun
> > >> >
> > >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>
> > >>
> >
>
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Till,
> Ok, so the plan is that finish() will flush all pending events and then send the MAX_WATERMARK.
> What I am missing is the connection between the lifecycle of the operator and signals the StreamTask and Task might receive (and also their life cycles).
> At the moment we do have the EndOfPartitionEvent that we send once the Task reaches its end. Based on what signals will
> StreamOperator.finish() be called? What kind of other signals for the StreamTask do we have to introduce in order to support stop-with-savepoint w/o --drain if we don't want to call finish()?
Currently the plan is to introduce a new event (like EndofInputEvent) to first notify all the tasks from sources to
sink to first call finish() (or skip finish() if stop-with-savepoint), then the tasks would wait for the last checkpoint and the finally emit EndOfPartition. We could
add a flag to `EndofInputEvent` to indicating if the finish() method would be called.
From the view of a single task, the life cycle of a source task would looks like
1. Open & Initialize
2. Emits all the records
3. Call finish() for all the operators
4. Emit MAX_WATERMARK
5. Emit EndofInputEvent {finish = true}
6. Wait for the checkpoint trigger, emit CheckpointBarrier and wait for the notification of the final checkpoint if needed.
7. Call close() for all the operators
8. Emit EndOfPartitionEvent
for normal finish. For stop-with-savepoint [--drain], it would looks like
1. Open & Initialize
2. Received stop-with-savepoint (--drain) request
3. Finish the task (e.g. interrupt the legacy source thread or suspend the mailbox for the new source).
4. Call finish() for all the operators.
5. Emit MAX_WATERMARK if --drain
6. Emit EndofInputEvent {finish = is drain}
7. Emit CheckpointBarrier for this savepoint and wait for the notification of the final savepoint if needed.
8. Call close() for all the operators
9. Emit EndOfPartitionEvent
Then the lifecycle of a non-source task would be
1. Open & Initialize
2. runMailboxLoop()
3. Received MAX_WATERMARK from all the channels. --> trigger all the timers.
4. Received EndofInputEvent from all the input channels --> Call finish() for all the operators.
5. Emit MAX_WATERMARK if --drain (this is nature with the watermark alignment mechanism)
6. Emit the received EndofInputEvent.
7. Wait for the checkpoint barrier aligned, emit CheckpointBarrier and wait for the notification of the final checkpoint / savepoint if needed.
8. Call close() for all the operators
9. Wait for received all EndOfPartitionEvent, release network resources and emit EndOfPartitionEvent
> If finish() sends MAX_WATERMARK after flushing all pending events and finish() itself is being triggered by some endOfInputEvent from the StreamTask, then we might actually send MAX_WATERMARK multiple times.
I think we might not send MAX_WATERMARK multiple times in that for non-source tasks, both MAX_WATERMARK and EndOfInputEvent would be processed only after they are received from event input channels.
Then after process the current task would emit MAX_WATERMARK and EndOfInputEvent again to the following tasks, thus both the two events would only be sent once for each task ?
Best,
Yun
------------------------------------------------------------------
From:Till Rohrmann <tr...@apache.org>
Send Time:2021 Jul. 16 (Fri.) 17:26
To:Yun Gao <yu...@aliyun.com>
Cc:dev <de...@flink.apache.org>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Ok, so the plan is that finish() will flush all pending events and then send the MAX_WATERMARK.
What I am missing is the connection between the lifecycle of the operator and signals the StreamTask and Task might receive (and also their life cycles). At the moment we do have the EndOfPartitionEvent that we send once the Task reaches its end. Based on what signals will StreamOperator.finish() be called? What kind of other signals for the StreamTask do we have to introduce in order to support stop-with-savepoint w/o --drain if we don't want to call finish()?
If finish() sends MAX_WATERMARK after flushing all pending events and finish() itself is being triggered by some endOfInputEvent from the StreamTask, then we might actually send MAX_WATERMARK multiple times.
I think what I am lacking a bit is how the StreamOperator lifecycle will fit together with the lifecycle of its owner and the corresponding signals we have to send.
Cheers,
Till
Cheers,
Till
On Fri, Jul 16, 2021 at 10:40 AM Yun Gao <yu...@aliyun.com> wrote:
Hi Till,
Sorry that I do not see the reply when sending the last mail, I think as a whole we are on the same page for the
1. For normal stop-with-savepoint, we do not call finish() and do not emit MAX_WATERMARK
2. For normal finish / stop-with-savepoint --drain, we would call finish() and emit MAX_WATERMARK
> But then there is the question, how do we signal the operator that the next checkpoint is supposed to stop the operator
> (how will the operator's lifecycle look in this case)? Maybe we simply don't tell the operator and handle this situation on the
> StreamTask level.
Logically I think in this case UDF seems do not need to know the next checkpoint is supposed to stop the operator since the final
checkpoint in this case have no difference with the ordinary checkpoints.
> So I guess the question is will finish() advance the time to the end or is this a separate mechanism (e.g. explicit watermarks).
I tend to have an explicit MAX_WATERMARK since it makes watermark processing to be unified with normal cases and make the meanings of
each event explicit. But this might be a private preference and both methods would work.
Very sorry for not making the whole thing clear in the FLIP again, if there are no other concerns I'll update the FLIP with the above conclusions
to make it precise in this part.
Best,
Yun
------------------------------------------------------------------
From:Till Rohrmann <tr...@apache.org>
Send Time:2021 Jul. 16 (Fri.) 16:00
To:dev <de...@flink.apache.org>
Cc:Yun Gao <yu...@aliyun.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
I think we should try to sort this out because it might affect how and when finish() will be called (or in general how the operator lifecycle looks like).
To give an example let's take a look at the stop-with-savepoint w/ and w/o --drain:
1) stop-with-savepoint w/o --drain: Conceptually what we would like to do is to stop processing w/o completing any windows or waiting for the AsyncOperator to finish its operations. All these unfinished operations should become part of the final checkpoint so that we can resume from it at a later point. Depending on what finish() does (flush unfinished windows or not), this method must or must not be called. Assuming that finish() flushes unfinished windows/waits for uncompleted async operations, we clearly shouldn't call it. But then there is the question, how do we signal the operator that the next checkpoint is supposed to stop the operator (how will the operator's lifecycle look in this case)? Maybe we simply don't tell the operator and handle this situation on the StreamTask level. If finish() does not flush unfinished windows, then it shouldn't be a problem.
2) stop-with-savepoint w/ --drain: Here we want to complete all pending operations and flush out all results because we don't intend to resume the job. Conceptually, we tell the system that we have reached MAX_WATERMARK. If finish() is defined so that it implicitly advances the watermark to MAX_WATERMARK, then there is no problem. If finish() does not have this semantic, then we need to send the MAX_WATERMARK before sending the endOfInput event to a downstream task. In fact, stop-with-savepoint /w --drain shouldn't be a lot different from a bounded source that reaches its end. It would also send MAX_WATERMARK and then signal the endOfInput event (note that endOfInput is decoupled from the event time here).
So I guess the question is will finish() advance the time to the end or is this a separate mechanism (e.g. explicit watermarks).
Concerning how to handle processing time, I am a bit unsure tbh. I can see arguments for completing processing time windows/firing processing time timers when calling stop-with-savepoint w/ --drain. On the other hand, I could also see that people want to define actions based on the wall clock time that are independent of the stream state and, thus, would want to ignore them if the Flink application is stopped before reaching this time.
Cheers,
Till
On Fri, Jul 16, 2021 at 7:48 AM Piotr Nowojski <pn...@apache.org> wrote:
Hi Till,
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
1) currently they are independent but parallel mechanisms. With event time,
they are basically the same.
2) it probably should for the sake of processing time windows.
Here you are touching the bit of the current design that I like the least.
We basically have now three different ways of conveying very similar things:
a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
processing time?)
b) endInput(), used for example by AsyncWaitOperator to flush it's internal
state
c) finish(), used for example by ContinuousFileReaderOperator
It's a bit messy and I'm not sure if this should be strengthened out? Each
one of those has a little bit different semantic/meaning, but at the same
time they are very similar. For single input operators `endInput()` and
`finish()` are actually the very same thing.
Piotrek
czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
> Thanks for updating the FLIP. Based on the new section about
> stop-with-savepoint [--drain] I got two other questions:
>
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
>
> Cheers,
> Till
>
> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Thanks a lot for your answers and clarifications Yun.
> >
> > 1+2) Agreed, this can be a future improvement if this becomes a problem.
> >
> > 3) Great, this will help a lot with understanding the FLIP.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > wrote:
> >
> >> Hi Till,
> >>
> >> Very thanks for the review and comments!
> >>
> >> 1) First I think in fact we could be able to do the computation outside
> >> of the main thread,
> >> and the current implementation mainly due to the computation is in
> >> general fast and we
> >> initially want to have a simplified first version.
> >>
> >> The main requirement here is to have a constant view of the state of the
> >> tasks, otherwise
> >> for example if we have A -> B, if A is running when we check if we need
> >> to trigger A, we will
> >> mark A as have to trigger, but if A gets to finished when we check B, we
> >> will also mark B as
> >> have to trigger, then B will receive both rpc trigger and checkpoint
> >> barrier, which would break
> >> some assumption on the task side and complicate the implementation.
> >>
> >> But to cope this issue, we in fact could first have a snapshot of the
> >> tasks' state and then do the
> >> computation, both the two step do not need to be in the main thread.
> >>
> >> 2) For the computation logic, in fact currently we benefit a lot from
> >> some shortcuts on all-to-all
> >> edges and job vertex with all tasks running, these shortcuts could do
> >> checks on the job vertex level
> >> first and skip some job vertices as a whole. With this optimization we
> >> have a O(V) algorithm, and the
> >> current running time of the worst case for a job with 320,000 tasks is
> >> less than 100ms. For
> >> daily graph sizes the time would be further reduced linearly.
> >>
> >> If we do the computation based on the last triggered tasks, we may not
> >> easily encode this information
> >> into the shortcuts on the job vertex level. And since the time seems to
> >> be short, perhaps it is enough
> >> to do re-computation from the scratch in consideration of the tradeoff
> >> between the performance and
> >> the complexity ?
> >>
> >> 3) We are going to emit the EndOfInput event exactly after the finish()
> >> method and before the last
> >> snapshotState() method so that we could shut down the whole topology
> with
> >> a single final checkpoint.
> >> Very sorry for not include enough details for this part and I'll
> >> complement the FLIP with the details on
> >> the process of the final checkpoint / savepoint.
> >>
> >> Best,
> >> Yun
> >>
> >>
> >>
> >> ------------------------------------------------------------------
> >> From:Till Rohrmann <tr...@apache.org>
> >> Send Time:2021 Jul. 14 (Wed.) 22:05
> >> To:dev <de...@flink.apache.org>
> >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >> Finished
> >>
> >> Hi everyone,
> >>
> >> I am a bit late to the voting party but let me ask three questions:
> >>
> >> 1) Why do we execute the trigger plan computation in the main thread if
> we
> >> cannot guarantee that all tasks are still running when triggering the
> >> checkpoint? Couldn't we do the computation in a different thread in
> order
> >> to relieve the main thread a bit.
> >>
> >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
> >> over the whole topology for every calculation. Wouldn't it be more
> >> efficient to maintain the set of current tasks to trigger and check
> >> whether
> >> anything has changed and if so check the succeeding tasks until we have
> >> found the current checkpoint trigger frontier?
> >>
> >> 3) When are we going to send the endOfInput events to a downstream task?
> >> If
> >> this happens after we call finish on the upstream operator but before
> >> snapshotState then it would be possible to shut down the whole topology
> >> with a single final checkpoint. I think this part could benefit from a
> bit
> >> more detailed description in the FLIP.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> >> wrote:
> >>
> >> > Hi there,
> >> >
> >> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
> >> now.
> >> >
> >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >> >
> >> > - Dawid Wysakowicz (binding)
> >> > - Piotr Nowojski(binding)
> >> > - Jiangang Liu (binding)
> >> > - Arvid Heise (binding)
> >> > - Jing Zhang (binding)
> >> > - Leonard Xu (non-binding)
> >> > - Guowei Ma (binding)
> >> >
> >> > Thus I'm happy to announce that the update to the FLIP-147 is
> accepted.
> >> >
> >> > Very thanks everyone!
> >> >
> >> > Best,
> >> > Yun
> >> >
> >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>
> >>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
Ok, so the plan is that finish() will flush all pending events and then
send the MAX_WATERMARK.
What I am missing is the connection between the lifecycle of the operator
and signals the StreamTask and Task might receive (and also their life
cycles). At the moment we do have the EndOfPartitionEvent that we send once
the Task reaches its end. Based on what signals will
StreamOperator.finish() be called? What kind of other signals for the
StreamTask do we have to introduce in order to support stop-with-savepoint
w/o --drain if we don't want to call finish()?
If finish() sends MAX_WATERMARK after flushing all pending events and
finish() itself is being triggered by some endOfInputEvent from the
StreamTask, then we might actually send MAX_WATERMARK multiple times.
I think what I am lacking a bit is how the StreamOperator lifecycle will
fit together with the lifecycle of its owner and the corresponding signals
we have to send.
Cheers,
Till
Cheers,
Till
On Fri, Jul 16, 2021 at 10:40 AM Yun Gao <yu...@aliyun.com> wrote:
> Hi Till,
>
> Sorry that I do not see the reply when sending the last mail, I think as a
> whole we are on the same page for the
> 1. For normal stop-with-savepoint, we do not call finish() and do not emit
> MAX_WATERMARK
> 2. For normal finish / stop-with-savepoint --drain, we would call finish()
> and emit MAX_WATERMARK
>
> > But then there is the question, how do we signal the operator that the
> next checkpoint is supposed to stop the operator
> > (how will the operator's lifecycle look in this case)? Maybe we simply
> don't tell the operator and handle this situation on the
> > StreamTask level.
>
> Logically I think in this case UDF seems do not need to know the next
> checkpoint is supposed to stop the operator since the final
> checkpoint in this case have no difference with the ordinary checkpoints.
>
> > So I guess the question is will finish() advance the time to the end or
> is this a separate mechanism (e.g. explicit watermarks).
>
> I tend to have an explicit MAX_WATERMARK since it makes watermark
> processing to be unified with normal cases and make the meanings of
> each event explicit. But this might be a private preference and both
> methods would work.
>
> Very sorry for not making the whole thing clear in the FLIP again, if
> there are no other concerns I'll update the FLIP with the above conclusions
> to make it precise in this part.
>
>
> Best,
> Yun
>
> ------------------------------------------------------------------
> From:Till Rohrmann <tr...@apache.org>
> Send Time:2021 Jul. 16 (Fri.) 16:00
> To:dev <de...@flink.apache.org>
> Cc:Yun Gao <yu...@aliyun.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> I think we should try to sort this out because it might affect how and
> when finish() will be called (or in general how the operator lifecycle
> looks like).
>
> To give an example let's take a look at the stop-with-savepoint w/ and w/o
> --drain:
>
> 1) stop-with-savepoint w/o --drain: Conceptually what we would like to do
> is to stop processing w/o completing any windows or waiting for the
> AsyncOperator to finish its operations. All these unfinished operations
> should become part of the final checkpoint so that we can resume from it at
> a later point. Depending on what finish() does (flush unfinished windows or
> not), this method must or must not be called. Assuming that finish()
> flushes unfinished windows/waits for uncompleted async operations, we
> clearly shouldn't call it. But then there is the question, how do we
> signal the operator that the next checkpoint is supposed to stop the
> operator (how will the operator's lifecycle look in this case)? Maybe we
> simply don't tell the operator and handle this situation on the StreamTask
> level. If finish() does not flush unfinished windows, then it shouldn't be
> a problem.
>
> 2) stop-with-savepoint w/ --drain: Here we want to complete all pending
> operations and flush out all results because we don't intend to resume the
> job. Conceptually, we tell the system that we have reached MAX_WATERMARK.
> If finish() is defined so that it implicitly advances the watermark to
> MAX_WATERMARK, then there is no problem. If finish() does not have this
> semantic, then we need to send the MAX_WATERMARK before sending the
> endOfInput event to a downstream task. In fact, stop-with-savepoint /w
> --drain shouldn't be a lot different from a bounded source that reaches its
> end. It would also send MAX_WATERMARK and then signal the endOfInput event
> (note that endOfInput is decoupled from the event time here).
>
> So I guess the question is will finish() advance the time to the end or is
> this a separate mechanism (e.g. explicit watermarks).
>
> Concerning how to handle processing time, I am a bit unsure tbh. I can see
> arguments for completing processing time windows/firing processing time
> timers when calling stop-with-savepoint w/ --drain. On the other hand, I
> could also see that people want to define actions based on the wall clock
> time that are independent of the stream state and, thus, would want to
> ignore them if the Flink application is stopped before reaching this time.
>
> Cheers,
> Till
>
> On Fri, Jul 16, 2021 at 7:48 AM Piotr Nowojski <pn...@apache.org>
> wrote:
> Hi Till,
>
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
>
> 1) currently they are independent but parallel mechanisms. With event time,
> they are basically the same.
> 2) it probably should for the sake of processing time windows.
>
> Here you are touching the bit of the current design that I like the least.
> We basically have now three different ways of conveying very similar
> things:
> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
> processing time?)
> b) endInput(), used for example by AsyncWaitOperator to flush it's internal
> state
> c) finish(), used for example by ContinuousFileReaderOperator
>
> It's a bit messy and I'm not sure if this should be strengthened out? Each
> one of those has a little bit different semantic/meaning, but at the same
> time they are very similar. For single input operators `endInput()` and
> `finish()` are actually the very same thing.
>
> Piotrek
>
> czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
>
> > Thanks for updating the FLIP. Based on the new section about
> > stop-with-savepoint [--drain] I got two other questions:
> >
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
> >
> > Cheers,
> > Till
> >
> > On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Thanks a lot for your answers and clarifications Yun.
> > >
> > > 1+2) Agreed, this can be a future improvement if this becomes a
> problem.
> > >
> > > 3) Great, this will help a lot with understanding the FLIP.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > > wrote:
> > >
> > >> Hi Till,
> > >>
> > >> Very thanks for the review and comments!
> > >>
> > >> 1) First I think in fact we could be able to do the computation
> outside
> > >> of the main thread,
> > >> and the current implementation mainly due to the computation is in
> > >> general fast and we
> > >> initially want to have a simplified first version.
> > >>
> > >> The main requirement here is to have a constant view of the state of
> the
> > >> tasks, otherwise
> > >> for example if we have A -> B, if A is running when we check if we
> need
> > >> to trigger A, we will
> > >> mark A as have to trigger, but if A gets to finished when we check B,
> we
> > >> will also mark B as
> > >> have to trigger, then B will receive both rpc trigger and checkpoint
> > >> barrier, which would break
> > >> some assumption on the task side and complicate the implementation.
> > >>
> > >> But to cope this issue, we in fact could first have a snapshot of the
> > >> tasks' state and then do the
> > >> computation, both the two step do not need to be in the main thread.
> > >>
> > >> 2) For the computation logic, in fact currently we benefit a lot from
> > >> some shortcuts on all-to-all
> > >> edges and job vertex with all tasks running, these shortcuts could do
> > >> checks on the job vertex level
> > >> first and skip some job vertices as a whole. With this optimization we
> > >> have a O(V) algorithm, and the
> > >> current running time of the worst case for a job with 320,000 tasks is
> > >> less than 100ms. For
> > >> daily graph sizes the time would be further reduced linearly.
> > >>
> > >> If we do the computation based on the last triggered tasks, we may not
> > >> easily encode this information
> > >> into the shortcuts on the job vertex level. And since the time seems
> to
> > >> be short, perhaps it is enough
> > >> to do re-computation from the scratch in consideration of the tradeoff
> > >> between the performance and
> > >> the complexity ?
> > >>
> > >> 3) We are going to emit the EndOfInput event exactly after the
> finish()
> > >> method and before the last
> > >> snapshotState() method so that we could shut down the whole topology
> > with
> > >> a single final checkpoint.
> > >> Very sorry for not include enough details for this part and I'll
> > >> complement the FLIP with the details on
> > >> the process of the final checkpoint / savepoint.
> > >>
> > >> Best,
> > >> Yun
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------------
> > >> From:Till Rohrmann <tr...@apache.org>
> > >> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >> To:dev <de...@flink.apache.org>
> > >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >> Finished
> > >>
> > >> Hi everyone,
> > >>
> > >> I am a bit late to the voting party but let me ask three questions:
> > >>
> > >> 1) Why do we execute the trigger plan computation in the main thread
> if
> > we
> > >> cannot guarantee that all tasks are still running when triggering the
> > >> checkpoint? Couldn't we do the computation in a different thread in
> > order
> > >> to relieve the main thread a bit.
> > >>
> > >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to
> go
> > >> over the whole topology for every calculation. Wouldn't it be more
> > >> efficient to maintain the set of current tasks to trigger and check
> > >> whether
> > >> anything has changed and if so check the succeeding tasks until we
> have
> > >> found the current checkpoint trigger frontier?
> > >>
> > >> 3) When are we going to send the endOfInput events to a downstream
> task?
> > >> If
> > >> this happens after we call finish on the upstream operator but before
> > >> snapshotState then it would be possible to shut down the whole
> topology
> > >> with a single final checkpoint. I think this part could benefit from a
> > bit
> > >> more detailed description in the FLIP.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> > >> wrote:
> > >>
> > >> > Hi there,
> > >> >
> > >> > Since the voting time of FLIP-147[1] has passed, I'm closing the
> vote
> > >> now.
> > >> >
> > >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >> >
> > >> > - Dawid Wysakowicz (binding)
> > >> > - Piotr Nowojski(binding)
> > >> > - Jiangang Liu (binding)
> > >> > - Arvid Heise (binding)
> > >> > - Jing Zhang (binding)
> > >> > - Leonard Xu (non-binding)
> > >> > - Guowei Ma (binding)
> > >> >
> > >> > Thus I'm happy to announce that the update to the FLIP-147 is
> > accepted.
> > >> >
> > >> > Very thanks everyone!
> > >> >
> > >> > Best,
> > >> > Yun
> > >> >
> > >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>
> > >>
> >
>
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Till,
Sorry that I do not see the reply when sending the last mail, I think as a whole we are on the same page for the
1. For normal stop-with-savepoint, we do not call finish() and do not emit MAX_WATERMARK
2. For normal finish / stop-with-savepoint --drain, we would call finish() and emit MAX_WATERMARK
> But then there is the question, how do we signal the operator that the next checkpoint is supposed to stop the operator
> (how will the operator's lifecycle look in this case)? Maybe we simply don't tell the operator and handle this situation on the
> StreamTask level.
Logically I think in this case UDF seems do not need to know the next checkpoint is supposed to stop the operator since the final
checkpoint in this case have no difference with the ordinary checkpoints.
> So I guess the question is will finish() advance the time to the end or is this a separate mechanism (e.g. explicit watermarks).
I tend to have an explicit MAX_WATERMARK since it makes watermark processing to be unified with normal cases and make the meanings of
each event explicit. But this might be a private preference and both methods would work.
Very sorry for not making the whole thing clear in the FLIP again, if there are no other concerns I'll update the FLIP with the above conclusions
to make it precise in this part.
Best,
Yun
------------------------------------------------------------------
From:Till Rohrmann <tr...@apache.org>
Send Time:2021 Jul. 16 (Fri.) 16:00
To:dev <de...@flink.apache.org>
Cc:Yun Gao <yu...@aliyun.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
I think we should try to sort this out because it might affect how and when finish() will be called (or in general how the operator lifecycle looks like).
To give an example let's take a look at the stop-with-savepoint w/ and w/o --drain:
1) stop-with-savepoint w/o --drain: Conceptually what we would like to do is to stop processing w/o completing any windows or waiting for the AsyncOperator to finish its operations. All these unfinished operations should become part of the final checkpoint so that we can resume from it at a later point. Depending on what finish() does (flush unfinished windows or not), this method must or must not be called. Assuming that finish() flushes unfinished windows/waits for uncompleted async operations, we clearly shouldn't call it. But then there is the question, how do we signal the operator that the next checkpoint is supposed to stop the operator (how will the operator's lifecycle look in this case)? Maybe we simply don't tell the operator and handle this situation on the StreamTask level. If finish() does not flush unfinished windows, then it shouldn't be a problem.
2) stop-with-savepoint w/ --drain: Here we want to complete all pending operations and flush out all results because we don't intend to resume the job. Conceptually, we tell the system that we have reached MAX_WATERMARK. If finish() is defined so that it implicitly advances the watermark to MAX_WATERMARK, then there is no problem. If finish() does not have this semantic, then we need to send the MAX_WATERMARK before sending the endOfInput event to a downstream task. In fact, stop-with-savepoint /w --drain shouldn't be a lot different from a bounded source that reaches its end. It would also send MAX_WATERMARK and then signal the endOfInput event (note that endOfInput is decoupled from the event time here).
So I guess the question is will finish() advance the time to the end or is this a separate mechanism (e.g. explicit watermarks).
Concerning how to handle processing time, I am a bit unsure tbh. I can see arguments for completing processing time windows/firing processing time timers when calling stop-with-savepoint w/ --drain. On the other hand, I could also see that people want to define actions based on the wall clock time that are independent of the stream state and, thus, would want to ignore them if the Flink application is stopped before reaching this time.
Cheers,
Till
On Fri, Jul 16, 2021 at 7:48 AM Piotr Nowojski <pn...@apache.org> wrote:
Hi Till,
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
1) currently they are independent but parallel mechanisms. With event time,
they are basically the same.
2) it probably should for the sake of processing time windows.
Here you are touching the bit of the current design that I like the least.
We basically have now three different ways of conveying very similar things:
a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
processing time?)
b) endInput(), used for example by AsyncWaitOperator to flush it's internal
state
c) finish(), used for example by ContinuousFileReaderOperator
It's a bit messy and I'm not sure if this should be strengthened out? Each
one of those has a little bit different semantic/meaning, but at the same
time they are very similar. For single input operators `endInput()` and
`finish()` are actually the very same thing.
Piotrek
czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
> Thanks for updating the FLIP. Based on the new section about
> stop-with-savepoint [--drain] I got two other questions:
>
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
>
> Cheers,
> Till
>
> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Thanks a lot for your answers and clarifications Yun.
> >
> > 1+2) Agreed, this can be a future improvement if this becomes a problem.
> >
> > 3) Great, this will help a lot with understanding the FLIP.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > wrote:
> >
> >> Hi Till,
> >>
> >> Very thanks for the review and comments!
> >>
> >> 1) First I think in fact we could be able to do the computation outside
> >> of the main thread,
> >> and the current implementation mainly due to the computation is in
> >> general fast and we
> >> initially want to have a simplified first version.
> >>
> >> The main requirement here is to have a constant view of the state of the
> >> tasks, otherwise
> >> for example if we have A -> B, if A is running when we check if we need
> >> to trigger A, we will
> >> mark A as have to trigger, but if A gets to finished when we check B, we
> >> will also mark B as
> >> have to trigger, then B will receive both rpc trigger and checkpoint
> >> barrier, which would break
> >> some assumption on the task side and complicate the implementation.
> >>
> >> But to cope this issue, we in fact could first have a snapshot of the
> >> tasks' state and then do the
> >> computation, both the two step do not need to be in the main thread.
> >>
> >> 2) For the computation logic, in fact currently we benefit a lot from
> >> some shortcuts on all-to-all
> >> edges and job vertex with all tasks running, these shortcuts could do
> >> checks on the job vertex level
> >> first and skip some job vertices as a whole. With this optimization we
> >> have a O(V) algorithm, and the
> >> current running time of the worst case for a job with 320,000 tasks is
> >> less than 100ms. For
> >> daily graph sizes the time would be further reduced linearly.
> >>
> >> If we do the computation based on the last triggered tasks, we may not
> >> easily encode this information
> >> into the shortcuts on the job vertex level. And since the time seems to
> >> be short, perhaps it is enough
> >> to do re-computation from the scratch in consideration of the tradeoff
> >> between the performance and
> >> the complexity ?
> >>
> >> 3) We are going to emit the EndOfInput event exactly after the finish()
> >> method and before the last
> >> snapshotState() method so that we could shut down the whole topology
> with
> >> a single final checkpoint.
> >> Very sorry for not include enough details for this part and I'll
> >> complement the FLIP with the details on
> >> the process of the final checkpoint / savepoint.
> >>
> >> Best,
> >> Yun
> >>
> >>
> >>
> >> ------------------------------------------------------------------
> >> From:Till Rohrmann <tr...@apache.org>
> >> Send Time:2021 Jul. 14 (Wed.) 22:05
> >> To:dev <de...@flink.apache.org>
> >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >> Finished
> >>
> >> Hi everyone,
> >>
> >> I am a bit late to the voting party but let me ask three questions:
> >>
> >> 1) Why do we execute the trigger plan computation in the main thread if
> we
> >> cannot guarantee that all tasks are still running when triggering the
> >> checkpoint? Couldn't we do the computation in a different thread in
> order
> >> to relieve the main thread a bit.
> >>
> >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
> >> over the whole topology for every calculation. Wouldn't it be more
> >> efficient to maintain the set of current tasks to trigger and check
> >> whether
> >> anything has changed and if so check the succeeding tasks until we have
> >> found the current checkpoint trigger frontier?
> >>
> >> 3) When are we going to send the endOfInput events to a downstream task?
> >> If
> >> this happens after we call finish on the upstream operator but before
> >> snapshotState then it would be possible to shut down the whole topology
> >> with a single final checkpoint. I think this part could benefit from a
> bit
> >> more detailed description in the FLIP.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> >> wrote:
> >>
> >> > Hi there,
> >> >
> >> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
> >> now.
> >> >
> >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >> >
> >> > - Dawid Wysakowicz (binding)
> >> > - Piotr Nowojski(binding)
> >> > - Jiangang Liu (binding)
> >> > - Arvid Heise (binding)
> >> > - Jing Zhang (binding)
> >> > - Leonard Xu (non-binding)
> >> > - Guowei Ma (binding)
> >> >
> >> > Thus I'm happy to announce that the update to the FLIP-147 is
> accepted.
> >> >
> >> > Very thanks everyone!
> >> >
> >> > Best,
> >> > Yun
> >> >
> >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>
> >>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
I think we should try to sort this out because it might affect how and when
finish() will be called (or in general how the operator lifecycle looks
like).
To give an example let's take a look at the stop-with-savepoint w/ and w/o
--drain:
1) stop-with-savepoint w/o --drain: Conceptually what we would like to do
is to stop processing w/o completing any windows or waiting for the
AsyncOperator to finish its operations. All these unfinished operations
should become part of the final checkpoint so that we can resume from it at
a later point. Depending on what finish() does (flush unfinished windows or
not), this method must or must not be called. Assuming that finish()
flushes unfinished windows/waits for uncompleted async operations, we
clearly shouldn't call it. But then there is the question, how do we
signal the operator that the next checkpoint is supposed to stop the
operator (how will the operator's lifecycle look in this case)? Maybe we
simply don't tell the operator and handle this situation on the StreamTask
level. If finish() does not flush unfinished windows, then it shouldn't be
a problem.
2) stop-with-savepoint w/ --drain: Here we want to complete all pending
operations and flush out all results because we don't intend to resume the
job. Conceptually, we tell the system that we have reached MAX_WATERMARK.
If finish() is defined so that it implicitly advances the watermark to
MAX_WATERMARK, then there is no problem. If finish() does not have this
semantic, then we need to send the MAX_WATERMARK before sending the
endOfInput event to a downstream task. In fact, stop-with-savepoint /w
--drain shouldn't be a lot different from a bounded source that reaches its
end. It would also send MAX_WATERMARK and then signal the endOfInput event
(note that endOfInput is decoupled from the event time here).
So I guess the question is will finish() advance the time to the end or is
this a separate mechanism (e.g. explicit watermarks).
Concerning how to handle processing time, I am a bit unsure tbh. I can see
arguments for completing processing time windows/firing processing time
timers when calling stop-with-savepoint w/ --drain. On the other hand, I
could also see that people want to define actions based on the wall clock
time that are independent of the stream state and, thus, would want to
ignore them if the Flink application is stopped before reaching this time.
Cheers,
Till
On Fri, Jul 16, 2021 at 7:48 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi Till,
>
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
>
> 1) currently they are independent but parallel mechanisms. With event time,
> they are basically the same.
> 2) it probably should for the sake of processing time windows.
>
> Here you are touching the bit of the current design that I like the least.
> We basically have now three different ways of conveying very similar
> things:
> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
> processing time?)
> b) endInput(), used for example by AsyncWaitOperator to flush it's internal
> state
> c) finish(), used for example by ContinuousFileReaderOperator
>
> It's a bit messy and I'm not sure if this should be strengthened out? Each
> one of those has a little bit different semantic/meaning, but at the same
> time they are very similar. For single input operators `endInput()` and
> `finish()` are actually the very same thing.
>
> Piotrek
>
> czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
>
> > Thanks for updating the FLIP. Based on the new section about
> > stop-with-savepoint [--drain] I got two other questions:
> >
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > 2) StreamOperator.finish says to flush all buffered events. Would a
> > WindowOperator close all windows and emit the results upon calling
> > finish, for example?
> >
> > Cheers,
> > Till
> >
> > On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Thanks a lot for your answers and clarifications Yun.
> > >
> > > 1+2) Agreed, this can be a future improvement if this becomes a
> problem.
> > >
> > > 3) Great, this will help a lot with understanding the FLIP.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > > wrote:
> > >
> > >> Hi Till,
> > >>
> > >> Very thanks for the review and comments!
> > >>
> > >> 1) First I think in fact we could be able to do the computation
> outside
> > >> of the main thread,
> > >> and the current implementation mainly due to the computation is in
> > >> general fast and we
> > >> initially want to have a simplified first version.
> > >>
> > >> The main requirement here is to have a constant view of the state of
> the
> > >> tasks, otherwise
> > >> for example if we have A -> B, if A is running when we check if we
> need
> > >> to trigger A, we will
> > >> mark A as have to trigger, but if A gets to finished when we check B,
> we
> > >> will also mark B as
> > >> have to trigger, then B will receive both rpc trigger and checkpoint
> > >> barrier, which would break
> > >> some assumption on the task side and complicate the implementation.
> > >>
> > >> But to cope this issue, we in fact could first have a snapshot of the
> > >> tasks' state and then do the
> > >> computation, both the two step do not need to be in the main thread.
> > >>
> > >> 2) For the computation logic, in fact currently we benefit a lot from
> > >> some shortcuts on all-to-all
> > >> edges and job vertex with all tasks running, these shortcuts could do
> > >> checks on the job vertex level
> > >> first and skip some job vertices as a whole. With this optimization we
> > >> have a O(V) algorithm, and the
> > >> current running time of the worst case for a job with 320,000 tasks is
> > >> less than 100ms. For
> > >> daily graph sizes the time would be further reduced linearly.
> > >>
> > >> If we do the computation based on the last triggered tasks, we may not
> > >> easily encode this information
> > >> into the shortcuts on the job vertex level. And since the time seems
> to
> > >> be short, perhaps it is enough
> > >> to do re-computation from the scratch in consideration of the tradeoff
> > >> between the performance and
> > >> the complexity ?
> > >>
> > >> 3) We are going to emit the EndOfInput event exactly after the
> finish()
> > >> method and before the last
> > >> snapshotState() method so that we could shut down the whole topology
> > with
> > >> a single final checkpoint.
> > >> Very sorry for not include enough details for this part and I'll
> > >> complement the FLIP with the details on
> > >> the process of the final checkpoint / savepoint.
> > >>
> > >> Best,
> > >> Yun
> > >>
> > >>
> > >>
> > >> ------------------------------------------------------------------
> > >> From:Till Rohrmann <tr...@apache.org>
> > >> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >> To:dev <de...@flink.apache.org>
> > >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >> Finished
> > >>
> > >> Hi everyone,
> > >>
> > >> I am a bit late to the voting party but let me ask three questions:
> > >>
> > >> 1) Why do we execute the trigger plan computation in the main thread
> if
> > we
> > >> cannot guarantee that all tasks are still running when triggering the
> > >> checkpoint? Couldn't we do the computation in a different thread in
> > order
> > >> to relieve the main thread a bit.
> > >>
> > >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to
> go
> > >> over the whole topology for every calculation. Wouldn't it be more
> > >> efficient to maintain the set of current tasks to trigger and check
> > >> whether
> > >> anything has changed and if so check the succeeding tasks until we
> have
> > >> found the current checkpoint trigger frontier?
> > >>
> > >> 3) When are we going to send the endOfInput events to a downstream
> task?
> > >> If
> > >> this happens after we call finish on the upstream operator but before
> > >> snapshotState then it would be possible to shut down the whole
> topology
> > >> with a single final checkpoint. I think this part could benefit from a
> > bit
> > >> more detailed description in the FLIP.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> > >> wrote:
> > >>
> > >> > Hi there,
> > >> >
> > >> > Since the voting time of FLIP-147[1] has passed, I'm closing the
> vote
> > >> now.
> > >> >
> > >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >> >
> > >> > - Dawid Wysakowicz (binding)
> > >> > - Piotr Nowojski(binding)
> > >> > - Jiangang Liu (binding)
> > >> > - Arvid Heise (binding)
> > >> > - Jing Zhang (binding)
> > >> > - Leonard Xu (non-binding)
> > >> > - Guowei Ma (binding)
> > >> >
> > >> > Thus I'm happy to announce that the update to the FLIP-147 is
> > accepted.
> > >> >
> > >> > Very thanks everyone!
> > >> >
> > >> > Best,
> > >> > Yun
> > >> >
> > >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>
> > >>
> >
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
I'd be fine to continue w/o a vote as long as the FLIP will contain all the
clarifications we talked about in this thread and in FLINK-21080.
Cheers,
Till
On Fri, Jul 23, 2021 at 8:34 AM Dawid Wysakowicz <dw...@apache.org>
wrote:
> How do you feel about proceeding without an additional vote?
>
> I agree, we added quite a bit in there, but none of the added content
> changes what has been voted so far. A big part of the added content can
> be considered as clarifying the implementation. I'd be inclined to
> proceed without a vote. WDYT?
>
> Best,
>
> Dawid
>
>
> On 22/07/2021 14:55, Yun Gao wrote:
> > Hi Till,
> >
> > Very thanks for the comments and tips! We will update the FLIP with the
> new details and we
> > should need a new vote since the FLIP is updated a large part, I think
> we would start the
> > vote right after we also solve the remaining UnionListState problem and
> have a double
> > check on that we have no other missing points for the design and FLIP~
> >
> > Best,
> > Yun
> >
> >
> > ------------------------------------------------------------------
> > From:Till Rohrmann <tr...@apache.org>
> > Send Time:2021 Jul. 22 (Thu.) 17:57
> > To:Yun Gao <yu...@aliyun.com>
> > Cc:Piotr Nowojski <pn...@apache.org>; dev <de...@flink.apache.org>;
> Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <
> piotr@ververica.com>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
> >
> > Thanks everyone for this discussion. I think this is very helpful.
> >
> > I do agree with Piotr's proposal to separate state and the lifecycle of
> a StreamOperator. That way the finished state can be used to recover a
> StreamOperator or to rescale the topology. I also believe that this will
> make the implementation a lot easier since we don't have to remember which
> StreamOperator has actually finished. Moreover, it solves the problem of
> rescaling.
> >
> > I also agree with the proposed solution for checkpoints after finish()
> has been called. I guess the first checkpoint that was triggered after
> finish() and sends notifyCheckpointComplete should be good enough to close
> the StreamTask.
> >
> > Shall we update the FLIP with these details? Do we need another vote for
> it or shall we continue w/o it?
> >
> > Cheers,
> > Till
> > On Thu, Jul 22, 2021 at 10:45 AM Yun Gao <yu...@aliyun.com> wrote:
> >
> > Hi Piotr,
> >
> > Very thanks for the explanation! and very sorry that initially I should
> wrongly understand the problem Dawid proposed.
> >
> >> And what should we do now? We can of course commit all transactions
> until checkpoint 43.
> >> But should we keep waiting for `notyifyCheckpointComplete(44)`?
> >> hat if in the meantime another checkpoint is triggered? We could end up
> waiting indefinitely.
> >> Our proposal is to shutdown the task immediately after seeing first
> `notifyCheckpointComplete(X)`,
> >> where X is any triggered checkpoint AFTER `finish()`. This should be
> fine, as:
> >> a) ideally there should be no new pending transactions opened after
> checkpoint 42
> >> b) even if operator/function is opening some transactions for
> checkpoint 43 and
> >> checkpoint 44 (`FlinkKafkaProducer`), those transactions after
> checkpoint 42 should be empty
> > Now I understand more on this issue: currently operators like the
> TwoPhaseCommitSinkFunctions would create
> > a new transaction when snapshotting state, thus for each checkpoint
> there would be one corresponding
> > transaction, thus after finish() and before notifyCheckpointComplete(),
> there might be some empty
> > transactions.
> >
> > I also agree with the proposal: we could finish the task after we
> received the first checkpoint complete
> > notification, and the checkpoint with the larger checkpoint id should be
> allowed to be finished. The
> > checkpoint complete notification should be able to commit all the
> non-empty transactions. A part of
> > the empty transactions would also get committed, but it should cause no
> harm. The other empty
> > transactions would be aborted in close(), which should also cause no
> harm.
> >
> >> If checkpoint 44 completes afterwards, it will still be valid. Ideally
> we would recommend that after
> > seeing `finish()` operators/functions should not be opening any new
> transactions, but that shouldn't be required.
> >
> > And from another of view, I think perhaps it might also be acceptable
> that we have some requirements for
> > the pattern of the operators, like the sink would be able to skip
> creating new transactions after finish() is called.
> > Perhaps we may treat the process as a protocol between the framework and
> the operators, and the operators
> > might need to follow the protocol. Of course it would be better that the
> framework could handle all the cases
> > elegantly, and put less implicit limitation to the operators, and at
> least we should guarantee not cause compatibility
> > problem after changing the process.
> >
> > Very thanks for the careful checks on the whole process.
> >
> > Best,
> > Yun
> >
> >
> > ------------------------------------------------------------------
> > From:Piotr Nowojski <pn...@apache.org>
> > Send Time:2021 Jul. 22 (Thu.) 15:33
> > To:dev <de...@flink.apache.org>; Yun Gao <yu...@aliyun.com>
> > Cc:Till Rohrmann <tr...@apache.org>; Yun Gao
> <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
> >
> > Hi Guowei,
> >
> >> Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish
> can becalled repeatedly.
> > Just to clarify. It's not about calling `finish()` and `endInput()`
> repeatedly, but about (from the perspective of operator's state)
> > 1. seeing `finish()`
> > 2. checkpoint X triggered and completed
> > 3. failure + recovery from X
> > 4. potentially processing more records
> > 5. another `finish()`
> >
> > But from the context of the remaining part of your message Guowei I
> presume that you have already got that point :)
> >
> > Yun:
> >
> >> For this issue perhaps we could explicitly requires the task to wait
> for a checkpoint triggered after finish()> method is called for all the
> operators ? We could be able to achieve this target by maintaining
> >> some state inside the task.
> > Isn't this exactly the "WAITING_FOR_FINAL_CP" from the FLIP document?
> That we always need to wait for a checkpoint triggered after `finish()` to
> complete, before shutting down a task?
> >
> > What Dawid was describing is a scenario where:
> > 1. task/operator received `finish()`
> > 2. checkpoint 42 triggered (not yet completed)
> > 3. checkpoint 43 triggered (not yet completed)
> > 4. checkpoint 44 triggered (not yet completed)
> > 5. notifyCheckpointComplete(43)
> >
> > And what should we do now? We can of course commit all transactions
> until checkpoint 43. But should we keep waiting for
> `notyifyCheckpointComplete(44)`? What if in the meantime another checkpoint
> is triggered? We could end up waiting indefinitely.
> >
> > Our proposal is to shutdown the task immediately after seeing first
> `notifyCheckpointComplete(X)`, where X is any triggered checkpoint AFTER
> `finish()`. This should be fine, as:
> > a) ideally there should be no new pending transactions opened after
> checkpoint 42
> > b) even if operator/function is opening some transactions for checkpoint
> 43 and checkpoint 44 (`FlinkKafkaProducer`), those transactions after
> checkpoint 42 should be empty
> >
> > Hence comment from Dawid
> >> We must make sure those checkpoints do not leave lingering external
> resources.
> > After seeing 5. (notifyCheckpointComplete(43)) It should be good enough
> to:
> > - commit transactions from checkpoint 42, (and 43 if they were created,
> depends on the user code)
> > - close operator, aborting any pending transactions (for checkpoint 44
> if they were opened, depends on the user code)
> >
> > If checkpoint 44 completes afterwards, it will still be valid. Ideally
> we would recommend that after seeing `finish()` operators/functions should
> not be opening any new transactions, but that shouldn't be required.
> >
> > Best,
> > Piotrek
> > czw., 22 lip 2021 o 09:00 Yun Gao <yu...@aliyun.com.invalid>
> napisał(a):
> > Hi Dawid, Piotr, Steven,
> >
> > Very thanks for pointing out these issues and very thanks for the
> discussion !
> >
> > Failure before notifyCheckpointComplete()
> >
> > For this issue I would agree with what Piotr has proposed. I tried to
> use some
> > operators like sink / window as example and currently I also do not
> found
> > explicit scenarios that might cause problems if records are processed
> by a
> > task that is assigned with states snapshotted after calling finish()
> before.
> > For the future cases it seems users should be able to implement their
> > target logic by explicitly add a flag regarding finished, and perhaps
> have
> > different logic if this part of states are referred to. Besides, this
> case would
> > only happen on rescaling or topology change, which embedded some kind
> of user
> > knowledge inside the action. Thus it looks acceptable that we still
> split the operators
> > state from the task lifecycle, and do not treat checkpoint after
> finish() differently.
> >
> > Finishing upon receiving notifyCheckpointComplete() of not the latest
> checkpoint
> >
> > For this issue perhaps we could explicitly requires the task to wait
> for a checkpoint triggered after finish()
> > method is called for all the operators ? We could be able to achieve
> this target by maintaining
> > some state inside the task.
> >
> > Checkpointing from a single subtask / UnionListState case
> >
> > This should indeed cause problems, and I also agree with that we could
> focus on this thread in the
> > https://issues.apache.org/jira/browse/FLINK-21080.
> >
> > Best,
> > Yun
> >
> >
> > ------------------------------------------------------------------
> > From:Piotr Nowojski <pn...@apache.org>
> > Send Time:2021 Jul. 22 (Thu.) 02:46
> > To:dev <de...@flink.apache.org>
> > Cc:Yun Gao <yu...@aliyun.com>; Till Rohrmann <tr...@apache.org>;
> Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <
> piotr@ververica.com>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
> >
> > Hi Steven,
> >
> > > I probably missed sth here. isn't this the case today already? Why is
> it a concern for the proposed change?
> >
> > The problem is with the newly added `finish()` method and the already
> existing `endInput()` call. Currently on master there are no issues,
> because we are not checkpointing any operators after some operators have
> finished. The purpose of this FLIP-147 is to exactly enable this and this
> opens a new problem described by Dawid.
> >
> > To paraphrase and to give a concrete example. Assume we have an
> operator with parallelism of two. Subtask 0 and subtask 1.
> >
> > 1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1
> hasn't (yet).
> > 2. Checkpoint 42 is triggered, and it completes.
> > 3. Job fails and is restarted, but at the same time it's rescaled. User
> has chosen to scale down his operator down to 1.
> >
> > Now we have a pickle. We don't know if `notifyCheckpointComplete(42)`
> has been processed or not, so while recovering to checkpoint 42, we have to
> recover both finished subtask 0 (#1) state and not yet finished subtask 1's
> (#1). But at the same time they are scaled down, so we only have a single
> subtask 0 (#2) that has a combined state from both of the previous
> instances. The potentially confusing issue is that the state from subtask 0
> (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's
> recovered to an operator that has still some records to process. In step 1.
> an user for example could store on the operator's state a bit of
> information "end input has been already called!", that after recovery would
> no longer be true.
> >
> > Hence the question about `finish()` and `endInput()` semantics. Should
> it be tied down to state, or just to an operator instance/execution attempt?
> >
> > Piotrek
> > śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> > > if a failure happens after sequence of finish() -> snapshotState(),
> but
> > before notifyCheckpointComplete(), we will restore such a state and we
> > might end up sending some more records to such an operator.
> >
> > I probably missed sth here. isn't this the case today already? Why is
> it a
> > concern for the proposed change?
> >
> > On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org>
> wrote:
> >
> > > Hi Dawid,
> > >
> > > Thanks for writing down those concerns.
> > >
> > > I think the first issue boils down what should be the contract of
> lifecycle
> > > methods like open(), close(), initializeState() etc and especially
> the new
> > > additions like finish() and endInput(). And what should be their
> relation
> > > with the operator state (regardless of it's type keyed, non-keyed,
> union,
> > > ...). Should those methods be tied to state or not? After thinking
> about it
> > > for a while (and discussing it offline with Dawid), I think the
> answer
> > > might be no, they shouldn't. I mean maybe we should just openly say
> that
> > > all of those methods relate to this single particular instance and
> > > execution of the operator. And if a job is recovered/rescaled, we
> would be
> > > allowed to freely resume consumption, ignoring a fact that maybe
> some parts
> > > of the state have previously seen `endInput()`. Why?
> > >
> > > 0. Yes, it might be confusing. Especially with `endInput()`. We call
> > > `endInput()`, we store something in a state and later after recovery
> > > combined with rescaling that state can see more records? Indeed
> weird,
> > > 1. I haven't come up yet with a counterexample that would break and
> make
> > > impossible to implement a real life use case. Theoretically yes, the
> user
> > > can store `endInput()` on state, and after rescaling this state
> would be
> > > inconsistent with what is actually happening with the operator, but I
> > > haven't found a use case that would break because of that.
> > > 2. Otherwise, implementation would be very difficult.
> > > 3. It's difficult to access keyed state from within
> `endInput()`/`finish()`
> > > calls, as they do not have key context.
> > > 4. After all, openly defining `endInput()` and `finish()` to be tied
> with
> > > it's operator execution instance lifecycle is not that strange and
> quite
> > > simple to explain. Sure, it can lead to a bit of confusion (0.), but
> that
> > > doesn't sound that bad in comparison with the alternatives that I'm
> aware
> > > of. Also currently methods like `open()` and `close()` are also tied
> to the
> > > operator execution instance, not to the state. Operators can be
> opened and
> > > closed multiple times, it doesn't mean that the state is lost after
> closing
> > > an operator.
> > >
> > > For the UnionListState problem I have posted my proposal in the
> ticket [1],
> > > so maybe let's move that particular discussion there?
> > >
> > > Piotrek
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-21080
> > >
> > > śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> > > napisał(a):
> > >
> > > > Hey all,
> > > >
> > > > To make the issues that were found transparent to the community, I
> want
> > > to
> > > > post an update:
> > > >
> > > > *1. Committing side-effects*
> > > > We do want to make sure that all side effects are committed before
> > > > bringing tasks down. Side effects are committed when calling
> > > > notifyCheckpointComplete. For the final checkpoint we introduced
> the
> > > method
> > > > finish(). This notifies the operator that we have consumed all
> incoming
> > > > records and we are preparing to close the Task. In turn we should
> flush
> > > any
> > > > pending buffered records and prepare to commit last transactions.
> The
> > > goal
> > > > is that after a successful sequence of finish() -> snapshotState()
> ->
> > > > notifyCheckpointComplete(), the remaining state can be considered
> > > > empty/finished and may be discarded.
> > > >
> > > > *Failure before notifyCheckpointComplete()*
> > > >
> > > > The question is what is the contract of the endInput()/finish()
> methods
> > > > and how do calling these methods affect the operators keyed,
> non-keyed
> > > > state and external state. Is it allowed to restore state snapshot
> taken
> > > > after calling endInput()/finish() and process more records? Or do
> we
> > > assume
> > > > that after a restore from such a state taken after finish() we
> should not
> > > > call any of the lifecycle methods or at least make sure those
> methods do
> > > > not emit records/interact with mailbox etc.
> > > >
> > > > Currently it is possible that if a failure happens after sequence
> of
> > > > finish() -> snapshotState(), but before
> notifyCheckpointComplete(), we
> > > will
> > > > restore such a state and we might end up sending some more records
> to
> > > such
> > > > an operator. It is possible if we rescale and this state is merged
> with a
> > > > state of a subtask that has not called finish() yet. It can also
> happen
> > > if
> > > > we rescale the upstream operator and the subtask of interest
> becomes
> > > > connected to a newly added non finished subtask.
> > > >
> > > > *Snapshotting StreamTasks that finish() has been called*
> > > >
> > > >
> > > > We thought about putting a flag into the snapshot of a subtask
> produced
> > > > after the finish() method. This would make it possible to skip
> execution
> > > of
> > > > certain lifecycle methods. Unfortunately this creates problems for
> > > > rescaling. How do we deal with a situation that subtask states
> with both
> > > > the feature flag set and unset end up in a single StreamTask.
> Additional
> > > > problem is that we merge those states into a single
> OperatorSubtaskState
> > > on
> > > > CheckpointCoordinator.
> > > >
> > > > *Finishing upon receiving notifyCheckpointComplete() of not the
> latest
> > > > checkpoint*
> > > >
> > > > We need to wait for a checkpoint to complete, that started after
> the
> > > > finish() method. However, we support concurrent checkpoints
> therefore,
> > > > there might be later checkpoints that completed, but the
> notification has
> > > > not arrived. We must make sure those checkpoints do not leave
> lingering
> > > > external resources.
> > > >
> > > > *Checkpointing from a single subtask / UnionListState case*
> > > > There are operators that checkpoint from a single subtask only.
> Usually
> > > > from the subtask index=0. If we let those subtasks finish,
> subsequent
> > > > checkpoints will miss this information.
> > > > Esp. Legacy sources problem:
> > > > https://issues.apache.org/jira/browse/FLINK-21080
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > > On 19/07/2021 15:10, Yun Gao wrote:
> > > >
> > > > Hi Till, Dawid,
> > > >
> > > > Very thanks for the comments and discussion! Glad that it seems we
> have
> > > > come to a convergence, and I also agree with that we could not
> include
> > > the
> > > > optimization in the first version.
> > > >
> > > > Best
> > > > Yun
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Dawid Wysakowicz <dw...@apache.org> <
> dwysakowicz@apache.org>
> > > > Send Time:2021 Jul. 19 (Mon.) 17:51
> > > > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till
> Rohrmann
> > > > <tr...@apache.org> <tr...@apache.org>
> > > > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > > > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > > Finished
> > > >
> > > > Small correction. I meant we need to adjust the EndOfInputEvent of
> > > course.
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > > > Hey Till,
> > > > >
> > > > > Yes, you're right we will have to adjust the current state of
> > > > > EndOfPartitionEvent and move the moment when we emit it to have
> what
> > > > > we're discussing here. We are aware of that.
> > > > >
> > > > > As for the MAX_WATERMARK vs finish(). My take is that we should
> always
> > > > > emit MAX_WATERMARK before calling finish() on an operator. At
> the same
> > > > > time finish() should not leave behind anything in state, as the
> > > > > intention is that we never restore from the taken
> savepoint/checkpoint
> > > > > (savepoint w drain or bounded data consumed).
> > > > >
> > > > > Best,
> > > > >
> > > > > Dawid
> > > > >
> > > > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > > > >> Hi Yun and Dawid,
> > > > >>
> > > >
> > > > >> Thanks for your comments. I do agree with your comments that
> finish()
> > > can
> > > > >> do more than MAX_WATERMARK. I guess we should then explain how
> > > > >> MAX_WATERMARK and finish() play together and what kind of
> > > > >> order guarantees we provide.
> > > > >>
> > > >
> > > > >> Concerning the EndOfPartitionEvent, I am not entirely sure
> whether it
> > > would
> > > >
> > > > >> work in its current state because we send this event when the
> Task is
> > > about
> > > > >> to shut down if I am not mistaken. What we want to have is to
> bring
> > > the
> > > >
> > > > >> StreamTasks into a state so that they shut down on the next
> > > checkpoint. For
> > > > >> this we need to keep the StreamTask running. In general, I am a
> fan of
> > > > >> making things explicit if possible. I think this helps
> maintenance and
> > > >
> > > > >> evolvability of code. That's why I think sending an
> EndOfInputEvent
> > > which
> > > > >> is a StreamTask level event and which says that there won't be
> any
> > > other
> > > > >> records coming only control events could make sense.
> > > > >>
> > > > >> I would leave the proposed optimization out of the first
> version. We
> > > can
> > > > >> still add it at a later point in time.
> > > > >>
> > > > >> Cheers,
> > > > >> Till
> > > > >>
> > > > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > > > <dw...@apache.org> <dw...@apache.org>
> > > > >> wrote:
> > > > >>
> > > >
> > > > >>> Personally I don't find this optimization important and I'd
> rather
> > > leave
> > > >
> > > > >>> it out not to complicate the codebase further. I doubt we save
> much
> > > there.
> > > > >>> I don't have a strong opinion though.
> > > > >>>
> > > > >>> Best,
> > > > >>>
> > > > >>> Dawid
> > > > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> Very thanks Dawid for the thoughts!
> > > > >>>
> > > > >>> Currently I also do not have different opinions regarding this
> part.
> > > > >>> But I have one more issue to confirm: during the previous
> discussion
> > > we
> > > > >>> have discussed that for the final checkpoint case, we might
> have an
> > > > >>> optmization
> > > >
> > > > >>> that if a task do not have operators using 2-pc, we might skip
> > > waiting for
> > > > >>> the
> > > >
> > > > >>> final checkpoint (but we could not skip the savepoint). To
> allow
> > > users to
> > > > >>> express
> > > > >>> the logic, we have proposed to add one more method to
> StreamOperator
> > > &
> > > > >>> CheckpointListener:
> > > > >>>
> > > > >>> interface StreamOperator {
> > > > >>> default boolean requiresFinalCheckpoint() {
> > > > >>> return true;
> > > > >>> }
> > > > >>> }
> > > > >>>
> > > > >>> interface CheckpointListener {
> > > > >>>
> > > > >>> default boolean requiresFinalCheckpoint() {
> > > > >>> return true;
> > > > >>> }
> > > > >>> }
> > > > >>>
> > > > >>> class AbstractUdfStreamOperator {
> > > > >>>
> > > > >>> @Override
> > > > >>> boolean requiresFinalCheckpoint() {
> > > > >>> return userFunction instanceof CheckpointListener &&
> > > >
> > > > >>> ((CheckpointListener)
> > > userFunction).requiresFinalCheckpoint();
> > > > >>> }
> > > > >>> }
> > > > >>>
> > > > >>>
> > > > >>> I think we should still keep the change ?
> > > > >>>
> > > > >>> Best,
> > > > >>> Yun
> > > > >>>
> > > > >>> ------------------Original Mail ------------------
> > > > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > > > <dw...@apache.org>
> > > > >>> <dw...@apache.org> <dw...@apache.org>
> > > > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > > > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <
> dev@flink.apache.org>
> > > > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > > > >>> Gao <yu...@aliyun.com.INVALID>
> <yu...@aliyun.com.INVALID>
> > > > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint
> After Tasks
> > > > >>> Finished
> > > > >>>
> > > > >>>> I think we're all really close to the same solution.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> I second Yun's thoughts that MAX_WATERMARK works well for
> time based
> > > > >>>>
> > > >
> > > > >>>> buffering, but it does not solve flushing other operations
> such as
> > > e.g.
> > > > >>>>
> > > > >>>> count windows or batching requests in Sinks. I'd prefer to
> treat the
> > > > >>>>
> > > > >>>> finish() as a message for Operator to "flush all records". The
> > > > >>>>
> > > >
> > > > >>>> MAX_WATERMARK in my opinion is mostly for backwards
> compatibility
> > > imo. I
> > > > >>>>
> > > > >>>> don't think operators need to get a signal "stop-processing"
> if they
> > > > >>>>
> > > > >>>> don't need to flush records. The "WHEN" records are emitted,
> should
> > > be
> > > > >>>>
> > > > >>>> in control of the StreamTask, by firing timers or by
> processing a
> > > next
> > > > >>>>
> > > > >>>> record from upstream.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > >
> > > > >>>> The only difference of my previous proposal compared to Yun's
> is
> > > that I
> > > > >>>>
> > > > >>>> did not want to send the EndOfUserRecords event in case of
> stop w/o
> > > > >>>>
> > > > >>>> drain. My thinking was that we could directly go from RUNNING
> to
> > > > >>>>
> > > > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could
> emit
> > > > >>>>
> > > > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop
> firing
> > > > >>>>
> > > >
> > > > >>>> timers and processing events (without calling finish() on
> > > Operator). In
> > > > >>>>
> > > > >>>> my initial suggestion I though we don't care about some events
> > > > >>>>
> > > > >>>> potentially being emitted after the savepoint was taken, as
> they
> > > would
> > > > >>>>
> > > >
> > > > >>>> anyway belong to the next after FINAL, which would be
> discarded. I
> > > think
> > > > >>>>
> > > > >>>> though the proposal to suspend records processing and timers
> is a
> > > > >>>>
> > > >
> > > > >>>> sensible thing to do and would go with the version that Yun
> put
> > > into the
> > > > >>>>
> > > > >>>> FLIP Wiki.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> What do you think Till?
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Best,
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Dawid
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > > > >>>>
> > > > >>>>> Hi Till, Piotr
> > > > >>>>> Very thanks for the comments!
> > > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >
> > > > >>>>> I also agree with Piotr that currently they are independent
> > > mechanisms,
> > > > >>>> and they are basically the same
> > > > >>>>
> > > > >>>>> for the event time.
> > > > >>>>> For more details, first there are some difference among the
> three
> > > > >>>> scenarios regarding the finish:
> > > > >>>>
> > > >
> > > > >>>>> For normal finish and stop-with-savepoint --drain, the job
> would
> > > not be
> > > > >>>> expected to be restarted,
> > > > >>>>
> > > > >>>>> and for stop-with-savepoint the job would be expected restart
> > > later.
> > > > >>>>> Then for finish / stop-with-savepoint --drain, currently
> Flink
> > > would
> > > > >>>> emit MAX_WATERMARK before the
> > > > >>>>
> > > >
> > > > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> > > endOfInput /
> > > > >>>> finish() should also only be called
> > > > >>>>
> > > > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> > > always
> > > > >>>> occurs at the same time. After the change,
> > > > >>>>
> > > > >>>>> we could emit MAX_WATERMARK before endOfInput event for the
> finish
> > > /
> > > > >>>> stop-with-savepoint --drain cases.
> > > > >>>>
> > > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > > a
> > > > >>>>>> WindowOperator close all windows and emit the results upon
> calling
> > > > >>>>>> finish, for example?
> > > > >>>>> As discussed above for stop-with-savepoint, we would always
> keep
> > > the
> > > > >>>> window as is, and restore them after restart.
> > > > >>>>
> > > > >>>>> Then for the finish / stop-with-savepoint --drain, I think
> perhaps
> > > it
> > > > >>>> depends on the Triggers. For
> > > > >>>>
> > > >
> > > > >>>>> event-time triggers / process time triggers, it would be
> > > reasonable to
> > > > >>>> flush all the windows since logically
> > > > >>>>
> > > >
> > > > >>>>> the time would always elapse and the window would always get
> > > triggered
> > > > >>>> in a logical future. But for triggers
> > > > >>>>
> > > >
> > > > >>>>> like CountTrigger, no matter how much time pass logically,
> the
> > > windows
> > > > >>>> would not trigger, thus we may not
> > > > >>>>
> > > > >>>>> flush these windows. If there are requirements we may provide
> > > > >>>> additional triggers.
> > > > >>>>
> > > >
> > > > >>>>>> It's a bit messy and I'm not sure if this should be
> strengthened
> > > out?
> > > > >>>> Each one of those has a little bit different semantic/meaning,
> > > > >>>>
> > > >
> > > > >>>>>> but at the same time they are very similar. For single input
> > > operators
> > > > >>>> `endInput()` and `finish()` are actually the very same thing.
> > > > >>>>
> > > >
> > > > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always
> happen
> > > at the
> > > > >>>> same time, and for single input operators `endInput()` and
> > > `finish()`
> > > > >>>>
> > > >
> > > > >>>>> are indeed the same thing. During the last discussion we ever
> > > mentioned
> > > > >>>> this issue and at then we thought that we might deprecate
> > > `endInput()`
> > > > >>>>
> > > > >>>>> in the future, then we would only have endInput(int input)
> and
> > > > >>>> finish().
> > > > >>>>
> > > > >>>>> Best,
> > > > >>>>> Yun
> > > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > > > >>>>>
> ------------------------------------------------------------------
> > > > >>>>> From:Piotr Nowojski
> > > > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > > > >>>>> To:dev
> > > > >>>>> Cc:Yun Gao
> > > > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint
> After Tasks
> > > > >>>> Finished
> > > > >>>>
> > > > >>>>> Hi Till,
> > > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > > a>
> > > > >>>> WindowOperator close all windows and emit the results upon
> calling
> > > > >>>>
> > > > >>>>>> finish, for example?
> > > > >>>>> 1) currently they are independent but parallel mechanisms.
> With
> > > event
> > > > >>>> time, they are basically the same.
> > > > >>>>
> > > > >>>>> 2) it probably should for the sake of processing time
> windows.
> > > > >>>>> Here you are touching the bit of the current design that I
> like the
> > > >
> > > > >>>> least. We basically have now three different ways of
> conveying very
> > > similar
> > > > >>>> things:
> > > > >>>>
> > > > >>>>> a) sending `MAX_WATERMARK`, used by event time
> WindowOperator (what
> > > > >>>> about processing time?)
> > > > >>>>
> > > > >>>>> b) endInput(), used for example by AsyncWaitOperator to
> flush it's
> > > > >>>> internal state
> > > > >>>>
> > > > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > > > >>>>> It's a bit messy and I'm not sure if this should be
> strengthened
> > > out?
> > > >
> > > > >>>> Each one of those has a little bit different
> semantic/meaning, but
> > > at the
> > > >
> > > > >>>> same time they are very similar. For single input operators
> > > `endInput()`
> > > > >>>> and `finish()` are actually the very same thing.
> > > > >>>>
> > > > >>>>> Piotrek
> > > > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > > > >>>>> Thanks for updating the FLIP. Based on the new section about
> > > > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > > > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > > >>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would a
> > > > >>>>> WindowOperator close all windows and emit the results upon
> calling
> > > > >>>>> finish, for example?
> > > > >>>>> Cheers,
> > > > >>>>> Till
> > > > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > > > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > > > >>>>>> 1+2) Agreed, this can be a future improvement if this
> becomes a
> > > > >>>> problem.
> > > > >>>>
> > > > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > > > >>>>>> Cheers,
> > > > >>>>>> Till
> > > > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > > > >>>>>> wrote:
> > > > >>>>>>> Hi Till,
> > > > >>>>>>> Very thanks for the review and comments!
> > > > >>>>>>> 1) First I think in fact we could be able to do the
> computation
> > > > >>>> outside
> > > > >>>>
> > > > >>>>>>> of the main thread,
> > > > >>>>>>> and the current implementation mainly due to the
> computation is
> > > in
> > > > >>>>>>> general fast and we
> > > > >>>>>>> initially want to have a simplified first version.
> > > >
> > > > >>>>>>> The main requirement here is to have a constant view of the
> > > state of
> > > > >>>> the
> > > > >>>>
> > > > >>>>>>> tasks, otherwise
> > > > >>>>>>> for example if we have A -> B, if A is running when we
> check if
> > > we
> > > > >>>> need
> > > > >>>>
> > > > >>>>>>> to trigger A, we will
> > > > >>>>>>> mark A as have to trigger, but if A gets to finished when
> we
> > > check
> > > > >>>> B, we
> > > > >>>>
> > > > >>>>>>> will also mark B as
> > > >
> > > > >>>>>>> have to trigger, then B will receive both rpc trigger and
> > > checkpoint
> > > > >>>>>>> barrier, which would break
> > > > >>>>>>> some assumption on the task side and complicate the
> > > implementation.
> > > >
> > > > >>>>>>> But to cope this issue, we in fact could first have a
> snapshot
> > > of the
> > > > >>>>>>> tasks' state and then do the
> > > >
> > > > >>>>>>> computation, both the two step do not need to be in the
> main
> > > thread.
> > > >
> > > > >>>>>>> 2) For the computation logic, in fact currently we benefit
> a lot
> > > from
> > > > >>>>>>> some shortcuts on all-to-all
> > > >
> > > > >>>>>>> edges and job vertex with all tasks running, these
> shortcuts
> > > could do
> > > > >>>>>>> checks on the job vertex level
> > > > >>>>>>> first and skip some job vertices as a whole. With this
> > > optimization
> > > > >>>> we
> > > > >>>>
> > > > >>>>>>> have a O(V) algorithm, and the
> > > > >>>>>>> current running time of the worst case for a job with
> 320,000
> > > tasks
> > > > >>>> is
> > > > >>>>
> > > > >>>>>>> less than 100ms. For
> > > > >>>>>>> daily graph sizes the time would be further reduced
> linearly.
> > > > >>>>>>> If we do the computation based on the last triggered
> tasks, we
> > > may
> > > > >>>> not
> > > > >>>>
> > > > >>>>>>> easily encode this information
> > > >
> > > > >>>>>>> into the shortcuts on the job vertex level. And since the
> time
> > > seems
> > > > >>>> to
> > > > >>>>
> > > > >>>>>>> be short, perhaps it is enough
> > > > >>>>>>> to do re-computation from the scratch in consideration of
> the
> > > > >>>> tradeoff
> > > > >>>>
> > > > >>>>>>> between the performance and
> > > > >>>>>>> the complexity ?
> > > > >>>>>>> 3) We are going to emit the EndOfInput event exactly after
> the
> > > > >>>> finish()
> > > > >>>>
> > > > >>>>>>> method and before the last
> > > >
> > > > >>>>>>> snapshotState() method so that we could shut down the whole
> > > topology
> > > > >>>> with
> > > > >>>>
> > > > >>>>>>> a single final checkpoint.
> > > > >>>>>>> Very sorry for not include enough details for this part
> and I'll
> > > > >>>>>>> complement the FLIP with the details on
> > > > >>>>>>> the process of the final checkpoint / savepoint.
> > > > >>>>>>> Best,
> > > > >>>>>>> Yun
> > > > >>>>>>>
> > > ------------------------------------------------------------------
> > > > >>>>>>> From:Till Rohrmann
> > > > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > > > >>>>>>> To:dev
> > > > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint
> After
> > > Tasks
> > > > >>>>>>> Finished
> > > > >>>>>>> Hi everyone,
> > > > >>>>>>> I am a bit late to the voting party but let me ask three
> > > questions:
> > > >
> > > > >>>>>>> 1) Why do we execute the trigger plan computation in the
> main
> > > thread
> > > > >>>> if we
> > > > >>>>
> > > >
> > > > >>>>>>> cannot guarantee that all tasks are still running when
> > > triggering the
> > > > >>>>>>> checkpoint? Couldn't we do the computation in a different
> thread
> > > in
> > > > >>>> order
> > > > >>>>
> > > > >>>>>>> to relieve the main thread a bit.
> > > > >>>>>>> 2) The implementation of the
> DefaultCheckpointPlanCalculator
> > > seems
> > > > >>>> to go
> > > > >>>>
> > > > >>>>>>> over the whole topology for every calculation. Wouldn't it
> be
> > > more
> > > > >>>>>>> efficient to maintain the set of current tasks to trigger
> and
> > > check
> > > > >>>>>>> whether
> > > > >>>>>>> anything has changed and if so check the succeeding tasks
> until
> > > we
> > > > >>>> have
> > > > >>>>
> > > > >>>>>>> found the current checkpoint trigger frontier?
> > > > >>>>>>> 3) When are we going to send the endOfInput events to a
> > > downstream
> > > > >>>> task?
> > > > >>>>
> > > > >>>>>>> If
> > > >
> > > > >>>>>>> this happens after we call finish on the upstream operator
> but
> > > before
> > > > >>>>>>> snapshotState then it would be possible to shut down the
> whole
> > > > >>>> topology
> > > > >>>>
> > > >
> > > > >>>>>>> with a single final checkpoint. I think this part could
> benefit
> > > from
> > > > >>>> a bit
> > > > >>>>
> > > > >>>>>>> more detailed description in the FLIP.
> > > > >>>>>>> Cheers,
> > > > >>>>>>> Till
> > > > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > > > >>>>>>> wrote:
> > > > >>>>>>>> Hi there,
> > > > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm
> closing the
> > > > >>>> vote
> > > > >>>>
> > > > >>>>>>> now.
> > > > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1
> votes:
> > > > >>>>>>>> - Dawid Wysakowicz (binding)
> > > > >>>>>>>> - Piotr Nowojski(binding)
> > > > >>>>>>>> - Jiangang Liu (binding)
> > > > >>>>>>>> - Arvid Heise (binding)
> > > > >>>>>>>> - Jing Zhang (binding)
> > > > >>>>>>>> - Leonard Xu (non-binding)
> > > > >>>>>>>> - Guowei Ma (binding)
> > > > >>>>>>>> Thus I'm happy to announce that the update to the
> FLIP-147 is
> > > > >>>> accepted.
> > > > >>>>
> > > > >>>>>>>> Very thanks everyone!
> > > > >>>>>>>> Best,
> > > > >>>>>>>> Yun
> > > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > > >>>>
> > > > >>>>
> > > >
> > > >
> > >
> >
> >
> >
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
How do you feel about proceeding without an additional vote?
I agree, we added quite a bit in there, but none of the added content
changes what has been voted so far. A big part of the added content can
be considered as clarifying the implementation. I'd be inclined to
proceed without a vote. WDYT?
Best,
Dawid
On 22/07/2021 14:55, Yun Gao wrote:
> Hi Till,
>
> Very thanks for the comments and tips! We will update the FLIP with the new details and we
> should need a new vote since the FLIP is updated a large part, I think we would start the
> vote right after we also solve the remaining UnionListState problem and have a double
> check on that we have no other missing points for the design and FLIP~
>
> Best,
> Yun
>
>
> ------------------------------------------------------------------
> From:Till Rohrmann <tr...@apache.org>
> Send Time:2021 Jul. 22 (Thu.) 17:57
> To:Yun Gao <yu...@aliyun.com>
> Cc:Piotr Nowojski <pn...@apache.org>; dev <de...@flink.apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
>
> Thanks everyone for this discussion. I think this is very helpful.
>
> I do agree with Piotr's proposal to separate state and the lifecycle of a StreamOperator. That way the finished state can be used to recover a StreamOperator or to rescale the topology. I also believe that this will make the implementation a lot easier since we don't have to remember which StreamOperator has actually finished. Moreover, it solves the problem of rescaling.
>
> I also agree with the proposed solution for checkpoints after finish() has been called. I guess the first checkpoint that was triggered after finish() and sends notifyCheckpointComplete should be good enough to close the StreamTask.
>
> Shall we update the FLIP with these details? Do we need another vote for it or shall we continue w/o it?
>
> Cheers,
> Till
> On Thu, Jul 22, 2021 at 10:45 AM Yun Gao <yu...@aliyun.com> wrote:
>
> Hi Piotr,
>
> Very thanks for the explanation! and very sorry that initially I should wrongly understand the problem Dawid proposed.
>
>> And what should we do now? We can of course commit all transactions until checkpoint 43.
>> But should we keep waiting for `notyifyCheckpointComplete(44)`?
>> hat if in the meantime another checkpoint is triggered? We could end up waiting indefinitely.
>> Our proposal is to shutdown the task immediately after seeing first `notifyCheckpointComplete(X)`,
>> where X is any triggered checkpoint AFTER `finish()`. This should be fine, as:
>> a) ideally there should be no new pending transactions opened after checkpoint 42
>> b) even if operator/function is opening some transactions for checkpoint 43 and
>> checkpoint 44 (`FlinkKafkaProducer`), those transactions after checkpoint 42 should be empty
> Now I understand more on this issue: currently operators like the TwoPhaseCommitSinkFunctions would create
> a new transaction when snapshotting state, thus for each checkpoint there would be one corresponding
> transaction, thus after finish() and before notifyCheckpointComplete(), there might be some empty
> transactions.
>
> I also agree with the proposal: we could finish the task after we received the first checkpoint complete
> notification, and the checkpoint with the larger checkpoint id should be allowed to be finished. The
> checkpoint complete notification should be able to commit all the non-empty transactions. A part of
> the empty transactions would also get committed, but it should cause no harm. The other empty
> transactions would be aborted in close(), which should also cause no harm.
>
>> If checkpoint 44 completes afterwards, it will still be valid. Ideally we would recommend that after
> seeing `finish()` operators/functions should not be opening any new transactions, but that shouldn't be required.
>
> And from another of view, I think perhaps it might also be acceptable that we have some requirements for
> the pattern of the operators, like the sink would be able to skip creating new transactions after finish() is called.
> Perhaps we may treat the process as a protocol between the framework and the operators, and the operators
> might need to follow the protocol. Of course it would be better that the framework could handle all the cases
> elegantly, and put less implicit limitation to the operators, and at least we should guarantee not cause compatibility
> problem after changing the process.
>
> Very thanks for the careful checks on the whole process.
>
> Best,
> Yun
>
>
> ------------------------------------------------------------------
> From:Piotr Nowojski <pn...@apache.org>
> Send Time:2021 Jul. 22 (Thu.) 15:33
> To:dev <de...@flink.apache.org>; Yun Gao <yu...@aliyun.com>
> Cc:Till Rohrmann <tr...@apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
>
> Hi Guowei,
>
>> Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can becalled repeatedly.
> Just to clarify. It's not about calling `finish()` and `endInput()` repeatedly, but about (from the perspective of operator's state)
> 1. seeing `finish()`
> 2. checkpoint X triggered and completed
> 3. failure + recovery from X
> 4. potentially processing more records
> 5. another `finish()`
>
> But from the context of the remaining part of your message Guowei I presume that you have already got that point :)
>
> Yun:
>
>> For this issue perhaps we could explicitly requires the task to wait for a checkpoint triggered after finish()> method is called for all the operators ? We could be able to achieve this target by maintaining
>> some state inside the task.
> Isn't this exactly the "WAITING_FOR_FINAL_CP" from the FLIP document? That we always need to wait for a checkpoint triggered after `finish()` to complete, before shutting down a task?
>
> What Dawid was describing is a scenario where:
> 1. task/operator received `finish()`
> 2. checkpoint 42 triggered (not yet completed)
> 3. checkpoint 43 triggered (not yet completed)
> 4. checkpoint 44 triggered (not yet completed)
> 5. notifyCheckpointComplete(43)
>
> And what should we do now? We can of course commit all transactions until checkpoint 43. But should we keep waiting for `notyifyCheckpointComplete(44)`? What if in the meantime another checkpoint is triggered? We could end up waiting indefinitely.
>
> Our proposal is to shutdown the task immediately after seeing first `notifyCheckpointComplete(X)`, where X is any triggered checkpoint AFTER `finish()`. This should be fine, as:
> a) ideally there should be no new pending transactions opened after checkpoint 42
> b) even if operator/function is opening some transactions for checkpoint 43 and checkpoint 44 (`FlinkKafkaProducer`), those transactions after checkpoint 42 should be empty
>
> Hence comment from Dawid
>> We must make sure those checkpoints do not leave lingering external resources.
> After seeing 5. (notifyCheckpointComplete(43)) It should be good enough to:
> - commit transactions from checkpoint 42, (and 43 if they were created, depends on the user code)
> - close operator, aborting any pending transactions (for checkpoint 44 if they were opened, depends on the user code)
>
> If checkpoint 44 completes afterwards, it will still be valid. Ideally we would recommend that after seeing `finish()` operators/functions should not be opening any new transactions, but that shouldn't be required.
>
> Best,
> Piotrek
> czw., 22 lip 2021 o 09:00 Yun Gao <yu...@aliyun.com.invalid> napisał(a):
> Hi Dawid, Piotr, Steven,
>
> Very thanks for pointing out these issues and very thanks for the discussion !
>
> Failure before notifyCheckpointComplete()
>
> For this issue I would agree with what Piotr has proposed. I tried to use some
> operators like sink / window as example and currently I also do not found
> explicit scenarios that might cause problems if records are processed by a
> task that is assigned with states snapshotted after calling finish() before.
> For the future cases it seems users should be able to implement their
> target logic by explicitly add a flag regarding finished, and perhaps have
> different logic if this part of states are referred to. Besides, this case would
> only happen on rescaling or topology change, which embedded some kind of user
> knowledge inside the action. Thus it looks acceptable that we still split the operators
> state from the task lifecycle, and do not treat checkpoint after finish() differently.
>
> Finishing upon receiving notifyCheckpointComplete() of not the latest checkpoint
>
> For this issue perhaps we could explicitly requires the task to wait for a checkpoint triggered after finish()
> method is called for all the operators ? We could be able to achieve this target by maintaining
> some state inside the task.
>
> Checkpointing from a single subtask / UnionListState case
>
> This should indeed cause problems, and I also agree with that we could focus on this thread in the
> https://issues.apache.org/jira/browse/FLINK-21080.
>
> Best,
> Yun
>
>
> ------------------------------------------------------------------
> From:Piotr Nowojski <pn...@apache.org>
> Send Time:2021 Jul. 22 (Thu.) 02:46
> To:dev <de...@flink.apache.org>
> Cc:Yun Gao <yu...@aliyun.com>; Till Rohrmann <tr...@apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
>
> Hi Steven,
>
> > I probably missed sth here. isn't this the case today already? Why is it a concern for the proposed change?
>
> The problem is with the newly added `finish()` method and the already existing `endInput()` call. Currently on master there are no issues, because we are not checkpointing any operators after some operators have finished. The purpose of this FLIP-147 is to exactly enable this and this opens a new problem described by Dawid.
>
> To paraphrase and to give a concrete example. Assume we have an operator with parallelism of two. Subtask 0 and subtask 1.
>
> 1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1 hasn't (yet).
> 2. Checkpoint 42 is triggered, and it completes.
> 3. Job fails and is restarted, but at the same time it's rescaled. User has chosen to scale down his operator down to 1.
>
> Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has been processed or not, so while recovering to checkpoint 42, we have to recover both finished subtask 0 (#1) state and not yet finished subtask 1's (#1). But at the same time they are scaled down, so we only have a single subtask 0 (#2) that has a combined state from both of the previous instances. The potentially confusing issue is that the state from subtask 0 (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's recovered to an operator that has still some records to process. In step 1. an user for example could store on the operator's state a bit of information "end input has been already called!", that after recovery would no longer be true.
>
> Hence the question about `finish()` and `endInput()` semantics. Should it be tied down to state, or just to an operator instance/execution attempt?
>
> Piotrek
> śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> > if a failure happens after sequence of finish() -> snapshotState(), but
> before notifyCheckpointComplete(), we will restore such a state and we
> might end up sending some more records to such an operator.
>
> I probably missed sth here. isn't this the case today already? Why is it a
> concern for the proposed change?
>
> On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org> wrote:
>
> > Hi Dawid,
> >
> > Thanks for writing down those concerns.
> >
> > I think the first issue boils down what should be the contract of lifecycle
> > methods like open(), close(), initializeState() etc and especially the new
> > additions like finish() and endInput(). And what should be their relation
> > with the operator state (regardless of it's type keyed, non-keyed, union,
> > ...). Should those methods be tied to state or not? After thinking about it
> > for a while (and discussing it offline with Dawid), I think the answer
> > might be no, they shouldn't. I mean maybe we should just openly say that
> > all of those methods relate to this single particular instance and
> > execution of the operator. And if a job is recovered/rescaled, we would be
> > allowed to freely resume consumption, ignoring a fact that maybe some parts
> > of the state have previously seen `endInput()`. Why?
> >
> > 0. Yes, it might be confusing. Especially with `endInput()`. We call
> > `endInput()`, we store something in a state and later after recovery
> > combined with rescaling that state can see more records? Indeed weird,
> > 1. I haven't come up yet with a counterexample that would break and make
> > impossible to implement a real life use case. Theoretically yes, the user
> > can store `endInput()` on state, and after rescaling this state would be
> > inconsistent with what is actually happening with the operator, but I
> > haven't found a use case that would break because of that.
> > 2. Otherwise, implementation would be very difficult.
> > 3. It's difficult to access keyed state from within `endInput()`/`finish()`
> > calls, as they do not have key context.
> > 4. After all, openly defining `endInput()` and `finish()` to be tied with
> > it's operator execution instance lifecycle is not that strange and quite
> > simple to explain. Sure, it can lead to a bit of confusion (0.), but that
> > doesn't sound that bad in comparison with the alternatives that I'm aware
> > of. Also currently methods like `open()` and `close()` are also tied to the
> > operator execution instance, not to the state. Operators can be opened and
> > closed multiple times, it doesn't mean that the state is lost after closing
> > an operator.
> >
> > For the UnionListState problem I have posted my proposal in the ticket [1],
> > so maybe let's move that particular discussion there?
> >
> > Piotrek
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-21080
> >
> > śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> > napisał(a):
> >
> > > Hey all,
> > >
> > > To make the issues that were found transparent to the community, I want
> > to
> > > post an update:
> > >
> > > *1. Committing side-effects*
> > > We do want to make sure that all side effects are committed before
> > > bringing tasks down. Side effects are committed when calling
> > > notifyCheckpointComplete. For the final checkpoint we introduced the
> > method
> > > finish(). This notifies the operator that we have consumed all incoming
> > > records and we are preparing to close the Task. In turn we should flush
> > any
> > > pending buffered records and prepare to commit last transactions. The
> > goal
> > > is that after a successful sequence of finish() -> snapshotState() ->
> > > notifyCheckpointComplete(), the remaining state can be considered
> > > empty/finished and may be discarded.
> > >
> > > *Failure before notifyCheckpointComplete()*
> > >
> > > The question is what is the contract of the endInput()/finish() methods
> > > and how do calling these methods affect the operators keyed, non-keyed
> > > state and external state. Is it allowed to restore state snapshot taken
> > > after calling endInput()/finish() and process more records? Or do we
> > assume
> > > that after a restore from such a state taken after finish() we should not
> > > call any of the lifecycle methods or at least make sure those methods do
> > > not emit records/interact with mailbox etc.
> > >
> > > Currently it is possible that if a failure happens after sequence of
> > > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> > will
> > > restore such a state and we might end up sending some more records to
> > such
> > > an operator. It is possible if we rescale and this state is merged with a
> > > state of a subtask that has not called finish() yet. It can also happen
> > if
> > > we rescale the upstream operator and the subtask of interest becomes
> > > connected to a newly added non finished subtask.
> > >
> > > *Snapshotting StreamTasks that finish() has been called*
> > >
> > >
> > > We thought about putting a flag into the snapshot of a subtask produced
> > > after the finish() method. This would make it possible to skip execution
> > of
> > > certain lifecycle methods. Unfortunately this creates problems for
> > > rescaling. How do we deal with a situation that subtask states with both
> > > the feature flag set and unset end up in a single StreamTask. Additional
> > > problem is that we merge those states into a single OperatorSubtaskState
> > on
> > > CheckpointCoordinator.
> > >
> > > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > > checkpoint*
> > >
> > > We need to wait for a checkpoint to complete, that started after the
> > > finish() method. However, we support concurrent checkpoints therefore,
> > > there might be later checkpoints that completed, but the notification has
> > > not arrived. We must make sure those checkpoints do not leave lingering
> > > external resources.
> > >
> > > *Checkpointing from a single subtask / UnionListState case*
> > > There are operators that checkpoint from a single subtask only. Usually
> > > from the subtask index=0. If we let those subtasks finish, subsequent
> > > checkpoints will miss this information.
> > > Esp. Legacy sources problem:
> > > https://issues.apache.org/jira/browse/FLINK-21080
> > >
> > > Best,
> > >
> > > Dawid
> > > On 19/07/2021 15:10, Yun Gao wrote:
> > >
> > > Hi Till, Dawid,
> > >
> > > Very thanks for the comments and discussion! Glad that it seems we have
> > > come to a convergence, and I also agree with that we could not include
> > the
> > > optimization in the first version.
> > >
> > > Best
> > > Yun
> > >
> > > ------------------------------------------------------------------
> > > From:Dawid Wysakowicz <dw...@apache.org> <dw...@apache.org>
> > > Send Time:2021 Jul. 19 (Mon.) 17:51
> > > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > > <tr...@apache.org> <tr...@apache.org>
> > > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > Finished
> > >
> > > Small correction. I meant we need to adjust the EndOfInputEvent of
> > course.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > > Hey Till,
> > > >
> > > > Yes, you're right we will have to adjust the current state of
> > > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > > we're discussing here. We are aware of that.
> > > >
> > > > As for the MAX_WATERMARK vs finish(). My take is that we should always
> > > > emit MAX_WATERMARK before calling finish() on an operator. At the same
> > > > time finish() should not leave behind anything in state, as the
> > > > intention is that we never restore from the taken savepoint/checkpoint
> > > > (savepoint w drain or bounded data consumed).
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > > >> Hi Yun and Dawid,
> > > >>
> > >
> > > >> Thanks for your comments. I do agree with your comments that finish()
> > can
> > > >> do more than MAX_WATERMARK. I guess we should then explain how
> > > >> MAX_WATERMARK and finish() play together and what kind of
> > > >> order guarantees we provide.
> > > >>
> > >
> > > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether it
> > would
> > >
> > > >> work in its current state because we send this event when the Task is
> > about
> > > >> to shut down if I am not mistaken. What we want to have is to bring
> > the
> > >
> > > >> StreamTasks into a state so that they shut down on the next
> > checkpoint. For
> > > >> this we need to keep the StreamTask running. In general, I am a fan of
> > > >> making things explicit if possible. I think this helps maintenance and
> > >
> > > >> evolvability of code. That's why I think sending an EndOfInputEvent
> > which
> > > >> is a StreamTask level event and which says that there won't be any
> > other
> > > >> records coming only control events could make sense.
> > > >>
> > > >> I would leave the proposed optimization out of the first version. We
> > can
> > > >> still add it at a later point in time.
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > > <dw...@apache.org> <dw...@apache.org>
> > > >> wrote:
> > > >>
> > >
> > > >>> Personally I don't find this optimization important and I'd rather
> > leave
> > >
> > > >>> it out not to complicate the codebase further. I doubt we save much
> > there.
> > > >>> I don't have a strong opinion though.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Dawid
> > > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> Very thanks Dawid for the thoughts!
> > > >>>
> > > >>> Currently I also do not have different opinions regarding this part.
> > > >>> But I have one more issue to confirm: during the previous discussion
> > we
> > > >>> have discussed that for the final checkpoint case, we might have an
> > > >>> optmization
> > >
> > > >>> that if a task do not have operators using 2-pc, we might skip
> > waiting for
> > > >>> the
> > >
> > > >>> final checkpoint (but we could not skip the savepoint). To allow
> > users to
> > > >>> express
> > > >>> the logic, we have proposed to add one more method to StreamOperator
> > &
> > > >>> CheckpointListener:
> > > >>>
> > > >>> interface StreamOperator {
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> interface CheckpointListener {
> > > >>>
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> class AbstractUdfStreamOperator {
> > > >>>
> > > >>> @Override
> > > >>> boolean requiresFinalCheckpoint() {
> > > >>> return userFunction instanceof CheckpointListener &&
> > >
> > > >>> ((CheckpointListener)
> > userFunction).requiresFinalCheckpoint();
> > > >>> }
> > > >>> }
> > > >>>
> > > >>>
> > > >>> I think we should still keep the change ?
> > > >>>
> > > >>> Best,
> > > >>> Yun
> > > >>>
> > > >>> ------------------Original Mail ------------------
> > > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > > <dw...@apache.org>
> > > >>> <dw...@apache.org> <dw...@apache.org>
> > > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>
> > > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > >>> Finished
> > > >>>
> > > >>>> I think we're all really close to the same solution.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
> > > >>>>
> > >
> > > >>>> buffering, but it does not solve flushing other operations such as
> > e.g.
> > > >>>>
> > > >>>> count windows or batching requests in Sinks. I'd prefer to treat the
> > > >>>>
> > > >>>> finish() as a message for Operator to "flush all records". The
> > > >>>>
> > >
> > > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> > imo. I
> > > >>>>
> > > >>>> don't think operators need to get a signal "stop-processing" if they
> > > >>>>
> > > >>>> don't need to flush records. The "WHEN" records are emitted, should
> > be
> > > >>>>
> > > >>>> in control of the StreamTask, by firing timers or by processing a
> > next
> > > >>>>
> > > >>>> record from upstream.
> > > >>>>
> > > >>>>
> > > >>>>
> > >
> > > >>>> The only difference of my previous proposal compared to Yun's is
> > that I
> > > >>>>
> > > >>>> did not want to send the EndOfUserRecords event in case of stop w/o
> > > >>>>
> > > >>>> drain. My thinking was that we could directly go from RUNNING to
> > > >>>>
> > > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> > > >>>>
> > > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> > > >>>>
> > >
> > > >>>> timers and processing events (without calling finish() on
> > Operator). In
> > > >>>>
> > > >>>> my initial suggestion I though we don't care about some events
> > > >>>>
> > > >>>> potentially being emitted after the savepoint was taken, as they
> > would
> > > >>>>
> > >
> > > >>>> anyway belong to the next after FINAL, which would be discarded. I
> > think
> > > >>>>
> > > >>>> though the proposal to suspend records processing and timers is a
> > > >>>>
> > >
> > > >>>> sensible thing to do and would go with the version that Yun put
> > into the
> > > >>>>
> > > >>>> FLIP Wiki.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> What do you think Till?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Dawid
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > > >>>>
> > > >>>>> Hi Till, Piotr
> > > >>>>> Very thanks for the comments!
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >
> > > >>>>> I also agree with Piotr that currently they are independent
> > mechanisms,
> > > >>>> and they are basically the same
> > > >>>>
> > > >>>>> for the event time.
> > > >>>>> For more details, first there are some difference among the three
> > > >>>> scenarios regarding the finish:
> > > >>>>
> > >
> > > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> > not be
> > > >>>> expected to be restarted,
> > > >>>>
> > > >>>>> and for stop-with-savepoint the job would be expected restart
> > later.
> > > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> > would
> > > >>>> emit MAX_WATERMARK before the
> > > >>>>
> > >
> > > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> > endOfInput /
> > > >>>> finish() should also only be called
> > > >>>>
> > > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> > always
> > > >>>> occurs at the same time. After the change,
> > > >>>>
> > > >>>>> we could emit MAX_WATERMARK before endOfInput event for the finish
> > /
> > > >>>> stop-with-savepoint --drain cases.
> > > >>>>
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> > a
> > > >>>>>> WindowOperator close all windows and emit the results upon calling
> > > >>>>>> finish, for example?
> > > >>>>> As discussed above for stop-with-savepoint, we would always keep
> > the
> > > >>>> window as is, and restore them after restart.
> > > >>>>
> > > >>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps
> > it
> > > >>>> depends on the Triggers. For
> > > >>>>
> > >
> > > >>>>> event-time triggers / process time triggers, it would be
> > reasonable to
> > > >>>> flush all the windows since logically
> > > >>>>
> > >
> > > >>>>> the time would always elapse and the window would always get
> > triggered
> > > >>>> in a logical future. But for triggers
> > > >>>>
> > >
> > > >>>>> like CountTrigger, no matter how much time pass logically, the
> > windows
> > > >>>> would not trigger, thus we may not
> > > >>>>
> > > >>>>> flush these windows. If there are requirements we may provide
> > > >>>> additional triggers.
> > > >>>>
> > >
> > > >>>>>> It's a bit messy and I'm not sure if this should be strengthened
> > out?
> > > >>>> Each one of those has a little bit different semantic/meaning,
> > > >>>>
> > >
> > > >>>>>> but at the same time they are very similar. For single input
> > operators
> > > >>>> `endInput()` and `finish()` are actually the very same thing.
> > > >>>>
> > >
> > > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> > at the
> > > >>>> same time, and for single input operators `endInput()` and
> > `finish()`
> > > >>>>
> > >
> > > >>>>> are indeed the same thing. During the last discussion we ever
> > mentioned
> > > >>>> this issue and at then we thought that we might deprecate
> > `endInput()`
> > > >>>>
> > > >>>>> in the future, then we would only have endInput(int input) and
> > > >>>> finish().
> > > >>>>
> > > >>>>> Best,
> > > >>>>> Yun
> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > > >>>>> ------------------------------------------------------------------
> > > >>>>> From:Piotr Nowojski
> > > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > > >>>>> To:dev
> > > >>>>> Cc:Yun Gao
> > > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > >>>> Finished
> > > >>>>
> > > >>>>> Hi Till,
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> > a>
> > > >>>> WindowOperator close all windows and emit the results upon calling
> > > >>>>
> > > >>>>>> finish, for example?
> > > >>>>> 1) currently they are independent but parallel mechanisms. With
> > event
> > > >>>> time, they are basically the same.
> > > >>>>
> > > >>>>> 2) it probably should for the sake of processing time windows.
> > > >>>>> Here you are touching the bit of the current design that I like the
> > >
> > > >>>> least. We basically have now three different ways of conveying very
> > similar
> > > >>>> things:
> > > >>>>
> > > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
> > > >>>> about processing time?)
> > > >>>>
> > > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
> > > >>>> internal state
> > > >>>>
> > > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> > out?
> > >
> > > >>>> Each one of those has a little bit different semantic/meaning, but
> > at the
> > >
> > > >>>> same time they are very similar. For single input operators
> > `endInput()`
> > > >>>> and `finish()` are actually the very same thing.
> > > >>>>
> > > >>>>> Piotrek
> > > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > > >>>>> Thanks for updating the FLIP. Based on the new section about
> > > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> > > >>>>> WindowOperator close all windows and emit the results upon calling
> > > >>>>> finish, for example?
> > > >>>>> Cheers,
> > > >>>>> Till
> > > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > > >>>> problem.
> > > >>>>
> > > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > > >>>>>> Cheers,
> > > >>>>>> Till
> > > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > > >>>>>> wrote:
> > > >>>>>>> Hi Till,
> > > >>>>>>> Very thanks for the review and comments!
> > > >>>>>>> 1) First I think in fact we could be able to do the computation
> > > >>>> outside
> > > >>>>
> > > >>>>>>> of the main thread,
> > > >>>>>>> and the current implementation mainly due to the computation is
> > in
> > > >>>>>>> general fast and we
> > > >>>>>>> initially want to have a simplified first version.
> > >
> > > >>>>>>> The main requirement here is to have a constant view of the
> > state of
> > > >>>> the
> > > >>>>
> > > >>>>>>> tasks, otherwise
> > > >>>>>>> for example if we have A -> B, if A is running when we check if
> > we
> > > >>>> need
> > > >>>>
> > > >>>>>>> to trigger A, we will
> > > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> > check
> > > >>>> B, we
> > > >>>>
> > > >>>>>>> will also mark B as
> > >
> > > >>>>>>> have to trigger, then B will receive both rpc trigger and
> > checkpoint
> > > >>>>>>> barrier, which would break
> > > >>>>>>> some assumption on the task side and complicate the
> > implementation.
> > >
> > > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> > of the
> > > >>>>>>> tasks' state and then do the
> > >
> > > >>>>>>> computation, both the two step do not need to be in the main
> > thread.
> > >
> > > >>>>>>> 2) For the computation logic, in fact currently we benefit a lot
> > from
> > > >>>>>>> some shortcuts on all-to-all
> > >
> > > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> > could do
> > > >>>>>>> checks on the job vertex level
> > > >>>>>>> first and skip some job vertices as a whole. With this
> > optimization
> > > >>>> we
> > > >>>>
> > > >>>>>>> have a O(V) algorithm, and the
> > > >>>>>>> current running time of the worst case for a job with 320,000
> > tasks
> > > >>>> is
> > > >>>>
> > > >>>>>>> less than 100ms. For
> > > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > > >>>>>>> If we do the computation based on the last triggered tasks, we
> > may
> > > >>>> not
> > > >>>>
> > > >>>>>>> easily encode this information
> > >
> > > >>>>>>> into the shortcuts on the job vertex level. And since the time
> > seems
> > > >>>> to
> > > >>>>
> > > >>>>>>> be short, perhaps it is enough
> > > >>>>>>> to do re-computation from the scratch in consideration of the
> > > >>>> tradeoff
> > > >>>>
> > > >>>>>>> between the performance and
> > > >>>>>>> the complexity ?
> > > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > > >>>> finish()
> > > >>>>
> > > >>>>>>> method and before the last
> > >
> > > >>>>>>> snapshotState() method so that we could shut down the whole
> > topology
> > > >>>> with
> > > >>>>
> > > >>>>>>> a single final checkpoint.
> > > >>>>>>> Very sorry for not include enough details for this part and I'll
> > > >>>>>>> complement the FLIP with the details on
> > > >>>>>>> the process of the final checkpoint / savepoint.
> > > >>>>>>> Best,
> > > >>>>>>> Yun
> > > >>>>>>>
> > ------------------------------------------------------------------
> > > >>>>>>> From:Till Rohrmann
> > > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > > >>>>>>> To:dev
> > > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> > Tasks
> > > >>>>>>> Finished
> > > >>>>>>> Hi everyone,
> > > >>>>>>> I am a bit late to the voting party but let me ask three
> > questions:
> > >
> > > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> > thread
> > > >>>> if we
> > > >>>>
> > >
> > > >>>>>>> cannot guarantee that all tasks are still running when
> > triggering the
> > > >>>>>>> checkpoint? Couldn't we do the computation in a different thread
> > in
> > > >>>> order
> > > >>>>
> > > >>>>>>> to relieve the main thread a bit.
> > > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> > seems
> > > >>>> to go
> > > >>>>
> > > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> > more
> > > >>>>>>> efficient to maintain the set of current tasks to trigger and
> > check
> > > >>>>>>> whether
> > > >>>>>>> anything has changed and if so check the succeeding tasks until
> > we
> > > >>>> have
> > > >>>>
> > > >>>>>>> found the current checkpoint trigger frontier?
> > > >>>>>>> 3) When are we going to send the endOfInput events to a
> > downstream
> > > >>>> task?
> > > >>>>
> > > >>>>>>> If
> > >
> > > >>>>>>> this happens after we call finish on the upstream operator but
> > before
> > > >>>>>>> snapshotState then it would be possible to shut down the whole
> > > >>>> topology
> > > >>>>
> > >
> > > >>>>>>> with a single final checkpoint. I think this part could benefit
> > from
> > > >>>> a bit
> > > >>>>
> > > >>>>>>> more detailed description in the FLIP.
> > > >>>>>>> Cheers,
> > > >>>>>>> Till
> > > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > > >>>>>>> wrote:
> > > >>>>>>>> Hi there,
> > > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
> > > >>>> vote
> > > >>>>
> > > >>>>>>> now.
> > > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > > >>>>>>>> - Dawid Wysakowicz (binding)
> > > >>>>>>>> - Piotr Nowojski(binding)
> > > >>>>>>>> - Jiangang Liu (binding)
> > > >>>>>>>> - Arvid Heise (binding)
> > > >>>>>>>> - Jing Zhang (binding)
> > > >>>>>>>> - Leonard Xu (non-binding)
> > > >>>>>>>> - Guowei Ma (binding)
> > > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > > >>>> accepted.
> > > >>>>
> > > >>>>>>>> Very thanks everyone!
> > > >>>>>>>> Best,
> > > >>>>>>>> Yun
> > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > >>>>
> > > >>>>
> > >
> > >
> >
>
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Till,
Very thanks for the comments and tips! We will update the FLIP with the new details and we
should need a new vote since the FLIP is updated a large part, I think we would start the
vote right after we also solve the remaining UnionListState problem and have a double
check on that we have no other missing points for the design and FLIP~
Best,
Yun
------------------------------------------------------------------
From:Till Rohrmann <tr...@apache.org>
Send Time:2021 Jul. 22 (Thu.) 17:57
To:Yun Gao <yu...@aliyun.com>
Cc:Piotr Nowojski <pn...@apache.org>; dev <de...@flink.apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Thanks everyone for this discussion. I think this is very helpful.
I do agree with Piotr's proposal to separate state and the lifecycle of a StreamOperator. That way the finished state can be used to recover a StreamOperator or to rescale the topology. I also believe that this will make the implementation a lot easier since we don't have to remember which StreamOperator has actually finished. Moreover, it solves the problem of rescaling.
I also agree with the proposed solution for checkpoints after finish() has been called. I guess the first checkpoint that was triggered after finish() and sends notifyCheckpointComplete should be good enough to close the StreamTask.
Shall we update the FLIP with these details? Do we need another vote for it or shall we continue w/o it?
Cheers,
Till
On Thu, Jul 22, 2021 at 10:45 AM Yun Gao <yu...@aliyun.com> wrote:
Hi Piotr,
Very thanks for the explanation! and very sorry that initially I should wrongly understand the problem Dawid proposed.
> And what should we do now? We can of course commit all transactions until checkpoint 43.
> But should we keep waiting for `notyifyCheckpointComplete(44)`?
> hat if in the meantime another checkpoint is triggered? We could end up waiting indefinitely.
> Our proposal is to shutdown the task immediately after seeing first `notifyCheckpointComplete(X)`,
> where X is any triggered checkpoint AFTER `finish()`. This should be fine, as:
> a) ideally there should be no new pending transactions opened after checkpoint 42
> b) even if operator/function is opening some transactions for checkpoint 43 and
> checkpoint 44 (`FlinkKafkaProducer`), those transactions after checkpoint 42 should be empty
Now I understand more on this issue: currently operators like the TwoPhaseCommitSinkFunctions would create
a new transaction when snapshotting state, thus for each checkpoint there would be one corresponding
transaction, thus after finish() and before notifyCheckpointComplete(), there might be some empty
transactions.
I also agree with the proposal: we could finish the task after we received the first checkpoint complete
notification, and the checkpoint with the larger checkpoint id should be allowed to be finished. The
checkpoint complete notification should be able to commit all the non-empty transactions. A part of
the empty transactions would also get committed, but it should cause no harm. The other empty
transactions would be aborted in close(), which should also cause no harm.
> If checkpoint 44 completes afterwards, it will still be valid. Ideally we would recommend that after
seeing `finish()` operators/functions should not be opening any new transactions, but that shouldn't be required.
And from another of view, I think perhaps it might also be acceptable that we have some requirements for
the pattern of the operators, like the sink would be able to skip creating new transactions after finish() is called.
Perhaps we may treat the process as a protocol between the framework and the operators, and the operators
might need to follow the protocol. Of course it would be better that the framework could handle all the cases
elegantly, and put less implicit limitation to the operators, and at least we should guarantee not cause compatibility
problem after changing the process.
Very thanks for the careful checks on the whole process.
Best,
Yun
------------------------------------------------------------------
From:Piotr Nowojski <pn...@apache.org>
Send Time:2021 Jul. 22 (Thu.) 15:33
To:dev <de...@flink.apache.org>; Yun Gao <yu...@aliyun.com>
Cc:Till Rohrmann <tr...@apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Guowei,
> Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can becalled repeatedly.
Just to clarify. It's not about calling `finish()` and `endInput()` repeatedly, but about (from the perspective of operator's state)
1. seeing `finish()`
2. checkpoint X triggered and completed
3. failure + recovery from X
4. potentially processing more records
5. another `finish()`
But from the context of the remaining part of your message Guowei I presume that you have already got that point :)
Yun:
> For this issue perhaps we could explicitly requires the task to wait for a checkpoint triggered after finish()> method is called for all the operators ? We could be able to achieve this target by maintaining
> some state inside the task.
Isn't this exactly the "WAITING_FOR_FINAL_CP" from the FLIP document? That we always need to wait for a checkpoint triggered after `finish()` to complete, before shutting down a task?
What Dawid was describing is a scenario where:
1. task/operator received `finish()`
2. checkpoint 42 triggered (not yet completed)
3. checkpoint 43 triggered (not yet completed)
4. checkpoint 44 triggered (not yet completed)
5. notifyCheckpointComplete(43)
And what should we do now? We can of course commit all transactions until checkpoint 43. But should we keep waiting for `notyifyCheckpointComplete(44)`? What if in the meantime another checkpoint is triggered? We could end up waiting indefinitely.
Our proposal is to shutdown the task immediately after seeing first `notifyCheckpointComplete(X)`, where X is any triggered checkpoint AFTER `finish()`. This should be fine, as:
a) ideally there should be no new pending transactions opened after checkpoint 42
b) even if operator/function is opening some transactions for checkpoint 43 and checkpoint 44 (`FlinkKafkaProducer`), those transactions after checkpoint 42 should be empty
Hence comment from Dawid
> We must make sure those checkpoints do not leave lingering external resources.
After seeing 5. (notifyCheckpointComplete(43)) It should be good enough to:
- commit transactions from checkpoint 42, (and 43 if they were created, depends on the user code)
- close operator, aborting any pending transactions (for checkpoint 44 if they were opened, depends on the user code)
If checkpoint 44 completes afterwards, it will still be valid. Ideally we would recommend that after seeing `finish()` operators/functions should not be opening any new transactions, but that shouldn't be required.
Best,
Piotrek
czw., 22 lip 2021 o 09:00 Yun Gao <yu...@aliyun.com.invalid> napisał(a):
Hi Dawid, Piotr, Steven,
Very thanks for pointing out these issues and very thanks for the discussion !
Failure before notifyCheckpointComplete()
For this issue I would agree with what Piotr has proposed. I tried to use some
operators like sink / window as example and currently I also do not found
explicit scenarios that might cause problems if records are processed by a
task that is assigned with states snapshotted after calling finish() before.
For the future cases it seems users should be able to implement their
target logic by explicitly add a flag regarding finished, and perhaps have
different logic if this part of states are referred to. Besides, this case would
only happen on rescaling or topology change, which embedded some kind of user
knowledge inside the action. Thus it looks acceptable that we still split the operators
state from the task lifecycle, and do not treat checkpoint after finish() differently.
Finishing upon receiving notifyCheckpointComplete() of not the latest checkpoint
For this issue perhaps we could explicitly requires the task to wait for a checkpoint triggered after finish()
method is called for all the operators ? We could be able to achieve this target by maintaining
some state inside the task.
Checkpointing from a single subtask / UnionListState case
This should indeed cause problems, and I also agree with that we could focus on this thread in the
https://issues.apache.org/jira/browse/FLINK-21080.
Best,
Yun
------------------------------------------------------------------
From:Piotr Nowojski <pn...@apache.org>
Send Time:2021 Jul. 22 (Thu.) 02:46
To:dev <de...@flink.apache.org>
Cc:Yun Gao <yu...@aliyun.com>; Till Rohrmann <tr...@apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Steven,
> I probably missed sth here. isn't this the case today already? Why is it a concern for the proposed change?
The problem is with the newly added `finish()` method and the already existing `endInput()` call. Currently on master there are no issues, because we are not checkpointing any operators after some operators have finished. The purpose of this FLIP-147 is to exactly enable this and this opens a new problem described by Dawid.
To paraphrase and to give a concrete example. Assume we have an operator with parallelism of two. Subtask 0 and subtask 1.
1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1 hasn't (yet).
2. Checkpoint 42 is triggered, and it completes.
3. Job fails and is restarted, but at the same time it's rescaled. User has chosen to scale down his operator down to 1.
Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has been processed or not, so while recovering to checkpoint 42, we have to recover both finished subtask 0 (#1) state and not yet finished subtask 1's (#1). But at the same time they are scaled down, so we only have a single subtask 0 (#2) that has a combined state from both of the previous instances. The potentially confusing issue is that the state from subtask 0 (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's recovered to an operator that has still some records to process. In step 1. an user for example could store on the operator's state a bit of information "end input has been already called!", that after recovery would no longer be true.
Hence the question about `finish()` and `endInput()` semantics. Should it be tied down to state, or just to an operator instance/execution attempt?
Piotrek
śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> if a failure happens after sequence of finish() -> snapshotState(), but
before notifyCheckpointComplete(), we will restore such a state and we
might end up sending some more records to such an operator.
I probably missed sth here. isn't this the case today already? Why is it a
concern for the proposed change?
On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi Dawid,
>
> Thanks for writing down those concerns.
>
> I think the first issue boils down what should be the contract of lifecycle
> methods like open(), close(), initializeState() etc and especially the new
> additions like finish() and endInput(). And what should be their relation
> with the operator state (regardless of it's type keyed, non-keyed, union,
> ...). Should those methods be tied to state or not? After thinking about it
> for a while (and discussing it offline with Dawid), I think the answer
> might be no, they shouldn't. I mean maybe we should just openly say that
> all of those methods relate to this single particular instance and
> execution of the operator. And if a job is recovered/rescaled, we would be
> allowed to freely resume consumption, ignoring a fact that maybe some parts
> of the state have previously seen `endInput()`. Why?
>
> 0. Yes, it might be confusing. Especially with `endInput()`. We call
> `endInput()`, we store something in a state and later after recovery
> combined with rescaling that state can see more records? Indeed weird,
> 1. I haven't come up yet with a counterexample that would break and make
> impossible to implement a real life use case. Theoretically yes, the user
> can store `endInput()` on state, and after rescaling this state would be
> inconsistent with what is actually happening with the operator, but I
> haven't found a use case that would break because of that.
> 2. Otherwise, implementation would be very difficult.
> 3. It's difficult to access keyed state from within `endInput()`/`finish()`
> calls, as they do not have key context.
> 4. After all, openly defining `endInput()` and `finish()` to be tied with
> it's operator execution instance lifecycle is not that strange and quite
> simple to explain. Sure, it can lead to a bit of confusion (0.), but that
> doesn't sound that bad in comparison with the alternatives that I'm aware
> of. Also currently methods like `open()` and `close()` are also tied to the
> operator execution instance, not to the state. Operators can be opened and
> closed multiple times, it doesn't mean that the state is lost after closing
> an operator.
>
> For the UnionListState problem I have posted my proposal in the ticket [1],
> so maybe let's move that particular discussion there?
>
> Piotrek
>
> [1] https://issues.apache.org/jira/browse/FLINK-21080
>
> śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> napisał(a):
>
> > Hey all,
> >
> > To make the issues that were found transparent to the community, I want
> to
> > post an update:
> >
> > *1. Committing side-effects*
> > We do want to make sure that all side effects are committed before
> > bringing tasks down. Side effects are committed when calling
> > notifyCheckpointComplete. For the final checkpoint we introduced the
> method
> > finish(). This notifies the operator that we have consumed all incoming
> > records and we are preparing to close the Task. In turn we should flush
> any
> > pending buffered records and prepare to commit last transactions. The
> goal
> > is that after a successful sequence of finish() -> snapshotState() ->
> > notifyCheckpointComplete(), the remaining state can be considered
> > empty/finished and may be discarded.
> >
> > *Failure before notifyCheckpointComplete()*
> >
> > The question is what is the contract of the endInput()/finish() methods
> > and how do calling these methods affect the operators keyed, non-keyed
> > state and external state. Is it allowed to restore state snapshot taken
> > after calling endInput()/finish() and process more records? Or do we
> assume
> > that after a restore from such a state taken after finish() we should not
> > call any of the lifecycle methods or at least make sure those methods do
> > not emit records/interact with mailbox etc.
> >
> > Currently it is possible that if a failure happens after sequence of
> > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> will
> > restore such a state and we might end up sending some more records to
> such
> > an operator. It is possible if we rescale and this state is merged with a
> > state of a subtask that has not called finish() yet. It can also happen
> if
> > we rescale the upstream operator and the subtask of interest becomes
> > connected to a newly added non finished subtask.
> >
> > *Snapshotting StreamTasks that finish() has been called*
> >
> >
> > We thought about putting a flag into the snapshot of a subtask produced
> > after the finish() method. This would make it possible to skip execution
> of
> > certain lifecycle methods. Unfortunately this creates problems for
> > rescaling. How do we deal with a situation that subtask states with both
> > the feature flag set and unset end up in a single StreamTask. Additional
> > problem is that we merge those states into a single OperatorSubtaskState
> on
> > CheckpointCoordinator.
> >
> > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > checkpoint*
> >
> > We need to wait for a checkpoint to complete, that started after the
> > finish() method. However, we support concurrent checkpoints therefore,
> > there might be later checkpoints that completed, but the notification has
> > not arrived. We must make sure those checkpoints do not leave lingering
> > external resources.
> >
> > *Checkpointing from a single subtask / UnionListState case*
> > There are operators that checkpoint from a single subtask only. Usually
> > from the subtask index=0. If we let those subtasks finish, subsequent
> > checkpoints will miss this information.
> > Esp. Legacy sources problem:
> > https://issues.apache.org/jira/browse/FLINK-21080
> >
> > Best,
> >
> > Dawid
> > On 19/07/2021 15:10, Yun Gao wrote:
> >
> > Hi Till, Dawid,
> >
> > Very thanks for the comments and discussion! Glad that it seems we have
> > come to a convergence, and I also agree with that we could not include
> the
> > optimization in the first version.
> >
> > Best
> > Yun
> >
> > ------------------------------------------------------------------
> > From:Dawid Wysakowicz <dw...@apache.org> <dw...@apache.org>
> > Send Time:2021 Jul. 19 (Mon.) 17:51
> > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > <tr...@apache.org> <tr...@apache.org>
> > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > Finished
> >
> > Small correction. I meant we need to adjust the EndOfInputEvent of
> course.
> >
> > Best,
> >
> > Dawid
> >
> > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > Hey Till,
> > >
> > > Yes, you're right we will have to adjust the current state of
> > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > we're discussing here. We are aware of that.
> > >
> > > As for the MAX_WATERMARK vs finish(). My take is that we should always
> > > emit MAX_WATERMARK before calling finish() on an operator. At the same
> > > time finish() should not leave behind anything in state, as the
> > > intention is that we never restore from the taken savepoint/checkpoint
> > > (savepoint w drain or bounded data consumed).
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > >> Hi Yun and Dawid,
> > >>
> >
> > >> Thanks for your comments. I do agree with your comments that finish()
> can
> > >> do more than MAX_WATERMARK. I guess we should then explain how
> > >> MAX_WATERMARK and finish() play together and what kind of
> > >> order guarantees we provide.
> > >>
> >
> > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether it
> would
> >
> > >> work in its current state because we send this event when the Task is
> about
> > >> to shut down if I am not mistaken. What we want to have is to bring
> the
> >
> > >> StreamTasks into a state so that they shut down on the next
> checkpoint. For
> > >> this we need to keep the StreamTask running. In general, I am a fan of
> > >> making things explicit if possible. I think this helps maintenance and
> >
> > >> evolvability of code. That's why I think sending an EndOfInputEvent
> which
> > >> is a StreamTask level event and which says that there won't be any
> other
> > >> records coming only control events could make sense.
> > >>
> > >> I would leave the proposed optimization out of the first version. We
> can
> > >> still add it at a later point in time.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > <dw...@apache.org> <dw...@apache.org>
> > >> wrote:
> > >>
> >
> > >>> Personally I don't find this optimization important and I'd rather
> leave
> >
> > >>> it out not to complicate the codebase further. I doubt we save much
> there.
> > >>> I don't have a strong opinion though.
> > >>>
> > >>> Best,
> > >>>
> > >>> Dawid
> > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> Very thanks Dawid for the thoughts!
> > >>>
> > >>> Currently I also do not have different opinions regarding this part.
> > >>> But I have one more issue to confirm: during the previous discussion
> we
> > >>> have discussed that for the final checkpoint case, we might have an
> > >>> optmization
> >
> > >>> that if a task do not have operators using 2-pc, we might skip
> waiting for
> > >>> the
> >
> > >>> final checkpoint (but we could not skip the savepoint). To allow
> users to
> > >>> express
> > >>> the logic, we have proposed to add one more method to StreamOperator
> &
> > >>> CheckpointListener:
> > >>>
> > >>> interface StreamOperator {
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> interface CheckpointListener {
> > >>>
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> class AbstractUdfStreamOperator {
> > >>>
> > >>> @Override
> > >>> boolean requiresFinalCheckpoint() {
> > >>> return userFunction instanceof CheckpointListener &&
> >
> > >>> ((CheckpointListener)
> userFunction).requiresFinalCheckpoint();
> > >>> }
> > >>> }
> > >>>
> > >>>
> > >>> I think we should still keep the change ?
> > >>>
> > >>> Best,
> > >>> Yun
> > >>>
> > >>> ------------------Original Mail ------------------
> > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > <dw...@apache.org>
> > >>> <dw...@apache.org> <dw...@apache.org>
> > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>
> > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>> Finished
> > >>>
> > >>>> I think we're all really close to the same solution.
> > >>>>
> > >>>>
> > >>>>
> > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
> > >>>>
> >
> > >>>> buffering, but it does not solve flushing other operations such as
> e.g.
> > >>>>
> > >>>> count windows or batching requests in Sinks. I'd prefer to treat the
> > >>>>
> > >>>> finish() as a message for Operator to "flush all records". The
> > >>>>
> >
> > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> imo. I
> > >>>>
> > >>>> don't think operators need to get a signal "stop-processing" if they
> > >>>>
> > >>>> don't need to flush records. The "WHEN" records are emitted, should
> be
> > >>>>
> > >>>> in control of the StreamTask, by firing timers or by processing a
> next
> > >>>>
> > >>>> record from upstream.
> > >>>>
> > >>>>
> > >>>>
> >
> > >>>> The only difference of my previous proposal compared to Yun's is
> that I
> > >>>>
> > >>>> did not want to send the EndOfUserRecords event in case of stop w/o
> > >>>>
> > >>>> drain. My thinking was that we could directly go from RUNNING to
> > >>>>
> > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> > >>>>
> > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> > >>>>
> >
> > >>>> timers and processing events (without calling finish() on
> Operator). In
> > >>>>
> > >>>> my initial suggestion I though we don't care about some events
> > >>>>
> > >>>> potentially being emitted after the savepoint was taken, as they
> would
> > >>>>
> >
> > >>>> anyway belong to the next after FINAL, which would be discarded. I
> think
> > >>>>
> > >>>> though the proposal to suspend records processing and timers is a
> > >>>>
> >
> > >>>> sensible thing to do and would go with the version that Yun put
> into the
> > >>>>
> > >>>> FLIP Wiki.
> > >>>>
> > >>>>
> > >>>>
> > >>>> What do you think Till?
> > >>>>
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>>
> > >>>>
> > >>>> Dawid
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > >>>>
> > >>>>> Hi Till, Piotr
> > >>>>> Very thanks for the comments!
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > >>>>> I also agree with Piotr that currently they are independent
> mechanisms,
> > >>>> and they are basically the same
> > >>>>
> > >>>>> for the event time.
> > >>>>> For more details, first there are some difference among the three
> > >>>> scenarios regarding the finish:
> > >>>>
> >
> > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> not be
> > >>>> expected to be restarted,
> > >>>>
> > >>>>> and for stop-with-savepoint the job would be expected restart
> later.
> > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> would
> > >>>> emit MAX_WATERMARK before the
> > >>>>
> >
> > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> endOfInput /
> > >>>> finish() should also only be called
> > >>>>
> > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> always
> > >>>> occurs at the same time. After the change,
> > >>>>
> > >>>>> we could emit MAX_WATERMARK before endOfInput event for the finish
> /
> > >>>> stop-with-savepoint --drain cases.
> > >>>>
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a
> > >>>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>>> finish, for example?
> > >>>>> As discussed above for stop-with-savepoint, we would always keep
> the
> > >>>> window as is, and restore them after restart.
> > >>>>
> > >>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps
> it
> > >>>> depends on the Triggers. For
> > >>>>
> >
> > >>>>> event-time triggers / process time triggers, it would be
> reasonable to
> > >>>> flush all the windows since logically
> > >>>>
> >
> > >>>>> the time would always elapse and the window would always get
> triggered
> > >>>> in a logical future. But for triggers
> > >>>>
> >
> > >>>>> like CountTrigger, no matter how much time pass logically, the
> windows
> > >>>> would not trigger, thus we may not
> > >>>>
> > >>>>> flush these windows. If there are requirements we may provide
> > >>>> additional triggers.
> > >>>>
> >
> > >>>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> > >>>> Each one of those has a little bit different semantic/meaning,
> > >>>>
> >
> > >>>>>> but at the same time they are very similar. For single input
> operators
> > >>>> `endInput()` and `finish()` are actually the very same thing.
> > >>>>
> >
> > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> at the
> > >>>> same time, and for single input operators `endInput()` and
> `finish()`
> > >>>>
> >
> > >>>>> are indeed the same thing. During the last discussion we ever
> mentioned
> > >>>> this issue and at then we thought that we might deprecate
> `endInput()`
> > >>>>
> > >>>>> in the future, then we would only have endInput(int input) and
> > >>>> finish().
> > >>>>
> > >>>>> Best,
> > >>>>> Yun
> > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > >>>>> ------------------------------------------------------------------
> > >>>>> From:Piotr Nowojski
> > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > >>>>> To:dev
> > >>>>> Cc:Yun Gao
> > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>>> Finished
> > >>>>
> > >>>>> Hi Till,
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a>
> > >>>> WindowOperator close all windows and emit the results upon calling
> > >>>>
> > >>>>>> finish, for example?
> > >>>>> 1) currently they are independent but parallel mechanisms. With
> event
> > >>>> time, they are basically the same.
> > >>>>
> > >>>>> 2) it probably should for the sake of processing time windows.
> > >>>>> Here you are touching the bit of the current design that I like the
> >
> > >>>> least. We basically have now three different ways of conveying very
> similar
> > >>>> things:
> > >>>>
> > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
> > >>>> about processing time?)
> > >>>>
> > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
> > >>>> internal state
> > >>>>
> > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> >
> > >>>> Each one of those has a little bit different semantic/meaning, but
> at the
> >
> > >>>> same time they are very similar. For single input operators
> `endInput()`
> > >>>> and `finish()` are actually the very same thing.
> > >>>>
> > >>>>> Piotrek
> > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > >>>>> Thanks for updating the FLIP. Based on the new section about
> > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> > >>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>> finish, for example?
> > >>>>> Cheers,
> > >>>>> Till
> > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > >>>> problem.
> > >>>>
> > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > >>>>>> Cheers,
> > >>>>>> Till
> > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > >>>>>> wrote:
> > >>>>>>> Hi Till,
> > >>>>>>> Very thanks for the review and comments!
> > >>>>>>> 1) First I think in fact we could be able to do the computation
> > >>>> outside
> > >>>>
> > >>>>>>> of the main thread,
> > >>>>>>> and the current implementation mainly due to the computation is
> in
> > >>>>>>> general fast and we
> > >>>>>>> initially want to have a simplified first version.
> >
> > >>>>>>> The main requirement here is to have a constant view of the
> state of
> > >>>> the
> > >>>>
> > >>>>>>> tasks, otherwise
> > >>>>>>> for example if we have A -> B, if A is running when we check if
> we
> > >>>> need
> > >>>>
> > >>>>>>> to trigger A, we will
> > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> check
> > >>>> B, we
> > >>>>
> > >>>>>>> will also mark B as
> >
> > >>>>>>> have to trigger, then B will receive both rpc trigger and
> checkpoint
> > >>>>>>> barrier, which would break
> > >>>>>>> some assumption on the task side and complicate the
> implementation.
> >
> > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> of the
> > >>>>>>> tasks' state and then do the
> >
> > >>>>>>> computation, both the two step do not need to be in the main
> thread.
> >
> > >>>>>>> 2) For the computation logic, in fact currently we benefit a lot
> from
> > >>>>>>> some shortcuts on all-to-all
> >
> > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> could do
> > >>>>>>> checks on the job vertex level
> > >>>>>>> first and skip some job vertices as a whole. With this
> optimization
> > >>>> we
> > >>>>
> > >>>>>>> have a O(V) algorithm, and the
> > >>>>>>> current running time of the worst case for a job with 320,000
> tasks
> > >>>> is
> > >>>>
> > >>>>>>> less than 100ms. For
> > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > >>>>>>> If we do the computation based on the last triggered tasks, we
> may
> > >>>> not
> > >>>>
> > >>>>>>> easily encode this information
> >
> > >>>>>>> into the shortcuts on the job vertex level. And since the time
> seems
> > >>>> to
> > >>>>
> > >>>>>>> be short, perhaps it is enough
> > >>>>>>> to do re-computation from the scratch in consideration of the
> > >>>> tradeoff
> > >>>>
> > >>>>>>> between the performance and
> > >>>>>>> the complexity ?
> > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > >>>> finish()
> > >>>>
> > >>>>>>> method and before the last
> >
> > >>>>>>> snapshotState() method so that we could shut down the whole
> topology
> > >>>> with
> > >>>>
> > >>>>>>> a single final checkpoint.
> > >>>>>>> Very sorry for not include enough details for this part and I'll
> > >>>>>>> complement the FLIP with the details on
> > >>>>>>> the process of the final checkpoint / savepoint.
> > >>>>>>> Best,
> > >>>>>>> Yun
> > >>>>>>>
> ------------------------------------------------------------------
> > >>>>>>> From:Till Rohrmann
> > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >>>>>>> To:dev
> > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > >>>>>>> Finished
> > >>>>>>> Hi everyone,
> > >>>>>>> I am a bit late to the voting party but let me ask three
> questions:
> >
> > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> thread
> > >>>> if we
> > >>>>
> >
> > >>>>>>> cannot guarantee that all tasks are still running when
> triggering the
> > >>>>>>> checkpoint? Couldn't we do the computation in a different thread
> in
> > >>>> order
> > >>>>
> > >>>>>>> to relieve the main thread a bit.
> > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> seems
> > >>>> to go
> > >>>>
> > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> more
> > >>>>>>> efficient to maintain the set of current tasks to trigger and
> check
> > >>>>>>> whether
> > >>>>>>> anything has changed and if so check the succeeding tasks until
> we
> > >>>> have
> > >>>>
> > >>>>>>> found the current checkpoint trigger frontier?
> > >>>>>>> 3) When are we going to send the endOfInput events to a
> downstream
> > >>>> task?
> > >>>>
> > >>>>>>> If
> >
> > >>>>>>> this happens after we call finish on the upstream operator but
> before
> > >>>>>>> snapshotState then it would be possible to shut down the whole
> > >>>> topology
> > >>>>
> >
> > >>>>>>> with a single final checkpoint. I think this part could benefit
> from
> > >>>> a bit
> > >>>>
> > >>>>>>> more detailed description in the FLIP.
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > >>>>>>> wrote:
> > >>>>>>>> Hi there,
> > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
> > >>>> vote
> > >>>>
> > >>>>>>> now.
> > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >>>>>>>> - Dawid Wysakowicz (binding)
> > >>>>>>>> - Piotr Nowojski(binding)
> > >>>>>>>> - Jiangang Liu (binding)
> > >>>>>>>> - Arvid Heise (binding)
> > >>>>>>>> - Jing Zhang (binding)
> > >>>>>>>> - Leonard Xu (non-binding)
> > >>>>>>>> - Guowei Ma (binding)
> > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > >>>> accepted.
> > >>>>
> > >>>>>>>> Very thanks everyone!
> > >>>>>>>> Best,
> > >>>>>>>> Yun
> > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>>>
> > >>>>
> >
> >
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
Thanks everyone for this discussion. I think this is very helpful.
I do agree with Piotr's proposal to separate state and the lifecycle of a
StreamOperator. That way the finished state can be used to recover a
StreamOperator or to rescale the topology. I also believe that this will
make the implementation a lot easier since we don't have to remember which
StreamOperator has actually finished. Moreover, it solves the problem of
rescaling.
I also agree with the proposed solution for checkpoints after finish() has
been called. I guess the first checkpoint that was triggered after finish()
and sends notifyCheckpointComplete should be good enough to close the
StreamTask.
Shall we update the FLIP with these details? Do we need another vote for it
or shall we continue w/o it?
Cheers,
Till
On Thu, Jul 22, 2021 at 10:45 AM Yun Gao <yu...@aliyun.com> wrote:
> Hi Piotr,
>
> Very thanks for the explanation! and very sorry that initially I should
> wrongly understand the problem Dawid proposed.
>
> > And what should we do now? We can of course commit all transactions
> until checkpoint 43.
> > But should we keep waiting for `notyifyCheckpointComplete(44)`?
> > hat if in the meantime another checkpoint is triggered? We could end up
> waiting indefinitely.
>
> > Our proposal is to shutdown the task immediately after seeing first
> `notifyCheckpointComplete(X)`,
> > where X is any triggered checkpoint AFTER `finish()`. This should be
> fine, as:
> > a) ideally there should be no new pending transactions opened after
> checkpoint 42
> > b) even if operator/function is opening some transactions for
> checkpoint 43 and
> > checkpoint 44 (`FlinkKafkaProducer`), those transactions after
> checkpoint 42 should be empty
>
> Now I understand more on this issue: currently operators like the
> TwoPhaseCommitSinkFunctions would create
> a new transaction when snapshotting state, thus for each checkpoint there
> would be one corresponding
> transaction, thus after finish() and before notifyCheckpointComplete(),
> there might be some empty
> transactions.
>
> I also agree with the proposal: we could finish the task after we received
> the first checkpoint complete
> notification, and the checkpoint with the larger checkpoint id should be
> allowed to be finished. The
> checkpoint complete notification should be able to commit all the
> non-empty transactions. A part of
> the empty transactions would also get committed, but it should cause no
> harm. The other empty
> transactions would be aborted in close(), which should also cause no harm.
>
> > If checkpoint 44 completes afterwards, it will still be valid. Ideally
> we would recommend that after
> seeing `finish()` operators/functions should not be opening any new
> transactions, but that shouldn't be required.
>
> And from another of view, I think perhaps it might also be acceptable that
> we have some requirements for
> the pattern of the operators, like the sink would be able to skip creating
> new transactions after finish() is called.
> Perhaps we may treat the process as a protocol between the framework and
> the operators, and the operators
> might need to follow the protocol. Of course it would be better that the
> framework could handle all the cases
> elegantly, and put less implicit limitation to the operators, and at least
> we should guarantee not cause compatibility
> problem after changing the process.
>
> Very thanks for the careful checks on the whole process.
>
> Best,
> Yun
>
>
> ------------------------------------------------------------------
> From:Piotr Nowojski <pn...@apache.org>
> Send Time:2021 Jul. 22 (Thu.) 15:33
> To:dev <de...@flink.apache.org>; Yun Gao <yu...@aliyun.com>
> Cc:Till Rohrmann <tr...@apache.org>; Yun Gao
> <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> Hi Guowei,
>
> > Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can
> be
> called repeatedly.
>
> Just to clarify. It's not about calling `finish()` and `endInput()`
> repeatedly, but about (from the perspective of operator's state)
> 1. seeing `finish()`
> 2. checkpoint X triggered and completed
> 3. failure + recovery from X
> 4. potentially processing more records
> 5. another `finish()`
>
> But from the context of the remaining part of your message Guowei I
> presume that you have already got that point :)
>
> Yun:
>
> > For this issue perhaps we could explicitly requires the task to wait for
> a checkpoint triggered after finish()
> > method is called for all the operators ? We could be able to achieve
> this target by maintaining
> > some state inside the task.
>
> Isn't this exactly the "WAITING_FOR_FINAL_CP" from the FLIP document? That
> we always need to wait for a checkpoint triggered after `finish()` to
> complete, before shutting down a task?
>
> What Dawid was describing is a scenario where:
> 1. task/operator received `finish()`
> 2. checkpoint 42 triggered (not yet completed)
> 3. checkpoint 43 triggered (not yet completed)
> 4. checkpoint 44 triggered (not yet completed)
> 5. notifyCheckpointComplete(43)
>
> And what should we do now? We can of course commit all transactions until
> checkpoint 43. But should we keep waiting for
> `notyifyCheckpointComplete(44)`? What if in the meantime another checkpoint
> is triggered? We could end up waiting indefinitely.
>
> Our proposal is to shutdown the task immediately after seeing first
> `notifyCheckpointComplete(X)`, where X is any triggered checkpoint AFTER
> `finish()`. This should be fine, as:
> a) ideally there should be no new pending transactions opened after
> checkpoint 42
> b) even if operator/function is opening some transactions for checkpoint
> 43 and checkpoint 44 (`FlinkKafkaProducer`), those transactions after
> checkpoint 42 should be empty
>
> Hence comment from Dawid
> > We must make sure those checkpoints do not leave lingering external
> resources.
>
> After seeing 5. (notifyCheckpointComplete(43)) It should be good enough to:
> - commit transactions from checkpoint 42, (and 43 if they were created,
> depends on the user code)
> - close operator, aborting any pending transactions (for checkpoint 44 if
> they were opened, depends on the user code)
>
> If checkpoint 44 completes afterwards, it will still be valid. Ideally we
> would recommend that after seeing `finish()` operators/functions should not
> be opening any new transactions, but that shouldn't be required.
>
> Best,
> Piotrek
>
> czw., 22 lip 2021 o 09:00 Yun Gao <yu...@aliyun.com.invalid>
> napisał(a):
> Hi Dawid, Piotr, Steven,
>
> Very thanks for pointing out these issues and very thanks for the
> discussion !
>
> Failure before notifyCheckpointComplete()
>
> For this issue I would agree with what Piotr has proposed. I tried to use
> some
> operators like sink / window as example and currently I also do not found
> explicit scenarios that might cause problems if records are processed by a
> task that is assigned with states snapshotted after calling finish()
> before.
> For the future cases it seems users should be able to implement their
> target logic by explicitly add a flag regarding finished, and perhaps have
> different logic if this part of states are referred to. Besides, this case
> would
> only happen on rescaling or topology change, which embedded some kind of
> user
> knowledge inside the action. Thus it looks acceptable that we still split
> the operators
> state from the task lifecycle, and do not treat checkpoint after finish()
> differently.
>
> Finishing upon receiving notifyCheckpointComplete() of not the latest
> checkpoint
>
> For this issue perhaps we could explicitly requires the task to wait for a
> checkpoint triggered after finish()
> method is called for all the operators ? We could be able to achieve this
> target by maintaining
> some state inside the task.
>
> Checkpointing from a single subtask / UnionListState case
>
> This should indeed cause problems, and I also agree with that we could
> focus on this thread in the
> https://issues.apache.org/jira/browse/FLINK-21080.
>
> Best,
> Yun
>
>
> ------------------------------------------------------------------
> From:Piotr Nowojski <pn...@apache.org>
> Send Time:2021 Jul. 22 (Thu.) 02:46
> To:dev <de...@flink.apache.org>
> Cc:Yun Gao <yu...@aliyun.com>; Till Rohrmann <tr...@apache.org>;
> Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <
> piotr@ververica.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> Hi Steven,
>
> > I probably missed sth here. isn't this the case today already? Why is it
> a concern for the proposed change?
>
> The problem is with the newly added `finish()` method and the already
> existing `endInput()` call. Currently on master there are no issues,
> because we are not checkpointing any operators after some operators have
> finished. The purpose of this FLIP-147 is to exactly enable this and this
> opens a new problem described by Dawid.
>
> To paraphrase and to give a concrete example. Assume we have an operator
> with parallelism of two. Subtask 0 and subtask 1.
>
> 1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1
> hasn't (yet).
> 2. Checkpoint 42 is triggered, and it completes.
> 3. Job fails and is restarted, but at the same time it's rescaled. User
> has chosen to scale down his operator down to 1.
>
> Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has
> been processed or not, so while recovering to checkpoint 42, we have to
> recover both finished subtask 0 (#1) state and not yet finished subtask 1's
> (#1). But at the same time they are scaled down, so we only have a single
> subtask 0 (#2) that has a combined state from both of the previous
> instances. The potentially confusing issue is that the state from subtask 0
> (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's
> recovered to an operator that has still some records to process. In step 1.
> an user for example could store on the operator's state a bit of
> information "end input has been already called!", that after recovery would
> no longer be true.
>
> Hence the question about `finish()` and `endInput()` semantics. Should it
> be tied down to state, or just to an operator instance/execution attempt?
>
> Piotrek
> śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> > if a failure happens after sequence of finish() -> snapshotState(), but
> before notifyCheckpointComplete(), we will restore such a state and we
> might end up sending some more records to such an operator.
>
> I probably missed sth here. isn't this the case today already? Why is it a
> concern for the proposed change?
>
> On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org>
> wrote:
>
> > Hi Dawid,
> >
> > Thanks for writing down those concerns.
> >
> > I think the first issue boils down what should be the contract of
> lifecycle
> > methods like open(), close(), initializeState() etc and especially the
> new
> > additions like finish() and endInput(). And what should be their
> relation
> > with the operator state (regardless of it's type keyed, non-keyed,
> union,
> > ...). Should those methods be tied to state or not? After thinking
> about it
> > for a while (and discussing it offline with Dawid), I think the answer
> > might be no, they shouldn't. I mean maybe we should just openly say that
> > all of those methods relate to this single particular instance and
> > execution of the operator. And if a job is recovered/rescaled, we would
> be
> > allowed to freely resume consumption, ignoring a fact that maybe some
> parts
> > of the state have previously seen `endInput()`. Why?
> >
> > 0. Yes, it might be confusing. Especially with `endInput()`. We call
> > `endInput()`, we store something in a state and later after recovery
> > combined with rescaling that state can see more records? Indeed weird,
> > 1. I haven't come up yet with a counterexample that would break and make
> > impossible to implement a real life use case. Theoretically yes, the
> user
> > can store `endInput()` on state, and after rescaling this state would be
> > inconsistent with what is actually happening with the operator, but I
> > haven't found a use case that would break because of that.
> > 2. Otherwise, implementation would be very difficult.
> > 3. It's difficult to access keyed state from within
> `endInput()`/`finish()`
> > calls, as they do not have key context.
> > 4. After all, openly defining `endInput()` and `finish()` to be tied
> with
> > it's operator execution instance lifecycle is not that strange and quite
> > simple to explain. Sure, it can lead to a bit of confusion (0.), but
> that
> > doesn't sound that bad in comparison with the alternatives that I'm
> aware
> > of. Also currently methods like `open()` and `close()` are also tied to
> the
> > operator execution instance, not to the state. Operators can be opened
> and
> > closed multiple times, it doesn't mean that the state is lost after
> closing
> > an operator.
> >
> > For the UnionListState problem I have posted my proposal in the ticket
> [1],
> > so maybe let's move that particular discussion there?
> >
> > Piotrek
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-21080
> >
> > śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> > napisał(a):
> >
> > > Hey all,
> > >
> > > To make the issues that were found transparent to the community, I
> want
> > to
> > > post an update:
> > >
> > > *1. Committing side-effects*
> > > We do want to make sure that all side effects are committed before
> > > bringing tasks down. Side effects are committed when calling
> > > notifyCheckpointComplete. For the final checkpoint we introduced the
> > method
> > > finish(). This notifies the operator that we have consumed all
> incoming
> > > records and we are preparing to close the Task. In turn we should
> flush
> > any
> > > pending buffered records and prepare to commit last transactions. The
> > goal
> > > is that after a successful sequence of finish() -> snapshotState() ->
> > > notifyCheckpointComplete(), the remaining state can be considered
> > > empty/finished and may be discarded.
> > >
> > > *Failure before notifyCheckpointComplete()*
> > >
> > > The question is what is the contract of the endInput()/finish()
> methods
> > > and how do calling these methods affect the operators keyed, non-keyed
> > > state and external state. Is it allowed to restore state snapshot
> taken
> > > after calling endInput()/finish() and process more records? Or do we
> > assume
> > > that after a restore from such a state taken after finish() we should
> not
> > > call any of the lifecycle methods or at least make sure those methods
> do
> > > not emit records/interact with mailbox etc.
> > >
> > > Currently it is possible that if a failure happens after sequence of
> > > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> > will
> > > restore such a state and we might end up sending some more records to
> > such
> > > an operator. It is possible if we rescale and this state is merged
> with a
> > > state of a subtask that has not called finish() yet. It can also
> happen
> > if
> > > we rescale the upstream operator and the subtask of interest becomes
> > > connected to a newly added non finished subtask.
> > >
> > > *Snapshotting StreamTasks that finish() has been called*
> > >
> > >
> > > We thought about putting a flag into the snapshot of a subtask
> produced
> > > after the finish() method. This would make it possible to skip
> execution
> > of
> > > certain lifecycle methods. Unfortunately this creates problems for
> > > rescaling. How do we deal with a situation that subtask states with
> both
> > > the feature flag set and unset end up in a single StreamTask.
> Additional
> > > problem is that we merge those states into a single
> OperatorSubtaskState
> > on
> > > CheckpointCoordinator.
> > >
> > > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > > checkpoint*
> > >
> > > We need to wait for a checkpoint to complete, that started after the
> > > finish() method. However, we support concurrent checkpoints therefore,
> > > there might be later checkpoints that completed, but the notification
> has
> > > not arrived. We must make sure those checkpoints do not leave
> lingering
> > > external resources.
> > >
> > > *Checkpointing from a single subtask / UnionListState case*
> > > There are operators that checkpoint from a single subtask only.
> Usually
> > > from the subtask index=0. If we let those subtasks finish, subsequent
> > > checkpoints will miss this information.
> > > Esp. Legacy sources problem:
> > > https://issues.apache.org/jira/browse/FLINK-21080
> > >
> > > Best,
> > >
> > > Dawid
> > > On 19/07/2021 15:10, Yun Gao wrote:
> > >
> > > Hi Till, Dawid,
> > >
> > > Very thanks for the comments and discussion! Glad that it seems we
> have
> > > come to a convergence, and I also agree with that we could not include
> > the
> > > optimization in the first version.
> > >
> > > Best
> > > Yun
> > >
> > > ------------------------------------------------------------------
> > > From:Dawid Wysakowicz <dw...@apache.org> <
> dwysakowicz@apache.org>
> > > Send Time:2021 Jul. 19 (Mon.) 17:51
> > > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > > <tr...@apache.org> <tr...@apache.org>
> > > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > Finished
> > >
> > > Small correction. I meant we need to adjust the EndOfInputEvent of
> > course.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > > Hey Till,
> > > >
> > > > Yes, you're right we will have to adjust the current state of
> > > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > > we're discussing here. We are aware of that.
> > > >
> > > > As for the MAX_WATERMARK vs finish(). My take is that we should
> always
> > > > emit MAX_WATERMARK before calling finish() on an operator. At the
> same
> > > > time finish() should not leave behind anything in state, as the
> > > > intention is that we never restore from the taken
> savepoint/checkpoint
> > > > (savepoint w drain or bounded data consumed).
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > > >> Hi Yun and Dawid,
> > > >>
> > >
> > > >> Thanks for your comments. I do agree with your comments that
> finish()
> > can
> > > >> do more than MAX_WATERMARK. I guess we should then explain how
> > > >> MAX_WATERMARK and finish() play together and what kind of
> > > >> order guarantees we provide.
> > > >>
> > >
> > > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether
> it
> > would
> > >
> > > >> work in its current state because we send this event when the Task
> is
> > about
> > > >> to shut down if I am not mistaken. What we want to have is to bring
> > the
> > >
> > > >> StreamTasks into a state so that they shut down on the next
> > checkpoint. For
> > > >> this we need to keep the StreamTask running. In general, I am a
> fan of
> > > >> making things explicit if possible. I think this helps maintenance
> and
> > >
> > > >> evolvability of code. That's why I think sending an EndOfInputEvent
> > which
> > > >> is a StreamTask level event and which says that there won't be any
> > other
> > > >> records coming only control events could make sense.
> > > >>
> > > >> I would leave the proposed optimization out of the first version.
> We
> > can
> > > >> still add it at a later point in time.
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > > <dw...@apache.org> <dw...@apache.org>
> > > >> wrote:
> > > >>
> > >
> > > >>> Personally I don't find this optimization important and I'd rather
> > leave
> > >
> > > >>> it out not to complicate the codebase further. I doubt we save
> much
> > there.
> > > >>> I don't have a strong opinion though.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Dawid
> > > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> Very thanks Dawid for the thoughts!
> > > >>>
> > > >>> Currently I also do not have different opinions regarding this
> part.
> > > >>> But I have one more issue to confirm: during the previous
> discussion
> > we
> > > >>> have discussed that for the final checkpoint case, we might have
> an
> > > >>> optmization
> > >
> > > >>> that if a task do not have operators using 2-pc, we might skip
> > waiting for
> > > >>> the
> > >
> > > >>> final checkpoint (but we could not skip the savepoint). To allow
> > users to
> > > >>> express
> > > >>> the logic, we have proposed to add one more method to
> StreamOperator
> > &
> > > >>> CheckpointListener:
> > > >>>
> > > >>> interface StreamOperator {
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> interface CheckpointListener {
> > > >>>
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> class AbstractUdfStreamOperator {
> > > >>>
> > > >>> @Override
> > > >>> boolean requiresFinalCheckpoint() {
> > > >>> return userFunction instanceof CheckpointListener &&
> > >
> > > >>> ((CheckpointListener)
> > userFunction).requiresFinalCheckpoint();
> > > >>> }
> > > >>> }
> > > >>>
> > > >>>
> > > >>> I think we should still keep the change ?
> > > >>>
> > > >>> Best,
> > > >>> Yun
> > > >>>
> > > >>> ------------------Original Mail ------------------
> > > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > > <dw...@apache.org>
> > > >>> <dw...@apache.org> <dw...@apache.org>
> > > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <
> dev@flink.apache.org>
> > > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > > >>> Finished
> > > >>>
> > > >>>> I think we're all really close to the same solution.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time
> based
> > > >>>>
> > >
> > > >>>> buffering, but it does not solve flushing other operations such
> as
> > e.g.
> > > >>>>
> > > >>>> count windows or batching requests in Sinks. I'd prefer to treat
> the
> > > >>>>
> > > >>>> finish() as a message for Operator to "flush all records". The
> > > >>>>
> > >
> > > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> > imo. I
> > > >>>>
> > > >>>> don't think operators need to get a signal "stop-processing" if
> they
> > > >>>>
> > > >>>> don't need to flush records. The "WHEN" records are emitted,
> should
> > be
> > > >>>>
> > > >>>> in control of the StreamTask, by firing timers or by processing a
> > next
> > > >>>>
> > > >>>> record from upstream.
> > > >>>>
> > > >>>>
> > > >>>>
> > >
> > > >>>> The only difference of my previous proposal compared to Yun's is
> > that I
> > > >>>>
> > > >>>> did not want to send the EndOfUserRecords event in case of stop
> w/o
> > > >>>>
> > > >>>> drain. My thinking was that we could directly go from RUNNING to
> > > >>>>
> > > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could
> emit
> > > >>>>
> > > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop
> firing
> > > >>>>
> > >
> > > >>>> timers and processing events (without calling finish() on
> > Operator). In
> > > >>>>
> > > >>>> my initial suggestion I though we don't care about some events
> > > >>>>
> > > >>>> potentially being emitted after the savepoint was taken, as they
> > would
> > > >>>>
> > >
> > > >>>> anyway belong to the next after FINAL, which would be discarded.
> I
> > think
> > > >>>>
> > > >>>> though the proposal to suspend records processing and timers is a
> > > >>>>
> > >
> > > >>>> sensible thing to do and would go with the version that Yun put
> > into the
> > > >>>>
> > > >>>> FLIP Wiki.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> What do you think Till?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Dawid
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > > >>>>
> > > >>>>> Hi Till, Piotr
> > > >>>>> Very thanks for the comments!
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >
> > > >>>>> I also agree with Piotr that currently they are independent
> > mechanisms,
> > > >>>> and they are basically the same
> > > >>>>
> > > >>>>> for the event time.
> > > >>>>> For more details, first there are some difference among the
> three
> > > >>>> scenarios regarding the finish:
> > > >>>>
> > >
> > > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> > not be
> > > >>>> expected to be restarted,
> > > >>>>
> > > >>>>> and for stop-with-savepoint the job would be expected restart
> > later.
> > > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> > would
> > > >>>> emit MAX_WATERMARK before the
> > > >>>>
> > >
> > > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> > endOfInput /
> > > >>>> finish() should also only be called
> > > >>>>
> > > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> > always
> > > >>>> occurs at the same time. After the change,
> > > >>>>
> > > >>>>> we could emit MAX_WATERMARK before endOfInput event for the
> finish
> > /
> > > >>>> stop-with-savepoint --drain cases.
> > > >>>>
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > a
> > > >>>>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>>> finish, for example?
> > > >>>>> As discussed above for stop-with-savepoint, we would always keep
> > the
> > > >>>> window as is, and restore them after restart.
> > > >>>>
> > > >>>>> Then for the finish / stop-with-savepoint --drain, I think
> perhaps
> > it
> > > >>>> depends on the Triggers. For
> > > >>>>
> > >
> > > >>>>> event-time triggers / process time triggers, it would be
> > reasonable to
> > > >>>> flush all the windows since logically
> > > >>>>
> > >
> > > >>>>> the time would always elapse and the window would always get
> > triggered
> > > >>>> in a logical future. But for triggers
> > > >>>>
> > >
> > > >>>>> like CountTrigger, no matter how much time pass logically, the
> > windows
> > > >>>> would not trigger, thus we may not
> > > >>>>
> > > >>>>> flush these windows. If there are requirements we may provide
> > > >>>> additional triggers.
> > > >>>>
> > >
> > > >>>>>> It's a bit messy and I'm not sure if this should be
> strengthened
> > out?
> > > >>>> Each one of those has a little bit different semantic/meaning,
> > > >>>>
> > >
> > > >>>>>> but at the same time they are very similar. For single input
> > operators
> > > >>>> `endInput()` and `finish()` are actually the very same thing.
> > > >>>>
> > >
> > > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> > at the
> > > >>>> same time, and for single input operators `endInput()` and
> > `finish()`
> > > >>>>
> > >
> > > >>>>> are indeed the same thing. During the last discussion we ever
> > mentioned
> > > >>>> this issue and at then we thought that we might deprecate
> > `endInput()`
> > > >>>>
> > > >>>>> in the future, then we would only have endInput(int input) and
> > > >>>> finish().
> > > >>>>
> > > >>>>> Best,
> > > >>>>> Yun
> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > > >>>>>
> ------------------------------------------------------------------
> > > >>>>> From:Piotr Nowojski
> > > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > > >>>>> To:dev
> > > >>>>> Cc:Yun Gao
> > > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > > >>>> Finished
> > > >>>>
> > > >>>>> Hi Till,
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > a>
> > > >>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>
> > > >>>>>> finish, for example?
> > > >>>>> 1) currently they are independent but parallel mechanisms. With
> > event
> > > >>>> time, they are basically the same.
> > > >>>>
> > > >>>>> 2) it probably should for the sake of processing time windows.
> > > >>>>> Here you are touching the bit of the current design that I like
> the
> > >
> > > >>>> least. We basically have now three different ways of conveying
> very
> > similar
> > > >>>> things:
> > > >>>>
> > > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator
> (what
> > > >>>> about processing time?)
> > > >>>>
> > > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush
> it's
> > > >>>> internal state
> > > >>>>
> > > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> > out?
> > >
> > > >>>> Each one of those has a little bit different semantic/meaning,
> but
> > at the
> > >
> > > >>>> same time they are very similar. For single input operators
> > `endInput()`
> > > >>>> and `finish()` are actually the very same thing.
> > > >>>>
> > > >>>>> Piotrek
> > > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > > >>>>> Thanks for updating the FLIP. Based on the new section about
> > > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would a
> > > >>>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>> finish, for example?
> > > >>>>> Cheers,
> > > >>>>> Till
> > > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > > >>>> problem.
> > > >>>>
> > > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > > >>>>>> Cheers,
> > > >>>>>> Till
> > > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > > >>>>>> wrote:
> > > >>>>>>> Hi Till,
> > > >>>>>>> Very thanks for the review and comments!
> > > >>>>>>> 1) First I think in fact we could be able to do the
> computation
> > > >>>> outside
> > > >>>>
> > > >>>>>>> of the main thread,
> > > >>>>>>> and the current implementation mainly due to the computation
> is
> > in
> > > >>>>>>> general fast and we
> > > >>>>>>> initially want to have a simplified first version.
> > >
> > > >>>>>>> The main requirement here is to have a constant view of the
> > state of
> > > >>>> the
> > > >>>>
> > > >>>>>>> tasks, otherwise
> > > >>>>>>> for example if we have A -> B, if A is running when we check
> if
> > we
> > > >>>> need
> > > >>>>
> > > >>>>>>> to trigger A, we will
> > > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> > check
> > > >>>> B, we
> > > >>>>
> > > >>>>>>> will also mark B as
> > >
> > > >>>>>>> have to trigger, then B will receive both rpc trigger and
> > checkpoint
> > > >>>>>>> barrier, which would break
> > > >>>>>>> some assumption on the task side and complicate the
> > implementation.
> > >
> > > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> > of the
> > > >>>>>>> tasks' state and then do the
> > >
> > > >>>>>>> computation, both the two step do not need to be in the main
> > thread.
> > >
> > > >>>>>>> 2) For the computation logic, in fact currently we benefit a
> lot
> > from
> > > >>>>>>> some shortcuts on all-to-all
> > >
> > > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> > could do
> > > >>>>>>> checks on the job vertex level
> > > >>>>>>> first and skip some job vertices as a whole. With this
> > optimization
> > > >>>> we
> > > >>>>
> > > >>>>>>> have a O(V) algorithm, and the
> > > >>>>>>> current running time of the worst case for a job with 320,000
> > tasks
> > > >>>> is
> > > >>>>
> > > >>>>>>> less than 100ms. For
> > > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > > >>>>>>> If we do the computation based on the last triggered tasks, we
> > may
> > > >>>> not
> > > >>>>
> > > >>>>>>> easily encode this information
> > >
> > > >>>>>>> into the shortcuts on the job vertex level. And since the time
> > seems
> > > >>>> to
> > > >>>>
> > > >>>>>>> be short, perhaps it is enough
> > > >>>>>>> to do re-computation from the scratch in consideration of the
> > > >>>> tradeoff
> > > >>>>
> > > >>>>>>> between the performance and
> > > >>>>>>> the complexity ?
> > > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > > >>>> finish()
> > > >>>>
> > > >>>>>>> method and before the last
> > >
> > > >>>>>>> snapshotState() method so that we could shut down the whole
> > topology
> > > >>>> with
> > > >>>>
> > > >>>>>>> a single final checkpoint.
> > > >>>>>>> Very sorry for not include enough details for this part and
> I'll
> > > >>>>>>> complement the FLIP with the details on
> > > >>>>>>> the process of the final checkpoint / savepoint.
> > > >>>>>>> Best,
> > > >>>>>>> Yun
> > > >>>>>>>
> > ------------------------------------------------------------------
> > > >>>>>>> From:Till Rohrmann
> > > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > > >>>>>>> To:dev
> > > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> > Tasks
> > > >>>>>>> Finished
> > > >>>>>>> Hi everyone,
> > > >>>>>>> I am a bit late to the voting party but let me ask three
> > questions:
> > >
> > > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> > thread
> > > >>>> if we
> > > >>>>
> > >
> > > >>>>>>> cannot guarantee that all tasks are still running when
> > triggering the
> > > >>>>>>> checkpoint? Couldn't we do the computation in a different
> thread
> > in
> > > >>>> order
> > > >>>>
> > > >>>>>>> to relieve the main thread a bit.
> > > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> > seems
> > > >>>> to go
> > > >>>>
> > > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> > more
> > > >>>>>>> efficient to maintain the set of current tasks to trigger and
> > check
> > > >>>>>>> whether
> > > >>>>>>> anything has changed and if so check the succeeding tasks
> until
> > we
> > > >>>> have
> > > >>>>
> > > >>>>>>> found the current checkpoint trigger frontier?
> > > >>>>>>> 3) When are we going to send the endOfInput events to a
> > downstream
> > > >>>> task?
> > > >>>>
> > > >>>>>>> If
> > >
> > > >>>>>>> this happens after we call finish on the upstream operator but
> > before
> > > >>>>>>> snapshotState then it would be possible to shut down the whole
> > > >>>> topology
> > > >>>>
> > >
> > > >>>>>>> with a single final checkpoint. I think this part could
> benefit
> > from
> > > >>>> a bit
> > > >>>>
> > > >>>>>>> more detailed description in the FLIP.
> > > >>>>>>> Cheers,
> > > >>>>>>> Till
> > > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > > >>>>>>> wrote:
> > > >>>>>>>> Hi there,
> > > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing
> the
> > > >>>> vote
> > > >>>>
> > > >>>>>>> now.
> > > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1
> votes:
> > > >>>>>>>> - Dawid Wysakowicz (binding)
> > > >>>>>>>> - Piotr Nowojski(binding)
> > > >>>>>>>> - Jiangang Liu (binding)
> > > >>>>>>>> - Arvid Heise (binding)
> > > >>>>>>>> - Jing Zhang (binding)
> > > >>>>>>>> - Leonard Xu (non-binding)
> > > >>>>>>>> - Guowei Ma (binding)
> > > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > > >>>> accepted.
> > > >>>>
> > > >>>>>>>> Very thanks everyone!
> > > >>>>>>>> Best,
> > > >>>>>>>> Yun
> > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > >>>>
> > > >>>>
> > >
> > >
> >
>
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Piotr,
Very thanks for the explanation! and very sorry that initially I should wrongly understand the problem Dawid proposed.
> And what should we do now? We can of course commit all transactions until checkpoint 43.
> But should we keep waiting for `notyifyCheckpointComplete(44)`?
> hat if in the meantime another checkpoint is triggered? We could end up waiting indefinitely.
> Our proposal is to shutdown the task immediately after seeing first `notifyCheckpointComplete(X)`,
> where X is any triggered checkpoint AFTER `finish()`. This should be fine, as:
> a) ideally there should be no new pending transactions opened after checkpoint 42
> b) even if operator/function is opening some transactions for checkpoint 43 and
> checkpoint 44 (`FlinkKafkaProducer`), those transactions after checkpoint 42 should be empty
Now I understand more on this issue: currently operators like the TwoPhaseCommitSinkFunctions would create
a new transaction when snapshotting state, thus for each checkpoint there would be one corresponding
transaction, thus after finish() and before notifyCheckpointComplete(), there might be some empty
transactions.
I also agree with the proposal: we could finish the task after we received the first checkpoint complete
notification, and the checkpoint with the larger checkpoint id should be allowed to be finished. The
checkpoint complete notification should be able to commit all the non-empty transactions. A part of
the empty transactions would also get committed, but it should cause no harm. The other empty
transactions would be aborted in close(), which should also cause no harm.
> If checkpoint 44 completes afterwards, it will still be valid. Ideally we would recommend that after
seeing `finish()` operators/functions should not be opening any new transactions, but that shouldn't be required.
And from another of view, I think perhaps it might also be acceptable that we have some requirements for
the pattern of the operators, like the sink would be able to skip creating new transactions after finish() is called.
Perhaps we may treat the process as a protocol between the framework and the operators, and the operators
might need to follow the protocol. Of course it would be better that the framework could handle all the cases
elegantly, and put less implicit limitation to the operators, and at least we should guarantee not cause compatibility
problem after changing the process.
Very thanks for the careful checks on the whole process.
Best,
Yun
------------------------------------------------------------------
From:Piotr Nowojski <pn...@apache.org>
Send Time:2021 Jul. 22 (Thu.) 15:33
To:dev <de...@flink.apache.org>; Yun Gao <yu...@aliyun.com>
Cc:Till Rohrmann <tr...@apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Guowei,
> Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can becalled repeatedly.
Just to clarify. It's not about calling `finish()` and `endInput()` repeatedly, but about (from the perspective of operator's state)
1. seeing `finish()`
2. checkpoint X triggered and completed
3. failure + recovery from X
4. potentially processing more records
5. another `finish()`
But from the context of the remaining part of your message Guowei I presume that you have already got that point :)
Yun:
> For this issue perhaps we could explicitly requires the task to wait for a checkpoint triggered after finish()> method is called for all the operators ? We could be able to achieve this target by maintaining
> some state inside the task.
Isn't this exactly the "WAITING_FOR_FINAL_CP" from the FLIP document? That we always need to wait for a checkpoint triggered after `finish()` to complete, before shutting down a task?
What Dawid was describing is a scenario where:
1. task/operator received `finish()`
2. checkpoint 42 triggered (not yet completed)
3. checkpoint 43 triggered (not yet completed)
4. checkpoint 44 triggered (not yet completed)
5. notifyCheckpointComplete(43)
And what should we do now? We can of course commit all transactions until checkpoint 43. But should we keep waiting for `notyifyCheckpointComplete(44)`? What if in the meantime another checkpoint is triggered? We could end up waiting indefinitely.
Our proposal is to shutdown the task immediately after seeing first `notifyCheckpointComplete(X)`, where X is any triggered checkpoint AFTER `finish()`. This should be fine, as:
a) ideally there should be no new pending transactions opened after checkpoint 42
b) even if operator/function is opening some transactions for checkpoint 43 and checkpoint 44 (`FlinkKafkaProducer`), those transactions after checkpoint 42 should be empty
Hence comment from Dawid
> We must make sure those checkpoints do not leave lingering external resources.
After seeing 5. (notifyCheckpointComplete(43)) It should be good enough to:
- commit transactions from checkpoint 42, (and 43 if they were created, depends on the user code)
- close operator, aborting any pending transactions (for checkpoint 44 if they were opened, depends on the user code)
If checkpoint 44 completes afterwards, it will still be valid. Ideally we would recommend that after seeing `finish()` operators/functions should not be opening any new transactions, but that shouldn't be required.
Best,
Piotrek
czw., 22 lip 2021 o 09:00 Yun Gao <yu...@aliyun.com.invalid> napisał(a):
Hi Dawid, Piotr, Steven,
Very thanks for pointing out these issues and very thanks for the discussion !
Failure before notifyCheckpointComplete()
For this issue I would agree with what Piotr has proposed. I tried to use some
operators like sink / window as example and currently I also do not found
explicit scenarios that might cause problems if records are processed by a
task that is assigned with states snapshotted after calling finish() before.
For the future cases it seems users should be able to implement their
target logic by explicitly add a flag regarding finished, and perhaps have
different logic if this part of states are referred to. Besides, this case would
only happen on rescaling or topology change, which embedded some kind of user
knowledge inside the action. Thus it looks acceptable that we still split the operators
state from the task lifecycle, and do not treat checkpoint after finish() differently.
Finishing upon receiving notifyCheckpointComplete() of not the latest checkpoint
For this issue perhaps we could explicitly requires the task to wait for a checkpoint triggered after finish()
method is called for all the operators ? We could be able to achieve this target by maintaining
some state inside the task.
Checkpointing from a single subtask / UnionListState case
This should indeed cause problems, and I also agree with that we could focus on this thread in the
https://issues.apache.org/jira/browse/FLINK-21080.
Best,
Yun
------------------------------------------------------------------
From:Piotr Nowojski <pn...@apache.org>
Send Time:2021 Jul. 22 (Thu.) 02:46
To:dev <de...@flink.apache.org>
Cc:Yun Gao <yu...@aliyun.com>; Till Rohrmann <tr...@apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Steven,
> I probably missed sth here. isn't this the case today already? Why is it a concern for the proposed change?
The problem is with the newly added `finish()` method and the already existing `endInput()` call. Currently on master there are no issues, because we are not checkpointing any operators after some operators have finished. The purpose of this FLIP-147 is to exactly enable this and this opens a new problem described by Dawid.
To paraphrase and to give a concrete example. Assume we have an operator with parallelism of two. Subtask 0 and subtask 1.
1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1 hasn't (yet).
2. Checkpoint 42 is triggered, and it completes.
3. Job fails and is restarted, but at the same time it's rescaled. User has chosen to scale down his operator down to 1.
Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has been processed or not, so while recovering to checkpoint 42, we have to recover both finished subtask 0 (#1) state and not yet finished subtask 1's (#1). But at the same time they are scaled down, so we only have a single subtask 0 (#2) that has a combined state from both of the previous instances. The potentially confusing issue is that the state from subtask 0 (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's recovered to an operator that has still some records to process. In step 1. an user for example could store on the operator's state a bit of information "end input has been already called!", that after recovery would no longer be true.
Hence the question about `finish()` and `endInput()` semantics. Should it be tied down to state, or just to an operator instance/execution attempt?
Piotrek
śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> if a failure happens after sequence of finish() -> snapshotState(), but
before notifyCheckpointComplete(), we will restore such a state and we
might end up sending some more records to such an operator.
I probably missed sth here. isn't this the case today already? Why is it a
concern for the proposed change?
On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi Dawid,
>
> Thanks for writing down those concerns.
>
> I think the first issue boils down what should be the contract of lifecycle
> methods like open(), close(), initializeState() etc and especially the new
> additions like finish() and endInput(). And what should be their relation
> with the operator state (regardless of it's type keyed, non-keyed, union,
> ...). Should those methods be tied to state or not? After thinking about it
> for a while (and discussing it offline with Dawid), I think the answer
> might be no, they shouldn't. I mean maybe we should just openly say that
> all of those methods relate to this single particular instance and
> execution of the operator. And if a job is recovered/rescaled, we would be
> allowed to freely resume consumption, ignoring a fact that maybe some parts
> of the state have previously seen `endInput()`. Why?
>
> 0. Yes, it might be confusing. Especially with `endInput()`. We call
> `endInput()`, we store something in a state and later after recovery
> combined with rescaling that state can see more records? Indeed weird,
> 1. I haven't come up yet with a counterexample that would break and make
> impossible to implement a real life use case. Theoretically yes, the user
> can store `endInput()` on state, and after rescaling this state would be
> inconsistent with what is actually happening with the operator, but I
> haven't found a use case that would break because of that.
> 2. Otherwise, implementation would be very difficult.
> 3. It's difficult to access keyed state from within `endInput()`/`finish()`
> calls, as they do not have key context.
> 4. After all, openly defining `endInput()` and `finish()` to be tied with
> it's operator execution instance lifecycle is not that strange and quite
> simple to explain. Sure, it can lead to a bit of confusion (0.), but that
> doesn't sound that bad in comparison with the alternatives that I'm aware
> of. Also currently methods like `open()` and `close()` are also tied to the
> operator execution instance, not to the state. Operators can be opened and
> closed multiple times, it doesn't mean that the state is lost after closing
> an operator.
>
> For the UnionListState problem I have posted my proposal in the ticket [1],
> so maybe let's move that particular discussion there?
>
> Piotrek
>
> [1] https://issues.apache.org/jira/browse/FLINK-21080
>
> śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> napisał(a):
>
> > Hey all,
> >
> > To make the issues that were found transparent to the community, I want
> to
> > post an update:
> >
> > *1. Committing side-effects*
> > We do want to make sure that all side effects are committed before
> > bringing tasks down. Side effects are committed when calling
> > notifyCheckpointComplete. For the final checkpoint we introduced the
> method
> > finish(). This notifies the operator that we have consumed all incoming
> > records and we are preparing to close the Task. In turn we should flush
> any
> > pending buffered records and prepare to commit last transactions. The
> goal
> > is that after a successful sequence of finish() -> snapshotState() ->
> > notifyCheckpointComplete(), the remaining state can be considered
> > empty/finished and may be discarded.
> >
> > *Failure before notifyCheckpointComplete()*
> >
> > The question is what is the contract of the endInput()/finish() methods
> > and how do calling these methods affect the operators keyed, non-keyed
> > state and external state. Is it allowed to restore state snapshot taken
> > after calling endInput()/finish() and process more records? Or do we
> assume
> > that after a restore from such a state taken after finish() we should not
> > call any of the lifecycle methods or at least make sure those methods do
> > not emit records/interact with mailbox etc.
> >
> > Currently it is possible that if a failure happens after sequence of
> > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> will
> > restore such a state and we might end up sending some more records to
> such
> > an operator. It is possible if we rescale and this state is merged with a
> > state of a subtask that has not called finish() yet. It can also happen
> if
> > we rescale the upstream operator and the subtask of interest becomes
> > connected to a newly added non finished subtask.
> >
> > *Snapshotting StreamTasks that finish() has been called*
> >
> >
> > We thought about putting a flag into the snapshot of a subtask produced
> > after the finish() method. This would make it possible to skip execution
> of
> > certain lifecycle methods. Unfortunately this creates problems for
> > rescaling. How do we deal with a situation that subtask states with both
> > the feature flag set and unset end up in a single StreamTask. Additional
> > problem is that we merge those states into a single OperatorSubtaskState
> on
> > CheckpointCoordinator.
> >
> > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > checkpoint*
> >
> > We need to wait for a checkpoint to complete, that started after the
> > finish() method. However, we support concurrent checkpoints therefore,
> > there might be later checkpoints that completed, but the notification has
> > not arrived. We must make sure those checkpoints do not leave lingering
> > external resources.
> >
> > *Checkpointing from a single subtask / UnionListState case*
> > There are operators that checkpoint from a single subtask only. Usually
> > from the subtask index=0. If we let those subtasks finish, subsequent
> > checkpoints will miss this information.
> > Esp. Legacy sources problem:
> > https://issues.apache.org/jira/browse/FLINK-21080
> >
> > Best,
> >
> > Dawid
> > On 19/07/2021 15:10, Yun Gao wrote:
> >
> > Hi Till, Dawid,
> >
> > Very thanks for the comments and discussion! Glad that it seems we have
> > come to a convergence, and I also agree with that we could not include
> the
> > optimization in the first version.
> >
> > Best
> > Yun
> >
> > ------------------------------------------------------------------
> > From:Dawid Wysakowicz <dw...@apache.org> <dw...@apache.org>
> > Send Time:2021 Jul. 19 (Mon.) 17:51
> > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > <tr...@apache.org> <tr...@apache.org>
> > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > Finished
> >
> > Small correction. I meant we need to adjust the EndOfInputEvent of
> course.
> >
> > Best,
> >
> > Dawid
> >
> > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > Hey Till,
> > >
> > > Yes, you're right we will have to adjust the current state of
> > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > we're discussing here. We are aware of that.
> > >
> > > As for the MAX_WATERMARK vs finish(). My take is that we should always
> > > emit MAX_WATERMARK before calling finish() on an operator. At the same
> > > time finish() should not leave behind anything in state, as the
> > > intention is that we never restore from the taken savepoint/checkpoint
> > > (savepoint w drain or bounded data consumed).
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > >> Hi Yun and Dawid,
> > >>
> >
> > >> Thanks for your comments. I do agree with your comments that finish()
> can
> > >> do more than MAX_WATERMARK. I guess we should then explain how
> > >> MAX_WATERMARK and finish() play together and what kind of
> > >> order guarantees we provide.
> > >>
> >
> > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether it
> would
> >
> > >> work in its current state because we send this event when the Task is
> about
> > >> to shut down if I am not mistaken. What we want to have is to bring
> the
> >
> > >> StreamTasks into a state so that they shut down on the next
> checkpoint. For
> > >> this we need to keep the StreamTask running. In general, I am a fan of
> > >> making things explicit if possible. I think this helps maintenance and
> >
> > >> evolvability of code. That's why I think sending an EndOfInputEvent
> which
> > >> is a StreamTask level event and which says that there won't be any
> other
> > >> records coming only control events could make sense.
> > >>
> > >> I would leave the proposed optimization out of the first version. We
> can
> > >> still add it at a later point in time.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > <dw...@apache.org> <dw...@apache.org>
> > >> wrote:
> > >>
> >
> > >>> Personally I don't find this optimization important and I'd rather
> leave
> >
> > >>> it out not to complicate the codebase further. I doubt we save much
> there.
> > >>> I don't have a strong opinion though.
> > >>>
> > >>> Best,
> > >>>
> > >>> Dawid
> > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> Very thanks Dawid for the thoughts!
> > >>>
> > >>> Currently I also do not have different opinions regarding this part.
> > >>> But I have one more issue to confirm: during the previous discussion
> we
> > >>> have discussed that for the final checkpoint case, we might have an
> > >>> optmization
> >
> > >>> that if a task do not have operators using 2-pc, we might skip
> waiting for
> > >>> the
> >
> > >>> final checkpoint (but we could not skip the savepoint). To allow
> users to
> > >>> express
> > >>> the logic, we have proposed to add one more method to StreamOperator
> &
> > >>> CheckpointListener:
> > >>>
> > >>> interface StreamOperator {
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> interface CheckpointListener {
> > >>>
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> class AbstractUdfStreamOperator {
> > >>>
> > >>> @Override
> > >>> boolean requiresFinalCheckpoint() {
> > >>> return userFunction instanceof CheckpointListener &&
> >
> > >>> ((CheckpointListener)
> userFunction).requiresFinalCheckpoint();
> > >>> }
> > >>> }
> > >>>
> > >>>
> > >>> I think we should still keep the change ?
> > >>>
> > >>> Best,
> > >>> Yun
> > >>>
> > >>> ------------------Original Mail ------------------
> > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > <dw...@apache.org>
> > >>> <dw...@apache.org> <dw...@apache.org>
> > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>
> > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>> Finished
> > >>>
> > >>>> I think we're all really close to the same solution.
> > >>>>
> > >>>>
> > >>>>
> > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
> > >>>>
> >
> > >>>> buffering, but it does not solve flushing other operations such as
> e.g.
> > >>>>
> > >>>> count windows or batching requests in Sinks. I'd prefer to treat the
> > >>>>
> > >>>> finish() as a message for Operator to "flush all records". The
> > >>>>
> >
> > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> imo. I
> > >>>>
> > >>>> don't think operators need to get a signal "stop-processing" if they
> > >>>>
> > >>>> don't need to flush records. The "WHEN" records are emitted, should
> be
> > >>>>
> > >>>> in control of the StreamTask, by firing timers or by processing a
> next
> > >>>>
> > >>>> record from upstream.
> > >>>>
> > >>>>
> > >>>>
> >
> > >>>> The only difference of my previous proposal compared to Yun's is
> that I
> > >>>>
> > >>>> did not want to send the EndOfUserRecords event in case of stop w/o
> > >>>>
> > >>>> drain. My thinking was that we could directly go from RUNNING to
> > >>>>
> > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> > >>>>
> > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> > >>>>
> >
> > >>>> timers and processing events (without calling finish() on
> Operator). In
> > >>>>
> > >>>> my initial suggestion I though we don't care about some events
> > >>>>
> > >>>> potentially being emitted after the savepoint was taken, as they
> would
> > >>>>
> >
> > >>>> anyway belong to the next after FINAL, which would be discarded. I
> think
> > >>>>
> > >>>> though the proposal to suspend records processing and timers is a
> > >>>>
> >
> > >>>> sensible thing to do and would go with the version that Yun put
> into the
> > >>>>
> > >>>> FLIP Wiki.
> > >>>>
> > >>>>
> > >>>>
> > >>>> What do you think Till?
> > >>>>
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>>
> > >>>>
> > >>>> Dawid
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > >>>>
> > >>>>> Hi Till, Piotr
> > >>>>> Very thanks for the comments!
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > >>>>> I also agree with Piotr that currently they are independent
> mechanisms,
> > >>>> and they are basically the same
> > >>>>
> > >>>>> for the event time.
> > >>>>> For more details, first there are some difference among the three
> > >>>> scenarios regarding the finish:
> > >>>>
> >
> > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> not be
> > >>>> expected to be restarted,
> > >>>>
> > >>>>> and for stop-with-savepoint the job would be expected restart
> later.
> > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> would
> > >>>> emit MAX_WATERMARK before the
> > >>>>
> >
> > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> endOfInput /
> > >>>> finish() should also only be called
> > >>>>
> > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> always
> > >>>> occurs at the same time. After the change,
> > >>>>
> > >>>>> we could emit MAX_WATERMARK before endOfInput event for the finish
> /
> > >>>> stop-with-savepoint --drain cases.
> > >>>>
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a
> > >>>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>>> finish, for example?
> > >>>>> As discussed above for stop-with-savepoint, we would always keep
> the
> > >>>> window as is, and restore them after restart.
> > >>>>
> > >>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps
> it
> > >>>> depends on the Triggers. For
> > >>>>
> >
> > >>>>> event-time triggers / process time triggers, it would be
> reasonable to
> > >>>> flush all the windows since logically
> > >>>>
> >
> > >>>>> the time would always elapse and the window would always get
> triggered
> > >>>> in a logical future. But for triggers
> > >>>>
> >
> > >>>>> like CountTrigger, no matter how much time pass logically, the
> windows
> > >>>> would not trigger, thus we may not
> > >>>>
> > >>>>> flush these windows. If there are requirements we may provide
> > >>>> additional triggers.
> > >>>>
> >
> > >>>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> > >>>> Each one of those has a little bit different semantic/meaning,
> > >>>>
> >
> > >>>>>> but at the same time they are very similar. For single input
> operators
> > >>>> `endInput()` and `finish()` are actually the very same thing.
> > >>>>
> >
> > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> at the
> > >>>> same time, and for single input operators `endInput()` and
> `finish()`
> > >>>>
> >
> > >>>>> are indeed the same thing. During the last discussion we ever
> mentioned
> > >>>> this issue and at then we thought that we might deprecate
> `endInput()`
> > >>>>
> > >>>>> in the future, then we would only have endInput(int input) and
> > >>>> finish().
> > >>>>
> > >>>>> Best,
> > >>>>> Yun
> > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > >>>>> ------------------------------------------------------------------
> > >>>>> From:Piotr Nowojski
> > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > >>>>> To:dev
> > >>>>> Cc:Yun Gao
> > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>>> Finished
> > >>>>
> > >>>>> Hi Till,
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a>
> > >>>> WindowOperator close all windows and emit the results upon calling
> > >>>>
> > >>>>>> finish, for example?
> > >>>>> 1) currently they are independent but parallel mechanisms. With
> event
> > >>>> time, they are basically the same.
> > >>>>
> > >>>>> 2) it probably should for the sake of processing time windows.
> > >>>>> Here you are touching the bit of the current design that I like the
> >
> > >>>> least. We basically have now three different ways of conveying very
> similar
> > >>>> things:
> > >>>>
> > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
> > >>>> about processing time?)
> > >>>>
> > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
> > >>>> internal state
> > >>>>
> > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> >
> > >>>> Each one of those has a little bit different semantic/meaning, but
> at the
> >
> > >>>> same time they are very similar. For single input operators
> `endInput()`
> > >>>> and `finish()` are actually the very same thing.
> > >>>>
> > >>>>> Piotrek
> > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > >>>>> Thanks for updating the FLIP. Based on the new section about
> > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> > >>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>> finish, for example?
> > >>>>> Cheers,
> > >>>>> Till
> > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > >>>> problem.
> > >>>>
> > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > >>>>>> Cheers,
> > >>>>>> Till
> > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > >>>>>> wrote:
> > >>>>>>> Hi Till,
> > >>>>>>> Very thanks for the review and comments!
> > >>>>>>> 1) First I think in fact we could be able to do the computation
> > >>>> outside
> > >>>>
> > >>>>>>> of the main thread,
> > >>>>>>> and the current implementation mainly due to the computation is
> in
> > >>>>>>> general fast and we
> > >>>>>>> initially want to have a simplified first version.
> >
> > >>>>>>> The main requirement here is to have a constant view of the
> state of
> > >>>> the
> > >>>>
> > >>>>>>> tasks, otherwise
> > >>>>>>> for example if we have A -> B, if A is running when we check if
> we
> > >>>> need
> > >>>>
> > >>>>>>> to trigger A, we will
> > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> check
> > >>>> B, we
> > >>>>
> > >>>>>>> will also mark B as
> >
> > >>>>>>> have to trigger, then B will receive both rpc trigger and
> checkpoint
> > >>>>>>> barrier, which would break
> > >>>>>>> some assumption on the task side and complicate the
> implementation.
> >
> > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> of the
> > >>>>>>> tasks' state and then do the
> >
> > >>>>>>> computation, both the two step do not need to be in the main
> thread.
> >
> > >>>>>>> 2) For the computation logic, in fact currently we benefit a lot
> from
> > >>>>>>> some shortcuts on all-to-all
> >
> > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> could do
> > >>>>>>> checks on the job vertex level
> > >>>>>>> first and skip some job vertices as a whole. With this
> optimization
> > >>>> we
> > >>>>
> > >>>>>>> have a O(V) algorithm, and the
> > >>>>>>> current running time of the worst case for a job with 320,000
> tasks
> > >>>> is
> > >>>>
> > >>>>>>> less than 100ms. For
> > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > >>>>>>> If we do the computation based on the last triggered tasks, we
> may
> > >>>> not
> > >>>>
> > >>>>>>> easily encode this information
> >
> > >>>>>>> into the shortcuts on the job vertex level. And since the time
> seems
> > >>>> to
> > >>>>
> > >>>>>>> be short, perhaps it is enough
> > >>>>>>> to do re-computation from the scratch in consideration of the
> > >>>> tradeoff
> > >>>>
> > >>>>>>> between the performance and
> > >>>>>>> the complexity ?
> > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > >>>> finish()
> > >>>>
> > >>>>>>> method and before the last
> >
> > >>>>>>> snapshotState() method so that we could shut down the whole
> topology
> > >>>> with
> > >>>>
> > >>>>>>> a single final checkpoint.
> > >>>>>>> Very sorry for not include enough details for this part and I'll
> > >>>>>>> complement the FLIP with the details on
> > >>>>>>> the process of the final checkpoint / savepoint.
> > >>>>>>> Best,
> > >>>>>>> Yun
> > >>>>>>>
> ------------------------------------------------------------------
> > >>>>>>> From:Till Rohrmann
> > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >>>>>>> To:dev
> > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > >>>>>>> Finished
> > >>>>>>> Hi everyone,
> > >>>>>>> I am a bit late to the voting party but let me ask three
> questions:
> >
> > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> thread
> > >>>> if we
> > >>>>
> >
> > >>>>>>> cannot guarantee that all tasks are still running when
> triggering the
> > >>>>>>> checkpoint? Couldn't we do the computation in a different thread
> in
> > >>>> order
> > >>>>
> > >>>>>>> to relieve the main thread a bit.
> > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> seems
> > >>>> to go
> > >>>>
> > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> more
> > >>>>>>> efficient to maintain the set of current tasks to trigger and
> check
> > >>>>>>> whether
> > >>>>>>> anything has changed and if so check the succeeding tasks until
> we
> > >>>> have
> > >>>>
> > >>>>>>> found the current checkpoint trigger frontier?
> > >>>>>>> 3) When are we going to send the endOfInput events to a
> downstream
> > >>>> task?
> > >>>>
> > >>>>>>> If
> >
> > >>>>>>> this happens after we call finish on the upstream operator but
> before
> > >>>>>>> snapshotState then it would be possible to shut down the whole
> > >>>> topology
> > >>>>
> >
> > >>>>>>> with a single final checkpoint. I think this part could benefit
> from
> > >>>> a bit
> > >>>>
> > >>>>>>> more detailed description in the FLIP.
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > >>>>>>> wrote:
> > >>>>>>>> Hi there,
> > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
> > >>>> vote
> > >>>>
> > >>>>>>> now.
> > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >>>>>>>> - Dawid Wysakowicz (binding)
> > >>>>>>>> - Piotr Nowojski(binding)
> > >>>>>>>> - Jiangang Liu (binding)
> > >>>>>>>> - Arvid Heise (binding)
> > >>>>>>>> - Jing Zhang (binding)
> > >>>>>>>> - Leonard Xu (non-binding)
> > >>>>>>>> - Guowei Ma (binding)
> > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > >>>> accepted.
> > >>>>
> > >>>>>>>> Very thanks everyone!
> > >>>>>>>> Best,
> > >>>>>>>> Yun
> > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>>>
> > >>>>
> >
> >
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Piotr Nowojski <pn...@apache.org>.
Hi Guowei,
> Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can
be
called repeatedly.
Just to clarify. It's not about calling `finish()` and `endInput()`
repeatedly, but about (from the perspective of operator's state)
1. seeing `finish()`
2. checkpoint X triggered and completed
3. failure + recovery from X
4. potentially processing more records
5. another `finish()`
But from the context of the remaining part of your message Guowei I presume
that you have already got that point :)
Yun:
> For this issue perhaps we could explicitly requires the task to wait for
a checkpoint triggered after finish()
> method is called for all the operators ? We could be able to achieve this
target by maintaining
> some state inside the task.
Isn't this exactly the "WAITING_FOR_FINAL_CP" from the FLIP document? That
we always need to wait for a checkpoint triggered after `finish()` to
complete, before shutting down a task?
What Dawid was describing is a scenario where:
1. task/operator received `finish()`
2. checkpoint 42 triggered (not yet completed)
3. checkpoint 43 triggered (not yet completed)
4. checkpoint 44 triggered (not yet completed)
5. notifyCheckpointComplete(43)
And what should we do now? We can of course commit all transactions until
checkpoint 43. But should we keep waiting for
`notyifyCheckpointComplete(44)`? What if in the meantime another checkpoint
is triggered? We could end up waiting indefinitely.
Our proposal is to shutdown the task immediately after seeing first
`notifyCheckpointComplete(X)`, where X is any triggered checkpoint AFTER
`finish()`. This should be fine, as:
a) ideally there should be no new pending transactions opened after
checkpoint 42
b) even if operator/function is opening some transactions for checkpoint 43
and checkpoint 44 (`FlinkKafkaProducer`), those transactions after
checkpoint 42 should be empty
Hence comment from Dawid
> We must make sure those checkpoints do not leave lingering external
resources.
After seeing 5. (notifyCheckpointComplete(43)) It should be good enough to:
- commit transactions from checkpoint 42, (and 43 if they were created,
depends on the user code)
- close operator, aborting any pending transactions (for checkpoint 44 if
they were opened, depends on the user code)
If checkpoint 44 completes afterwards, it will still be valid. Ideally we
would recommend that after seeing `finish()` operators/functions should not
be opening any new transactions, but that shouldn't be required.
Best,
Piotrek
czw., 22 lip 2021 o 09:00 Yun Gao <yu...@aliyun.com.invalid> napisał(a):
> Hi Dawid, Piotr, Steven,
>
> Very thanks for pointing out these issues and very thanks for the
> discussion !
>
> Failure before notifyCheckpointComplete()
>
> For this issue I would agree with what Piotr has proposed. I tried to use
> some
> operators like sink / window as example and currently I also do not found
> explicit scenarios that might cause problems if records are processed by a
> task that is assigned with states snapshotted after calling finish()
> before.
> For the future cases it seems users should be able to implement their
> target logic by explicitly add a flag regarding finished, and perhaps have
> different logic if this part of states are referred to. Besides, this case
> would
> only happen on rescaling or topology change, which embedded some kind of
> user
> knowledge inside the action. Thus it looks acceptable that we still split
> the operators
> state from the task lifecycle, and do not treat checkpoint after finish()
> differently.
>
> Finishing upon receiving notifyCheckpointComplete() of not the latest
> checkpoint
>
> For this issue perhaps we could explicitly requires the task to wait for a
> checkpoint triggered after finish()
> method is called for all the operators ? We could be able to achieve this
> target by maintaining
> some state inside the task.
>
> Checkpointing from a single subtask / UnionListState case
>
> This should indeed cause problems, and I also agree with that we could
> focus on this thread in the
> https://issues.apache.org/jira/browse/FLINK-21080.
>
> Best,
> Yun
>
>
> ------------------------------------------------------------------
> From:Piotr Nowojski <pn...@apache.org>
> Send Time:2021 Jul. 22 (Thu.) 02:46
> To:dev <de...@flink.apache.org>
> Cc:Yun Gao <yu...@aliyun.com>; Till Rohrmann <tr...@apache.org>;
> Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <
> piotr@ververica.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> Hi Steven,
>
> > I probably missed sth here. isn't this the case today already? Why is it
> a concern for the proposed change?
>
> The problem is with the newly added `finish()` method and the already
> existing `endInput()` call. Currently on master there are no issues,
> because we are not checkpointing any operators after some operators have
> finished. The purpose of this FLIP-147 is to exactly enable this and this
> opens a new problem described by Dawid.
>
> To paraphrase and to give a concrete example. Assume we have an operator
> with parallelism of two. Subtask 0 and subtask 1.
>
> 1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1
> hasn't (yet).
> 2. Checkpoint 42 is triggered, and it completes.
> 3. Job fails and is restarted, but at the same time it's rescaled. User
> has chosen to scale down his operator down to 1.
>
> Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has
> been processed or not, so while recovering to checkpoint 42, we have to
> recover both finished subtask 0 (#1) state and not yet finished subtask 1's
> (#1). But at the same time they are scaled down, so we only have a single
> subtask 0 (#2) that has a combined state from both of the previous
> instances. The potentially confusing issue is that the state from subtask 0
> (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's
> recovered to an operator that has still some records to process. In step 1.
> an user for example could store on the operator's state a bit of
> information "end input has been already called!", that after recovery would
> no longer be true.
>
> Hence the question about `finish()` and `endInput()` semantics. Should it
> be tied down to state, or just to an operator instance/execution attempt?
>
> Piotrek
> śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> > if a failure happens after sequence of finish() -> snapshotState(), but
> before notifyCheckpointComplete(), we will restore such a state and we
> might end up sending some more records to such an operator.
>
> I probably missed sth here. isn't this the case today already? Why is it a
> concern for the proposed change?
>
> On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org>
> wrote:
>
> > Hi Dawid,
> >
> > Thanks for writing down those concerns.
> >
> > I think the first issue boils down what should be the contract of
> lifecycle
> > methods like open(), close(), initializeState() etc and especially the
> new
> > additions like finish() and endInput(). And what should be their
> relation
> > with the operator state (regardless of it's type keyed, non-keyed,
> union,
> > ...). Should those methods be tied to state or not? After thinking
> about it
> > for a while (and discussing it offline with Dawid), I think the answer
> > might be no, they shouldn't. I mean maybe we should just openly say that
> > all of those methods relate to this single particular instance and
> > execution of the operator. And if a job is recovered/rescaled, we would
> be
> > allowed to freely resume consumption, ignoring a fact that maybe some
> parts
> > of the state have previously seen `endInput()`. Why?
> >
> > 0. Yes, it might be confusing. Especially with `endInput()`. We call
> > `endInput()`, we store something in a state and later after recovery
> > combined with rescaling that state can see more records? Indeed weird,
> > 1. I haven't come up yet with a counterexample that would break and make
> > impossible to implement a real life use case. Theoretically yes, the
> user
> > can store `endInput()` on state, and after rescaling this state would be
> > inconsistent with what is actually happening with the operator, but I
> > haven't found a use case that would break because of that.
> > 2. Otherwise, implementation would be very difficult.
> > 3. It's difficult to access keyed state from within
> `endInput()`/`finish()`
> > calls, as they do not have key context.
> > 4. After all, openly defining `endInput()` and `finish()` to be tied
> with
> > it's operator execution instance lifecycle is not that strange and quite
> > simple to explain. Sure, it can lead to a bit of confusion (0.), but
> that
> > doesn't sound that bad in comparison with the alternatives that I'm
> aware
> > of. Also currently methods like `open()` and `close()` are also tied to
> the
> > operator execution instance, not to the state. Operators can be opened
> and
> > closed multiple times, it doesn't mean that the state is lost after
> closing
> > an operator.
> >
> > For the UnionListState problem I have posted my proposal in the ticket
> [1],
> > so maybe let's move that particular discussion there?
> >
> > Piotrek
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-21080
> >
> > śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> > napisał(a):
> >
> > > Hey all,
> > >
> > > To make the issues that were found transparent to the community, I
> want
> > to
> > > post an update:
> > >
> > > *1. Committing side-effects*
> > > We do want to make sure that all side effects are committed before
> > > bringing tasks down. Side effects are committed when calling
> > > notifyCheckpointComplete. For the final checkpoint we introduced the
> > method
> > > finish(). This notifies the operator that we have consumed all
> incoming
> > > records and we are preparing to close the Task. In turn we should
> flush
> > any
> > > pending buffered records and prepare to commit last transactions. The
> > goal
> > > is that after a successful sequence of finish() -> snapshotState() ->
> > > notifyCheckpointComplete(), the remaining state can be considered
> > > empty/finished and may be discarded.
> > >
> > > *Failure before notifyCheckpointComplete()*
> > >
> > > The question is what is the contract of the endInput()/finish()
> methods
> > > and how do calling these methods affect the operators keyed, non-keyed
> > > state and external state. Is it allowed to restore state snapshot
> taken
> > > after calling endInput()/finish() and process more records? Or do we
> > assume
> > > that after a restore from such a state taken after finish() we should
> not
> > > call any of the lifecycle methods or at least make sure those methods
> do
> > > not emit records/interact with mailbox etc.
> > >
> > > Currently it is possible that if a failure happens after sequence of
> > > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> > will
> > > restore such a state and we might end up sending some more records to
> > such
> > > an operator. It is possible if we rescale and this state is merged
> with a
> > > state of a subtask that has not called finish() yet. It can also
> happen
> > if
> > > we rescale the upstream operator and the subtask of interest becomes
> > > connected to a newly added non finished subtask.
> > >
> > > *Snapshotting StreamTasks that finish() has been called*
> > >
> > >
> > > We thought about putting a flag into the snapshot of a subtask
> produced
> > > after the finish() method. This would make it possible to skip
> execution
> > of
> > > certain lifecycle methods. Unfortunately this creates problems for
> > > rescaling. How do we deal with a situation that subtask states with
> both
> > > the feature flag set and unset end up in a single StreamTask.
> Additional
> > > problem is that we merge those states into a single
> OperatorSubtaskState
> > on
> > > CheckpointCoordinator.
> > >
> > > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > > checkpoint*
> > >
> > > We need to wait for a checkpoint to complete, that started after the
> > > finish() method. However, we support concurrent checkpoints therefore,
> > > there might be later checkpoints that completed, but the notification
> has
> > > not arrived. We must make sure those checkpoints do not leave
> lingering
> > > external resources.
> > >
> > > *Checkpointing from a single subtask / UnionListState case*
> > > There are operators that checkpoint from a single subtask only.
> Usually
> > > from the subtask index=0. If we let those subtasks finish, subsequent
> > > checkpoints will miss this information.
> > > Esp. Legacy sources problem:
> > > https://issues.apache.org/jira/browse/FLINK-21080
> > >
> > > Best,
> > >
> > > Dawid
> > > On 19/07/2021 15:10, Yun Gao wrote:
> > >
> > > Hi Till, Dawid,
> > >
> > > Very thanks for the comments and discussion! Glad that it seems we
> have
> > > come to a convergence, and I also agree with that we could not include
> > the
> > > optimization in the first version.
> > >
> > > Best
> > > Yun
> > >
> > > ------------------------------------------------------------------
> > > From:Dawid Wysakowicz <dw...@apache.org> <
> dwysakowicz@apache.org>
> > > Send Time:2021 Jul. 19 (Mon.) 17:51
> > > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > > <tr...@apache.org> <tr...@apache.org>
> > > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > Finished
> > >
> > > Small correction. I meant we need to adjust the EndOfInputEvent of
> > course.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > > Hey Till,
> > > >
> > > > Yes, you're right we will have to adjust the current state of
> > > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > > we're discussing here. We are aware of that.
> > > >
> > > > As for the MAX_WATERMARK vs finish(). My take is that we should
> always
> > > > emit MAX_WATERMARK before calling finish() on an operator. At the
> same
> > > > time finish() should not leave behind anything in state, as the
> > > > intention is that we never restore from the taken
> savepoint/checkpoint
> > > > (savepoint w drain or bounded data consumed).
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > > >> Hi Yun and Dawid,
> > > >>
> > >
> > > >> Thanks for your comments. I do agree with your comments that
> finish()
> > can
> > > >> do more than MAX_WATERMARK. I guess we should then explain how
> > > >> MAX_WATERMARK and finish() play together and what kind of
> > > >> order guarantees we provide.
> > > >>
> > >
> > > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether
> it
> > would
> > >
> > > >> work in its current state because we send this event when the Task
> is
> > about
> > > >> to shut down if I am not mistaken. What we want to have is to bring
> > the
> > >
> > > >> StreamTasks into a state so that they shut down on the next
> > checkpoint. For
> > > >> this we need to keep the StreamTask running. In general, I am a
> fan of
> > > >> making things explicit if possible. I think this helps maintenance
> and
> > >
> > > >> evolvability of code. That's why I think sending an EndOfInputEvent
> > which
> > > >> is a StreamTask level event and which says that there won't be any
> > other
> > > >> records coming only control events could make sense.
> > > >>
> > > >> I would leave the proposed optimization out of the first version.
> We
> > can
> > > >> still add it at a later point in time.
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > > <dw...@apache.org> <dw...@apache.org>
> > > >> wrote:
> > > >>
> > >
> > > >>> Personally I don't find this optimization important and I'd rather
> > leave
> > >
> > > >>> it out not to complicate the codebase further. I doubt we save
> much
> > there.
> > > >>> I don't have a strong opinion though.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Dawid
> > > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> Very thanks Dawid for the thoughts!
> > > >>>
> > > >>> Currently I also do not have different opinions regarding this
> part.
> > > >>> But I have one more issue to confirm: during the previous
> discussion
> > we
> > > >>> have discussed that for the final checkpoint case, we might have
> an
> > > >>> optmization
> > >
> > > >>> that if a task do not have operators using 2-pc, we might skip
> > waiting for
> > > >>> the
> > >
> > > >>> final checkpoint (but we could not skip the savepoint). To allow
> > users to
> > > >>> express
> > > >>> the logic, we have proposed to add one more method to
> StreamOperator
> > &
> > > >>> CheckpointListener:
> > > >>>
> > > >>> interface StreamOperator {
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> interface CheckpointListener {
> > > >>>
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> class AbstractUdfStreamOperator {
> > > >>>
> > > >>> @Override
> > > >>> boolean requiresFinalCheckpoint() {
> > > >>> return userFunction instanceof CheckpointListener &&
> > >
> > > >>> ((CheckpointListener)
> > userFunction).requiresFinalCheckpoint();
> > > >>> }
> > > >>> }
> > > >>>
> > > >>>
> > > >>> I think we should still keep the change ?
> > > >>>
> > > >>> Best,
> > > >>> Yun
> > > >>>
> > > >>> ------------------Original Mail ------------------
> > > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > > <dw...@apache.org>
> > > >>> <dw...@apache.org> <dw...@apache.org>
> > > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <
> dev@flink.apache.org>
> > > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > > >>> Finished
> > > >>>
> > > >>>> I think we're all really close to the same solution.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time
> based
> > > >>>>
> > >
> > > >>>> buffering, but it does not solve flushing other operations such
> as
> > e.g.
> > > >>>>
> > > >>>> count windows or batching requests in Sinks. I'd prefer to treat
> the
> > > >>>>
> > > >>>> finish() as a message for Operator to "flush all records". The
> > > >>>>
> > >
> > > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> > imo. I
> > > >>>>
> > > >>>> don't think operators need to get a signal "stop-processing" if
> they
> > > >>>>
> > > >>>> don't need to flush records. The "WHEN" records are emitted,
> should
> > be
> > > >>>>
> > > >>>> in control of the StreamTask, by firing timers or by processing a
> > next
> > > >>>>
> > > >>>> record from upstream.
> > > >>>>
> > > >>>>
> > > >>>>
> > >
> > > >>>> The only difference of my previous proposal compared to Yun's is
> > that I
> > > >>>>
> > > >>>> did not want to send the EndOfUserRecords event in case of stop
> w/o
> > > >>>>
> > > >>>> drain. My thinking was that we could directly go from RUNNING to
> > > >>>>
> > > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could
> emit
> > > >>>>
> > > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop
> firing
> > > >>>>
> > >
> > > >>>> timers and processing events (without calling finish() on
> > Operator). In
> > > >>>>
> > > >>>> my initial suggestion I though we don't care about some events
> > > >>>>
> > > >>>> potentially being emitted after the savepoint was taken, as they
> > would
> > > >>>>
> > >
> > > >>>> anyway belong to the next after FINAL, which would be discarded.
> I
> > think
> > > >>>>
> > > >>>> though the proposal to suspend records processing and timers is a
> > > >>>>
> > >
> > > >>>> sensible thing to do and would go with the version that Yun put
> > into the
> > > >>>>
> > > >>>> FLIP Wiki.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> What do you think Till?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Dawid
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > > >>>>
> > > >>>>> Hi Till, Piotr
> > > >>>>> Very thanks for the comments!
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >
> > > >>>>> I also agree with Piotr that currently they are independent
> > mechanisms,
> > > >>>> and they are basically the same
> > > >>>>
> > > >>>>> for the event time.
> > > >>>>> For more details, first there are some difference among the
> three
> > > >>>> scenarios regarding the finish:
> > > >>>>
> > >
> > > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> > not be
> > > >>>> expected to be restarted,
> > > >>>>
> > > >>>>> and for stop-with-savepoint the job would be expected restart
> > later.
> > > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> > would
> > > >>>> emit MAX_WATERMARK before the
> > > >>>>
> > >
> > > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> > endOfInput /
> > > >>>> finish() should also only be called
> > > >>>>
> > > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> > always
> > > >>>> occurs at the same time. After the change,
> > > >>>>
> > > >>>>> we could emit MAX_WATERMARK before endOfInput event for the
> finish
> > /
> > > >>>> stop-with-savepoint --drain cases.
> > > >>>>
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > a
> > > >>>>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>>> finish, for example?
> > > >>>>> As discussed above for stop-with-savepoint, we would always keep
> > the
> > > >>>> window as is, and restore them after restart.
> > > >>>>
> > > >>>>> Then for the finish / stop-with-savepoint --drain, I think
> perhaps
> > it
> > > >>>> depends on the Triggers. For
> > > >>>>
> > >
> > > >>>>> event-time triggers / process time triggers, it would be
> > reasonable to
> > > >>>> flush all the windows since logically
> > > >>>>
> > >
> > > >>>>> the time would always elapse and the window would always get
> > triggered
> > > >>>> in a logical future. But for triggers
> > > >>>>
> > >
> > > >>>>> like CountTrigger, no matter how much time pass logically, the
> > windows
> > > >>>> would not trigger, thus we may not
> > > >>>>
> > > >>>>> flush these windows. If there are requirements we may provide
> > > >>>> additional triggers.
> > > >>>>
> > >
> > > >>>>>> It's a bit messy and I'm not sure if this should be
> strengthened
> > out?
> > > >>>> Each one of those has a little bit different semantic/meaning,
> > > >>>>
> > >
> > > >>>>>> but at the same time they are very similar. For single input
> > operators
> > > >>>> `endInput()` and `finish()` are actually the very same thing.
> > > >>>>
> > >
> > > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> > at the
> > > >>>> same time, and for single input operators `endInput()` and
> > `finish()`
> > > >>>>
> > >
> > > >>>>> are indeed the same thing. During the last discussion we ever
> > mentioned
> > > >>>> this issue and at then we thought that we might deprecate
> > `endInput()`
> > > >>>>
> > > >>>>> in the future, then we would only have endInput(int input) and
> > > >>>> finish().
> > > >>>>
> > > >>>>> Best,
> > > >>>>> Yun
> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > > >>>>>
> ------------------------------------------------------------------
> > > >>>>> From:Piotr Nowojski
> > > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > > >>>>> To:dev
> > > >>>>> Cc:Yun Gao
> > > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > > >>>> Finished
> > > >>>>
> > > >>>>> Hi Till,
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > a>
> > > >>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>
> > > >>>>>> finish, for example?
> > > >>>>> 1) currently they are independent but parallel mechanisms. With
> > event
> > > >>>> time, they are basically the same.
> > > >>>>
> > > >>>>> 2) it probably should for the sake of processing time windows.
> > > >>>>> Here you are touching the bit of the current design that I like
> the
> > >
> > > >>>> least. We basically have now three different ways of conveying
> very
> > similar
> > > >>>> things:
> > > >>>>
> > > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator
> (what
> > > >>>> about processing time?)
> > > >>>>
> > > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush
> it's
> > > >>>> internal state
> > > >>>>
> > > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> > out?
> > >
> > > >>>> Each one of those has a little bit different semantic/meaning,
> but
> > at the
> > >
> > > >>>> same time they are very similar. For single input operators
> > `endInput()`
> > > >>>> and `finish()` are actually the very same thing.
> > > >>>>
> > > >>>>> Piotrek
> > > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > > >>>>> Thanks for updating the FLIP. Based on the new section about
> > > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would a
> > > >>>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>> finish, for example?
> > > >>>>> Cheers,
> > > >>>>> Till
> > > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > > >>>> problem.
> > > >>>>
> > > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > > >>>>>> Cheers,
> > > >>>>>> Till
> > > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > > >>>>>> wrote:
> > > >>>>>>> Hi Till,
> > > >>>>>>> Very thanks for the review and comments!
> > > >>>>>>> 1) First I think in fact we could be able to do the
> computation
> > > >>>> outside
> > > >>>>
> > > >>>>>>> of the main thread,
> > > >>>>>>> and the current implementation mainly due to the computation
> is
> > in
> > > >>>>>>> general fast and we
> > > >>>>>>> initially want to have a simplified first version.
> > >
> > > >>>>>>> The main requirement here is to have a constant view of the
> > state of
> > > >>>> the
> > > >>>>
> > > >>>>>>> tasks, otherwise
> > > >>>>>>> for example if we have A -> B, if A is running when we check
> if
> > we
> > > >>>> need
> > > >>>>
> > > >>>>>>> to trigger A, we will
> > > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> > check
> > > >>>> B, we
> > > >>>>
> > > >>>>>>> will also mark B as
> > >
> > > >>>>>>> have to trigger, then B will receive both rpc trigger and
> > checkpoint
> > > >>>>>>> barrier, which would break
> > > >>>>>>> some assumption on the task side and complicate the
> > implementation.
> > >
> > > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> > of the
> > > >>>>>>> tasks' state and then do the
> > >
> > > >>>>>>> computation, both the two step do not need to be in the main
> > thread.
> > >
> > > >>>>>>> 2) For the computation logic, in fact currently we benefit a
> lot
> > from
> > > >>>>>>> some shortcuts on all-to-all
> > >
> > > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> > could do
> > > >>>>>>> checks on the job vertex level
> > > >>>>>>> first and skip some job vertices as a whole. With this
> > optimization
> > > >>>> we
> > > >>>>
> > > >>>>>>> have a O(V) algorithm, and the
> > > >>>>>>> current running time of the worst case for a job with 320,000
> > tasks
> > > >>>> is
> > > >>>>
> > > >>>>>>> less than 100ms. For
> > > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > > >>>>>>> If we do the computation based on the last triggered tasks, we
> > may
> > > >>>> not
> > > >>>>
> > > >>>>>>> easily encode this information
> > >
> > > >>>>>>> into the shortcuts on the job vertex level. And since the time
> > seems
> > > >>>> to
> > > >>>>
> > > >>>>>>> be short, perhaps it is enough
> > > >>>>>>> to do re-computation from the scratch in consideration of the
> > > >>>> tradeoff
> > > >>>>
> > > >>>>>>> between the performance and
> > > >>>>>>> the complexity ?
> > > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > > >>>> finish()
> > > >>>>
> > > >>>>>>> method and before the last
> > >
> > > >>>>>>> snapshotState() method so that we could shut down the whole
> > topology
> > > >>>> with
> > > >>>>
> > > >>>>>>> a single final checkpoint.
> > > >>>>>>> Very sorry for not include enough details for this part and
> I'll
> > > >>>>>>> complement the FLIP with the details on
> > > >>>>>>> the process of the final checkpoint / savepoint.
> > > >>>>>>> Best,
> > > >>>>>>> Yun
> > > >>>>>>>
> > ------------------------------------------------------------------
> > > >>>>>>> From:Till Rohrmann
> > > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > > >>>>>>> To:dev
> > > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> > Tasks
> > > >>>>>>> Finished
> > > >>>>>>> Hi everyone,
> > > >>>>>>> I am a bit late to the voting party but let me ask three
> > questions:
> > >
> > > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> > thread
> > > >>>> if we
> > > >>>>
> > >
> > > >>>>>>> cannot guarantee that all tasks are still running when
> > triggering the
> > > >>>>>>> checkpoint? Couldn't we do the computation in a different
> thread
> > in
> > > >>>> order
> > > >>>>
> > > >>>>>>> to relieve the main thread a bit.
> > > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> > seems
> > > >>>> to go
> > > >>>>
> > > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> > more
> > > >>>>>>> efficient to maintain the set of current tasks to trigger and
> > check
> > > >>>>>>> whether
> > > >>>>>>> anything has changed and if so check the succeeding tasks
> until
> > we
> > > >>>> have
> > > >>>>
> > > >>>>>>> found the current checkpoint trigger frontier?
> > > >>>>>>> 3) When are we going to send the endOfInput events to a
> > downstream
> > > >>>> task?
> > > >>>>
> > > >>>>>>> If
> > >
> > > >>>>>>> this happens after we call finish on the upstream operator but
> > before
> > > >>>>>>> snapshotState then it would be possible to shut down the whole
> > > >>>> topology
> > > >>>>
> > >
> > > >>>>>>> with a single final checkpoint. I think this part could
> benefit
> > from
> > > >>>> a bit
> > > >>>>
> > > >>>>>>> more detailed description in the FLIP.
> > > >>>>>>> Cheers,
> > > >>>>>>> Till
> > > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > > >>>>>>> wrote:
> > > >>>>>>>> Hi there,
> > > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing
> the
> > > >>>> vote
> > > >>>>
> > > >>>>>>> now.
> > > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1
> votes:
> > > >>>>>>>> - Dawid Wysakowicz (binding)
> > > >>>>>>>> - Piotr Nowojski(binding)
> > > >>>>>>>> - Jiangang Liu (binding)
> > > >>>>>>>> - Arvid Heise (binding)
> > > >>>>>>>> - Jing Zhang (binding)
> > > >>>>>>>> - Leonard Xu (non-binding)
> > > >>>>>>>> - Guowei Ma (binding)
> > > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > > >>>> accepted.
> > > >>>>
> > > >>>>>>>> Very thanks everyone!
> > > >>>>>>>> Best,
> > > >>>>>>>> Yun
> > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > >>>>
> > > >>>>
> > >
> > >
> >
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Dawid, Piotr, Steven,
Very thanks for pointing out these issues and very thanks for the discussion !
Failure before notifyCheckpointComplete()
For this issue I would agree with what Piotr has proposed. I tried to use some
operators like sink / window as example and currently I also do not found
explicit scenarios that might cause problems if records are processed by a
task that is assigned with states snapshotted after calling finish() before.
For the future cases it seems users should be able to implement their
target logic by explicitly add a flag regarding finished, and perhaps have
different logic if this part of states are referred to. Besides, this case would
only happen on rescaling or topology change, which embedded some kind of user
knowledge inside the action. Thus it looks acceptable that we still split the operators
state from the task lifecycle, and do not treat checkpoint after finish() differently.
Finishing upon receiving notifyCheckpointComplete() of not the latest checkpoint
For this issue perhaps we could explicitly requires the task to wait for a checkpoint triggered after finish()
method is called for all the operators ? We could be able to achieve this target by maintaining
some state inside the task.
Checkpointing from a single subtask / UnionListState case
This should indeed cause problems, and I also agree with that we could focus on this thread in the
https://issues.apache.org/jira/browse/FLINK-21080.
Best,
Yun
------------------------------------------------------------------
From:Piotr Nowojski <pn...@apache.org>
Send Time:2021 Jul. 22 (Thu.) 02:46
To:dev <de...@flink.apache.org>
Cc:Yun Gao <yu...@aliyun.com>; Till Rohrmann <tr...@apache.org>; Yun Gao <yu...@aliyun.com.invalid>; Piotr Nowojski <pi...@ververica.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Steven,
> I probably missed sth here. isn't this the case today already? Why is it a concern for the proposed change?
The problem is with the newly added `finish()` method and the already existing `endInput()` call. Currently on master there are no issues, because we are not checkpointing any operators after some operators have finished. The purpose of this FLIP-147 is to exactly enable this and this opens a new problem described by Dawid.
To paraphrase and to give a concrete example. Assume we have an operator with parallelism of two. Subtask 0 and subtask 1.
1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1 hasn't (yet).
2. Checkpoint 42 is triggered, and it completes.
3. Job fails and is restarted, but at the same time it's rescaled. User has chosen to scale down his operator down to 1.
Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has been processed or not, so while recovering to checkpoint 42, we have to recover both finished subtask 0 (#1) state and not yet finished subtask 1's (#1). But at the same time they are scaled down, so we only have a single subtask 0 (#2) that has a combined state from both of the previous instances. The potentially confusing issue is that the state from subtask 0 (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's recovered to an operator that has still some records to process. In step 1. an user for example could store on the operator's state a bit of information "end input has been already called!", that after recovery would no longer be true.
Hence the question about `finish()` and `endInput()` semantics. Should it be tied down to state, or just to an operator instance/execution attempt?
Piotrek
śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> if a failure happens after sequence of finish() -> snapshotState(), but
before notifyCheckpointComplete(), we will restore such a state and we
might end up sending some more records to such an operator.
I probably missed sth here. isn't this the case today already? Why is it a
concern for the proposed change?
On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi Dawid,
>
> Thanks for writing down those concerns.
>
> I think the first issue boils down what should be the contract of lifecycle
> methods like open(), close(), initializeState() etc and especially the new
> additions like finish() and endInput(). And what should be their relation
> with the operator state (regardless of it's type keyed, non-keyed, union,
> ...). Should those methods be tied to state or not? After thinking about it
> for a while (and discussing it offline with Dawid), I think the answer
> might be no, they shouldn't. I mean maybe we should just openly say that
> all of those methods relate to this single particular instance and
> execution of the operator. And if a job is recovered/rescaled, we would be
> allowed to freely resume consumption, ignoring a fact that maybe some parts
> of the state have previously seen `endInput()`. Why?
>
> 0. Yes, it might be confusing. Especially with `endInput()`. We call
> `endInput()`, we store something in a state and later after recovery
> combined with rescaling that state can see more records? Indeed weird,
> 1. I haven't come up yet with a counterexample that would break and make
> impossible to implement a real life use case. Theoretically yes, the user
> can store `endInput()` on state, and after rescaling this state would be
> inconsistent with what is actually happening with the operator, but I
> haven't found a use case that would break because of that.
> 2. Otherwise, implementation would be very difficult.
> 3. It's difficult to access keyed state from within `endInput()`/`finish()`
> calls, as they do not have key context.
> 4. After all, openly defining `endInput()` and `finish()` to be tied with
> it's operator execution instance lifecycle is not that strange and quite
> simple to explain. Sure, it can lead to a bit of confusion (0.), but that
> doesn't sound that bad in comparison with the alternatives that I'm aware
> of. Also currently methods like `open()` and `close()` are also tied to the
> operator execution instance, not to the state. Operators can be opened and
> closed multiple times, it doesn't mean that the state is lost after closing
> an operator.
>
> For the UnionListState problem I have posted my proposal in the ticket [1],
> so maybe let's move that particular discussion there?
>
> Piotrek
>
> [1] https://issues.apache.org/jira/browse/FLINK-21080
>
> śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> napisał(a):
>
> > Hey all,
> >
> > To make the issues that were found transparent to the community, I want
> to
> > post an update:
> >
> > *1. Committing side-effects*
> > We do want to make sure that all side effects are committed before
> > bringing tasks down. Side effects are committed when calling
> > notifyCheckpointComplete. For the final checkpoint we introduced the
> method
> > finish(). This notifies the operator that we have consumed all incoming
> > records and we are preparing to close the Task. In turn we should flush
> any
> > pending buffered records and prepare to commit last transactions. The
> goal
> > is that after a successful sequence of finish() -> snapshotState() ->
> > notifyCheckpointComplete(), the remaining state can be considered
> > empty/finished and may be discarded.
> >
> > *Failure before notifyCheckpointComplete()*
> >
> > The question is what is the contract of the endInput()/finish() methods
> > and how do calling these methods affect the operators keyed, non-keyed
> > state and external state. Is it allowed to restore state snapshot taken
> > after calling endInput()/finish() and process more records? Or do we
> assume
> > that after a restore from such a state taken after finish() we should not
> > call any of the lifecycle methods or at least make sure those methods do
> > not emit records/interact with mailbox etc.
> >
> > Currently it is possible that if a failure happens after sequence of
> > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> will
> > restore such a state and we might end up sending some more records to
> such
> > an operator. It is possible if we rescale and this state is merged with a
> > state of a subtask that has not called finish() yet. It can also happen
> if
> > we rescale the upstream operator and the subtask of interest becomes
> > connected to a newly added non finished subtask.
> >
> > *Snapshotting StreamTasks that finish() has been called*
> >
> >
> > We thought about putting a flag into the snapshot of a subtask produced
> > after the finish() method. This would make it possible to skip execution
> of
> > certain lifecycle methods. Unfortunately this creates problems for
> > rescaling. How do we deal with a situation that subtask states with both
> > the feature flag set and unset end up in a single StreamTask. Additional
> > problem is that we merge those states into a single OperatorSubtaskState
> on
> > CheckpointCoordinator.
> >
> > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > checkpoint*
> >
> > We need to wait for a checkpoint to complete, that started after the
> > finish() method. However, we support concurrent checkpoints therefore,
> > there might be later checkpoints that completed, but the notification has
> > not arrived. We must make sure those checkpoints do not leave lingering
> > external resources.
> >
> > *Checkpointing from a single subtask / UnionListState case*
> > There are operators that checkpoint from a single subtask only. Usually
> > from the subtask index=0. If we let those subtasks finish, subsequent
> > checkpoints will miss this information.
> > Esp. Legacy sources problem:
> > https://issues.apache.org/jira/browse/FLINK-21080
> >
> > Best,
> >
> > Dawid
> > On 19/07/2021 15:10, Yun Gao wrote:
> >
> > Hi Till, Dawid,
> >
> > Very thanks for the comments and discussion! Glad that it seems we have
> > come to a convergence, and I also agree with that we could not include
> the
> > optimization in the first version.
> >
> > Best
> > Yun
> >
> > ------------------------------------------------------------------
> > From:Dawid Wysakowicz <dw...@apache.org> <dw...@apache.org>
> > Send Time:2021 Jul. 19 (Mon.) 17:51
> > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > <tr...@apache.org> <tr...@apache.org>
> > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > Finished
> >
> > Small correction. I meant we need to adjust the EndOfInputEvent of
> course.
> >
> > Best,
> >
> > Dawid
> >
> > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > Hey Till,
> > >
> > > Yes, you're right we will have to adjust the current state of
> > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > we're discussing here. We are aware of that.
> > >
> > > As for the MAX_WATERMARK vs finish(). My take is that we should always
> > > emit MAX_WATERMARK before calling finish() on an operator. At the same
> > > time finish() should not leave behind anything in state, as the
> > > intention is that we never restore from the taken savepoint/checkpoint
> > > (savepoint w drain or bounded data consumed).
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > >> Hi Yun and Dawid,
> > >>
> >
> > >> Thanks for your comments. I do agree with your comments that finish()
> can
> > >> do more than MAX_WATERMARK. I guess we should then explain how
> > >> MAX_WATERMARK and finish() play together and what kind of
> > >> order guarantees we provide.
> > >>
> >
> > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether it
> would
> >
> > >> work in its current state because we send this event when the Task is
> about
> > >> to shut down if I am not mistaken. What we want to have is to bring
> the
> >
> > >> StreamTasks into a state so that they shut down on the next
> checkpoint. For
> > >> this we need to keep the StreamTask running. In general, I am a fan of
> > >> making things explicit if possible. I think this helps maintenance and
> >
> > >> evolvability of code. That's why I think sending an EndOfInputEvent
> which
> > >> is a StreamTask level event and which says that there won't be any
> other
> > >> records coming only control events could make sense.
> > >>
> > >> I would leave the proposed optimization out of the first version. We
> can
> > >> still add it at a later point in time.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > <dw...@apache.org> <dw...@apache.org>
> > >> wrote:
> > >>
> >
> > >>> Personally I don't find this optimization important and I'd rather
> leave
> >
> > >>> it out not to complicate the codebase further. I doubt we save much
> there.
> > >>> I don't have a strong opinion though.
> > >>>
> > >>> Best,
> > >>>
> > >>> Dawid
> > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> Very thanks Dawid for the thoughts!
> > >>>
> > >>> Currently I also do not have different opinions regarding this part.
> > >>> But I have one more issue to confirm: during the previous discussion
> we
> > >>> have discussed that for the final checkpoint case, we might have an
> > >>> optmization
> >
> > >>> that if a task do not have operators using 2-pc, we might skip
> waiting for
> > >>> the
> >
> > >>> final checkpoint (but we could not skip the savepoint). To allow
> users to
> > >>> express
> > >>> the logic, we have proposed to add one more method to StreamOperator
> &
> > >>> CheckpointListener:
> > >>>
> > >>> interface StreamOperator {
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> interface CheckpointListener {
> > >>>
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> class AbstractUdfStreamOperator {
> > >>>
> > >>> @Override
> > >>> boolean requiresFinalCheckpoint() {
> > >>> return userFunction instanceof CheckpointListener &&
> >
> > >>> ((CheckpointListener)
> userFunction).requiresFinalCheckpoint();
> > >>> }
> > >>> }
> > >>>
> > >>>
> > >>> I think we should still keep the change ?
> > >>>
> > >>> Best,
> > >>> Yun
> > >>>
> > >>> ------------------Original Mail ------------------
> > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > <dw...@apache.org>
> > >>> <dw...@apache.org> <dw...@apache.org>
> > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>
> > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>> Finished
> > >>>
> > >>>> I think we're all really close to the same solution.
> > >>>>
> > >>>>
> > >>>>
> > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
> > >>>>
> >
> > >>>> buffering, but it does not solve flushing other operations such as
> e.g.
> > >>>>
> > >>>> count windows or batching requests in Sinks. I'd prefer to treat the
> > >>>>
> > >>>> finish() as a message for Operator to "flush all records". The
> > >>>>
> >
> > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> imo. I
> > >>>>
> > >>>> don't think operators need to get a signal "stop-processing" if they
> > >>>>
> > >>>> don't need to flush records. The "WHEN" records are emitted, should
> be
> > >>>>
> > >>>> in control of the StreamTask, by firing timers or by processing a
> next
> > >>>>
> > >>>> record from upstream.
> > >>>>
> > >>>>
> > >>>>
> >
> > >>>> The only difference of my previous proposal compared to Yun's is
> that I
> > >>>>
> > >>>> did not want to send the EndOfUserRecords event in case of stop w/o
> > >>>>
> > >>>> drain. My thinking was that we could directly go from RUNNING to
> > >>>>
> > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> > >>>>
> > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> > >>>>
> >
> > >>>> timers and processing events (without calling finish() on
> Operator). In
> > >>>>
> > >>>> my initial suggestion I though we don't care about some events
> > >>>>
> > >>>> potentially being emitted after the savepoint was taken, as they
> would
> > >>>>
> >
> > >>>> anyway belong to the next after FINAL, which would be discarded. I
> think
> > >>>>
> > >>>> though the proposal to suspend records processing and timers is a
> > >>>>
> >
> > >>>> sensible thing to do and would go with the version that Yun put
> into the
> > >>>>
> > >>>> FLIP Wiki.
> > >>>>
> > >>>>
> > >>>>
> > >>>> What do you think Till?
> > >>>>
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>>
> > >>>>
> > >>>> Dawid
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > >>>>
> > >>>>> Hi Till, Piotr
> > >>>>> Very thanks for the comments!
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > >>>>> I also agree with Piotr that currently they are independent
> mechanisms,
> > >>>> and they are basically the same
> > >>>>
> > >>>>> for the event time.
> > >>>>> For more details, first there are some difference among the three
> > >>>> scenarios regarding the finish:
> > >>>>
> >
> > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> not be
> > >>>> expected to be restarted,
> > >>>>
> > >>>>> and for stop-with-savepoint the job would be expected restart
> later.
> > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> would
> > >>>> emit MAX_WATERMARK before the
> > >>>>
> >
> > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> endOfInput /
> > >>>> finish() should also only be called
> > >>>>
> > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> always
> > >>>> occurs at the same time. After the change,
> > >>>>
> > >>>>> we could emit MAX_WATERMARK before endOfInput event for the finish
> /
> > >>>> stop-with-savepoint --drain cases.
> > >>>>
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a
> > >>>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>>> finish, for example?
> > >>>>> As discussed above for stop-with-savepoint, we would always keep
> the
> > >>>> window as is, and restore them after restart.
> > >>>>
> > >>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps
> it
> > >>>> depends on the Triggers. For
> > >>>>
> >
> > >>>>> event-time triggers / process time triggers, it would be
> reasonable to
> > >>>> flush all the windows since logically
> > >>>>
> >
> > >>>>> the time would always elapse and the window would always get
> triggered
> > >>>> in a logical future. But for triggers
> > >>>>
> >
> > >>>>> like CountTrigger, no matter how much time pass logically, the
> windows
> > >>>> would not trigger, thus we may not
> > >>>>
> > >>>>> flush these windows. If there are requirements we may provide
> > >>>> additional triggers.
> > >>>>
> >
> > >>>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> > >>>> Each one of those has a little bit different semantic/meaning,
> > >>>>
> >
> > >>>>>> but at the same time they are very similar. For single input
> operators
> > >>>> `endInput()` and `finish()` are actually the very same thing.
> > >>>>
> >
> > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> at the
> > >>>> same time, and for single input operators `endInput()` and
> `finish()`
> > >>>>
> >
> > >>>>> are indeed the same thing. During the last discussion we ever
> mentioned
> > >>>> this issue and at then we thought that we might deprecate
> `endInput()`
> > >>>>
> > >>>>> in the future, then we would only have endInput(int input) and
> > >>>> finish().
> > >>>>
> > >>>>> Best,
> > >>>>> Yun
> > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > >>>>> ------------------------------------------------------------------
> > >>>>> From:Piotr Nowojski
> > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > >>>>> To:dev
> > >>>>> Cc:Yun Gao
> > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>>> Finished
> > >>>>
> > >>>>> Hi Till,
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a>
> > >>>> WindowOperator close all windows and emit the results upon calling
> > >>>>
> > >>>>>> finish, for example?
> > >>>>> 1) currently they are independent but parallel mechanisms. With
> event
> > >>>> time, they are basically the same.
> > >>>>
> > >>>>> 2) it probably should for the sake of processing time windows.
> > >>>>> Here you are touching the bit of the current design that I like the
> >
> > >>>> least. We basically have now three different ways of conveying very
> similar
> > >>>> things:
> > >>>>
> > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
> > >>>> about processing time?)
> > >>>>
> > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
> > >>>> internal state
> > >>>>
> > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> >
> > >>>> Each one of those has a little bit different semantic/meaning, but
> at the
> >
> > >>>> same time they are very similar. For single input operators
> `endInput()`
> > >>>> and `finish()` are actually the very same thing.
> > >>>>
> > >>>>> Piotrek
> > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > >>>>> Thanks for updating the FLIP. Based on the new section about
> > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> > >>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>> finish, for example?
> > >>>>> Cheers,
> > >>>>> Till
> > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > >>>> problem.
> > >>>>
> > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > >>>>>> Cheers,
> > >>>>>> Till
> > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > >>>>>> wrote:
> > >>>>>>> Hi Till,
> > >>>>>>> Very thanks for the review and comments!
> > >>>>>>> 1) First I think in fact we could be able to do the computation
> > >>>> outside
> > >>>>
> > >>>>>>> of the main thread,
> > >>>>>>> and the current implementation mainly due to the computation is
> in
> > >>>>>>> general fast and we
> > >>>>>>> initially want to have a simplified first version.
> >
> > >>>>>>> The main requirement here is to have a constant view of the
> state of
> > >>>> the
> > >>>>
> > >>>>>>> tasks, otherwise
> > >>>>>>> for example if we have A -> B, if A is running when we check if
> we
> > >>>> need
> > >>>>
> > >>>>>>> to trigger A, we will
> > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> check
> > >>>> B, we
> > >>>>
> > >>>>>>> will also mark B as
> >
> > >>>>>>> have to trigger, then B will receive both rpc trigger and
> checkpoint
> > >>>>>>> barrier, which would break
> > >>>>>>> some assumption on the task side and complicate the
> implementation.
> >
> > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> of the
> > >>>>>>> tasks' state and then do the
> >
> > >>>>>>> computation, both the two step do not need to be in the main
> thread.
> >
> > >>>>>>> 2) For the computation logic, in fact currently we benefit a lot
> from
> > >>>>>>> some shortcuts on all-to-all
> >
> > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> could do
> > >>>>>>> checks on the job vertex level
> > >>>>>>> first and skip some job vertices as a whole. With this
> optimization
> > >>>> we
> > >>>>
> > >>>>>>> have a O(V) algorithm, and the
> > >>>>>>> current running time of the worst case for a job with 320,000
> tasks
> > >>>> is
> > >>>>
> > >>>>>>> less than 100ms. For
> > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > >>>>>>> If we do the computation based on the last triggered tasks, we
> may
> > >>>> not
> > >>>>
> > >>>>>>> easily encode this information
> >
> > >>>>>>> into the shortcuts on the job vertex level. And since the time
> seems
> > >>>> to
> > >>>>
> > >>>>>>> be short, perhaps it is enough
> > >>>>>>> to do re-computation from the scratch in consideration of the
> > >>>> tradeoff
> > >>>>
> > >>>>>>> between the performance and
> > >>>>>>> the complexity ?
> > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > >>>> finish()
> > >>>>
> > >>>>>>> method and before the last
> >
> > >>>>>>> snapshotState() method so that we could shut down the whole
> topology
> > >>>> with
> > >>>>
> > >>>>>>> a single final checkpoint.
> > >>>>>>> Very sorry for not include enough details for this part and I'll
> > >>>>>>> complement the FLIP with the details on
> > >>>>>>> the process of the final checkpoint / savepoint.
> > >>>>>>> Best,
> > >>>>>>> Yun
> > >>>>>>>
> ------------------------------------------------------------------
> > >>>>>>> From:Till Rohrmann
> > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >>>>>>> To:dev
> > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > >>>>>>> Finished
> > >>>>>>> Hi everyone,
> > >>>>>>> I am a bit late to the voting party but let me ask three
> questions:
> >
> > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> thread
> > >>>> if we
> > >>>>
> >
> > >>>>>>> cannot guarantee that all tasks are still running when
> triggering the
> > >>>>>>> checkpoint? Couldn't we do the computation in a different thread
> in
> > >>>> order
> > >>>>
> > >>>>>>> to relieve the main thread a bit.
> > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> seems
> > >>>> to go
> > >>>>
> > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> more
> > >>>>>>> efficient to maintain the set of current tasks to trigger and
> check
> > >>>>>>> whether
> > >>>>>>> anything has changed and if so check the succeeding tasks until
> we
> > >>>> have
> > >>>>
> > >>>>>>> found the current checkpoint trigger frontier?
> > >>>>>>> 3) When are we going to send the endOfInput events to a
> downstream
> > >>>> task?
> > >>>>
> > >>>>>>> If
> >
> > >>>>>>> this happens after we call finish on the upstream operator but
> before
> > >>>>>>> snapshotState then it would be possible to shut down the whole
> > >>>> topology
> > >>>>
> >
> > >>>>>>> with a single final checkpoint. I think this part could benefit
> from
> > >>>> a bit
> > >>>>
> > >>>>>>> more detailed description in the FLIP.
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > >>>>>>> wrote:
> > >>>>>>>> Hi there,
> > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
> > >>>> vote
> > >>>>
> > >>>>>>> now.
> > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >>>>>>>> - Dawid Wysakowicz (binding)
> > >>>>>>>> - Piotr Nowojski(binding)
> > >>>>>>>> - Jiangang Liu (binding)
> > >>>>>>>> - Arvid Heise (binding)
> > >>>>>>>> - Jing Zhang (binding)
> > >>>>>>>> - Leonard Xu (non-binding)
> > >>>>>>>> - Guowei Ma (binding)
> > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > >>>> accepted.
> > >>>>
> > >>>>>>>> Very thanks everyone!
> > >>>>>>>> Best,
> > >>>>>>>> Yun
> > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>>>
> > >>>>
> >
> >
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Guowei Ma <gu...@gmail.com>.
Hi,
Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can be
called repeatedly.
Let me share some of my understanding of this problem, hoping to help
people with a similar problem.
At the beginning, what bothered me the most was: if the framework notified
a task `Finish & Snapshot & CheckpointComplete` message,
why would it notify the task `EndInput` message again and even have new
data? The reason I found it strange at first was: I thought
CheckpointComplete (X) semantics is: the data before X will not come again,
if this X represents the end "position", then
it means that all the data has been processed, and then also There will be
no more data.
After having some offline discussions with Yun, I think the "location"
represented by X is actually related to the topology of the job.
CheckpointComplete(X) only means that the data has been processed before
the X position of an operator in a certain topology,
not all data of the entire job will never come again (especially when we
introduced Final Checkpoint Point).
Therefore, if the topology changes, under the new topology, the meaning of
the previous position has changed,
so it may be reasonable to continue to receive messages.
Best,
Guowei
On Thu, Jul 22, 2021 at 2:46 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi Steven,
>
> > I probably missed sth here. isn't this the case today already? Why is it
> a concern for the proposed change?
>
> The problem is with the newly added `finish()` method and the already
> existing `endInput()` call. Currently on master there are no issues,
> because we are not checkpointing any operators after some operators have
> finished. The purpose of this FLIP-147 is to exactly enable this and this
> opens a new problem described by Dawid.
>
> To paraphrase and to give a concrete example. Assume we have an operator
> with parallelism of two. Subtask 0 and subtask 1.
>
> 1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1
> hasn't (yet).
> 2. Checkpoint 42 is triggered, and it completes.
> 3. Job fails and is restarted, but at the same time it's rescaled. User has
> chosen to scale down his operator down to 1.
>
> Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has
> been processed or not, so while recovering to checkpoint 42, we have to
> recover both finished subtask 0 (#1) state and not yet finished subtask 1's
> (#1). But at the same time they are scaled down, so we only have a single
> subtask 0 (#2) that has a combined state from both of the previous
> instances. The potentially confusing issue is that the state from subtask 0
> (#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's
> recovered to an operator that has still some records to process. In step 1.
> an user for example could store on the operator's state a bit of
> information "end input has been already called!", that after recovery would
> no longer be true.
>
> Hence the question about `finish()` and `endInput()` semantics. Should it
> be tied down to state, or just to an operator instance/execution attempt?
>
> Piotrek
>
> śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
>
> > > if a failure happens after sequence of finish() -> snapshotState(), but
> > before notifyCheckpointComplete(), we will restore such a state and we
> > might end up sending some more records to such an operator.
> >
> > I probably missed sth here. isn't this the case today already? Why is it
> a
> > concern for the proposed change?
> >
> > On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org>
> > wrote:
> >
> > > Hi Dawid,
> > >
> > > Thanks for writing down those concerns.
> > >
> > > I think the first issue boils down what should be the contract of
> > lifecycle
> > > methods like open(), close(), initializeState() etc and especially the
> > new
> > > additions like finish() and endInput(). And what should be their
> relation
> > > with the operator state (regardless of it's type keyed, non-keyed,
> union,
> > > ...). Should those methods be tied to state or not? After thinking
> about
> > it
> > > for a while (and discussing it offline with Dawid), I think the answer
> > > might be no, they shouldn't. I mean maybe we should just openly say
> that
> > > all of those methods relate to this single particular instance and
> > > execution of the operator. And if a job is recovered/rescaled, we would
> > be
> > > allowed to freely resume consumption, ignoring a fact that maybe some
> > parts
> > > of the state have previously seen `endInput()`. Why?
> > >
> > > 0. Yes, it might be confusing. Especially with `endInput()`. We call
> > > `endInput()`, we store something in a state and later after recovery
> > > combined with rescaling that state can see more records? Indeed weird,
> > > 1. I haven't come up yet with a counterexample that would break and
> make
> > > impossible to implement a real life use case. Theoretically yes, the
> user
> > > can store `endInput()` on state, and after rescaling this state would
> be
> > > inconsistent with what is actually happening with the operator, but I
> > > haven't found a use case that would break because of that.
> > > 2. Otherwise, implementation would be very difficult.
> > > 3. It's difficult to access keyed state from within
> > `endInput()`/`finish()`
> > > calls, as they do not have key context.
> > > 4. After all, openly defining `endInput()` and `finish()` to be tied
> with
> > > it's operator execution instance lifecycle is not that strange and
> quite
> > > simple to explain. Sure, it can lead to a bit of confusion (0.), but
> that
> > > doesn't sound that bad in comparison with the alternatives that I'm
> aware
> > > of. Also currently methods like `open()` and `close()` are also tied to
> > the
> > > operator execution instance, not to the state. Operators can be opened
> > and
> > > closed multiple times, it doesn't mean that the state is lost after
> > closing
> > > an operator.
> > >
> > > For the UnionListState problem I have posted my proposal in the ticket
> > [1],
> > > so maybe let's move that particular discussion there?
> > >
> > > Piotrek
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-21080
> > >
> > > śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> > > napisał(a):
> > >
> > > > Hey all,
> > > >
> > > > To make the issues that were found transparent to the community, I
> want
> > > to
> > > > post an update:
> > > >
> > > > *1. Committing side-effects*
> > > > We do want to make sure that all side effects are committed before
> > > > bringing tasks down. Side effects are committed when calling
> > > > notifyCheckpointComplete. For the final checkpoint we introduced the
> > > method
> > > > finish(). This notifies the operator that we have consumed all
> incoming
> > > > records and we are preparing to close the Task. In turn we should
> flush
> > > any
> > > > pending buffered records and prepare to commit last transactions. The
> > > goal
> > > > is that after a successful sequence of finish() -> snapshotState() ->
> > > > notifyCheckpointComplete(), the remaining state can be considered
> > > > empty/finished and may be discarded.
> > > >
> > > > *Failure before notifyCheckpointComplete()*
> > > >
> > > > The question is what is the contract of the endInput()/finish()
> methods
> > > > and how do calling these methods affect the operators keyed,
> non-keyed
> > > > state and external state. Is it allowed to restore state snapshot
> taken
> > > > after calling endInput()/finish() and process more records? Or do we
> > > assume
> > > > that after a restore from such a state taken after finish() we should
> > not
> > > > call any of the lifecycle methods or at least make sure those methods
> > do
> > > > not emit records/interact with mailbox etc.
> > > >
> > > > Currently it is possible that if a failure happens after sequence of
> > > > finish() -> snapshotState(), but before notifyCheckpointComplete(),
> we
> > > will
> > > > restore such a state and we might end up sending some more records to
> > > such
> > > > an operator. It is possible if we rescale and this state is merged
> > with a
> > > > state of a subtask that has not called finish() yet. It can also
> happen
> > > if
> > > > we rescale the upstream operator and the subtask of interest becomes
> > > > connected to a newly added non finished subtask.
> > > >
> > > > *Snapshotting StreamTasks that finish() has been called*
> > > >
> > > >
> > > > We thought about putting a flag into the snapshot of a subtask
> produced
> > > > after the finish() method. This would make it possible to skip
> > execution
> > > of
> > > > certain lifecycle methods. Unfortunately this creates problems for
> > > > rescaling. How do we deal with a situation that subtask states with
> > both
> > > > the feature flag set and unset end up in a single StreamTask.
> > Additional
> > > > problem is that we merge those states into a single
> > OperatorSubtaskState
> > > on
> > > > CheckpointCoordinator.
> > > >
> > > > *Finishing upon receiving notifyCheckpointComplete() of not the
> latest
> > > > checkpoint*
> > > >
> > > > We need to wait for a checkpoint to complete, that started after the
> > > > finish() method. However, we support concurrent checkpoints
> therefore,
> > > > there might be later checkpoints that completed, but the notification
> > has
> > > > not arrived. We must make sure those checkpoints do not leave
> lingering
> > > > external resources.
> > > >
> > > > *Checkpointing from a single subtask / UnionListState case*
> > > > There are operators that checkpoint from a single subtask only.
> Usually
> > > > from the subtask index=0. If we let those subtasks finish, subsequent
> > > > checkpoints will miss this information.
> > > > Esp. Legacy sources problem:
> > > > https://issues.apache.org/jira/browse/FLINK-21080
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > > On 19/07/2021 15:10, Yun Gao wrote:
> > > >
> > > > Hi Till, Dawid,
> > > >
> > > > Very thanks for the comments and discussion! Glad that it seems we
> have
> > > > come to a convergence, and I also agree with that we could not
> include
> > > the
> > > > optimization in the first version.
> > > >
> > > > Best
> > > > Yun
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Dawid Wysakowicz <dw...@apache.org> <
> dwysakowicz@apache.org
> > >
> > > > Send Time:2021 Jul. 19 (Mon.) 17:51
> > > > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > > > <tr...@apache.org> <tr...@apache.org>
> > > > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > > > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > > Finished
> > > >
> > > > Small correction. I meant we need to adjust the EndOfInputEvent of
> > > course.
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > > > Hey Till,
> > > > >
> > > > > Yes, you're right we will have to adjust the current state of
> > > > > EndOfPartitionEvent and move the moment when we emit it to have
> what
> > > > > we're discussing here. We are aware of that.
> > > > >
> > > > > As for the MAX_WATERMARK vs finish(). My take is that we should
> > always
> > > > > emit MAX_WATERMARK before calling finish() on an operator. At the
> > same
> > > > > time finish() should not leave behind anything in state, as the
> > > > > intention is that we never restore from the taken
> > savepoint/checkpoint
> > > > > (savepoint w drain or bounded data consumed).
> > > > >
> > > > > Best,
> > > > >
> > > > > Dawid
> > > > >
> > > > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > > > >> Hi Yun and Dawid,
> > > > >>
> > > >
> > > > >> Thanks for your comments. I do agree with your comments that
> > finish()
> > > can
> > > > >> do more than MAX_WATERMARK. I guess we should then explain how
> > > > >> MAX_WATERMARK and finish() play together and what kind of
> > > > >> order guarantees we provide.
> > > > >>
> > > >
> > > > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether
> > it
> > > would
> > > >
> > > > >> work in its current state because we send this event when the Task
> > is
> > > about
> > > > >> to shut down if I am not mistaken. What we want to have is to
> bring
> > > the
> > > >
> > > > >> StreamTasks into a state so that they shut down on the next
> > > checkpoint. For
> > > > >> this we need to keep the StreamTask running. In general, I am a
> fan
> > of
> > > > >> making things explicit if possible. I think this helps maintenance
> > and
> > > >
> > > > >> evolvability of code. That's why I think sending an
> EndOfInputEvent
> > > which
> > > > >> is a StreamTask level event and which says that there won't be any
> > > other
> > > > >> records coming only control events could make sense.
> > > > >>
> > > > >> I would leave the proposed optimization out of the first version.
> We
> > > can
> > > > >> still add it at a later point in time.
> > > > >>
> > > > >> Cheers,
> > > > >> Till
> > > > >>
> > > > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > > > <dw...@apache.org> <dw...@apache.org>
> > > > >> wrote:
> > > > >>
> > > >
> > > > >>> Personally I don't find this optimization important and I'd
> rather
> > > leave
> > > >
> > > > >>> it out not to complicate the codebase further. I doubt we save
> much
> > > there.
> > > > >>> I don't have a strong opinion though.
> > > > >>>
> > > > >>> Best,
> > > > >>>
> > > > >>> Dawid
> > > > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> Very thanks Dawid for the thoughts!
> > > > >>>
> > > > >>> Currently I also do not have different opinions regarding this
> > part.
> > > > >>> But I have one more issue to confirm: during the previous
> > discussion
> > > we
> > > > >>> have discussed that for the final checkpoint case, we might have
> an
> > > > >>> optmization
> > > >
> > > > >>> that if a task do not have operators using 2-pc, we might skip
> > > waiting for
> > > > >>> the
> > > >
> > > > >>> final checkpoint (but we could not skip the savepoint). To allow
> > > users to
> > > > >>> express
> > > > >>> the logic, we have proposed to add one more method to
> > StreamOperator
> > > &
> > > > >>> CheckpointListener:
> > > > >>>
> > > > >>> interface StreamOperator {
> > > > >>> default boolean requiresFinalCheckpoint() {
> > > > >>> return true;
> > > > >>> }
> > > > >>> }
> > > > >>>
> > > > >>> interface CheckpointListener {
> > > > >>>
> > > > >>> default boolean requiresFinalCheckpoint() {
> > > > >>> return true;
> > > > >>> }
> > > > >>> }
> > > > >>>
> > > > >>> class AbstractUdfStreamOperator {
> > > > >>>
> > > > >>> @Override
> > > > >>> boolean requiresFinalCheckpoint() {
> > > > >>> return userFunction instanceof CheckpointListener &&
> > > >
> > > > >>> ((CheckpointListener)
> > > userFunction).requiresFinalCheckpoint();
> > > > >>> }
> > > > >>> }
> > > > >>>
> > > > >>>
> > > > >>> I think we should still keep the change ?
> > > > >>>
> > > > >>> Best,
> > > > >>> Yun
> > > > >>>
> > > > >>> ------------------Original Mail ------------------
> > > > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > > > <dw...@apache.org>
> > > > >>> <dw...@apache.org> <dw...@apache.org>
> > > > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > > > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <
> > dev@flink.apache.org>
> > > > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > > > >>> Gao <yu...@aliyun.com.INVALID> <yungao.gy@aliyun.com.INVALID
> >
> > > > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> > Tasks
> > > > >>> Finished
> > > > >>>
> > > > >>>> I think we're all really close to the same solution.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time
> > based
> > > > >>>>
> > > >
> > > > >>>> buffering, but it does not solve flushing other operations such
> as
> > > e.g.
> > > > >>>>
> > > > >>>> count windows or batching requests in Sinks. I'd prefer to treat
> > the
> > > > >>>>
> > > > >>>> finish() as a message for Operator to "flush all records". The
> > > > >>>>
> > > >
> > > > >>>> MAX_WATERMARK in my opinion is mostly for backwards
> compatibility
> > > imo. I
> > > > >>>>
> > > > >>>> don't think operators need to get a signal "stop-processing" if
> > they
> > > > >>>>
> > > > >>>> don't need to flush records. The "WHEN" records are emitted,
> > should
> > > be
> > > > >>>>
> > > > >>>> in control of the StreamTask, by firing timers or by processing
> a
> > > next
> > > > >>>>
> > > > >>>> record from upstream.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > >
> > > > >>>> The only difference of my previous proposal compared to Yun's is
> > > that I
> > > > >>>>
> > > > >>>> did not want to send the EndOfUserRecords event in case of stop
> > w/o
> > > > >>>>
> > > > >>>> drain. My thinking was that we could directly go from RUNNING to
> > > > >>>>
> > > > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could
> emit
> > > > >>>>
> > > > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop
> firing
> > > > >>>>
> > > >
> > > > >>>> timers and processing events (without calling finish() on
> > > Operator). In
> > > > >>>>
> > > > >>>> my initial suggestion I though we don't care about some events
> > > > >>>>
> > > > >>>> potentially being emitted after the savepoint was taken, as they
> > > would
> > > > >>>>
> > > >
> > > > >>>> anyway belong to the next after FINAL, which would be
> discarded. I
> > > think
> > > > >>>>
> > > > >>>> though the proposal to suspend records processing and timers is
> a
> > > > >>>>
> > > >
> > > > >>>> sensible thing to do and would go with the version that Yun put
> > > into the
> > > > >>>>
> > > > >>>> FLIP Wiki.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> What do you think Till?
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Best,
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Dawid
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > > > >>>>
> > > > >>>>> Hi Till, Piotr
> > > > >>>>> Very thanks for the comments!
> > > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >
> > > > >>>>> I also agree with Piotr that currently they are independent
> > > mechanisms,
> > > > >>>> and they are basically the same
> > > > >>>>
> > > > >>>>> for the event time.
> > > > >>>>> For more details, first there are some difference among the
> three
> > > > >>>> scenarios regarding the finish:
> > > > >>>>
> > > >
> > > > >>>>> For normal finish and stop-with-savepoint --drain, the job
> would
> > > not be
> > > > >>>> expected to be restarted,
> > > > >>>>
> > > > >>>>> and for stop-with-savepoint the job would be expected restart
> > > later.
> > > > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> > > would
> > > > >>>> emit MAX_WATERMARK before the
> > > > >>>>
> > > >
> > > > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> > > endOfInput /
> > > > >>>> finish() should also only be called
> > > > >>>>
> > > > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> > > always
> > > > >>>> occurs at the same time. After the change,
> > > > >>>>
> > > > >>>>> we could emit MAX_WATERMARK before endOfInput event for the
> > finish
> > > /
> > > > >>>> stop-with-savepoint --drain cases.
> > > > >>>>
> > > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> > Would
> > > a
> > > > >>>>>> WindowOperator close all windows and emit the results upon
> > calling
> > > > >>>>>> finish, for example?
> > > > >>>>> As discussed above for stop-with-savepoint, we would always
> keep
> > > the
> > > > >>>> window as is, and restore them after restart.
> > > > >>>>
> > > > >>>>> Then for the finish / stop-with-savepoint --drain, I think
> > perhaps
> > > it
> > > > >>>> depends on the Triggers. For
> > > > >>>>
> > > >
> > > > >>>>> event-time triggers / process time triggers, it would be
> > > reasonable to
> > > > >>>> flush all the windows since logically
> > > > >>>>
> > > >
> > > > >>>>> the time would always elapse and the window would always get
> > > triggered
> > > > >>>> in a logical future. But for triggers
> > > > >>>>
> > > >
> > > > >>>>> like CountTrigger, no matter how much time pass logically, the
> > > windows
> > > > >>>> would not trigger, thus we may not
> > > > >>>>
> > > > >>>>> flush these windows. If there are requirements we may provide
> > > > >>>> additional triggers.
> > > > >>>>
> > > >
> > > > >>>>>> It's a bit messy and I'm not sure if this should be
> strengthened
> > > out?
> > > > >>>> Each one of those has a little bit different semantic/meaning,
> > > > >>>>
> > > >
> > > > >>>>>> but at the same time they are very similar. For single input
> > > operators
> > > > >>>> `endInput()` and `finish()` are actually the very same thing.
> > > > >>>>
> > > >
> > > > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always
> happen
> > > at the
> > > > >>>> same time, and for single input operators `endInput()` and
> > > `finish()`
> > > > >>>>
> > > >
> > > > >>>>> are indeed the same thing. During the last discussion we ever
> > > mentioned
> > > > >>>> this issue and at then we thought that we might deprecate
> > > `endInput()`
> > > > >>>>
> > > > >>>>> in the future, then we would only have endInput(int input) and
> > > > >>>> finish().
> > > > >>>>
> > > > >>>>> Best,
> > > > >>>>> Yun
> > > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > > > >>>>>
> > ------------------------------------------------------------------
> > > > >>>>> From:Piotr Nowojski
> > > > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > > > >>>>> To:dev
> > > > >>>>> Cc:Yun Gao
> > > > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> > Tasks
> > > > >>>> Finished
> > > > >>>>
> > > > >>>>> Hi Till,
> > > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> > Would
> > > a>
> > > > >>>> WindowOperator close all windows and emit the results upon
> calling
> > > > >>>>
> > > > >>>>>> finish, for example?
> > > > >>>>> 1) currently they are independent but parallel mechanisms. With
> > > event
> > > > >>>> time, they are basically the same.
> > > > >>>>
> > > > >>>>> 2) it probably should for the sake of processing time windows.
> > > > >>>>> Here you are touching the bit of the current design that I like
> > the
> > > >
> > > > >>>> least. We basically have now three different ways of conveying
> > very
> > > similar
> > > > >>>> things:
> > > > >>>>
> > > > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator
> > (what
> > > > >>>> about processing time?)
> > > > >>>>
> > > > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush
> > it's
> > > > >>>> internal state
> > > > >>>>
> > > > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > > > >>>>> It's a bit messy and I'm not sure if this should be
> strengthened
> > > out?
> > > >
> > > > >>>> Each one of those has a little bit different semantic/meaning,
> but
> > > at the
> > > >
> > > > >>>> same time they are very similar. For single input operators
> > > `endInput()`
> > > > >>>> and `finish()` are actually the very same thing.
> > > > >>>>
> > > > >>>>> Piotrek
> > > > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > > > >>>>> Thanks for updating the FLIP. Based on the new section about
> > > > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > > > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > > >>>>> 2) StreamOperator.finish says to flush all buffered events.
> > Would a
> > > > >>>>> WindowOperator close all windows and emit the results upon
> > calling
> > > > >>>>> finish, for example?
> > > > >>>>> Cheers,
> > > > >>>>> Till
> > > > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > > > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > > > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes
> a
> > > > >>>> problem.
> > > > >>>>
> > > > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > > > >>>>>> Cheers,
> > > > >>>>>> Till
> > > > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > > > >>>>>> wrote:
> > > > >>>>>>> Hi Till,
> > > > >>>>>>> Very thanks for the review and comments!
> > > > >>>>>>> 1) First I think in fact we could be able to do the
> computation
> > > > >>>> outside
> > > > >>>>
> > > > >>>>>>> of the main thread,
> > > > >>>>>>> and the current implementation mainly due to the computation
> is
> > > in
> > > > >>>>>>> general fast and we
> > > > >>>>>>> initially want to have a simplified first version.
> > > >
> > > > >>>>>>> The main requirement here is to have a constant view of the
> > > state of
> > > > >>>> the
> > > > >>>>
> > > > >>>>>>> tasks, otherwise
> > > > >>>>>>> for example if we have A -> B, if A is running when we check
> if
> > > we
> > > > >>>> need
> > > > >>>>
> > > > >>>>>>> to trigger A, we will
> > > > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> > > check
> > > > >>>> B, we
> > > > >>>>
> > > > >>>>>>> will also mark B as
> > > >
> > > > >>>>>>> have to trigger, then B will receive both rpc trigger and
> > > checkpoint
> > > > >>>>>>> barrier, which would break
> > > > >>>>>>> some assumption on the task side and complicate the
> > > implementation.
> > > >
> > > > >>>>>>> But to cope this issue, we in fact could first have a
> snapshot
> > > of the
> > > > >>>>>>> tasks' state and then do the
> > > >
> > > > >>>>>>> computation, both the two step do not need to be in the main
> > > thread.
> > > >
> > > > >>>>>>> 2) For the computation logic, in fact currently we benefit a
> > lot
> > > from
> > > > >>>>>>> some shortcuts on all-to-all
> > > >
> > > > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> > > could do
> > > > >>>>>>> checks on the job vertex level
> > > > >>>>>>> first and skip some job vertices as a whole. With this
> > > optimization
> > > > >>>> we
> > > > >>>>
> > > > >>>>>>> have a O(V) algorithm, and the
> > > > >>>>>>> current running time of the worst case for a job with 320,000
> > > tasks
> > > > >>>> is
> > > > >>>>
> > > > >>>>>>> less than 100ms. For
> > > > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > > > >>>>>>> If we do the computation based on the last triggered tasks,
> we
> > > may
> > > > >>>> not
> > > > >>>>
> > > > >>>>>>> easily encode this information
> > > >
> > > > >>>>>>> into the shortcuts on the job vertex level. And since the
> time
> > > seems
> > > > >>>> to
> > > > >>>>
> > > > >>>>>>> be short, perhaps it is enough
> > > > >>>>>>> to do re-computation from the scratch in consideration of the
> > > > >>>> tradeoff
> > > > >>>>
> > > > >>>>>>> between the performance and
> > > > >>>>>>> the complexity ?
> > > > >>>>>>> 3) We are going to emit the EndOfInput event exactly after
> the
> > > > >>>> finish()
> > > > >>>>
> > > > >>>>>>> method and before the last
> > > >
> > > > >>>>>>> snapshotState() method so that we could shut down the whole
> > > topology
> > > > >>>> with
> > > > >>>>
> > > > >>>>>>> a single final checkpoint.
> > > > >>>>>>> Very sorry for not include enough details for this part and
> > I'll
> > > > >>>>>>> complement the FLIP with the details on
> > > > >>>>>>> the process of the final checkpoint / savepoint.
> > > > >>>>>>> Best,
> > > > >>>>>>> Yun
> > > > >>>>>>>
> > > ------------------------------------------------------------------
> > > > >>>>>>> From:Till Rohrmann
> > > > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > > > >>>>>>> To:dev
> > > > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> > > Tasks
> > > > >>>>>>> Finished
> > > > >>>>>>> Hi everyone,
> > > > >>>>>>> I am a bit late to the voting party but let me ask three
> > > questions:
> > > >
> > > > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> > > thread
> > > > >>>> if we
> > > > >>>>
> > > >
> > > > >>>>>>> cannot guarantee that all tasks are still running when
> > > triggering the
> > > > >>>>>>> checkpoint? Couldn't we do the computation in a different
> > thread
> > > in
> > > > >>>> order
> > > > >>>>
> > > > >>>>>>> to relieve the main thread a bit.
> > > > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> > > seems
> > > > >>>> to go
> > > > >>>>
> > > > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> > > more
> > > > >>>>>>> efficient to maintain the set of current tasks to trigger and
> > > check
> > > > >>>>>>> whether
> > > > >>>>>>> anything has changed and if so check the succeeding tasks
> until
> > > we
> > > > >>>> have
> > > > >>>>
> > > > >>>>>>> found the current checkpoint trigger frontier?
> > > > >>>>>>> 3) When are we going to send the endOfInput events to a
> > > downstream
> > > > >>>> task?
> > > > >>>>
> > > > >>>>>>> If
> > > >
> > > > >>>>>>> this happens after we call finish on the upstream operator
> but
> > > before
> > > > >>>>>>> snapshotState then it would be possible to shut down the
> whole
> > > > >>>> topology
> > > > >>>>
> > > >
> > > > >>>>>>> with a single final checkpoint. I think this part could
> benefit
> > > from
> > > > >>>> a bit
> > > > >>>>
> > > > >>>>>>> more detailed description in the FLIP.
> > > > >>>>>>> Cheers,
> > > > >>>>>>> Till
> > > > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > > > >>>>>>> wrote:
> > > > >>>>>>>> Hi there,
> > > > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing
> > the
> > > > >>>> vote
> > > > >>>>
> > > > >>>>>>> now.
> > > > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1
> > votes:
> > > > >>>>>>>> - Dawid Wysakowicz (binding)
> > > > >>>>>>>> - Piotr Nowojski(binding)
> > > > >>>>>>>> - Jiangang Liu (binding)
> > > > >>>>>>>> - Arvid Heise (binding)
> > > > >>>>>>>> - Jing Zhang (binding)
> > > > >>>>>>>> - Leonard Xu (non-binding)
> > > > >>>>>>>> - Guowei Ma (binding)
> > > > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147
> is
> > > > >>>> accepted.
> > > > >>>>
> > > > >>>>>>>> Very thanks everyone!
> > > > >>>>>>>> Best,
> > > > >>>>>>>> Yun
> > > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > > >>>>
> > > > >>>>
> > > >
> > > >
> > >
> >
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Piotr Nowojski <pn...@apache.org>.
Hi Steven,
> I probably missed sth here. isn't this the case today already? Why is it
a concern for the proposed change?
The problem is with the newly added `finish()` method and the already
existing `endInput()` call. Currently on master there are no issues,
because we are not checkpointing any operators after some operators have
finished. The purpose of this FLIP-147 is to exactly enable this and this
opens a new problem described by Dawid.
To paraphrase and to give a concrete example. Assume we have an operator
with parallelism of two. Subtask 0 and subtask 1.
1. Subtask 0 received both `endInput()` and `finish()`, but subtask 1
hasn't (yet).
2. Checkpoint 42 is triggered, and it completes.
3. Job fails and is restarted, but at the same time it's rescaled. User has
chosen to scale down his operator down to 1.
Now we have a pickle. We don't know if `notifyCheckpointComplete(42)` has
been processed or not, so while recovering to checkpoint 42, we have to
recover both finished subtask 0 (#1) state and not yet finished subtask 1's
(#1). But at the same time they are scaled down, so we only have a single
subtask 0 (#2) that has a combined state from both of the previous
instances. The potentially confusing issue is that the state from subtask 0
(#1) was checkpointed AFTER `endInput()` and `finish()` calls, but it's
recovered to an operator that has still some records to process. In step 1.
an user for example could store on the operator's state a bit of
information "end input has been already called!", that after recovery would
no longer be true.
Hence the question about `finish()` and `endInput()` semantics. Should it
be tied down to state, or just to an operator instance/execution attempt?
Piotrek
śr., 21 lip 2021 o 19:00 Steven Wu <st...@gmail.com> napisał(a):
> > if a failure happens after sequence of finish() -> snapshotState(), but
> before notifyCheckpointComplete(), we will restore such a state and we
> might end up sending some more records to such an operator.
>
> I probably missed sth here. isn't this the case today already? Why is it a
> concern for the proposed change?
>
> On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org>
> wrote:
>
> > Hi Dawid,
> >
> > Thanks for writing down those concerns.
> >
> > I think the first issue boils down what should be the contract of
> lifecycle
> > methods like open(), close(), initializeState() etc and especially the
> new
> > additions like finish() and endInput(). And what should be their relation
> > with the operator state (regardless of it's type keyed, non-keyed, union,
> > ...). Should those methods be tied to state or not? After thinking about
> it
> > for a while (and discussing it offline with Dawid), I think the answer
> > might be no, they shouldn't. I mean maybe we should just openly say that
> > all of those methods relate to this single particular instance and
> > execution of the operator. And if a job is recovered/rescaled, we would
> be
> > allowed to freely resume consumption, ignoring a fact that maybe some
> parts
> > of the state have previously seen `endInput()`. Why?
> >
> > 0. Yes, it might be confusing. Especially with `endInput()`. We call
> > `endInput()`, we store something in a state and later after recovery
> > combined with rescaling that state can see more records? Indeed weird,
> > 1. I haven't come up yet with a counterexample that would break and make
> > impossible to implement a real life use case. Theoretically yes, the user
> > can store `endInput()` on state, and after rescaling this state would be
> > inconsistent with what is actually happening with the operator, but I
> > haven't found a use case that would break because of that.
> > 2. Otherwise, implementation would be very difficult.
> > 3. It's difficult to access keyed state from within
> `endInput()`/`finish()`
> > calls, as they do not have key context.
> > 4. After all, openly defining `endInput()` and `finish()` to be tied with
> > it's operator execution instance lifecycle is not that strange and quite
> > simple to explain. Sure, it can lead to a bit of confusion (0.), but that
> > doesn't sound that bad in comparison with the alternatives that I'm aware
> > of. Also currently methods like `open()` and `close()` are also tied to
> the
> > operator execution instance, not to the state. Operators can be opened
> and
> > closed multiple times, it doesn't mean that the state is lost after
> closing
> > an operator.
> >
> > For the UnionListState problem I have posted my proposal in the ticket
> [1],
> > so maybe let's move that particular discussion there?
> >
> > Piotrek
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-21080
> >
> > śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> > napisał(a):
> >
> > > Hey all,
> > >
> > > To make the issues that were found transparent to the community, I want
> > to
> > > post an update:
> > >
> > > *1. Committing side-effects*
> > > We do want to make sure that all side effects are committed before
> > > bringing tasks down. Side effects are committed when calling
> > > notifyCheckpointComplete. For the final checkpoint we introduced the
> > method
> > > finish(). This notifies the operator that we have consumed all incoming
> > > records and we are preparing to close the Task. In turn we should flush
> > any
> > > pending buffered records and prepare to commit last transactions. The
> > goal
> > > is that after a successful sequence of finish() -> snapshotState() ->
> > > notifyCheckpointComplete(), the remaining state can be considered
> > > empty/finished and may be discarded.
> > >
> > > *Failure before notifyCheckpointComplete()*
> > >
> > > The question is what is the contract of the endInput()/finish() methods
> > > and how do calling these methods affect the operators keyed, non-keyed
> > > state and external state. Is it allowed to restore state snapshot taken
> > > after calling endInput()/finish() and process more records? Or do we
> > assume
> > > that after a restore from such a state taken after finish() we should
> not
> > > call any of the lifecycle methods or at least make sure those methods
> do
> > > not emit records/interact with mailbox etc.
> > >
> > > Currently it is possible that if a failure happens after sequence of
> > > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> > will
> > > restore such a state and we might end up sending some more records to
> > such
> > > an operator. It is possible if we rescale and this state is merged
> with a
> > > state of a subtask that has not called finish() yet. It can also happen
> > if
> > > we rescale the upstream operator and the subtask of interest becomes
> > > connected to a newly added non finished subtask.
> > >
> > > *Snapshotting StreamTasks that finish() has been called*
> > >
> > >
> > > We thought about putting a flag into the snapshot of a subtask produced
> > > after the finish() method. This would make it possible to skip
> execution
> > of
> > > certain lifecycle methods. Unfortunately this creates problems for
> > > rescaling. How do we deal with a situation that subtask states with
> both
> > > the feature flag set and unset end up in a single StreamTask.
> Additional
> > > problem is that we merge those states into a single
> OperatorSubtaskState
> > on
> > > CheckpointCoordinator.
> > >
> > > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > > checkpoint*
> > >
> > > We need to wait for a checkpoint to complete, that started after the
> > > finish() method. However, we support concurrent checkpoints therefore,
> > > there might be later checkpoints that completed, but the notification
> has
> > > not arrived. We must make sure those checkpoints do not leave lingering
> > > external resources.
> > >
> > > *Checkpointing from a single subtask / UnionListState case*
> > > There are operators that checkpoint from a single subtask only. Usually
> > > from the subtask index=0. If we let those subtasks finish, subsequent
> > > checkpoints will miss this information.
> > > Esp. Legacy sources problem:
> > > https://issues.apache.org/jira/browse/FLINK-21080
> > >
> > > Best,
> > >
> > > Dawid
> > > On 19/07/2021 15:10, Yun Gao wrote:
> > >
> > > Hi Till, Dawid,
> > >
> > > Very thanks for the comments and discussion! Glad that it seems we have
> > > come to a convergence, and I also agree with that we could not include
> > the
> > > optimization in the first version.
> > >
> > > Best
> > > Yun
> > >
> > > ------------------------------------------------------------------
> > > From:Dawid Wysakowicz <dw...@apache.org> <dwysakowicz@apache.org
> >
> > > Send Time:2021 Jul. 19 (Mon.) 17:51
> > > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > > <tr...@apache.org> <tr...@apache.org>
> > > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > > Finished
> > >
> > > Small correction. I meant we need to adjust the EndOfInputEvent of
> > course.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > > Hey Till,
> > > >
> > > > Yes, you're right we will have to adjust the current state of
> > > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > > we're discussing here. We are aware of that.
> > > >
> > > > As for the MAX_WATERMARK vs finish(). My take is that we should
> always
> > > > emit MAX_WATERMARK before calling finish() on an operator. At the
> same
> > > > time finish() should not leave behind anything in state, as the
> > > > intention is that we never restore from the taken
> savepoint/checkpoint
> > > > (savepoint w drain or bounded data consumed).
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > > >> Hi Yun and Dawid,
> > > >>
> > >
> > > >> Thanks for your comments. I do agree with your comments that
> finish()
> > can
> > > >> do more than MAX_WATERMARK. I guess we should then explain how
> > > >> MAX_WATERMARK and finish() play together and what kind of
> > > >> order guarantees we provide.
> > > >>
> > >
> > > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether
> it
> > would
> > >
> > > >> work in its current state because we send this event when the Task
> is
> > about
> > > >> to shut down if I am not mistaken. What we want to have is to bring
> > the
> > >
> > > >> StreamTasks into a state so that they shut down on the next
> > checkpoint. For
> > > >> this we need to keep the StreamTask running. In general, I am a fan
> of
> > > >> making things explicit if possible. I think this helps maintenance
> and
> > >
> > > >> evolvability of code. That's why I think sending an EndOfInputEvent
> > which
> > > >> is a StreamTask level event and which says that there won't be any
> > other
> > > >> records coming only control events could make sense.
> > > >>
> > > >> I would leave the proposed optimization out of the first version. We
> > can
> > > >> still add it at a later point in time.
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > > <dw...@apache.org> <dw...@apache.org>
> > > >> wrote:
> > > >>
> > >
> > > >>> Personally I don't find this optimization important and I'd rather
> > leave
> > >
> > > >>> it out not to complicate the codebase further. I doubt we save much
> > there.
> > > >>> I don't have a strong opinion though.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Dawid
> > > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> Very thanks Dawid for the thoughts!
> > > >>>
> > > >>> Currently I also do not have different opinions regarding this
> part.
> > > >>> But I have one more issue to confirm: during the previous
> discussion
> > we
> > > >>> have discussed that for the final checkpoint case, we might have an
> > > >>> optmization
> > >
> > > >>> that if a task do not have operators using 2-pc, we might skip
> > waiting for
> > > >>> the
> > >
> > > >>> final checkpoint (but we could not skip the savepoint). To allow
> > users to
> > > >>> express
> > > >>> the logic, we have proposed to add one more method to
> StreamOperator
> > &
> > > >>> CheckpointListener:
> > > >>>
> > > >>> interface StreamOperator {
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> interface CheckpointListener {
> > > >>>
> > > >>> default boolean requiresFinalCheckpoint() {
> > > >>> return true;
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> class AbstractUdfStreamOperator {
> > > >>>
> > > >>> @Override
> > > >>> boolean requiresFinalCheckpoint() {
> > > >>> return userFunction instanceof CheckpointListener &&
> > >
> > > >>> ((CheckpointListener)
> > userFunction).requiresFinalCheckpoint();
> > > >>> }
> > > >>> }
> > > >>>
> > > >>>
> > > >>> I think we should still keep the change ?
> > > >>>
> > > >>> Best,
> > > >>> Yun
> > > >>>
> > > >>> ------------------Original Mail ------------------
> > > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > > <dw...@apache.org>
> > > >>> <dw...@apache.org> <dw...@apache.org>
> > > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <
> dev@flink.apache.org>
> > > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > > >>> Finished
> > > >>>
> > > >>>> I think we're all really close to the same solution.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time
> based
> > > >>>>
> > >
> > > >>>> buffering, but it does not solve flushing other operations such as
> > e.g.
> > > >>>>
> > > >>>> count windows or batching requests in Sinks. I'd prefer to treat
> the
> > > >>>>
> > > >>>> finish() as a message for Operator to "flush all records". The
> > > >>>>
> > >
> > > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> > imo. I
> > > >>>>
> > > >>>> don't think operators need to get a signal "stop-processing" if
> they
> > > >>>>
> > > >>>> don't need to flush records. The "WHEN" records are emitted,
> should
> > be
> > > >>>>
> > > >>>> in control of the StreamTask, by firing timers or by processing a
> > next
> > > >>>>
> > > >>>> record from upstream.
> > > >>>>
> > > >>>>
> > > >>>>
> > >
> > > >>>> The only difference of my previous proposal compared to Yun's is
> > that I
> > > >>>>
> > > >>>> did not want to send the EndOfUserRecords event in case of stop
> w/o
> > > >>>>
> > > >>>> drain. My thinking was that we could directly go from RUNNING to
> > > >>>>
> > > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> > > >>>>
> > > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> > > >>>>
> > >
> > > >>>> timers and processing events (without calling finish() on
> > Operator). In
> > > >>>>
> > > >>>> my initial suggestion I though we don't care about some events
> > > >>>>
> > > >>>> potentially being emitted after the savepoint was taken, as they
> > would
> > > >>>>
> > >
> > > >>>> anyway belong to the next after FINAL, which would be discarded. I
> > think
> > > >>>>
> > > >>>> though the proposal to suspend records processing and timers is a
> > > >>>>
> > >
> > > >>>> sensible thing to do and would go with the version that Yun put
> > into the
> > > >>>>
> > > >>>> FLIP Wiki.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> What do you think Till?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Dawid
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > > >>>>
> > > >>>>> Hi Till, Piotr
> > > >>>>> Very thanks for the comments!
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >
> > > >>>>> I also agree with Piotr that currently they are independent
> > mechanisms,
> > > >>>> and they are basically the same
> > > >>>>
> > > >>>>> for the event time.
> > > >>>>> For more details, first there are some difference among the three
> > > >>>> scenarios regarding the finish:
> > > >>>>
> > >
> > > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> > not be
> > > >>>> expected to be restarted,
> > > >>>>
> > > >>>>> and for stop-with-savepoint the job would be expected restart
> > later.
> > > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> > would
> > > >>>> emit MAX_WATERMARK before the
> > > >>>>
> > >
> > > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> > endOfInput /
> > > >>>> finish() should also only be called
> > > >>>>
> > > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> > always
> > > >>>> occurs at the same time. After the change,
> > > >>>>
> > > >>>>> we could emit MAX_WATERMARK before endOfInput event for the
> finish
> > /
> > > >>>> stop-with-savepoint --drain cases.
> > > >>>>
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > a
> > > >>>>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>>> finish, for example?
> > > >>>>> As discussed above for stop-with-savepoint, we would always keep
> > the
> > > >>>> window as is, and restore them after restart.
> > > >>>>
> > > >>>>> Then for the finish / stop-with-savepoint --drain, I think
> perhaps
> > it
> > > >>>> depends on the Triggers. For
> > > >>>>
> > >
> > > >>>>> event-time triggers / process time triggers, it would be
> > reasonable to
> > > >>>> flush all the windows since logically
> > > >>>>
> > >
> > > >>>>> the time would always elapse and the window would always get
> > triggered
> > > >>>> in a logical future. But for triggers
> > > >>>>
> > >
> > > >>>>> like CountTrigger, no matter how much time pass logically, the
> > windows
> > > >>>> would not trigger, thus we may not
> > > >>>>
> > > >>>>> flush these windows. If there are requirements we may provide
> > > >>>> additional triggers.
> > > >>>>
> > >
> > > >>>>>> It's a bit messy and I'm not sure if this should be strengthened
> > out?
> > > >>>> Each one of those has a little bit different semantic/meaning,
> > > >>>>
> > >
> > > >>>>>> but at the same time they are very similar. For single input
> > operators
> > > >>>> `endInput()` and `finish()` are actually the very same thing.
> > > >>>>
> > >
> > > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> > at the
> > > >>>> same time, and for single input operators `endInput()` and
> > `finish()`
> > > >>>>
> > >
> > > >>>>> are indeed the same thing. During the last discussion we ever
> > mentioned
> > > >>>> this issue and at then we thought that we might deprecate
> > `endInput()`
> > > >>>>
> > > >>>>> in the future, then we would only have endInput(int input) and
> > > >>>> finish().
> > > >>>>
> > > >>>>> Best,
> > > >>>>> Yun
> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > > >>>>>
> ------------------------------------------------------------------
> > > >>>>> From:Piotr Nowojski
> > > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > > >>>>> To:dev
> > > >>>>> Cc:Yun Gao
> > > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > > >>>> Finished
> > > >>>>
> > > >>>>> Hi Till,
> > > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would
> > a>
> > > >>>> WindowOperator close all windows and emit the results upon calling
> > > >>>>
> > > >>>>>> finish, for example?
> > > >>>>> 1) currently they are independent but parallel mechanisms. With
> > event
> > > >>>> time, they are basically the same.
> > > >>>>
> > > >>>>> 2) it probably should for the sake of processing time windows.
> > > >>>>> Here you are touching the bit of the current design that I like
> the
> > >
> > > >>>> least. We basically have now three different ways of conveying
> very
> > similar
> > > >>>> things:
> > > >>>>
> > > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator
> (what
> > > >>>> about processing time?)
> > > >>>>
> > > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush
> it's
> > > >>>> internal state
> > > >>>>
> > > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> > out?
> > >
> > > >>>> Each one of those has a little bit different semantic/meaning, but
> > at the
> > >
> > > >>>> same time they are very similar. For single input operators
> > `endInput()`
> > > >>>> and `finish()` are actually the very same thing.
> > > >>>>
> > > >>>>> Piotrek
> > > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > > >>>>> Thanks for updating the FLIP. Based on the new section about
> > > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > > >>>>> 2) StreamOperator.finish says to flush all buffered events.
> Would a
> > > >>>>> WindowOperator close all windows and emit the results upon
> calling
> > > >>>>> finish, for example?
> > > >>>>> Cheers,
> > > >>>>> Till
> > > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > > >>>> problem.
> > > >>>>
> > > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > > >>>>>> Cheers,
> > > >>>>>> Till
> > > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > > >>>>>> wrote:
> > > >>>>>>> Hi Till,
> > > >>>>>>> Very thanks for the review and comments!
> > > >>>>>>> 1) First I think in fact we could be able to do the computation
> > > >>>> outside
> > > >>>>
> > > >>>>>>> of the main thread,
> > > >>>>>>> and the current implementation mainly due to the computation is
> > in
> > > >>>>>>> general fast and we
> > > >>>>>>> initially want to have a simplified first version.
> > >
> > > >>>>>>> The main requirement here is to have a constant view of the
> > state of
> > > >>>> the
> > > >>>>
> > > >>>>>>> tasks, otherwise
> > > >>>>>>> for example if we have A -> B, if A is running when we check if
> > we
> > > >>>> need
> > > >>>>
> > > >>>>>>> to trigger A, we will
> > > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> > check
> > > >>>> B, we
> > > >>>>
> > > >>>>>>> will also mark B as
> > >
> > > >>>>>>> have to trigger, then B will receive both rpc trigger and
> > checkpoint
> > > >>>>>>> barrier, which would break
> > > >>>>>>> some assumption on the task side and complicate the
> > implementation.
> > >
> > > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> > of the
> > > >>>>>>> tasks' state and then do the
> > >
> > > >>>>>>> computation, both the two step do not need to be in the main
> > thread.
> > >
> > > >>>>>>> 2) For the computation logic, in fact currently we benefit a
> lot
> > from
> > > >>>>>>> some shortcuts on all-to-all
> > >
> > > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> > could do
> > > >>>>>>> checks on the job vertex level
> > > >>>>>>> first and skip some job vertices as a whole. With this
> > optimization
> > > >>>> we
> > > >>>>
> > > >>>>>>> have a O(V) algorithm, and the
> > > >>>>>>> current running time of the worst case for a job with 320,000
> > tasks
> > > >>>> is
> > > >>>>
> > > >>>>>>> less than 100ms. For
> > > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > > >>>>>>> If we do the computation based on the last triggered tasks, we
> > may
> > > >>>> not
> > > >>>>
> > > >>>>>>> easily encode this information
> > >
> > > >>>>>>> into the shortcuts on the job vertex level. And since the time
> > seems
> > > >>>> to
> > > >>>>
> > > >>>>>>> be short, perhaps it is enough
> > > >>>>>>> to do re-computation from the scratch in consideration of the
> > > >>>> tradeoff
> > > >>>>
> > > >>>>>>> between the performance and
> > > >>>>>>> the complexity ?
> > > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > > >>>> finish()
> > > >>>>
> > > >>>>>>> method and before the last
> > >
> > > >>>>>>> snapshotState() method so that we could shut down the whole
> > topology
> > > >>>> with
> > > >>>>
> > > >>>>>>> a single final checkpoint.
> > > >>>>>>> Very sorry for not include enough details for this part and
> I'll
> > > >>>>>>> complement the FLIP with the details on
> > > >>>>>>> the process of the final checkpoint / savepoint.
> > > >>>>>>> Best,
> > > >>>>>>> Yun
> > > >>>>>>>
> > ------------------------------------------------------------------
> > > >>>>>>> From:Till Rohrmann
> > > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > > >>>>>>> To:dev
> > > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> > Tasks
> > > >>>>>>> Finished
> > > >>>>>>> Hi everyone,
> > > >>>>>>> I am a bit late to the voting party but let me ask three
> > questions:
> > >
> > > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> > thread
> > > >>>> if we
> > > >>>>
> > >
> > > >>>>>>> cannot guarantee that all tasks are still running when
> > triggering the
> > > >>>>>>> checkpoint? Couldn't we do the computation in a different
> thread
> > in
> > > >>>> order
> > > >>>>
> > > >>>>>>> to relieve the main thread a bit.
> > > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> > seems
> > > >>>> to go
> > > >>>>
> > > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> > more
> > > >>>>>>> efficient to maintain the set of current tasks to trigger and
> > check
> > > >>>>>>> whether
> > > >>>>>>> anything has changed and if so check the succeeding tasks until
> > we
> > > >>>> have
> > > >>>>
> > > >>>>>>> found the current checkpoint trigger frontier?
> > > >>>>>>> 3) When are we going to send the endOfInput events to a
> > downstream
> > > >>>> task?
> > > >>>>
> > > >>>>>>> If
> > >
> > > >>>>>>> this happens after we call finish on the upstream operator but
> > before
> > > >>>>>>> snapshotState then it would be possible to shut down the whole
> > > >>>> topology
> > > >>>>
> > >
> > > >>>>>>> with a single final checkpoint. I think this part could benefit
> > from
> > > >>>> a bit
> > > >>>>
> > > >>>>>>> more detailed description in the FLIP.
> > > >>>>>>> Cheers,
> > > >>>>>>> Till
> > > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > > >>>>>>> wrote:
> > > >>>>>>>> Hi there,
> > > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing
> the
> > > >>>> vote
> > > >>>>
> > > >>>>>>> now.
> > > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1
> votes:
> > > >>>>>>>> - Dawid Wysakowicz (binding)
> > > >>>>>>>> - Piotr Nowojski(binding)
> > > >>>>>>>> - Jiangang Liu (binding)
> > > >>>>>>>> - Arvid Heise (binding)
> > > >>>>>>>> - Jing Zhang (binding)
> > > >>>>>>>> - Leonard Xu (non-binding)
> > > >>>>>>>> - Guowei Ma (binding)
> > > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > > >>>> accepted.
> > > >>>>
> > > >>>>>>>> Very thanks everyone!
> > > >>>>>>>> Best,
> > > >>>>>>>> Yun
> > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > >>>>
> > > >>>>
> > >
> > >
> >
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Steven Wu <st...@gmail.com>.
> if a failure happens after sequence of finish() -> snapshotState(), but
before notifyCheckpointComplete(), we will restore such a state and we
might end up sending some more records to such an operator.
I probably missed sth here. isn't this the case today already? Why is it a
concern for the proposed change?
On Wed, Jul 21, 2021 at 4:39 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi Dawid,
>
> Thanks for writing down those concerns.
>
> I think the first issue boils down what should be the contract of lifecycle
> methods like open(), close(), initializeState() etc and especially the new
> additions like finish() and endInput(). And what should be their relation
> with the operator state (regardless of it's type keyed, non-keyed, union,
> ...). Should those methods be tied to state or not? After thinking about it
> for a while (and discussing it offline with Dawid), I think the answer
> might be no, they shouldn't. I mean maybe we should just openly say that
> all of those methods relate to this single particular instance and
> execution of the operator. And if a job is recovered/rescaled, we would be
> allowed to freely resume consumption, ignoring a fact that maybe some parts
> of the state have previously seen `endInput()`. Why?
>
> 0. Yes, it might be confusing. Especially with `endInput()`. We call
> `endInput()`, we store something in a state and later after recovery
> combined with rescaling that state can see more records? Indeed weird,
> 1. I haven't come up yet with a counterexample that would break and make
> impossible to implement a real life use case. Theoretically yes, the user
> can store `endInput()` on state, and after rescaling this state would be
> inconsistent with what is actually happening with the operator, but I
> haven't found a use case that would break because of that.
> 2. Otherwise, implementation would be very difficult.
> 3. It's difficult to access keyed state from within `endInput()`/`finish()`
> calls, as they do not have key context.
> 4. After all, openly defining `endInput()` and `finish()` to be tied with
> it's operator execution instance lifecycle is not that strange and quite
> simple to explain. Sure, it can lead to a bit of confusion (0.), but that
> doesn't sound that bad in comparison with the alternatives that I'm aware
> of. Also currently methods like `open()` and `close()` are also tied to the
> operator execution instance, not to the state. Operators can be opened and
> closed multiple times, it doesn't mean that the state is lost after closing
> an operator.
>
> For the UnionListState problem I have posted my proposal in the ticket [1],
> so maybe let's move that particular discussion there?
>
> Piotrek
>
> [1] https://issues.apache.org/jira/browse/FLINK-21080
>
> śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
> napisał(a):
>
> > Hey all,
> >
> > To make the issues that were found transparent to the community, I want
> to
> > post an update:
> >
> > *1. Committing side-effects*
> > We do want to make sure that all side effects are committed before
> > bringing tasks down. Side effects are committed when calling
> > notifyCheckpointComplete. For the final checkpoint we introduced the
> method
> > finish(). This notifies the operator that we have consumed all incoming
> > records and we are preparing to close the Task. In turn we should flush
> any
> > pending buffered records and prepare to commit last transactions. The
> goal
> > is that after a successful sequence of finish() -> snapshotState() ->
> > notifyCheckpointComplete(), the remaining state can be considered
> > empty/finished and may be discarded.
> >
> > *Failure before notifyCheckpointComplete()*
> >
> > The question is what is the contract of the endInput()/finish() methods
> > and how do calling these methods affect the operators keyed, non-keyed
> > state and external state. Is it allowed to restore state snapshot taken
> > after calling endInput()/finish() and process more records? Or do we
> assume
> > that after a restore from such a state taken after finish() we should not
> > call any of the lifecycle methods or at least make sure those methods do
> > not emit records/interact with mailbox etc.
> >
> > Currently it is possible that if a failure happens after sequence of
> > finish() -> snapshotState(), but before notifyCheckpointComplete(), we
> will
> > restore such a state and we might end up sending some more records to
> such
> > an operator. It is possible if we rescale and this state is merged with a
> > state of a subtask that has not called finish() yet. It can also happen
> if
> > we rescale the upstream operator and the subtask of interest becomes
> > connected to a newly added non finished subtask.
> >
> > *Snapshotting StreamTasks that finish() has been called*
> >
> >
> > We thought about putting a flag into the snapshot of a subtask produced
> > after the finish() method. This would make it possible to skip execution
> of
> > certain lifecycle methods. Unfortunately this creates problems for
> > rescaling. How do we deal with a situation that subtask states with both
> > the feature flag set and unset end up in a single StreamTask. Additional
> > problem is that we merge those states into a single OperatorSubtaskState
> on
> > CheckpointCoordinator.
> >
> > *Finishing upon receiving notifyCheckpointComplete() of not the latest
> > checkpoint*
> >
> > We need to wait for a checkpoint to complete, that started after the
> > finish() method. However, we support concurrent checkpoints therefore,
> > there might be later checkpoints that completed, but the notification has
> > not arrived. We must make sure those checkpoints do not leave lingering
> > external resources.
> >
> > *Checkpointing from a single subtask / UnionListState case*
> > There are operators that checkpoint from a single subtask only. Usually
> > from the subtask index=0. If we let those subtasks finish, subsequent
> > checkpoints will miss this information.
> > Esp. Legacy sources problem:
> > https://issues.apache.org/jira/browse/FLINK-21080
> >
> > Best,
> >
> > Dawid
> > On 19/07/2021 15:10, Yun Gao wrote:
> >
> > Hi Till, Dawid,
> >
> > Very thanks for the comments and discussion! Glad that it seems we have
> > come to a convergence, and I also agree with that we could not include
> the
> > optimization in the first version.
> >
> > Best
> > Yun
> >
> > ------------------------------------------------------------------
> > From:Dawid Wysakowicz <dw...@apache.org> <dw...@apache.org>
> > Send Time:2021 Jul. 19 (Mon.) 17:51
> > To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> > <tr...@apache.org> <tr...@apache.org>
> > Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> > <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > Finished
> >
> > Small correction. I meant we need to adjust the EndOfInputEvent of
> course.
> >
> > Best,
> >
> > Dawid
> >
> > On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > > Hey Till,
> > >
> > > Yes, you're right we will have to adjust the current state of
> > > EndOfPartitionEvent and move the moment when we emit it to have what
> > > we're discussing here. We are aware of that.
> > >
> > > As for the MAX_WATERMARK vs finish(). My take is that we should always
> > > emit MAX_WATERMARK before calling finish() on an operator. At the same
> > > time finish() should not leave behind anything in state, as the
> > > intention is that we never restore from the taken savepoint/checkpoint
> > > (savepoint w drain or bounded data consumed).
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/07/2021 11:33, Till Rohrmann wrote:
> > >> Hi Yun and Dawid,
> > >>
> >
> > >> Thanks for your comments. I do agree with your comments that finish()
> can
> > >> do more than MAX_WATERMARK. I guess we should then explain how
> > >> MAX_WATERMARK and finish() play together and what kind of
> > >> order guarantees we provide.
> > >>
> >
> > >> Concerning the EndOfPartitionEvent, I am not entirely sure whether it
> would
> >
> > >> work in its current state because we send this event when the Task is
> about
> > >> to shut down if I am not mistaken. What we want to have is to bring
> the
> >
> > >> StreamTasks into a state so that they shut down on the next
> checkpoint. For
> > >> this we need to keep the StreamTask running. In general, I am a fan of
> > >> making things explicit if possible. I think this helps maintenance and
> >
> > >> evolvability of code. That's why I think sending an EndOfInputEvent
> which
> > >> is a StreamTask level event and which says that there won't be any
> other
> > >> records coming only control events could make sense.
> > >>
> > >> I would leave the proposed optimization out of the first version. We
> can
> > >> still add it at a later point in time.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> > <dw...@apache.org> <dw...@apache.org>
> > >> wrote:
> > >>
> >
> > >>> Personally I don't find this optimization important and I'd rather
> leave
> >
> > >>> it out not to complicate the codebase further. I doubt we save much
> there.
> > >>> I don't have a strong opinion though.
> > >>>
> > >>> Best,
> > >>>
> > >>> Dawid
> > >>> On 19/07/2021 10:31, Yun Gao wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> Very thanks Dawid for the thoughts!
> > >>>
> > >>> Currently I also do not have different opinions regarding this part.
> > >>> But I have one more issue to confirm: during the previous discussion
> we
> > >>> have discussed that for the final checkpoint case, we might have an
> > >>> optmization
> >
> > >>> that if a task do not have operators using 2-pc, we might skip
> waiting for
> > >>> the
> >
> > >>> final checkpoint (but we could not skip the savepoint). To allow
> users to
> > >>> express
> > >>> the logic, we have proposed to add one more method to StreamOperator
> &
> > >>> CheckpointListener:
> > >>>
> > >>> interface StreamOperator {
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> interface CheckpointListener {
> > >>>
> > >>> default boolean requiresFinalCheckpoint() {
> > >>> return true;
> > >>> }
> > >>> }
> > >>>
> > >>> class AbstractUdfStreamOperator {
> > >>>
> > >>> @Override
> > >>> boolean requiresFinalCheckpoint() {
> > >>> return userFunction instanceof CheckpointListener &&
> >
> > >>> ((CheckpointListener)
> userFunction).requiresFinalCheckpoint();
> > >>> }
> > >>> }
> > >>>
> > >>>
> > >>> I think we should still keep the change ?
> > >>>
> > >>> Best,
> > >>> Yun
> > >>>
> > >>> ------------------Original Mail ------------------
> > >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> > <dw...@apache.org>
> > >>> <dw...@apache.org> <dw...@apache.org>
> > >>> *Send Date:*Sun Jul 18 18:44:50 2021
> > >>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>
> > <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> > >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> > >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>> Finished
> > >>>
> > >>>> I think we're all really close to the same solution.
> > >>>>
> > >>>>
> > >>>>
> > >>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
> > >>>>
> >
> > >>>> buffering, but it does not solve flushing other operations such as
> e.g.
> > >>>>
> > >>>> count windows or batching requests in Sinks. I'd prefer to treat the
> > >>>>
> > >>>> finish() as a message for Operator to "flush all records". The
> > >>>>
> >
> > >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility
> imo. I
> > >>>>
> > >>>> don't think operators need to get a signal "stop-processing" if they
> > >>>>
> > >>>> don't need to flush records. The "WHEN" records are emitted, should
> be
> > >>>>
> > >>>> in control of the StreamTask, by firing timers or by processing a
> next
> > >>>>
> > >>>> record from upstream.
> > >>>>
> > >>>>
> > >>>>
> >
> > >>>> The only difference of my previous proposal compared to Yun's is
> that I
> > >>>>
> > >>>> did not want to send the EndOfUserRecords event in case of stop w/o
> > >>>>
> > >>>> drain. My thinking was that we could directly go from RUNNING to
> > >>>>
> > >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> > >>>>
> > >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> > >>>>
> >
> > >>>> timers and processing events (without calling finish() on
> Operator). In
> > >>>>
> > >>>> my initial suggestion I though we don't care about some events
> > >>>>
> > >>>> potentially being emitted after the savepoint was taken, as they
> would
> > >>>>
> >
> > >>>> anyway belong to the next after FINAL, which would be discarded. I
> think
> > >>>>
> > >>>> though the proposal to suspend records processing and timers is a
> > >>>>
> >
> > >>>> sensible thing to do and would go with the version that Yun put
> into the
> > >>>>
> > >>>> FLIP Wiki.
> > >>>>
> > >>>>
> > >>>>
> > >>>> What do you think Till?
> > >>>>
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>>
> > >>>>
> > >>>> Dawid
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 16/07/2021 10:03, Yun Gao wrote:
> > >>>>
> > >>>>> Hi Till, Piotr
> > >>>>> Very thanks for the comments!
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >
> > >>>>> I also agree with Piotr that currently they are independent
> mechanisms,
> > >>>> and they are basically the same
> > >>>>
> > >>>>> for the event time.
> > >>>>> For more details, first there are some difference among the three
> > >>>> scenarios regarding the finish:
> > >>>>
> >
> > >>>>> For normal finish and stop-with-savepoint --drain, the job would
> not be
> > >>>> expected to be restarted,
> > >>>>
> > >>>>> and for stop-with-savepoint the job would be expected restart
> later.
> > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> would
> > >>>> emit MAX_WATERMARK before the
> > >>>>
> >
> > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> endOfInput /
> > >>>> finish() should also only be called
> > >>>>
> > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> always
> > >>>> occurs at the same time. After the change,
> > >>>>
> > >>>>> we could emit MAX_WATERMARK before endOfInput event for the finish
> /
> > >>>> stop-with-savepoint --drain cases.
> > >>>>
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a
> > >>>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>>> finish, for example?
> > >>>>> As discussed above for stop-with-savepoint, we would always keep
> the
> > >>>> window as is, and restore them after restart.
> > >>>>
> > >>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps
> it
> > >>>> depends on the Triggers. For
> > >>>>
> >
> > >>>>> event-time triggers / process time triggers, it would be
> reasonable to
> > >>>> flush all the windows since logically
> > >>>>
> >
> > >>>>> the time would always elapse and the window would always get
> triggered
> > >>>> in a logical future. But for triggers
> > >>>>
> >
> > >>>>> like CountTrigger, no matter how much time pass logically, the
> windows
> > >>>> would not trigger, thus we may not
> > >>>>
> > >>>>> flush these windows. If there are requirements we may provide
> > >>>> additional triggers.
> > >>>>
> >
> > >>>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> > >>>> Each one of those has a little bit different semantic/meaning,
> > >>>>
> >
> > >>>>>> but at the same time they are very similar. For single input
> operators
> > >>>> `endInput()` and `finish()` are actually the very same thing.
> > >>>>
> >
> > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen
> at the
> > >>>> same time, and for single input operators `endInput()` and
> `finish()`
> > >>>>
> >
> > >>>>> are indeed the same thing. During the last discussion we ever
> mentioned
> > >>>> this issue and at then we thought that we might deprecate
> `endInput()`
> > >>>>
> > >>>>> in the future, then we would only have endInput(int input) and
> > >>>> finish().
> > >>>>
> > >>>>> Best,
> > >>>>> Yun
> > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > >>>>> ------------------------------------------------------------------
> > >>>>> From:Piotr Nowojski
> > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > >>>>> To:dev
> > >>>>> Cc:Yun Gao
> > >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> > >>>> Finished
> > >>>>
> > >>>>> Hi Till,
> > >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would
> a>
> > >>>> WindowOperator close all windows and emit the results upon calling
> > >>>>
> > >>>>>> finish, for example?
> > >>>>> 1) currently they are independent but parallel mechanisms. With
> event
> > >>>> time, they are basically the same.
> > >>>>
> > >>>>> 2) it probably should for the sake of processing time windows.
> > >>>>> Here you are touching the bit of the current design that I like the
> >
> > >>>> least. We basically have now three different ways of conveying very
> similar
> > >>>> things:
> > >>>>
> > >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
> > >>>> about processing time?)
> > >>>>
> > >>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
> > >>>> internal state
> > >>>>
> > >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> > >>>>> It's a bit messy and I'm not sure if this should be strengthened
> out?
> >
> > >>>> Each one of those has a little bit different semantic/meaning, but
> at the
> >
> > >>>> same time they are very similar. For single input operators
> `endInput()`
> > >>>> and `finish()` are actually the very same thing.
> > >>>>
> > >>>>> Piotrek
> > >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> > >>>>> Thanks for updating the FLIP. Based on the new section about
> > >>>>> stop-with-savepoint [--drain] I got two other questions:
> > >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> > >>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> > >>>>> WindowOperator close all windows and emit the results upon calling
> > >>>>> finish, for example?
> > >>>>> Cheers,
> > >>>>> Till
> > >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> > >>>>>> Thanks a lot for your answers and clarifications Yun.
> > >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> > >>>> problem.
> > >>>>
> > >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> > >>>>>> Cheers,
> > >>>>>> Till
> > >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > >>>>>> wrote:
> > >>>>>>> Hi Till,
> > >>>>>>> Very thanks for the review and comments!
> > >>>>>>> 1) First I think in fact we could be able to do the computation
> > >>>> outside
> > >>>>
> > >>>>>>> of the main thread,
> > >>>>>>> and the current implementation mainly due to the computation is
> in
> > >>>>>>> general fast and we
> > >>>>>>> initially want to have a simplified first version.
> >
> > >>>>>>> The main requirement here is to have a constant view of the
> state of
> > >>>> the
> > >>>>
> > >>>>>>> tasks, otherwise
> > >>>>>>> for example if we have A -> B, if A is running when we check if
> we
> > >>>> need
> > >>>>
> > >>>>>>> to trigger A, we will
> > >>>>>>> mark A as have to trigger, but if A gets to finished when we
> check
> > >>>> B, we
> > >>>>
> > >>>>>>> will also mark B as
> >
> > >>>>>>> have to trigger, then B will receive both rpc trigger and
> checkpoint
> > >>>>>>> barrier, which would break
> > >>>>>>> some assumption on the task side and complicate the
> implementation.
> >
> > >>>>>>> But to cope this issue, we in fact could first have a snapshot
> of the
> > >>>>>>> tasks' state and then do the
> >
> > >>>>>>> computation, both the two step do not need to be in the main
> thread.
> >
> > >>>>>>> 2) For the computation logic, in fact currently we benefit a lot
> from
> > >>>>>>> some shortcuts on all-to-all
> >
> > >>>>>>> edges and job vertex with all tasks running, these shortcuts
> could do
> > >>>>>>> checks on the job vertex level
> > >>>>>>> first and skip some job vertices as a whole. With this
> optimization
> > >>>> we
> > >>>>
> > >>>>>>> have a O(V) algorithm, and the
> > >>>>>>> current running time of the worst case for a job with 320,000
> tasks
> > >>>> is
> > >>>>
> > >>>>>>> less than 100ms. For
> > >>>>>>> daily graph sizes the time would be further reduced linearly.
> > >>>>>>> If we do the computation based on the last triggered tasks, we
> may
> > >>>> not
> > >>>>
> > >>>>>>> easily encode this information
> >
> > >>>>>>> into the shortcuts on the job vertex level. And since the time
> seems
> > >>>> to
> > >>>>
> > >>>>>>> be short, perhaps it is enough
> > >>>>>>> to do re-computation from the scratch in consideration of the
> > >>>> tradeoff
> > >>>>
> > >>>>>>> between the performance and
> > >>>>>>> the complexity ?
> > >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> > >>>> finish()
> > >>>>
> > >>>>>>> method and before the last
> >
> > >>>>>>> snapshotState() method so that we could shut down the whole
> topology
> > >>>> with
> > >>>>
> > >>>>>>> a single final checkpoint.
> > >>>>>>> Very sorry for not include enough details for this part and I'll
> > >>>>>>> complement the FLIP with the details on
> > >>>>>>> the process of the final checkpoint / savepoint.
> > >>>>>>> Best,
> > >>>>>>> Yun
> > >>>>>>>
> ------------------------------------------------------------------
> > >>>>>>> From:Till Rohrmann
> > >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> > >>>>>>> To:dev
> > >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks
> > >>>>>>> Finished
> > >>>>>>> Hi everyone,
> > >>>>>>> I am a bit late to the voting party but let me ask three
> questions:
> >
> > >>>>>>> 1) Why do we execute the trigger plan computation in the main
> thread
> > >>>> if we
> > >>>>
> >
> > >>>>>>> cannot guarantee that all tasks are still running when
> triggering the
> > >>>>>>> checkpoint? Couldn't we do the computation in a different thread
> in
> > >>>> order
> > >>>>
> > >>>>>>> to relieve the main thread a bit.
> > >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator
> seems
> > >>>> to go
> > >>>>
> > >>>>>>> over the whole topology for every calculation. Wouldn't it be
> more
> > >>>>>>> efficient to maintain the set of current tasks to trigger and
> check
> > >>>>>>> whether
> > >>>>>>> anything has changed and if so check the succeeding tasks until
> we
> > >>>> have
> > >>>>
> > >>>>>>> found the current checkpoint trigger frontier?
> > >>>>>>> 3) When are we going to send the endOfInput events to a
> downstream
> > >>>> task?
> > >>>>
> > >>>>>>> If
> >
> > >>>>>>> this happens after we call finish on the upstream operator but
> before
> > >>>>>>> snapshotState then it would be possible to shut down the whole
> > >>>> topology
> > >>>>
> >
> > >>>>>>> with a single final checkpoint. I think this part could benefit
> from
> > >>>> a bit
> > >>>>
> > >>>>>>> more detailed description in the FLIP.
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> > >>>>>>> wrote:
> > >>>>>>>> Hi there,
> > >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
> > >>>> vote
> > >>>>
> > >>>>>>> now.
> > >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> > >>>>>>>> - Dawid Wysakowicz (binding)
> > >>>>>>>> - Piotr Nowojski(binding)
> > >>>>>>>> - Jiangang Liu (binding)
> > >>>>>>>> - Arvid Heise (binding)
> > >>>>>>>> - Jing Zhang (binding)
> > >>>>>>>> - Leonard Xu (non-binding)
> > >>>>>>>> - Guowei Ma (binding)
> > >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> > >>>> accepted.
> > >>>>
> > >>>>>>>> Very thanks everyone!
> > >>>>>>>> Best,
> > >>>>>>>> Yun
> > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > >>>>
> > >>>>
> >
> >
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Piotr Nowojski <pn...@apache.org>.
Hi Dawid,
Thanks for writing down those concerns.
I think the first issue boils down what should be the contract of lifecycle
methods like open(), close(), initializeState() etc and especially the new
additions like finish() and endInput(). And what should be their relation
with the operator state (regardless of it's type keyed, non-keyed, union,
...). Should those methods be tied to state or not? After thinking about it
for a while (and discussing it offline with Dawid), I think the answer
might be no, they shouldn't. I mean maybe we should just openly say that
all of those methods relate to this single particular instance and
execution of the operator. And if a job is recovered/rescaled, we would be
allowed to freely resume consumption, ignoring a fact that maybe some parts
of the state have previously seen `endInput()`. Why?
0. Yes, it might be confusing. Especially with `endInput()`. We call
`endInput()`, we store something in a state and later after recovery
combined with rescaling that state can see more records? Indeed weird,
1. I haven't come up yet with a counterexample that would break and make
impossible to implement a real life use case. Theoretically yes, the user
can store `endInput()` on state, and after rescaling this state would be
inconsistent with what is actually happening with the operator, but I
haven't found a use case that would break because of that.
2. Otherwise, implementation would be very difficult.
3. It's difficult to access keyed state from within `endInput()`/`finish()`
calls, as they do not have key context.
4. After all, openly defining `endInput()` and `finish()` to be tied with
it's operator execution instance lifecycle is not that strange and quite
simple to explain. Sure, it can lead to a bit of confusion (0.), but that
doesn't sound that bad in comparison with the alternatives that I'm aware
of. Also currently methods like `open()` and `close()` are also tied to the
operator execution instance, not to the state. Operators can be opened and
closed multiple times, it doesn't mean that the state is lost after closing
an operator.
For the UnionListState problem I have posted my proposal in the ticket [1],
so maybe let's move that particular discussion there?
Piotrek
[1] https://issues.apache.org/jira/browse/FLINK-21080
śr., 21 lip 2021 o 12:39 Dawid Wysakowicz <dw...@apache.org>
napisał(a):
> Hey all,
>
> To make the issues that were found transparent to the community, I want to
> post an update:
>
> *1. Committing side-effects*
> We do want to make sure that all side effects are committed before
> bringing tasks down. Side effects are committed when calling
> notifyCheckpointComplete. For the final checkpoint we introduced the method
> finish(). This notifies the operator that we have consumed all incoming
> records and we are preparing to close the Task. In turn we should flush any
> pending buffered records and prepare to commit last transactions. The goal
> is that after a successful sequence of finish() -> snapshotState() ->
> notifyCheckpointComplete(), the remaining state can be considered
> empty/finished and may be discarded.
>
> *Failure before notifyCheckpointComplete()*
>
> The question is what is the contract of the endInput()/finish() methods
> and how do calling these methods affect the operators keyed, non-keyed
> state and external state. Is it allowed to restore state snapshot taken
> after calling endInput()/finish() and process more records? Or do we assume
> that after a restore from such a state taken after finish() we should not
> call any of the lifecycle methods or at least make sure those methods do
> not emit records/interact with mailbox etc.
>
> Currently it is possible that if a failure happens after sequence of
> finish() -> snapshotState(), but before notifyCheckpointComplete(), we will
> restore such a state and we might end up sending some more records to such
> an operator. It is possible if we rescale and this state is merged with a
> state of a subtask that has not called finish() yet. It can also happen if
> we rescale the upstream operator and the subtask of interest becomes
> connected to a newly added non finished subtask.
>
> *Snapshotting StreamTasks that finish() has been called*
>
>
> We thought about putting a flag into the snapshot of a subtask produced
> after the finish() method. This would make it possible to skip execution of
> certain lifecycle methods. Unfortunately this creates problems for
> rescaling. How do we deal with a situation that subtask states with both
> the feature flag set and unset end up in a single StreamTask. Additional
> problem is that we merge those states into a single OperatorSubtaskState on
> CheckpointCoordinator.
>
> *Finishing upon receiving notifyCheckpointComplete() of not the latest
> checkpoint*
>
> We need to wait for a checkpoint to complete, that started after the
> finish() method. However, we support concurrent checkpoints therefore,
> there might be later checkpoints that completed, but the notification has
> not arrived. We must make sure those checkpoints do not leave lingering
> external resources.
>
> *Checkpointing from a single subtask / UnionListState case*
> There are operators that checkpoint from a single subtask only. Usually
> from the subtask index=0. If we let those subtasks finish, subsequent
> checkpoints will miss this information.
> Esp. Legacy sources problem:
> https://issues.apache.org/jira/browse/FLINK-21080
>
> Best,
>
> Dawid
> On 19/07/2021 15:10, Yun Gao wrote:
>
> Hi Till, Dawid,
>
> Very thanks for the comments and discussion! Glad that it seems we have
> come to a convergence, and I also agree with that we could not include the
> optimization in the first version.
>
> Best
> Yun
>
> ------------------------------------------------------------------
> From:Dawid Wysakowicz <dw...@apache.org> <dw...@apache.org>
> Send Time:2021 Jul. 19 (Mon.) 17:51
> To:dev <de...@flink.apache.org> <de...@flink.apache.org>; Till Rohrmann
> <tr...@apache.org> <tr...@apache.org>
> Cc:Yun Gao <yu...@aliyun.com> <yu...@aliyun.com>; Yun Gao
> <yu...@aliyun.com.invalid> <yu...@aliyun.com.invalid>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> Small correction. I meant we need to adjust the EndOfInputEvent of course.
>
> Best,
>
> Dawid
>
> On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > Hey Till,
> >
> > Yes, you're right we will have to adjust the current state of
> > EndOfPartitionEvent and move the moment when we emit it to have what
> > we're discussing here. We are aware of that.
> >
> > As for the MAX_WATERMARK vs finish(). My take is that we should always
> > emit MAX_WATERMARK before calling finish() on an operator. At the same
> > time finish() should not leave behind anything in state, as the
> > intention is that we never restore from the taken savepoint/checkpoint
> > (savepoint w drain or bounded data consumed).
> >
> > Best,
> >
> > Dawid
> >
> > On 19/07/2021 11:33, Till Rohrmann wrote:
> >> Hi Yun and Dawid,
> >>
>
> >> Thanks for your comments. I do agree with your comments that finish() can
> >> do more than MAX_WATERMARK. I guess we should then explain how
> >> MAX_WATERMARK and finish() play together and what kind of
> >> order guarantees we provide.
> >>
>
> >> Concerning the EndOfPartitionEvent, I am not entirely sure whether it would
>
> >> work in its current state because we send this event when the Task is about
> >> to shut down if I am not mistaken. What we want to have is to bring the
>
> >> StreamTasks into a state so that they shut down on the next checkpoint. For
> >> this we need to keep the StreamTask running. In general, I am a fan of
> >> making things explicit if possible. I think this helps maintenance and
>
> >> evolvability of code. That's why I think sending an EndOfInputEvent which
> >> is a StreamTask level event and which says that there won't be any other
> >> records coming only control events could make sense.
> >>
> >> I would leave the proposed optimization out of the first version. We can
> >> still add it at a later point in time.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz
> <dw...@apache.org> <dw...@apache.org>
> >> wrote:
> >>
>
> >>> Personally I don't find this optimization important and I'd rather leave
>
> >>> it out not to complicate the codebase further. I doubt we save much there.
> >>> I don't have a strong opinion though.
> >>>
> >>> Best,
> >>>
> >>> Dawid
> >>> On 19/07/2021 10:31, Yun Gao wrote:
> >>>
> >>> Hi,
> >>>
> >>> Very thanks Dawid for the thoughts!
> >>>
> >>> Currently I also do not have different opinions regarding this part.
> >>> But I have one more issue to confirm: during the previous discussion we
> >>> have discussed that for the final checkpoint case, we might have an
> >>> optmization
>
> >>> that if a task do not have operators using 2-pc, we might skip waiting for
> >>> the
>
> >>> final checkpoint (but we could not skip the savepoint). To allow users to
> >>> express
> >>> the logic, we have proposed to add one more method to StreamOperator &
> >>> CheckpointListener:
> >>>
> >>> interface StreamOperator {
> >>> default boolean requiresFinalCheckpoint() {
> >>> return true;
> >>> }
> >>> }
> >>>
> >>> interface CheckpointListener {
> >>>
> >>> default boolean requiresFinalCheckpoint() {
> >>> return true;
> >>> }
> >>> }
> >>>
> >>> class AbstractUdfStreamOperator {
> >>>
> >>> @Override
> >>> boolean requiresFinalCheckpoint() {
> >>> return userFunction instanceof CheckpointListener &&
>
> >>> ((CheckpointListener) userFunction).requiresFinalCheckpoint();
> >>> }
> >>> }
> >>>
> >>>
> >>> I think we should still keep the change ?
> >>>
> >>> Best,
> >>> Yun
> >>>
> >>> ------------------Original Mail ------------------
> >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> <dw...@apache.org>
> >>> <dw...@apache.org> <dw...@apache.org>
> >>> *Send Date:*Sun Jul 18 18:44:50 2021
> >>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>
> <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >>> Finished
> >>>
> >>>> I think we're all really close to the same solution.
> >>>>
> >>>>
> >>>>
> >>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
> >>>>
>
> >>>> buffering, but it does not solve flushing other operations such as e.g.
> >>>>
> >>>> count windows or batching requests in Sinks. I'd prefer to treat the
> >>>>
> >>>> finish() as a message for Operator to "flush all records". The
> >>>>
>
> >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
> >>>>
> >>>> don't think operators need to get a signal "stop-processing" if they
> >>>>
> >>>> don't need to flush records. The "WHEN" records are emitted, should be
> >>>>
> >>>> in control of the StreamTask, by firing timers or by processing a next
> >>>>
> >>>> record from upstream.
> >>>>
> >>>>
> >>>>
>
> >>>> The only difference of my previous proposal compared to Yun's is that I
> >>>>
> >>>> did not want to send the EndOfUserRecords event in case of stop w/o
> >>>>
> >>>> drain. My thinking was that we could directly go from RUNNING to
> >>>>
> >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> >>>>
> >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> >>>>
>
> >>>> timers and processing events (without calling finish() on Operator). In
> >>>>
> >>>> my initial suggestion I though we don't care about some events
> >>>>
> >>>> potentially being emitted after the savepoint was taken, as they would
> >>>>
>
> >>>> anyway belong to the next after FINAL, which would be discarded. I think
> >>>>
> >>>> though the proposal to suspend records processing and timers is a
> >>>>
>
> >>>> sensible thing to do and would go with the version that Yun put into the
> >>>>
> >>>> FLIP Wiki.
> >>>>
> >>>>
> >>>>
> >>>> What do you think Till?
> >>>>
> >>>>
> >>>>
> >>>> Best,
> >>>>
> >>>>
> >>>>
> >>>> Dawid
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 16/07/2021 10:03, Yun Gao wrote:
> >>>>
> >>>>> Hi Till, Piotr
> >>>>> Very thanks for the comments!
> >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> >>>>> I also agree with Piotr that currently they are independent mechanisms,
> >>>> and they are basically the same
> >>>>
> >>>>> for the event time.
> >>>>> For more details, first there are some difference among the three
> >>>> scenarios regarding the finish:
> >>>>
>
> >>>>> For normal finish and stop-with-savepoint --drain, the job would not be
> >>>> expected to be restarted,
> >>>>
> >>>>> and for stop-with-savepoint the job would be expected restart later.
> >>>>> Then for finish / stop-with-savepoint --drain, currently Flink would
> >>>> emit MAX_WATERMARK before the
> >>>>
>
> >>>>> EndOfPartition. Besides, as we have discussed before [1], endOfInput /
> >>>> finish() should also only be called
> >>>>
> >>>>> for finish / stop-with-savepoint --drain. Thus currently they always
> >>>> occurs at the same time. After the change,
> >>>>
> >>>>> we could emit MAX_WATERMARK before endOfInput event for the finish /
> >>>> stop-with-savepoint --drain cases.
> >>>>
> >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> >>>>>> WindowOperator close all windows and emit the results upon calling
> >>>>>> finish, for example?
> >>>>> As discussed above for stop-with-savepoint, we would always keep the
> >>>> window as is, and restore them after restart.
> >>>>
> >>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps it
> >>>> depends on the Triggers. For
> >>>>
>
> >>>>> event-time triggers / process time triggers, it would be reasonable to
> >>>> flush all the windows since logically
> >>>>
>
> >>>>> the time would always elapse and the window would always get triggered
> >>>> in a logical future. But for triggers
> >>>>
>
> >>>>> like CountTrigger, no matter how much time pass logically, the windows
> >>>> would not trigger, thus we may not
> >>>>
> >>>>> flush these windows. If there are requirements we may provide
> >>>> additional triggers.
> >>>>
>
> >>>>>> It's a bit messy and I'm not sure if this should be strengthened out?
> >>>> Each one of those has a little bit different semantic/meaning,
> >>>>
>
> >>>>>> but at the same time they are very similar. For single input operators
> >>>> `endInput()` and `finish()` are actually the very same thing.
> >>>>
>
> >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen at the
> >>>> same time, and for single input operators `endInput()` and `finish()`
> >>>>
>
> >>>>> are indeed the same thing. During the last discussion we ever mentioned
> >>>> this issue and at then we thought that we might deprecate `endInput()`
> >>>>
> >>>>> in the future, then we would only have endInput(int input) and
> >>>> finish().
> >>>>
> >>>>> Best,
> >>>>> Yun
> >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> >>>>> ------------------------------------------------------------------
> >>>>> From:Piotr Nowojski
> >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> >>>>> To:dev
> >>>>> Cc:Yun Gao
> >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >>>> Finished
> >>>>
> >>>>> Hi Till,
> >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a>
> >>>> WindowOperator close all windows and emit the results upon calling
> >>>>
> >>>>>> finish, for example?
> >>>>> 1) currently they are independent but parallel mechanisms. With event
> >>>> time, they are basically the same.
> >>>>
> >>>>> 2) it probably should for the sake of processing time windows.
> >>>>> Here you are touching the bit of the current design that I like the
>
> >>>> least. We basically have now three different ways of conveying very similar
> >>>> things:
> >>>>
> >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
> >>>> about processing time?)
> >>>>
> >>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
> >>>> internal state
> >>>>
> >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> >>>>> It's a bit messy and I'm not sure if this should be strengthened out?
>
> >>>> Each one of those has a little bit different semantic/meaning, but at the
>
> >>>> same time they are very similar. For single input operators `endInput()`
> >>>> and `finish()` are actually the very same thing.
> >>>>
> >>>>> Piotrek
> >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> >>>>> Thanks for updating the FLIP. Based on the new section about
> >>>>> stop-with-savepoint [--drain] I got two other questions:
> >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> >>>>> WindowOperator close all windows and emit the results upon calling
> >>>>> finish, for example?
> >>>>> Cheers,
> >>>>> Till
> >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> >>>>>> Thanks a lot for your answers and clarifications Yun.
> >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> >>>> problem.
> >>>>
> >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> >>>>>> Cheers,
> >>>>>> Till
> >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> >>>>>> wrote:
> >>>>>>> Hi Till,
> >>>>>>> Very thanks for the review and comments!
> >>>>>>> 1) First I think in fact we could be able to do the computation
> >>>> outside
> >>>>
> >>>>>>> of the main thread,
> >>>>>>> and the current implementation mainly due to the computation is in
> >>>>>>> general fast and we
> >>>>>>> initially want to have a simplified first version.
>
> >>>>>>> The main requirement here is to have a constant view of the state of
> >>>> the
> >>>>
> >>>>>>> tasks, otherwise
> >>>>>>> for example if we have A -> B, if A is running when we check if we
> >>>> need
> >>>>
> >>>>>>> to trigger A, we will
> >>>>>>> mark A as have to trigger, but if A gets to finished when we check
> >>>> B, we
> >>>>
> >>>>>>> will also mark B as
>
> >>>>>>> have to trigger, then B will receive both rpc trigger and checkpoint
> >>>>>>> barrier, which would break
> >>>>>>> some assumption on the task side and complicate the implementation.
>
> >>>>>>> But to cope this issue, we in fact could first have a snapshot of the
> >>>>>>> tasks' state and then do the
>
> >>>>>>> computation, both the two step do not need to be in the main thread.
>
> >>>>>>> 2) For the computation logic, in fact currently we benefit a lot from
> >>>>>>> some shortcuts on all-to-all
>
> >>>>>>> edges and job vertex with all tasks running, these shortcuts could do
> >>>>>>> checks on the job vertex level
> >>>>>>> first and skip some job vertices as a whole. With this optimization
> >>>> we
> >>>>
> >>>>>>> have a O(V) algorithm, and the
> >>>>>>> current running time of the worst case for a job with 320,000 tasks
> >>>> is
> >>>>
> >>>>>>> less than 100ms. For
> >>>>>>> daily graph sizes the time would be further reduced linearly.
> >>>>>>> If we do the computation based on the last triggered tasks, we may
> >>>> not
> >>>>
> >>>>>>> easily encode this information
>
> >>>>>>> into the shortcuts on the job vertex level. And since the time seems
> >>>> to
> >>>>
> >>>>>>> be short, perhaps it is enough
> >>>>>>> to do re-computation from the scratch in consideration of the
> >>>> tradeoff
> >>>>
> >>>>>>> between the performance and
> >>>>>>> the complexity ?
> >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> >>>> finish()
> >>>>
> >>>>>>> method and before the last
>
> >>>>>>> snapshotState() method so that we could shut down the whole topology
> >>>> with
> >>>>
> >>>>>>> a single final checkpoint.
> >>>>>>> Very sorry for not include enough details for this part and I'll
> >>>>>>> complement the FLIP with the details on
> >>>>>>> the process of the final checkpoint / savepoint.
> >>>>>>> Best,
> >>>>>>> Yun
> >>>>>>> ------------------------------------------------------------------
> >>>>>>> From:Till Rohrmann
> >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> >>>>>>> To:dev
> >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >>>>>>> Finished
> >>>>>>> Hi everyone,
> >>>>>>> I am a bit late to the voting party but let me ask three questions:
>
> >>>>>>> 1) Why do we execute the trigger plan computation in the main thread
> >>>> if we
> >>>>
>
> >>>>>>> cannot guarantee that all tasks are still running when triggering the
> >>>>>>> checkpoint? Couldn't we do the computation in a different thread in
> >>>> order
> >>>>
> >>>>>>> to relieve the main thread a bit.
> >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator seems
> >>>> to go
> >>>>
> >>>>>>> over the whole topology for every calculation. Wouldn't it be more
> >>>>>>> efficient to maintain the set of current tasks to trigger and check
> >>>>>>> whether
> >>>>>>> anything has changed and if so check the succeeding tasks until we
> >>>> have
> >>>>
> >>>>>>> found the current checkpoint trigger frontier?
> >>>>>>> 3) When are we going to send the endOfInput events to a downstream
> >>>> task?
> >>>>
> >>>>>>> If
>
> >>>>>>> this happens after we call finish on the upstream operator but before
> >>>>>>> snapshotState then it would be possible to shut down the whole
> >>>> topology
> >>>>
>
> >>>>>>> with a single final checkpoint. I think this part could benefit from
> >>>> a bit
> >>>>
> >>>>>>> more detailed description in the FLIP.
> >>>>>>> Cheers,
> >>>>>>> Till
> >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> >>>>>>> wrote:
> >>>>>>>> Hi there,
> >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
> >>>> vote
> >>>>
> >>>>>>> now.
> >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >>>>>>>> - Dawid Wysakowicz (binding)
> >>>>>>>> - Piotr Nowojski(binding)
> >>>>>>>> - Jiangang Liu (binding)
> >>>>>>>> - Arvid Heise (binding)
> >>>>>>>> - Jing Zhang (binding)
> >>>>>>>> - Leonard Xu (non-binding)
> >>>>>>>> - Guowei Ma (binding)
> >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> >>>> accepted.
> >>>>
> >>>>>>>> Very thanks everyone!
> >>>>>>>> Best,
> >>>>>>>> Yun
> >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>>>
> >>>>
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hey all,
To make the issues that were found transparent to the community, I want
to post an update:
*1. Committing side-effects*
We do want to make sure that all side effects are committed before
bringing tasks down. Side effects are committed when calling
notifyCheckpointComplete. For the final checkpoint we introduced the
method finish(). This notifies the operator that we have consumed all
incoming records and we are preparing to close the Task. In turn we
should flush any pending buffered records and prepare to commit last
transactions. The goal is that after a successful sequence of finish()
-> snapshotState() -> notifyCheckpointComplete(), the remaining state
can be considered empty/finished and may be discarded.
_/Failure before notifyCheckpointComplete()/_
The question is what is the contract of the endInput()/finish() methods
and how do calling these methods affect the operators keyed, non-keyed
state and external state. Is it allowed to restore state snapshot taken
after calling endInput()/finish() and process more records? Or do we
assume that after a restore from such a state taken after finish() we
should not call any of the lifecycle methods or at least make sure those
methods do not emit records/interact with mailbox etc.
Currently it is possible that if a failure happens after sequence of
finish() -> snapshotState(), but before notifyCheckpointComplete(), we
will restore such a state and we might end up sending some more records
to such an operator. It is possible if we rescale and this state is
merged with a state of a subtask that has not called finish() yet. It
can also happen if we rescale the upstream operator and the subtask of
interest becomes connected to a newly added non finished subtask.
/Snapshotting StreamTasks that finish() has been called/
We thought about putting a flag into the snapshot of a subtask produced
after the finish() method. This would make it possible to skip execution
of certain lifecycle methods. Unfortunately this creates problems for
rescaling. How do we deal with a situation that subtask states with both
the feature flag set and unset end up in a single StreamTask. Additional
problem is that we merge those states into a single OperatorSubtaskState
on CheckpointCoordinator.
_/Finishing upon receiving notifyCheckpointComplete() of not the latest
checkpoint/_
We need to wait for a checkpoint to complete, that started after the
finish() method. However, we support concurrent checkpoints therefore,
there might be later checkpoints that completed, but the notification
has not arrived. We must make sure those checkpoints do not leave
lingering external resources.
*Checkpointing from a single subtask / UnionListState case*
There are operators that checkpoint from a single subtask only. Usually
from the subtask index=0. If we let those subtasks finish, subsequent
checkpoints will miss this information.
Esp. Legacy sources problem:
https://issues.apache.org/jira/browse/FLINK-21080
Best,
Dawid
On 19/07/2021 15:10, Yun Gao wrote:
> Hi Till, Dawid,
>
> Very thanks for the comments and discussion! Glad that it seems we
> have come to a convergence, and I also agree with that we could not
> include the optimization in the first version.
>
> Best
> Yun
>
> ------------------------------------------------------------------
> From:Dawid Wysakowicz <dw...@apache.org>
> Send Time:2021 Jul. 19 (Mon.) 17:51
> To:dev <de...@flink.apache.org>; Till Rohrmann <tr...@apache.org>
> Cc:Yun Gao <yu...@aliyun.com>; Yun Gao
> <yu...@aliyun.com.invalid>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks Finished
>
> Small correction. I meant we need to adjust the EndOfInputEvent of course.
>
> Best,
>
> Dawid
>
> On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> > Hey Till,
> >
> > Yes, you're right we will have to adjust the current state of
> > EndOfPartitionEvent and move the moment when we emit it to have what
> > we're discussing here. We are aware of that.
> >
> > As for the MAX_WATERMARK vs finish(). My take is that we should always
> > emit MAX_WATERMARK before calling finish() on an operator. At the same
> > time finish() should not leave behind anything in state, as the
> > intention is that we never restore from the taken savepoint/checkpoint
> > (savepoint w drain or bounded data consumed).
> >
> > Best,
> >
> > Dawid
> >
> > On 19/07/2021 11:33, Till Rohrmann wrote:
> >> Hi Yun and Dawid,
> >>
> >> Thanks for your comments. I do agree with your comments that finish() can
> >> do more than MAX_WATERMARK. I guess we should then explain how
> >> MAX_WATERMARK and finish() play together and what kind of
> >> order guarantees we provide.
> >>
> >> Concerning the EndOfPartitionEvent, I am not entirely sure whether it would
> >> work in its current state because we send this event when the Task is about
> >> to shut down if I am not mistaken. What we want to have is to bring the
> >> StreamTasks into a state so that they shut down on the next checkpoint. For
> >> this we need to keep the StreamTask running. In general, I am a fan of
> >> making things explicit if possible. I think this helps maintenance and
> >> evolvability of code. That's why I think sending an EndOfInputEvent which
> >> is a StreamTask level event and which says that there won't be any other
> >> records coming only control events could make sense.
> >>
> >> I would leave the proposed optimization out of the first version. We can
> >> still add it at a later point in time.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz <dw...@apache.org>
> >> wrote:
> >>
> >>> Personally I don't find this optimization important and I'd rather leave
> >>> it out not to complicate the codebase further. I doubt we save much there.
> >>> I don't have a strong opinion though.
> >>>
> >>> Best,
> >>>
> >>> Dawid
> >>> On 19/07/2021 10:31, Yun Gao wrote:
> >>>
> >>> Hi,
> >>>
> >>> Very thanks Dawid for the thoughts!
> >>>
> >>> Currently I also do not have different opinions regarding this part.
> >>> But I have one more issue to confirm: during the previous discussion we
> >>> have discussed that for the final checkpoint case, we might have an
> >>> optmization
> >>> that if a task do not have operators using 2-pc, we might skip waiting for
> >>> the
> >>> final checkpoint (but we could not skip the savepoint). To allow users to
> >>> express
> >>> the logic, we have proposed to add one more method to StreamOperator &
> >>> CheckpointListener:
> >>>
> >>> interface StreamOperator {
> >>> default boolean requiresFinalCheckpoint() {
> >>> return true;
> >>> }
> >>> }
> >>>
> >>> interface CheckpointListener {
> >>>
> >>> default boolean requiresFinalCheckpoint() {
> >>> return true;
> >>> }
> >>> }
> >>>
> >>> class AbstractUdfStreamOperator {
> >>>
> >>> @Override
> >>> boolean requiresFinalCheckpoint() {
> >>> return userFunction instanceof CheckpointListener &&
> >>> ((CheckpointListener) userFunction).requiresFinalCheckpoint();
> >>> }
> >>> }
> >>>
> >>>
> >>> I think we should still keep the change ?
> >>>
> >>> Best,
> >>> Yun
> >>>
> >>> ------------------Original Mail ------------------
> >>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> >>> <dw...@apache.org>
> >>> *Send Date:*Sun Jul 18 18:44:50 2021
> >>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> >>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> >>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >>> Finished
> >>>
> >>>> I think we're all really close to the same solution.
> >>>>
> >>>>
> >>>>
> >>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
> >>>>
> >>>> buffering, but it does not solve flushing other operations such as e.g.
> >>>>
> >>>> count windows or batching requests in Sinks. I'd prefer to treat the
> >>>>
> >>>> finish() as a message for Operator to "flush all records". The
> >>>>
> >>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
> >>>>
> >>>> don't think operators need to get a signal "stop-processing" if they
> >>>>
> >>>> don't need to flush records. The "WHEN" records are emitted, should be
> >>>>
> >>>> in control of the StreamTask, by firing timers or by processing a next
> >>>>
> >>>> record from upstream.
> >>>>
> >>>>
> >>>>
> >>>> The only difference of my previous proposal compared to Yun's is that I
> >>>>
> >>>> did not want to send the EndOfUserRecords event in case of stop w/o
> >>>>
> >>>> drain. My thinking was that we could directly go from RUNNING to
> >>>>
> >>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
> >>>>
> >>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
> >>>>
> >>>> timers and processing events (without calling finish() on Operator). In
> >>>>
> >>>> my initial suggestion I though we don't care about some events
> >>>>
> >>>> potentially being emitted after the savepoint was taken, as they would
> >>>>
> >>>> anyway belong to the next after FINAL, which would be discarded. I think
> >>>>
> >>>> though the proposal to suspend records processing and timers is a
> >>>>
> >>>> sensible thing to do and would go with the version that Yun put into the
> >>>>
> >>>> FLIP Wiki.
> >>>>
> >>>>
> >>>>
> >>>> What do you think Till?
> >>>>
> >>>>
> >>>>
> >>>> Best,
> >>>>
> >>>>
> >>>>
> >>>> Dawid
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 16/07/2021 10:03, Yun Gao wrote:
> >>>>
> >>>>> Hi Till, Piotr
> >>>>> Very thanks for the comments!
> >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >>>>> I also agree with Piotr that currently they are independent mechanisms,
> >>>> and they are basically the same
> >>>>
> >>>>> for the event time.
> >>>>> For more details, first there are some difference among the three
> >>>> scenarios regarding the finish:
> >>>>
> >>>>> For normal finish and stop-with-savepoint --drain, the job would not be
> >>>> expected to be restarted,
> >>>>
> >>>>> and for stop-with-savepoint the job would be expected restart later.
> >>>>> Then for finish / stop-with-savepoint --drain, currently Flink would
> >>>> emit MAX_WATERMARK before the
> >>>>
> >>>>> EndOfPartition. Besides, as we have discussed before [1], endOfInput /
> >>>> finish() should also only be called
> >>>>
> >>>>> for finish / stop-with-savepoint --drain. Thus currently they always
> >>>> occurs at the same time. After the change,
> >>>>
> >>>>> we could emit MAX_WATERMARK before endOfInput event for the finish /
> >>>> stop-with-savepoint --drain cases.
> >>>>
> >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> >>>>>> WindowOperator close all windows and emit the results upon calling
> >>>>>> finish, for example?
> >>>>> As discussed above for stop-with-savepoint, we would always keep the
> >>>> window as is, and restore them after restart.
> >>>>
> >>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps it
> >>>> depends on the Triggers. For
> >>>>
> >>>>> event-time triggers / process time triggers, it would be reasonable to
> >>>> flush all the windows since logically
> >>>>
> >>>>> the time would always elapse and the window would always get triggered
> >>>> in a logical future. But for triggers
> >>>>
> >>>>> like CountTrigger, no matter how much time pass logically, the windows
> >>>> would not trigger, thus we may not
> >>>>
> >>>>> flush these windows. If there are requirements we may provide
> >>>> additional triggers.
> >>>>
> >>>>>> It's a bit messy and I'm not sure if this should be strengthened out?
> >>>> Each one of those has a little bit different semantic/meaning,
> >>>>
> >>>>>> but at the same time they are very similar. For single input operators
> >>>> `endInput()` and `finish()` are actually the very same thing.
> >>>>
> >>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen at the
> >>>> same time, and for single input operators `endInput()` and `finish()`
> >>>>
> >>>>> are indeed the same thing. During the last discussion we ever mentioned
> >>>> this issue and at then we thought that we might deprecate `endInput()`
> >>>>
> >>>>> in the future, then we would only have endInput(int input) and
> >>>> finish().
> >>>>
> >>>>> Best,
> >>>>> Yun
> >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> >>>>> ------------------------------------------------------------------
> >>>>> From:Piotr Nowojski
> >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> >>>>> To:dev
> >>>>> Cc:Yun Gao
> >>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >>>> Finished
> >>>>
> >>>>> Hi Till,
> >>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a>
> >>>> WindowOperator close all windows and emit the results upon calling
> >>>>
> >>>>>> finish, for example?
> >>>>> 1) currently they are independent but parallel mechanisms. With event
> >>>> time, they are basically the same.
> >>>>
> >>>>> 2) it probably should for the sake of processing time windows.
> >>>>> Here you are touching the bit of the current design that I like the
> >>>> least. We basically have now three different ways of conveying very similar
> >>>> things:
> >>>>
> >>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
> >>>> about processing time?)
> >>>>
> >>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
> >>>> internal state
> >>>>
> >>>>> c) finish(), used for example by ContinuousFileReaderOperator
> >>>>> It's a bit messy and I'm not sure if this should be strengthened out?
> >>>> Each one of those has a little bit different semantic/meaning, but at the
> >>>> same time they are very similar. For single input operators `endInput()`
> >>>> and `finish()` are actually the very same thing.
> >>>>
> >>>>> Piotrek
> >>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> >>>>> Thanks for updating the FLIP. Based on the new section about
> >>>>> stop-with-savepoint [--drain] I got two other questions:
> >>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> >>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
> >>>>> WindowOperator close all windows and emit the results upon calling
> >>>>> finish, for example?
> >>>>> Cheers,
> >>>>> Till
> >>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
> >>>>>> Thanks a lot for your answers and clarifications Yun.
> >>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
> >>>> problem.
> >>>>
> >>>>>> 3) Great, this will help a lot with understanding the FLIP.
> >>>>>> Cheers,
> >>>>>> Till
> >>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> >>>>>> wrote:
> >>>>>>> Hi Till,
> >>>>>>> Very thanks for the review and comments!
> >>>>>>> 1) First I think in fact we could be able to do the computation
> >>>> outside
> >>>>
> >>>>>>> of the main thread,
> >>>>>>> and the current implementation mainly due to the computation is in
> >>>>>>> general fast and we
> >>>>>>> initially want to have a simplified first version.
> >>>>>>> The main requirement here is to have a constant view of the state of
> >>>> the
> >>>>
> >>>>>>> tasks, otherwise
> >>>>>>> for example if we have A -> B, if A is running when we check if we
> >>>> need
> >>>>
> >>>>>>> to trigger A, we will
> >>>>>>> mark A as have to trigger, but if A gets to finished when we check
> >>>> B, we
> >>>>
> >>>>>>> will also mark B as
> >>>>>>> have to trigger, then B will receive both rpc trigger and checkpoint
> >>>>>>> barrier, which would break
> >>>>>>> some assumption on the task side and complicate the implementation.
> >>>>>>> But to cope this issue, we in fact could first have a snapshot of the
> >>>>>>> tasks' state and then do the
> >>>>>>> computation, both the two step do not need to be in the main thread.
> >>>>>>> 2) For the computation logic, in fact currently we benefit a lot from
> >>>>>>> some shortcuts on all-to-all
> >>>>>>> edges and job vertex with all tasks running, these shortcuts could do
> >>>>>>> checks on the job vertex level
> >>>>>>> first and skip some job vertices as a whole. With this optimization
> >>>> we
> >>>>
> >>>>>>> have a O(V) algorithm, and the
> >>>>>>> current running time of the worst case for a job with 320,000 tasks
> >>>> is
> >>>>
> >>>>>>> less than 100ms. For
> >>>>>>> daily graph sizes the time would be further reduced linearly.
> >>>>>>> If we do the computation based on the last triggered tasks, we may
> >>>> not
> >>>>
> >>>>>>> easily encode this information
> >>>>>>> into the shortcuts on the job vertex level. And since the time seems
> >>>> to
> >>>>
> >>>>>>> be short, perhaps it is enough
> >>>>>>> to do re-computation from the scratch in consideration of the
> >>>> tradeoff
> >>>>
> >>>>>>> between the performance and
> >>>>>>> the complexity ?
> >>>>>>> 3) We are going to emit the EndOfInput event exactly after the
> >>>> finish()
> >>>>
> >>>>>>> method and before the last
> >>>>>>> snapshotState() method so that we could shut down the whole topology
> >>>> with
> >>>>
> >>>>>>> a single final checkpoint.
> >>>>>>> Very sorry for not include enough details for this part and I'll
> >>>>>>> complement the FLIP with the details on
> >>>>>>> the process of the final checkpoint / savepoint.
> >>>>>>> Best,
> >>>>>>> Yun
> >>>>>>> ------------------------------------------------------------------
> >>>>>>> From:Till Rohrmann
> >>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
> >>>>>>> To:dev
> >>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >>>>>>> Finished
> >>>>>>> Hi everyone,
> >>>>>>> I am a bit late to the voting party but let me ask three questions:
> >>>>>>> 1) Why do we execute the trigger plan computation in the main thread
> >>>> if we
> >>>>
> >>>>>>> cannot guarantee that all tasks are still running when triggering the
> >>>>>>> checkpoint? Couldn't we do the computation in a different thread in
> >>>> order
> >>>>
> >>>>>>> to relieve the main thread a bit.
> >>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator seems
> >>>> to go
> >>>>
> >>>>>>> over the whole topology for every calculation. Wouldn't it be more
> >>>>>>> efficient to maintain the set of current tasks to trigger and check
> >>>>>>> whether
> >>>>>>> anything has changed and if so check the succeeding tasks until we
> >>>> have
> >>>>
> >>>>>>> found the current checkpoint trigger frontier?
> >>>>>>> 3) When are we going to send the endOfInput events to a downstream
> >>>> task?
> >>>>
> >>>>>>> If
> >>>>>>> this happens after we call finish on the upstream operator but before
> >>>>>>> snapshotState then it would be possible to shut down the whole
> >>>> topology
> >>>>
> >>>>>>> with a single final checkpoint. I think this part could benefit from
> >>>> a bit
> >>>>
> >>>>>>> more detailed description in the FLIP.
> >>>>>>> Cheers,
> >>>>>>> Till
> >>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> >>>>>>> wrote:
> >>>>>>>> Hi there,
> >>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
> >>>> vote
> >>>>
> >>>>>>> now.
> >>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >>>>>>>> - Dawid Wysakowicz (binding)
> >>>>>>>> - Piotr Nowojski(binding)
> >>>>>>>> - Jiangang Liu (binding)
> >>>>>>>> - Arvid Heise (binding)
> >>>>>>>> - Jing Zhang (binding)
> >>>>>>>> - Leonard Xu (non-binding)
> >>>>>>>> - Guowei Ma (binding)
> >>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
> >>>> accepted.
> >>>>
> >>>>>>>> Very thanks everyone!
> >>>>>>>> Best,
> >>>>>>>> Yun
> >>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>>>
> >>>>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Till, Dawid,
Very thanks for the comments and discussion! Glad that it seems we have come to a convergence, and I also agree with that we could not include the optimization in the first version.
Best
Yun
------------------------------------------------------------------
From:Dawid Wysakowicz <dw...@apache.org>
Send Time:2021 Jul. 19 (Mon.) 17:51
To:dev <de...@flink.apache.org>; Till Rohrmann <tr...@apache.org>
Cc:Yun Gao <yu...@aliyun.com>; Yun Gao <yu...@aliyun.com.invalid>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Small correction. I meant we need to adjust the EndOfInputEvent of course.
Best,
Dawid
On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> Hey Till,
>
> Yes, you're right we will have to adjust the current state of
> EndOfPartitionEvent and move the moment when we emit it to have what
> we're discussing here. We are aware of that.
>
> As for the MAX_WATERMARK vs finish(). My take is that we should always
> emit MAX_WATERMARK before calling finish() on an operator. At the same
> time finish() should not leave behind anything in state, as the
> intention is that we never restore from the taken savepoint/checkpoint
> (savepoint w drain or bounded data consumed).
>
> Best,
>
> Dawid
>
> On 19/07/2021 11:33, Till Rohrmann wrote:
>> Hi Yun and Dawid,
>>
>> Thanks for your comments. I do agree with your comments that finish() can
>> do more than MAX_WATERMARK. I guess we should then explain how
>> MAX_WATERMARK and finish() play together and what kind of
>> order guarantees we provide.
>>
>> Concerning the EndOfPartitionEvent, I am not entirely sure whether it would
>> work in its current state because we send this event when the Task is about
>> to shut down if I am not mistaken. What we want to have is to bring the
>> StreamTasks into a state so that they shut down on the next checkpoint. For
>> this we need to keep the StreamTask running. In general, I am a fan of
>> making things explicit if possible. I think this helps maintenance and
>> evolvability of code. That's why I think sending an EndOfInputEvent which
>> is a StreamTask level event and which says that there won't be any other
>> records coming only control events could make sense.
>>
>> I would leave the proposed optimization out of the first version. We can
>> still add it at a later point in time.
>>
>> Cheers,
>> Till
>>
>> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz <dw...@apache.org>
>> wrote:
>>
>>> Personally I don't find this optimization important and I'd rather leave
>>> it out not to complicate the codebase further. I doubt we save much there.
>>> I don't have a strong opinion though.
>>>
>>> Best,
>>>
>>> Dawid
>>> On 19/07/2021 10:31, Yun Gao wrote:
>>>
>>> Hi,
>>>
>>> Very thanks Dawid for the thoughts!
>>>
>>> Currently I also do not have different opinions regarding this part.
>>> But I have one more issue to confirm: during the previous discussion we
>>> have discussed that for the final checkpoint case, we might have an
>>> optmization
>>> that if a task do not have operators using 2-pc, we might skip waiting for
>>> the
>>> final checkpoint (but we could not skip the savepoint). To allow users to
>>> express
>>> the logic, we have proposed to add one more method to StreamOperator &
>>> CheckpointListener:
>>>
>>> interface StreamOperator {
>>> default boolean requiresFinalCheckpoint() {
>>> return true;
>>> }
>>> }
>>>
>>> interface CheckpointListener {
>>>
>>> default boolean requiresFinalCheckpoint() {
>>> return true;
>>> }
>>> }
>>>
>>> class AbstractUdfStreamOperator {
>>>
>>> @Override
>>> boolean requiresFinalCheckpoint() {
>>> return userFunction instanceof CheckpointListener &&
>>> ((CheckpointListener) userFunction).requiresFinalCheckpoint();
>>> }
>>> }
>>>
>>>
>>> I think we should still keep the change ?
>>>
>>> Best,
>>> Yun
>>>
>>> ------------------Original Mail ------------------
>>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
>>> <dw...@apache.org>
>>> *Send Date:*Sun Jul 18 18:44:50 2021
>>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>, Yun
>>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
>>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>> Finished
>>>
>>>> I think we're all really close to the same solution.
>>>>
>>>>
>>>>
>>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
>>>>
>>>> buffering, but it does not solve flushing other operations such as e.g.
>>>>
>>>> count windows or batching requests in Sinks. I'd prefer to treat the
>>>>
>>>> finish() as a message for Operator to "flush all records". The
>>>>
>>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
>>>>
>>>> don't think operators need to get a signal "stop-processing" if they
>>>>
>>>> don't need to flush records. The "WHEN" records are emitted, should be
>>>>
>>>> in control of the StreamTask, by firing timers or by processing a next
>>>>
>>>> record from upstream.
>>>>
>>>>
>>>>
>>>> The only difference of my previous proposal compared to Yun's is that I
>>>>
>>>> did not want to send the EndOfUserRecords event in case of stop w/o
>>>>
>>>> drain. My thinking was that we could directly go from RUNNING to
>>>>
>>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
>>>>
>>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
>>>>
>>>> timers and processing events (without calling finish() on Operator). In
>>>>
>>>> my initial suggestion I though we don't care about some events
>>>>
>>>> potentially being emitted after the savepoint was taken, as they would
>>>>
>>>> anyway belong to the next after FINAL, which would be discarded. I think
>>>>
>>>> though the proposal to suspend records processing and timers is a
>>>>
>>>> sensible thing to do and would go with the version that Yun put into the
>>>>
>>>> FLIP Wiki.
>>>>
>>>>
>>>>
>>>> What do you think Till?
>>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>>
>>>>
>>>> Dawid
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 16/07/2021 10:03, Yun Gao wrote:
>>>>
>>>>> Hi Till, Piotr
>>>>> Very thanks for the comments!
>>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>>> I also agree with Piotr that currently they are independent mechanisms,
>>>> and they are basically the same
>>>>
>>>>> for the event time.
>>>>> For more details, first there are some difference among the three
>>>> scenarios regarding the finish:
>>>>
>>>>> For normal finish and stop-with-savepoint --drain, the job would not be
>>>> expected to be restarted,
>>>>
>>>>> and for stop-with-savepoint the job would be expected restart later.
>>>>> Then for finish / stop-with-savepoint --drain, currently Flink would
>>>> emit MAX_WATERMARK before the
>>>>
>>>>> EndOfPartition. Besides, as we have discussed before [1], endOfInput /
>>>> finish() should also only be called
>>>>
>>>>> for finish / stop-with-savepoint --drain. Thus currently they always
>>>> occurs at the same time. After the change,
>>>>
>>>>> we could emit MAX_WATERMARK before endOfInput event for the finish /
>>>> stop-with-savepoint --drain cases.
>>>>
>>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>>>>> WindowOperator close all windows and emit the results upon calling
>>>>>> finish, for example?
>>>>> As discussed above for stop-with-savepoint, we would always keep the
>>>> window as is, and restore them after restart.
>>>>
>>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps it
>>>> depends on the Triggers. For
>>>>
>>>>> event-time triggers / process time triggers, it would be reasonable to
>>>> flush all the windows since logically
>>>>
>>>>> the time would always elapse and the window would always get triggered
>>>> in a logical future. But for triggers
>>>>
>>>>> like CountTrigger, no matter how much time pass logically, the windows
>>>> would not trigger, thus we may not
>>>>
>>>>> flush these windows. If there are requirements we may provide
>>>> additional triggers.
>>>>
>>>>>> It's a bit messy and I'm not sure if this should be strengthened out?
>>>> Each one of those has a little bit different semantic/meaning,
>>>>
>>>>>> but at the same time they are very similar. For single input operators
>>>> `endInput()` and `finish()` are actually the very same thing.
>>>>
>>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen at the
>>>> same time, and for single input operators `endInput()` and `finish()`
>>>>
>>>>> are indeed the same thing. During the last discussion we ever mentioned
>>>> this issue and at then we thought that we might deprecate `endInput()`
>>>>
>>>>> in the future, then we would only have endInput(int input) and
>>>> finish().
>>>>
>>>>> Best,
>>>>> Yun
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
>>>>> ------------------------------------------------------------------
>>>>> From:Piotr Nowojski
>>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
>>>>> To:dev
>>>>> Cc:Yun Gao
>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>>> Finished
>>>>
>>>>> Hi Till,
>>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a>
>>>> WindowOperator close all windows and emit the results upon calling
>>>>
>>>>>> finish, for example?
>>>>> 1) currently they are independent but parallel mechanisms. With event
>>>> time, they are basically the same.
>>>>
>>>>> 2) it probably should for the sake of processing time windows.
>>>>> Here you are touching the bit of the current design that I like the
>>>> least. We basically have now three different ways of conveying very similar
>>>> things:
>>>>
>>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
>>>> about processing time?)
>>>>
>>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
>>>> internal state
>>>>
>>>>> c) finish(), used for example by ContinuousFileReaderOperator
>>>>> It's a bit messy and I'm not sure if this should be strengthened out?
>>>> Each one of those has a little bit different semantic/meaning, but at the
>>>> same time they are very similar. For single input operators `endInput()`
>>>> and `finish()` are actually the very same thing.
>>>>
>>>>> Piotrek
>>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
>>>>> Thanks for updating the FLIP. Based on the new section about
>>>>> stop-with-savepoint [--drain] I got two other questions:
>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>>>> WindowOperator close all windows and emit the results upon calling
>>>>> finish, for example?
>>>>> Cheers,
>>>>> Till
>>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
>>>>>> Thanks a lot for your answers and clarifications Yun.
>>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
>>>> problem.
>>>>
>>>>>> 3) Great, this will help a lot with understanding the FLIP.
>>>>>> Cheers,
>>>>>> Till
>>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
>>>>>> wrote:
>>>>>>> Hi Till,
>>>>>>> Very thanks for the review and comments!
>>>>>>> 1) First I think in fact we could be able to do the computation
>>>> outside
>>>>
>>>>>>> of the main thread,
>>>>>>> and the current implementation mainly due to the computation is in
>>>>>>> general fast and we
>>>>>>> initially want to have a simplified first version.
>>>>>>> The main requirement here is to have a constant view of the state of
>>>> the
>>>>
>>>>>>> tasks, otherwise
>>>>>>> for example if we have A -> B, if A is running when we check if we
>>>> need
>>>>
>>>>>>> to trigger A, we will
>>>>>>> mark A as have to trigger, but if A gets to finished when we check
>>>> B, we
>>>>
>>>>>>> will also mark B as
>>>>>>> have to trigger, then B will receive both rpc trigger and checkpoint
>>>>>>> barrier, which would break
>>>>>>> some assumption on the task side and complicate the implementation.
>>>>>>> But to cope this issue, we in fact could first have a snapshot of the
>>>>>>> tasks' state and then do the
>>>>>>> computation, both the two step do not need to be in the main thread.
>>>>>>> 2) For the computation logic, in fact currently we benefit a lot from
>>>>>>> some shortcuts on all-to-all
>>>>>>> edges and job vertex with all tasks running, these shortcuts could do
>>>>>>> checks on the job vertex level
>>>>>>> first and skip some job vertices as a whole. With this optimization
>>>> we
>>>>
>>>>>>> have a O(V) algorithm, and the
>>>>>>> current running time of the worst case for a job with 320,000 tasks
>>>> is
>>>>
>>>>>>> less than 100ms. For
>>>>>>> daily graph sizes the time would be further reduced linearly.
>>>>>>> If we do the computation based on the last triggered tasks, we may
>>>> not
>>>>
>>>>>>> easily encode this information
>>>>>>> into the shortcuts on the job vertex level. And since the time seems
>>>> to
>>>>
>>>>>>> be short, perhaps it is enough
>>>>>>> to do re-computation from the scratch in consideration of the
>>>> tradeoff
>>>>
>>>>>>> between the performance and
>>>>>>> the complexity ?
>>>>>>> 3) We are going to emit the EndOfInput event exactly after the
>>>> finish()
>>>>
>>>>>>> method and before the last
>>>>>>> snapshotState() method so that we could shut down the whole topology
>>>> with
>>>>
>>>>>>> a single final checkpoint.
>>>>>>> Very sorry for not include enough details for this part and I'll
>>>>>>> complement the FLIP with the details on
>>>>>>> the process of the final checkpoint / savepoint.
>>>>>>> Best,
>>>>>>> Yun
>>>>>>> ------------------------------------------------------------------
>>>>>>> From:Till Rohrmann
>>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
>>>>>>> To:dev
>>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>>>>>> Finished
>>>>>>> Hi everyone,
>>>>>>> I am a bit late to the voting party but let me ask three questions:
>>>>>>> 1) Why do we execute the trigger plan computation in the main thread
>>>> if we
>>>>
>>>>>>> cannot guarantee that all tasks are still running when triggering the
>>>>>>> checkpoint? Couldn't we do the computation in a different thread in
>>>> order
>>>>
>>>>>>> to relieve the main thread a bit.
>>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator seems
>>>> to go
>>>>
>>>>>>> over the whole topology for every calculation. Wouldn't it be more
>>>>>>> efficient to maintain the set of current tasks to trigger and check
>>>>>>> whether
>>>>>>> anything has changed and if so check the succeeding tasks until we
>>>> have
>>>>
>>>>>>> found the current checkpoint trigger frontier?
>>>>>>> 3) When are we going to send the endOfInput events to a downstream
>>>> task?
>>>>
>>>>>>> If
>>>>>>> this happens after we call finish on the upstream operator but before
>>>>>>> snapshotState then it would be possible to shut down the whole
>>>> topology
>>>>
>>>>>>> with a single final checkpoint. I think this part could benefit from
>>>> a bit
>>>>
>>>>>>> more detailed description in the FLIP.
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
>>>>>>> wrote:
>>>>>>>> Hi there,
>>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
>>>> vote
>>>>
>>>>>>> now.
>>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>>>>>>>> - Dawid Wysakowicz (binding)
>>>>>>>> - Piotr Nowojski(binding)
>>>>>>>> - Jiangang Liu (binding)
>>>>>>>> - Arvid Heise (binding)
>>>>>>>> - Jing Zhang (binding)
>>>>>>>> - Leonard Xu (non-binding)
>>>>>>>> - Guowei Ma (binding)
>>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
>>>> accepted.
>>>>
>>>>>>>> Very thanks everyone!
>>>>>>>> Best,
>>>>>>>> Yun
>>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>>>
>>>>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
Small correction. I meant we need to adjust the EndOfInputEvent of course.
Best,
Dawid
On 19/07/2021 11:48, Dawid Wysakowicz wrote:
> Hey Till,
>
> Yes, you're right we will have to adjust the current state of
> EndOfPartitionEvent and move the moment when we emit it to have what
> we're discussing here. We are aware of that.
>
> As for the MAX_WATERMARK vs finish(). My take is that we should always
> emit MAX_WATERMARK before calling finish() on an operator. At the same
> time finish() should not leave behind anything in state, as the
> intention is that we never restore from the taken savepoint/checkpoint
> (savepoint w drain or bounded data consumed).
>
> Best,
>
> Dawid
>
> On 19/07/2021 11:33, Till Rohrmann wrote:
>> Hi Yun and Dawid,
>>
>> Thanks for your comments. I do agree with your comments that finish() can
>> do more than MAX_WATERMARK. I guess we should then explain how
>> MAX_WATERMARK and finish() play together and what kind of
>> order guarantees we provide.
>>
>> Concerning the EndOfPartitionEvent, I am not entirely sure whether it would
>> work in its current state because we send this event when the Task is about
>> to shut down if I am not mistaken. What we want to have is to bring the
>> StreamTasks into a state so that they shut down on the next checkpoint. For
>> this we need to keep the StreamTask running. In general, I am a fan of
>> making things explicit if possible. I think this helps maintenance and
>> evolvability of code. That's why I think sending an EndOfInputEvent which
>> is a StreamTask level event and which says that there won't be any other
>> records coming only control events could make sense.
>>
>> I would leave the proposed optimization out of the first version. We can
>> still add it at a later point in time.
>>
>> Cheers,
>> Till
>>
>> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz <dw...@apache.org>
>> wrote:
>>
>>> Personally I don't find this optimization important and I'd rather leave
>>> it out not to complicate the codebase further. I doubt we save much there.
>>> I don't have a strong opinion though.
>>>
>>> Best,
>>>
>>> Dawid
>>> On 19/07/2021 10:31, Yun Gao wrote:
>>>
>>> Hi,
>>>
>>> Very thanks Dawid for the thoughts!
>>>
>>> Currently I also do not have different opinions regarding this part.
>>> But I have one more issue to confirm: during the previous discussion we
>>> have discussed that for the final checkpoint case, we might have an
>>> optmization
>>> that if a task do not have operators using 2-pc, we might skip waiting for
>>> the
>>> final checkpoint (but we could not skip the savepoint). To allow users to
>>> express
>>> the logic, we have proposed to add one more method to StreamOperator &
>>> CheckpointListener:
>>>
>>> interface StreamOperator {
>>> default boolean requiresFinalCheckpoint() {
>>> return true;
>>> }
>>> }
>>>
>>> interface CheckpointListener {
>>>
>>> default boolean requiresFinalCheckpoint() {
>>> return true;
>>> }
>>> }
>>>
>>> class AbstractUdfStreamOperator {
>>>
>>> @Override
>>> boolean requiresFinalCheckpoint() {
>>> return userFunction instanceof CheckpointListener &&
>>> ((CheckpointListener) userFunction).requiresFinalCheckpoint();
>>> }
>>> }
>>>
>>>
>>> I think we should still keep the change ?
>>>
>>> Best,
>>> Yun
>>>
>>> ------------------Original Mail ------------------
>>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
>>> <dw...@apache.org>
>>> *Send Date:*Sun Jul 18 18:44:50 2021
>>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>, Yun
>>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
>>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>> Finished
>>>
>>>> I think we're all really close to the same solution.
>>>>
>>>>
>>>>
>>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
>>>>
>>>> buffering, but it does not solve flushing other operations such as e.g.
>>>>
>>>> count windows or batching requests in Sinks. I'd prefer to treat the
>>>>
>>>> finish() as a message for Operator to "flush all records". The
>>>>
>>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
>>>>
>>>> don't think operators need to get a signal "stop-processing" if they
>>>>
>>>> don't need to flush records. The "WHEN" records are emitted, should be
>>>>
>>>> in control of the StreamTask, by firing timers or by processing a next
>>>>
>>>> record from upstream.
>>>>
>>>>
>>>>
>>>> The only difference of my previous proposal compared to Yun's is that I
>>>>
>>>> did not want to send the EndOfUserRecords event in case of stop w/o
>>>>
>>>> drain. My thinking was that we could directly go from RUNNING to
>>>>
>>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
>>>>
>>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
>>>>
>>>> timers and processing events (without calling finish() on Operator). In
>>>>
>>>> my initial suggestion I though we don't care about some events
>>>>
>>>> potentially being emitted after the savepoint was taken, as they would
>>>>
>>>> anyway belong to the next after FINAL, which would be discarded. I think
>>>>
>>>> though the proposal to suspend records processing and timers is a
>>>>
>>>> sensible thing to do and would go with the version that Yun put into the
>>>>
>>>> FLIP Wiki.
>>>>
>>>>
>>>>
>>>> What do you think Till?
>>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>>
>>>>
>>>> Dawid
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 16/07/2021 10:03, Yun Gao wrote:
>>>>
>>>>> Hi Till, Piotr
>>>>> Very thanks for the comments!
>>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>>> I also agree with Piotr that currently they are independent mechanisms,
>>>> and they are basically the same
>>>>
>>>>> for the event time.
>>>>> For more details, first there are some difference among the three
>>>> scenarios regarding the finish:
>>>>
>>>>> For normal finish and stop-with-savepoint --drain, the job would not be
>>>> expected to be restarted,
>>>>
>>>>> and for stop-with-savepoint the job would be expected restart later.
>>>>> Then for finish / stop-with-savepoint --drain, currently Flink would
>>>> emit MAX_WATERMARK before the
>>>>
>>>>> EndOfPartition. Besides, as we have discussed before [1], endOfInput /
>>>> finish() should also only be called
>>>>
>>>>> for finish / stop-with-savepoint --drain. Thus currently they always
>>>> occurs at the same time. After the change,
>>>>
>>>>> we could emit MAX_WATERMARK before endOfInput event for the finish /
>>>> stop-with-savepoint --drain cases.
>>>>
>>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>>>>> WindowOperator close all windows and emit the results upon calling
>>>>>> finish, for example?
>>>>> As discussed above for stop-with-savepoint, we would always keep the
>>>> window as is, and restore them after restart.
>>>>
>>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps it
>>>> depends on the Triggers. For
>>>>
>>>>> event-time triggers / process time triggers, it would be reasonable to
>>>> flush all the windows since logically
>>>>
>>>>> the time would always elapse and the window would always get triggered
>>>> in a logical future. But for triggers
>>>>
>>>>> like CountTrigger, no matter how much time pass logically, the windows
>>>> would not trigger, thus we may not
>>>>
>>>>> flush these windows. If there are requirements we may provide
>>>> additional triggers.
>>>>
>>>>>> It's a bit messy and I'm not sure if this should be strengthened out?
>>>> Each one of those has a little bit different semantic/meaning,
>>>>
>>>>>> but at the same time they are very similar. For single input operators
>>>> `endInput()` and `finish()` are actually the very same thing.
>>>>
>>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen at the
>>>> same time, and for single input operators `endInput()` and `finish()`
>>>>
>>>>> are indeed the same thing. During the last discussion we ever mentioned
>>>> this issue and at then we thought that we might deprecate `endInput()`
>>>>
>>>>> in the future, then we would only have endInput(int input) and
>>>> finish().
>>>>
>>>>> Best,
>>>>> Yun
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
>>>>> ------------------------------------------------------------------
>>>>> From:Piotr Nowojski
>>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
>>>>> To:dev
>>>>> Cc:Yun Gao
>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>>> Finished
>>>>
>>>>> Hi Till,
>>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a>
>>>> WindowOperator close all windows and emit the results upon calling
>>>>
>>>>>> finish, for example?
>>>>> 1) currently they are independent but parallel mechanisms. With event
>>>> time, they are basically the same.
>>>>
>>>>> 2) it probably should for the sake of processing time windows.
>>>>> Here you are touching the bit of the current design that I like the
>>>> least. We basically have now three different ways of conveying very similar
>>>> things:
>>>>
>>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
>>>> about processing time?)
>>>>
>>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
>>>> internal state
>>>>
>>>>> c) finish(), used for example by ContinuousFileReaderOperator
>>>>> It's a bit messy and I'm not sure if this should be strengthened out?
>>>> Each one of those has a little bit different semantic/meaning, but at the
>>>> same time they are very similar. For single input operators `endInput()`
>>>> and `finish()` are actually the very same thing.
>>>>
>>>>> Piotrek
>>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
>>>>> Thanks for updating the FLIP. Based on the new section about
>>>>> stop-with-savepoint [--drain] I got two other questions:
>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>>>> WindowOperator close all windows and emit the results upon calling
>>>>> finish, for example?
>>>>> Cheers,
>>>>> Till
>>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
>>>>>> Thanks a lot for your answers and clarifications Yun.
>>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
>>>> problem.
>>>>
>>>>>> 3) Great, this will help a lot with understanding the FLIP.
>>>>>> Cheers,
>>>>>> Till
>>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
>>>>>> wrote:
>>>>>>> Hi Till,
>>>>>>> Very thanks for the review and comments!
>>>>>>> 1) First I think in fact we could be able to do the computation
>>>> outside
>>>>
>>>>>>> of the main thread,
>>>>>>> and the current implementation mainly due to the computation is in
>>>>>>> general fast and we
>>>>>>> initially want to have a simplified first version.
>>>>>>> The main requirement here is to have a constant view of the state of
>>>> the
>>>>
>>>>>>> tasks, otherwise
>>>>>>> for example if we have A -> B, if A is running when we check if we
>>>> need
>>>>
>>>>>>> to trigger A, we will
>>>>>>> mark A as have to trigger, but if A gets to finished when we check
>>>> B, we
>>>>
>>>>>>> will also mark B as
>>>>>>> have to trigger, then B will receive both rpc trigger and checkpoint
>>>>>>> barrier, which would break
>>>>>>> some assumption on the task side and complicate the implementation.
>>>>>>> But to cope this issue, we in fact could first have a snapshot of the
>>>>>>> tasks' state and then do the
>>>>>>> computation, both the two step do not need to be in the main thread.
>>>>>>> 2) For the computation logic, in fact currently we benefit a lot from
>>>>>>> some shortcuts on all-to-all
>>>>>>> edges and job vertex with all tasks running, these shortcuts could do
>>>>>>> checks on the job vertex level
>>>>>>> first and skip some job vertices as a whole. With this optimization
>>>> we
>>>>
>>>>>>> have a O(V) algorithm, and the
>>>>>>> current running time of the worst case for a job with 320,000 tasks
>>>> is
>>>>
>>>>>>> less than 100ms. For
>>>>>>> daily graph sizes the time would be further reduced linearly.
>>>>>>> If we do the computation based on the last triggered tasks, we may
>>>> not
>>>>
>>>>>>> easily encode this information
>>>>>>> into the shortcuts on the job vertex level. And since the time seems
>>>> to
>>>>
>>>>>>> be short, perhaps it is enough
>>>>>>> to do re-computation from the scratch in consideration of the
>>>> tradeoff
>>>>
>>>>>>> between the performance and
>>>>>>> the complexity ?
>>>>>>> 3) We are going to emit the EndOfInput event exactly after the
>>>> finish()
>>>>
>>>>>>> method and before the last
>>>>>>> snapshotState() method so that we could shut down the whole topology
>>>> with
>>>>
>>>>>>> a single final checkpoint.
>>>>>>> Very sorry for not include enough details for this part and I'll
>>>>>>> complement the FLIP with the details on
>>>>>>> the process of the final checkpoint / savepoint.
>>>>>>> Best,
>>>>>>> Yun
>>>>>>> ------------------------------------------------------------------
>>>>>>> From:Till Rohrmann
>>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
>>>>>>> To:dev
>>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>>>>>> Finished
>>>>>>> Hi everyone,
>>>>>>> I am a bit late to the voting party but let me ask three questions:
>>>>>>> 1) Why do we execute the trigger plan computation in the main thread
>>>> if we
>>>>
>>>>>>> cannot guarantee that all tasks are still running when triggering the
>>>>>>> checkpoint? Couldn't we do the computation in a different thread in
>>>> order
>>>>
>>>>>>> to relieve the main thread a bit.
>>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator seems
>>>> to go
>>>>
>>>>>>> over the whole topology for every calculation. Wouldn't it be more
>>>>>>> efficient to maintain the set of current tasks to trigger and check
>>>>>>> whether
>>>>>>> anything has changed and if so check the succeeding tasks until we
>>>> have
>>>>
>>>>>>> found the current checkpoint trigger frontier?
>>>>>>> 3) When are we going to send the endOfInput events to a downstream
>>>> task?
>>>>
>>>>>>> If
>>>>>>> this happens after we call finish on the upstream operator but before
>>>>>>> snapshotState then it would be possible to shut down the whole
>>>> topology
>>>>
>>>>>>> with a single final checkpoint. I think this part could benefit from
>>>> a bit
>>>>
>>>>>>> more detailed description in the FLIP.
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
>>>>>>> wrote:
>>>>>>>> Hi there,
>>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
>>>> vote
>>>>
>>>>>>> now.
>>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>>>>>>>> - Dawid Wysakowicz (binding)
>>>>>>>> - Piotr Nowojski(binding)
>>>>>>>> - Jiangang Liu (binding)
>>>>>>>> - Arvid Heise (binding)
>>>>>>>> - Jing Zhang (binding)
>>>>>>>> - Leonard Xu (non-binding)
>>>>>>>> - Guowei Ma (binding)
>>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
>>>> accepted.
>>>>
>>>>>>>> Very thanks everyone!
>>>>>>>> Best,
>>>>>>>> Yun
>>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>>>
>>>>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hey Till,
Yes, you're right we will have to adjust the current state of
EndOfPartitionEvent and move the moment when we emit it to have what
we're discussing here. We are aware of that.
As for the MAX_WATERMARK vs finish(). My take is that we should always
emit MAX_WATERMARK before calling finish() on an operator. At the same
time finish() should not leave behind anything in state, as the
intention is that we never restore from the taken savepoint/checkpoint
(savepoint w drain or bounded data consumed).
Best,
Dawid
On 19/07/2021 11:33, Till Rohrmann wrote:
> Hi Yun and Dawid,
>
> Thanks for your comments. I do agree with your comments that finish() can
> do more than MAX_WATERMARK. I guess we should then explain how
> MAX_WATERMARK and finish() play together and what kind of
> order guarantees we provide.
>
> Concerning the EndOfPartitionEvent, I am not entirely sure whether it would
> work in its current state because we send this event when the Task is about
> to shut down if I am not mistaken. What we want to have is to bring the
> StreamTasks into a state so that they shut down on the next checkpoint. For
> this we need to keep the StreamTask running. In general, I am a fan of
> making things explicit if possible. I think this helps maintenance and
> evolvability of code. That's why I think sending an EndOfInputEvent which
> is a StreamTask level event and which says that there won't be any other
> records coming only control events could make sense.
>
> I would leave the proposed optimization out of the first version. We can
> still add it at a later point in time.
>
> Cheers,
> Till
>
> On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz <dw...@apache.org>
> wrote:
>
>> Personally I don't find this optimization important and I'd rather leave
>> it out not to complicate the codebase further. I doubt we save much there.
>> I don't have a strong opinion though.
>>
>> Best,
>>
>> Dawid
>> On 19/07/2021 10:31, Yun Gao wrote:
>>
>> Hi,
>>
>> Very thanks Dawid for the thoughts!
>>
>> Currently I also do not have different opinions regarding this part.
>> But I have one more issue to confirm: during the previous discussion we
>> have discussed that for the final checkpoint case, we might have an
>> optmization
>> that if a task do not have operators using 2-pc, we might skip waiting for
>> the
>> final checkpoint (but we could not skip the savepoint). To allow users to
>> express
>> the logic, we have proposed to add one more method to StreamOperator &
>> CheckpointListener:
>>
>> interface StreamOperator {
>> default boolean requiresFinalCheckpoint() {
>> return true;
>> }
>> }
>>
>> interface CheckpointListener {
>>
>> default boolean requiresFinalCheckpoint() {
>> return true;
>> }
>> }
>>
>> class AbstractUdfStreamOperator {
>>
>> @Override
>> boolean requiresFinalCheckpoint() {
>> return userFunction instanceof CheckpointListener &&
>> ((CheckpointListener) userFunction).requiresFinalCheckpoint();
>> }
>> }
>>
>>
>> I think we should still keep the change ?
>>
>> Best,
>> Yun
>>
>> ------------------Original Mail ------------------
>> *Sender:*Dawid Wysakowicz <dw...@apache.org>
>> <dw...@apache.org>
>> *Send Date:*Sun Jul 18 18:44:50 2021
>> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>, Yun
>> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
>> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>> Finished
>>
>>> I think we're all really close to the same solution.
>>>
>>>
>>>
>>> I second Yun's thoughts that MAX_WATERMARK works well for time based
>>>
>>> buffering, but it does not solve flushing other operations such as e.g.
>>>
>>> count windows or batching requests in Sinks. I'd prefer to treat the
>>>
>>> finish() as a message for Operator to "flush all records". The
>>>
>>> MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
>>>
>>> don't think operators need to get a signal "stop-processing" if they
>>>
>>> don't need to flush records. The "WHEN" records are emitted, should be
>>>
>>> in control of the StreamTask, by firing timers or by processing a next
>>>
>>> record from upstream.
>>>
>>>
>>>
>>> The only difference of my previous proposal compared to Yun's is that I
>>>
>>> did not want to send the EndOfUserRecords event in case of stop w/o
>>>
>>> drain. My thinking was that we could directly go from RUNNING to
>>>
>>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
>>>
>>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
>>>
>>> timers and processing events (without calling finish() on Operator). In
>>>
>>> my initial suggestion I though we don't care about some events
>>>
>>> potentially being emitted after the savepoint was taken, as they would
>>>
>>> anyway belong to the next after FINAL, which would be discarded. I think
>>>
>>> though the proposal to suspend records processing and timers is a
>>>
>>> sensible thing to do and would go with the version that Yun put into the
>>>
>>> FLIP Wiki.
>>>
>>>
>>>
>>> What do you think Till?
>>>
>>>
>>>
>>> Best,
>>>
>>>
>>>
>>> Dawid
>>>
>>>
>>>
>>>
>>>
>>> On 16/07/2021 10:03, Yun Gao wrote:
>>>
>>>> Hi Till, Piotr
>>>> Very thanks for the comments!
>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>> I also agree with Piotr that currently they are independent mechanisms,
>>> and they are basically the same
>>>
>>>> for the event time.
>>>> For more details, first there are some difference among the three
>>> scenarios regarding the finish:
>>>
>>>> For normal finish and stop-with-savepoint --drain, the job would not be
>>> expected to be restarted,
>>>
>>>> and for stop-with-savepoint the job would be expected restart later.
>>>> Then for finish / stop-with-savepoint --drain, currently Flink would
>>> emit MAX_WATERMARK before the
>>>
>>>> EndOfPartition. Besides, as we have discussed before [1], endOfInput /
>>> finish() should also only be called
>>>
>>>> for finish / stop-with-savepoint --drain. Thus currently they always
>>> occurs at the same time. After the change,
>>>
>>>> we could emit MAX_WATERMARK before endOfInput event for the finish /
>>> stop-with-savepoint --drain cases.
>>>
>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>>>> WindowOperator close all windows and emit the results upon calling
>>>>> finish, for example?
>>>> As discussed above for stop-with-savepoint, we would always keep the
>>> window as is, and restore them after restart.
>>>
>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps it
>>> depends on the Triggers. For
>>>
>>>> event-time triggers / process time triggers, it would be reasonable to
>>> flush all the windows since logically
>>>
>>>> the time would always elapse and the window would always get triggered
>>> in a logical future. But for triggers
>>>
>>>> like CountTrigger, no matter how much time pass logically, the windows
>>> would not trigger, thus we may not
>>>
>>>> flush these windows. If there are requirements we may provide
>>> additional triggers.
>>>
>>>>> It's a bit messy and I'm not sure if this should be strengthened out?
>>> Each one of those has a little bit different semantic/meaning,
>>>
>>>>> but at the same time they are very similar. For single input operators
>>> `endInput()` and `finish()` are actually the very same thing.
>>>
>>>> Currently MAX_WATERMARK / endInput / finish indeed always happen at the
>>> same time, and for single input operators `endInput()` and `finish()`
>>>
>>>> are indeed the same thing. During the last discussion we ever mentioned
>>> this issue and at then we thought that we might deprecate `endInput()`
>>>
>>>> in the future, then we would only have endInput(int input) and
>>> finish().
>>>
>>>> Best,
>>>> Yun
>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
>>>> ------------------------------------------------------------------
>>>> From:Piotr Nowojski
>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
>>>> To:dev
>>>> Cc:Yun Gao
>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>> Finished
>>>
>>>> Hi Till,
>>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>>> 2) StreamOperator.finish says to flush all buffered events. Would a>
>>> WindowOperator close all windows and emit the results upon calling
>>>
>>>>> finish, for example?
>>>> 1) currently they are independent but parallel mechanisms. With event
>>> time, they are basically the same.
>>>
>>>> 2) it probably should for the sake of processing time windows.
>>>> Here you are touching the bit of the current design that I like the
>>> least. We basically have now three different ways of conveying very similar
>>> things:
>>>
>>>> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
>>> about processing time?)
>>>
>>>> b) endInput(), used for example by AsyncWaitOperator to flush it's
>>> internal state
>>>
>>>> c) finish(), used for example by ContinuousFileReaderOperator
>>>> It's a bit messy and I'm not sure if this should be strengthened out?
>>> Each one of those has a little bit different semantic/meaning, but at the
>>> same time they are very similar. For single input operators `endInput()`
>>> and `finish()` are actually the very same thing.
>>>
>>>> Piotrek
>>>> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
>>>> Thanks for updating the FLIP. Based on the new section about
>>>> stop-with-savepoint [--drain] I got two other questions:
>>>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>>> 2) StreamOperator.finish says to flush all buffered events. Would a
>>>> WindowOperator close all windows and emit the results upon calling
>>>> finish, for example?
>>>> Cheers,
>>>> Till
>>>> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
>>>>> Thanks a lot for your answers and clarifications Yun.
>>>>> 1+2) Agreed, this can be a future improvement if this becomes a
>>> problem.
>>>
>>>>> 3) Great, this will help a lot with understanding the FLIP.
>>>>> Cheers,
>>>>> Till
>>>>> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
>>>>> wrote:
>>>>>> Hi Till,
>>>>>> Very thanks for the review and comments!
>>>>>> 1) First I think in fact we could be able to do the computation
>>> outside
>>>
>>>>>> of the main thread,
>>>>>> and the current implementation mainly due to the computation is in
>>>>>> general fast and we
>>>>>> initially want to have a simplified first version.
>>>>>> The main requirement here is to have a constant view of the state of
>>> the
>>>
>>>>>> tasks, otherwise
>>>>>> for example if we have A -> B, if A is running when we check if we
>>> need
>>>
>>>>>> to trigger A, we will
>>>>>> mark A as have to trigger, but if A gets to finished when we check
>>> B, we
>>>
>>>>>> will also mark B as
>>>>>> have to trigger, then B will receive both rpc trigger and checkpoint
>>>>>> barrier, which would break
>>>>>> some assumption on the task side and complicate the implementation.
>>>>>> But to cope this issue, we in fact could first have a snapshot of the
>>>>>> tasks' state and then do the
>>>>>> computation, both the two step do not need to be in the main thread.
>>>>>> 2) For the computation logic, in fact currently we benefit a lot from
>>>>>> some shortcuts on all-to-all
>>>>>> edges and job vertex with all tasks running, these shortcuts could do
>>>>>> checks on the job vertex level
>>>>>> first and skip some job vertices as a whole. With this optimization
>>> we
>>>
>>>>>> have a O(V) algorithm, and the
>>>>>> current running time of the worst case for a job with 320,000 tasks
>>> is
>>>
>>>>>> less than 100ms. For
>>>>>> daily graph sizes the time would be further reduced linearly.
>>>>>> If we do the computation based on the last triggered tasks, we may
>>> not
>>>
>>>>>> easily encode this information
>>>>>> into the shortcuts on the job vertex level. And since the time seems
>>> to
>>>
>>>>>> be short, perhaps it is enough
>>>>>> to do re-computation from the scratch in consideration of the
>>> tradeoff
>>>
>>>>>> between the performance and
>>>>>> the complexity ?
>>>>>> 3) We are going to emit the EndOfInput event exactly after the
>>> finish()
>>>
>>>>>> method and before the last
>>>>>> snapshotState() method so that we could shut down the whole topology
>>> with
>>>
>>>>>> a single final checkpoint.
>>>>>> Very sorry for not include enough details for this part and I'll
>>>>>> complement the FLIP with the details on
>>>>>> the process of the final checkpoint / savepoint.
>>>>>> Best,
>>>>>> Yun
>>>>>> ------------------------------------------------------------------
>>>>>> From:Till Rohrmann
>>>>>> Send Time:2021 Jul. 14 (Wed.) 22:05
>>>>>> To:dev
>>>>>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>>>>> Finished
>>>>>> Hi everyone,
>>>>>> I am a bit late to the voting party but let me ask three questions:
>>>>>> 1) Why do we execute the trigger plan computation in the main thread
>>> if we
>>>
>>>>>> cannot guarantee that all tasks are still running when triggering the
>>>>>> checkpoint? Couldn't we do the computation in a different thread in
>>> order
>>>
>>>>>> to relieve the main thread a bit.
>>>>>> 2) The implementation of the DefaultCheckpointPlanCalculator seems
>>> to go
>>>
>>>>>> over the whole topology for every calculation. Wouldn't it be more
>>>>>> efficient to maintain the set of current tasks to trigger and check
>>>>>> whether
>>>>>> anything has changed and if so check the succeeding tasks until we
>>> have
>>>
>>>>>> found the current checkpoint trigger frontier?
>>>>>> 3) When are we going to send the endOfInput events to a downstream
>>> task?
>>>
>>>>>> If
>>>>>> this happens after we call finish on the upstream operator but before
>>>>>> snapshotState then it would be possible to shut down the whole
>>> topology
>>>
>>>>>> with a single final checkpoint. I think this part could benefit from
>>> a bit
>>>
>>>>>> more detailed description in the FLIP.
>>>>>> Cheers,
>>>>>> Till
>>>>>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
>>>>>> wrote:
>>>>>>> Hi there,
>>>>>>> Since the voting time of FLIP-147[1] has passed, I'm closing the
>>> vote
>>>
>>>>>> now.
>>>>>>> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>>>>>>> - Dawid Wysakowicz (binding)
>>>>>>> - Piotr Nowojski(binding)
>>>>>>> - Jiangang Liu (binding)
>>>>>>> - Arvid Heise (binding)
>>>>>>> - Jing Zhang (binding)
>>>>>>> - Leonard Xu (non-binding)
>>>>>>> - Guowei Ma (binding)
>>>>>>> Thus I'm happy to announce that the update to the FLIP-147 is
>>> accepted.
>>>
>>>>>>> Very thanks everyone!
>>>>>>> Best,
>>>>>>> Yun
>>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>>
>>>
>>>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
Hi Yun and Dawid,
Thanks for your comments. I do agree with your comments that finish() can
do more than MAX_WATERMARK. I guess we should then explain how
MAX_WATERMARK and finish() play together and what kind of
order guarantees we provide.
Concerning the EndOfPartitionEvent, I am not entirely sure whether it would
work in its current state because we send this event when the Task is about
to shut down if I am not mistaken. What we want to have is to bring the
StreamTasks into a state so that they shut down on the next checkpoint. For
this we need to keep the StreamTask running. In general, I am a fan of
making things explicit if possible. I think this helps maintenance and
evolvability of code. That's why I think sending an EndOfInputEvent which
is a StreamTask level event and which says that there won't be any other
records coming only control events could make sense.
I would leave the proposed optimization out of the first version. We can
still add it at a later point in time.
Cheers,
Till
On Mon, Jul 19, 2021 at 10:35 AM Dawid Wysakowicz <dw...@apache.org>
wrote:
> Personally I don't find this optimization important and I'd rather leave
> it out not to complicate the codebase further. I doubt we save much there.
> I don't have a strong opinion though.
>
> Best,
>
> Dawid
> On 19/07/2021 10:31, Yun Gao wrote:
>
> Hi,
>
> Very thanks Dawid for the thoughts!
>
> Currently I also do not have different opinions regarding this part.
> But I have one more issue to confirm: during the previous discussion we
> have discussed that for the final checkpoint case, we might have an
> optmization
> that if a task do not have operators using 2-pc, we might skip waiting for
> the
> final checkpoint (but we could not skip the savepoint). To allow users to
> express
> the logic, we have proposed to add one more method to StreamOperator &
> CheckpointListener:
>
> interface StreamOperator {
> default boolean requiresFinalCheckpoint() {
> return true;
> }
> }
>
> interface CheckpointListener {
>
> default boolean requiresFinalCheckpoint() {
> return true;
> }
> }
>
> class AbstractUdfStreamOperator {
>
> @Override
> boolean requiresFinalCheckpoint() {
> return userFunction instanceof CheckpointListener &&
> ((CheckpointListener) userFunction).requiresFinalCheckpoint();
> }
> }
>
>
> I think we should still keep the change ?
>
> Best,
> Yun
>
> ------------------Original Mail ------------------
> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> <dw...@apache.org>
> *Send Date:*Sun Jul 18 18:44:50 2021
> *Recipients:*Flink Dev <de...@flink.apache.org> <de...@flink.apache.org>, Yun
> Gao <yu...@aliyun.com.INVALID> <yu...@aliyun.com.INVALID>
> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
>> I think we're all really close to the same solution.
>>
>>
>>
>> I second Yun's thoughts that MAX_WATERMARK works well for time based
>>
>> buffering, but it does not solve flushing other operations such as e.g.
>>
>> count windows or batching requests in Sinks. I'd prefer to treat the
>>
>> finish() as a message for Operator to "flush all records". The
>>
>> MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
>>
>> don't think operators need to get a signal "stop-processing" if they
>>
>> don't need to flush records. The "WHEN" records are emitted, should be
>>
>> in control of the StreamTask, by firing timers or by processing a next
>>
>> record from upstream.
>>
>>
>>
>> The only difference of my previous proposal compared to Yun's is that I
>>
>> did not want to send the EndOfUserRecords event in case of stop w/o
>>
>> drain. My thinking was that we could directly go from RUNNING to
>>
>> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
>>
>> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
>>
>> timers and processing events (without calling finish() on Operator). In
>>
>> my initial suggestion I though we don't care about some events
>>
>> potentially being emitted after the savepoint was taken, as they would
>>
>> anyway belong to the next after FINAL, which would be discarded. I think
>>
>> though the proposal to suspend records processing and timers is a
>>
>> sensible thing to do and would go with the version that Yun put into the
>>
>> FLIP Wiki.
>>
>>
>>
>> What do you think Till?
>>
>>
>>
>> Best,
>>
>>
>>
>> Dawid
>>
>>
>>
>>
>>
>> On 16/07/2021 10:03, Yun Gao wrote:
>>
>> > Hi Till, Piotr
>>
>> >
>>
>> > Very thanks for the comments!
>>
>> >
>>
>> >> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>
>> > I also agree with Piotr that currently they are independent mechanisms,
>> and they are basically the same
>>
>> > for the event time.
>>
>> >
>>
>> > For more details, first there are some difference among the three
>> scenarios regarding the finish:
>>
>> > For normal finish and stop-with-savepoint --drain, the job would not be
>> expected to be restarted,
>>
>> > and for stop-with-savepoint the job would be expected restart later.
>>
>> >
>>
>> > Then for finish / stop-with-savepoint --drain, currently Flink would
>> emit MAX_WATERMARK before the
>>
>> > EndOfPartition. Besides, as we have discussed before [1], endOfInput /
>> finish() should also only be called
>>
>> > for finish / stop-with-savepoint --drain. Thus currently they always
>> occurs at the same time. After the change,
>>
>> > we could emit MAX_WATERMARK before endOfInput event for the finish /
>> stop-with-savepoint --drain cases.
>>
>> >
>>
>> >> 2) StreamOperator.finish says to flush all buffered events. Would a
>>
>> >> WindowOperator close all windows and emit the results upon calling
>>
>> >> finish, for example?
>>
>> > As discussed above for stop-with-savepoint, we would always keep the
>> window as is, and restore them after restart.
>>
>> > Then for the finish / stop-with-savepoint --drain, I think perhaps it
>> depends on the Triggers. For
>>
>> > event-time triggers / process time triggers, it would be reasonable to
>> flush all the windows since logically
>>
>> > the time would always elapse and the window would always get triggered
>> in a logical future. But for triggers
>>
>> > like CountTrigger, no matter how much time pass logically, the windows
>> would not trigger, thus we may not
>>
>> > flush these windows. If there are requirements we may provide
>> additional triggers.
>>
>> >
>>
>> >> It's a bit messy and I'm not sure if this should be strengthened out?
>> Each one of those has a little bit different semantic/meaning,
>>
>> >> but at the same time they are very similar. For single input operators
>> `endInput()` and `finish()` are actually the very same thing.
>>
>> > Currently MAX_WATERMARK / endInput / finish indeed always happen at the
>> same time, and for single input operators `endInput()` and `finish()`
>>
>> > are indeed the same thing. During the last discussion we ever mentioned
>> this issue and at then we thought that we might deprecate `endInput()`
>>
>> > in the future, then we would only have endInput(int input) and
>> finish().
>>
>> >
>>
>> > Best,
>>
>> > Yun
>>
>> >
>>
>> >
>>
>> > [1] https://issues.apache.org/jira/browse/FLINK-21132
>>
>> >
>>
>> >
>>
>> >
>>
>> > ------------------------------------------------------------------
>>
>> > From:Piotr Nowojski
>>
>> > Send Time:2021 Jul. 16 (Fri.) 13:48
>>
>> > To:dev
>>
>> > Cc:Yun Gao
>>
>> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>> Finished
>>
>> >
>>
>> > Hi Till,
>>
>> >
>>
>> >> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>
>> >>
>>
>> >> 2) StreamOperator.finish says to flush all buffered events. Would a>
>> WindowOperator close all windows and emit the results upon calling
>>
>> >> finish, for example?
>>
>> > 1) currently they are independent but parallel mechanisms. With event
>> time, they are basically the same.
>>
>> > 2) it probably should for the sake of processing time windows.
>>
>> >
>>
>> > Here you are touching the bit of the current design that I like the
>> least. We basically have now three different ways of conveying very similar
>> things:
>>
>> > a) sending `MAX_WATERMARK`, used by event time WindowOperator (what
>> about processing time?)
>>
>> > b) endInput(), used for example by AsyncWaitOperator to flush it's
>> internal state
>>
>> > c) finish(), used for example by ContinuousFileReaderOperator
>>
>> >
>>
>> > It's a bit messy and I'm not sure if this should be strengthened out?
>> Each one of those has a little bit different semantic/meaning, but at the
>> same time they are very similar. For single input operators `endInput()`
>> and `finish()` are actually the very same thing.
>>
>> >
>>
>> > Piotrek
>>
>> > czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
>>
>> > Thanks for updating the FLIP. Based on the new section about
>>
>> > stop-with-savepoint [--drain] I got two other questions:
>>
>> >
>>
>> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>
>> >
>>
>> > 2) StreamOperator.finish says to flush all buffered events. Would a
>>
>> > WindowOperator close all windows and emit the results upon calling
>>
>> > finish, for example?
>>
>> >
>>
>> > Cheers,
>>
>> > Till
>>
>> >
>>
>> > On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
>>
>> >
>>
>> > > Thanks a lot for your answers and clarifications Yun.
>>
>> > >
>>
>> > > 1+2) Agreed, this can be a future improvement if this becomes a
>> problem.
>>
>> > >
>>
>> > > 3) Great, this will help a lot with understanding the FLIP.
>>
>> > >
>>
>> > > Cheers,
>>
>> > > Till
>>
>> > >
>>
>> > > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
>>
>> > > wrote:
>>
>> > >
>>
>> > >> Hi Till,
>>
>> > >>
>>
>> > >> Very thanks for the review and comments!
>>
>> > >>
>>
>> > >> 1) First I think in fact we could be able to do the computation
>> outside
>>
>> > >> of the main thread,
>>
>> > >> and the current implementation mainly due to the computation is in
>>
>> > >> general fast and we
>>
>> > >> initially want to have a simplified first version.
>>
>> > >>
>>
>> > >> The main requirement here is to have a constant view of the state of
>> the
>>
>> > >> tasks, otherwise
>>
>> > >> for example if we have A -> B, if A is running when we check if we
>> need
>>
>> > >> to trigger A, we will
>>
>> > >> mark A as have to trigger, but if A gets to finished when we check
>> B, we
>>
>> > >> will also mark B as
>>
>> > >> have to trigger, then B will receive both rpc trigger and checkpoint
>>
>> > >> barrier, which would break
>>
>> > >> some assumption on the task side and complicate the implementation.
>>
>> > >>
>>
>> > >> But to cope this issue, we in fact could first have a snapshot of the
>>
>> > >> tasks' state and then do the
>>
>> > >> computation, both the two step do not need to be in the main thread.
>>
>> > >>
>>
>> > >> 2) For the computation logic, in fact currently we benefit a lot from
>>
>> > >> some shortcuts on all-to-all
>>
>> > >> edges and job vertex with all tasks running, these shortcuts could do
>>
>> > >> checks on the job vertex level
>>
>> > >> first and skip some job vertices as a whole. With this optimization
>> we
>>
>> > >> have a O(V) algorithm, and the
>>
>> > >> current running time of the worst case for a job with 320,000 tasks
>> is
>>
>> > >> less than 100ms. For
>>
>> > >> daily graph sizes the time would be further reduced linearly.
>>
>> > >>
>>
>> > >> If we do the computation based on the last triggered tasks, we may
>> not
>>
>> > >> easily encode this information
>>
>> > >> into the shortcuts on the job vertex level. And since the time seems
>> to
>>
>> > >> be short, perhaps it is enough
>>
>> > >> to do re-computation from the scratch in consideration of the
>> tradeoff
>>
>> > >> between the performance and
>>
>> > >> the complexity ?
>>
>> > >>
>>
>> > >> 3) We are going to emit the EndOfInput event exactly after the
>> finish()
>>
>> > >> method and before the last
>>
>> > >> snapshotState() method so that we could shut down the whole topology
>> with
>>
>> > >> a single final checkpoint.
>>
>> > >> Very sorry for not include enough details for this part and I'll
>>
>> > >> complement the FLIP with the details on
>>
>> > >> the process of the final checkpoint / savepoint.
>>
>> > >>
>>
>> > >> Best,
>>
>> > >> Yun
>>
>> > >>
>>
>> > >>
>>
>> > >>
>>
>> > >> ------------------------------------------------------------------
>>
>> > >> From:Till Rohrmann
>>
>> > >> Send Time:2021 Jul. 14 (Wed.) 22:05
>>
>> > >> To:dev
>>
>> > >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>>
>> > >> Finished
>>
>> > >>
>>
>> > >> Hi everyone,
>>
>> > >>
>>
>> > >> I am a bit late to the voting party but let me ask three questions:
>>
>> > >>
>>
>> > >> 1) Why do we execute the trigger plan computation in the main thread
>> if we
>>
>> > >> cannot guarantee that all tasks are still running when triggering the
>>
>> > >> checkpoint? Couldn't we do the computation in a different thread in
>> order
>>
>> > >> to relieve the main thread a bit.
>>
>> > >>
>>
>> > >> 2) The implementation of the DefaultCheckpointPlanCalculator seems
>> to go
>>
>> > >> over the whole topology for every calculation. Wouldn't it be more
>>
>> > >> efficient to maintain the set of current tasks to trigger and check
>>
>> > >> whether
>>
>> > >> anything has changed and if so check the succeeding tasks until we
>> have
>>
>> > >> found the current checkpoint trigger frontier?
>>
>> > >>
>>
>> > >> 3) When are we going to send the endOfInput events to a downstream
>> task?
>>
>> > >> If
>>
>> > >> this happens after we call finish on the upstream operator but before
>>
>> > >> snapshotState then it would be possible to shut down the whole
>> topology
>>
>> > >> with a single final checkpoint. I think this part could benefit from
>> a bit
>>
>> > >> more detailed description in the FLIP.
>>
>> > >>
>>
>> > >> Cheers,
>>
>> > >> Till
>>
>> > >>
>>
>> > >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
>>
>> > >> wrote:
>>
>> > >>
>>
>> > >> > Hi there,
>>
>> > >> >
>>
>> > >> > Since the voting time of FLIP-147[1] has passed, I'm closing the
>> vote
>>
>> > >> now.
>>
>> > >> >
>>
>> > >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>>
>> > >> >
>>
>> > >> > - Dawid Wysakowicz (binding)
>>
>> > >> > - Piotr Nowojski(binding)
>>
>> > >> > - Jiangang Liu (binding)
>>
>> > >> > - Arvid Heise (binding)
>>
>> > >> > - Jing Zhang (binding)
>>
>> > >> > - Leonard Xu (non-binding)
>>
>> > >> > - Guowei Ma (binding)
>>
>> > >> >
>>
>> > >> > Thus I'm happy to announce that the update to the FLIP-147 is
>> accepted.
>>
>> > >> >
>>
>> > >> > Very thanks everyone!
>>
>> > >> >
>>
>> > >> > Best,
>>
>> > >> > Yun
>>
>> > >> >
>>
>> > >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>
>> > >>
>>
>> > >>
>>
>> >
>>
>>
>>
>>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
Personally I don't find this optimization important and I'd rather leave
it out not to complicate the codebase further. I doubt we save much
there. I don't have a strong opinion though.
Best,
Dawid
On 19/07/2021 10:31, Yun Gao wrote:
> Hi,
>
> Very thanks Dawid for the thoughts!
>
> Currently I also do not have different opinions regarding this part.
> But I have one more issue to confirm: during the previous discussion we
> have discussed that for the final checkpoint case, we might have an
> optmization
> that if a task do not have operators using 2-pc, we might skip waiting
> for the
> final checkpoint (but we could not skip the savepoint). To allow users
> to express
> the logic, we have proposed to add one more method to StreamOperator &
> CheckpointListener:
>
> |interface| |StreamOperator {|
> | ||default| |boolean| |requiresFinalCheckpoint() {|
> | ||return| |true||;|
> | ||}|
> |}|
>
> |interface| |CheckpointListener {|
>
> | ||default| |boolean| |requiresFinalCheckpoint() {|
> | ||return| |true||;|
> | ||}|
> |}|
>
> |class| |AbstractUdfStreamOperator {|
> | |
> | ||@Override|
> | ||boolean| |requiresFinalCheckpoint() {|
> | ||return| |userFunction ||instanceof| |CheckpointListener &&|
> | ||((CheckpointListener) userFunction).requiresFinalCheckpoint();|
> | ||}|
> |}|
> |
> |
>
> I think we should still keep the change ?
>
> Best,
> Yun
>
> ------------------Original Mail ------------------
> *Sender:*Dawid Wysakowicz <dw...@apache.org>
> *Send Date:*Sun Jul 18 18:44:50 2021
> *Recipients:*Flink Dev <de...@flink.apache.org>, Yun Gao
> <yu...@aliyun.com.INVALID>
> *Subject:*Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
> Tasks Finished
>
> I think we're all really close to the same solution.
>
>
>
> I second Yun's thoughts that MAX_WATERMARK works well for time
> based
>
> buffering, but it does not solve flushing other operations
> such as e.g.
>
> count windows or batching requests in Sinks. I'd prefer to
> treat the
>
> finish() as a message for Operator to "flush all records". The
>
> MAX_WATERMARK in my opinion is mostly for backwards
> compatibility imo. I
>
> don't think operators need to get a signal "stop-processing"
> if they
>
> don't need to flush records. The "WHEN" records are emitted,
> should be
>
> in control of the StreamTask, by firing timers or by
> processing a next
>
> record from upstream.
>
>
>
> The only difference of my previous proposal compared to Yun's
> is that I
>
> did not want to send the EndOfUserRecords event in case of
> stop w/o
>
> drain. My thinking was that we could directly go from RUNNING to
>
> WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
>
> EndOfUserRecordsEvent with an additional flag and e.g. stop firing
>
> timers and processing events (without calling finish() on
> Operator). In
>
> my initial suggestion I though we don't care about some events
>
> potentially being emitted after the savepoint was taken, as
> they would
>
> anyway belong to the next after FINAL, which would be
> discarded. I think
>
> though the proposal to suspend records processing and timers is a
>
> sensible thing to do and would go with the version that Yun
> put into the
>
> FLIP Wiki.
>
>
>
> What do you think Till?
>
>
>
> Best,
>
>
>
> Dawid
>
>
>
>
>
> On 16/07/2021 10:03, Yun Gao wrote:
>
> > Hi Till, Piotr
>
> >
>
> > Very thanks for the comments!
>
> >
>
> >> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> > I also agree with Piotr that currently they are independent
> mechanisms, and they are basically the same
>
> > for the event time.
>
> >
>
> > For more details, first there are some difference among the
> three scenarios regarding the finish:
>
> > For normal finish and stop-with-savepoint --drain, the job
> would not be expected to be restarted,
>
> > and for stop-with-savepoint the job would be expected
> restart later.
>
> >
>
> > Then for finish / stop-with-savepoint --drain, currently
> Flink would emit MAX_WATERMARK before the
>
> > EndOfPartition. Besides, as we have discussed before [1],
> endOfInput / finish() should also only be called
>
> > for finish / stop-with-savepoint --drain. Thus currently
> they always occurs at the same time. After the change,
>
> > we could emit MAX_WATERMARK before endOfInput event for the
> finish / stop-with-savepoint --drain cases.
>
> >
>
> >> 2) StreamOperator.finish says to flush all buffered events.
> Would a
>
> >> WindowOperator close all windows and emit the results upon
> calling
>
> >> finish, for example?
>
> > As discussed above for stop-with-savepoint, we would always
> keep the window as is, and restore them after restart.
>
> > Then for the finish / stop-with-savepoint --drain, I think
> perhaps it depends on the Triggers. For
>
> > event-time triggers / process time triggers, it would be
> reasonable to flush all the windows since logically
>
> > the time would always elapse and the window would always get
> triggered in a logical future. But for triggers
>
> > like CountTrigger, no matter how much time pass logically,
> the windows would not trigger, thus we may not
>
> > flush these windows. If there are requirements we may
> provide additional triggers.
>
> >
>
> >> It's a bit messy and I'm not sure if this should be
> strengthened out? Each one of those has a little bit different
> semantic/meaning,
>
> >> but at the same time they are very similar. For single
> input operators `endInput()` and `finish()` are actually the
> very same thing.
>
> > Currently MAX_WATERMARK / endInput / finish indeed always
> happen at the same time, and for single input operators
> `endInput()` and `finish()`
>
> > are indeed the same thing. During the last discussion we
> ever mentioned this issue and at then we thought that we might
> deprecate `endInput()`
>
> > in the future, then we would only have endInput(int input)
> and finish().
>
> >
>
> > Best,
>
> > Yun
>
> >
>
> >
>
> > [1] https://issues.apache.org/jira/browse/FLINK-21132
>
> >
>
> >
>
> >
>
> >
> ------------------------------------------------------------------
>
> > From:Piotr Nowojski
>
> > Send Time:2021 Jul. 16 (Fri.) 13:48
>
> > To:dev
>
> > Cc:Yun Gao
>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint
> After Tasks Finished
>
> >
>
> > Hi Till,
>
> >
>
> >> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> >>
>
> >> 2) StreamOperator.finish says to flush all buffered events.
> Would a> WindowOperator close all windows and emit the results
> upon calling
>
> >> finish, for example?
>
> > 1) currently they are independent but parallel mechanisms.
> With event time, they are basically the same.
>
> > 2) it probably should for the sake of processing time windows.
>
> >
>
> > Here you are touching the bit of the current design that I
> like the least. We basically have now three different ways of
> conveying very similar things:
>
> > a) sending `MAX_WATERMARK`, used by event time
> WindowOperator (what about processing time?)
>
> > b) endInput(), used for example by AsyncWaitOperator to
> flush it's internal state
>
> > c) finish(), used for example by ContinuousFileReaderOperator
>
> >
>
> > It's a bit messy and I'm not sure if this should be
> strengthened out? Each one of those has a little bit different
> semantic/meaning, but at the same time they are very similar.
> For single input operators `endInput()` and `finish()` are
> actually the very same thing.
>
> >
>
> > Piotrek
>
> > czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
>
> > Thanks for updating the FLIP. Based on the new section about
>
> > stop-with-savepoint [--drain] I got two other questions:
>
> >
>
> > 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> >
>
> > 2) StreamOperator.finish says to flush all buffered events.
> Would a
>
> > WindowOperator close all windows and emit the results upon
> calling
>
> > finish, for example?
>
> >
>
> > Cheers,
>
> > Till
>
> >
>
> > On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
>
> >
>
> > > Thanks a lot for your answers and clarifications Yun.
>
> > >
>
> > > 1+2) Agreed, this can be a future improvement if this
> becomes a problem.
>
> > >
>
> > > 3) Great, this will help a lot with understanding the FLIP.
>
> > >
>
> > > Cheers,
>
> > > Till
>
> > >
>
> > > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
>
> > > wrote:
>
> > >
>
> > >> Hi Till,
>
> > >>
>
> > >> Very thanks for the review and comments!
>
> > >>
>
> > >> 1) First I think in fact we could be able to do the
> computation outside
>
> > >> of the main thread,
>
> > >> and the current implementation mainly due to the
> computation is in
>
> > >> general fast and we
>
> > >> initially want to have a simplified first version.
>
> > >>
>
> > >> The main requirement here is to have a constant view of
> the state of the
>
> > >> tasks, otherwise
>
> > >> for example if we have A -> B, if A is running when we
> check if we need
>
> > >> to trigger A, we will
>
> > >> mark A as have to trigger, but if A gets to finished when
> we check B, we
>
> > >> will also mark B as
>
> > >> have to trigger, then B will receive both rpc trigger and
> checkpoint
>
> > >> barrier, which would break
>
> > >> some assumption on the task side and complicate the
> implementation.
>
> > >>
>
> > >> But to cope this issue, we in fact could first have a
> snapshot of the
>
> > >> tasks' state and then do the
>
> > >> computation, both the two step do not need to be in the
> main thread.
>
> > >>
>
> > >> 2) For the computation logic, in fact currently we
> benefit a lot from
>
> > >> some shortcuts on all-to-all
>
> > >> edges and job vertex with all tasks running, these
> shortcuts could do
>
> > >> checks on the job vertex level
>
> > >> first and skip some job vertices as a whole. With this
> optimization we
>
> > >> have a O(V) algorithm, and the
>
> > >> current running time of the worst case for a job with
> 320,000 tasks is
>
> > >> less than 100ms. For
>
> > >> daily graph sizes the time would be further reduced linearly.
>
> > >>
>
> > >> If we do the computation based on the last triggered
> tasks, we may not
>
> > >> easily encode this information
>
> > >> into the shortcuts on the job vertex level. And since the
> time seems to
>
> > >> be short, perhaps it is enough
>
> > >> to do re-computation from the scratch in consideration of
> the tradeoff
>
> > >> between the performance and
>
> > >> the complexity ?
>
> > >>
>
> > >> 3) We are going to emit the EndOfInput event exactly
> after the finish()
>
> > >> method and before the last
>
> > >> snapshotState() method so that we could shut down the
> whole topology with
>
> > >> a single final checkpoint.
>
> > >> Very sorry for not include enough details for this part
> and I'll
>
> > >> complement the FLIP with the details on
>
> > >> the process of the final checkpoint / savepoint.
>
> > >>
>
> > >> Best,
>
> > >> Yun
>
> > >>
>
> > >>
>
> > >>
>
> > >>
> ------------------------------------------------------------------
>
> > >> From:Till Rohrmann
>
> > >> Send Time:2021 Jul. 14 (Wed.) 22:05
>
> > >> To:dev
>
> > >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint
> After Tasks
>
> > >> Finished
>
> > >>
>
> > >> Hi everyone,
>
> > >>
>
> > >> I am a bit late to the voting party but let me ask three
> questions:
>
> > >>
>
> > >> 1) Why do we execute the trigger plan computation in the
> main thread if we
>
> > >> cannot guarantee that all tasks are still running when
> triggering the
>
> > >> checkpoint? Couldn't we do the computation in a different
> thread in order
>
> > >> to relieve the main thread a bit.
>
> > >>
>
> > >> 2) The implementation of the
> DefaultCheckpointPlanCalculator seems to go
>
> > >> over the whole topology for every calculation. Wouldn't
> it be more
>
> > >> efficient to maintain the set of current tasks to trigger
> and check
>
> > >> whether
>
> > >> anything has changed and if so check the succeeding tasks
> until we have
>
> > >> found the current checkpoint trigger frontier?
>
> > >>
>
> > >> 3) When are we going to send the endOfInput events to a
> downstream task?
>
> > >> If
>
> > >> this happens after we call finish on the upstream
> operator but before
>
> > >> snapshotState then it would be possible to shut down the
> whole topology
>
> > >> with a single final checkpoint. I think this part could
> benefit from a bit
>
> > >> more detailed description in the FLIP.
>
> > >>
>
> > >> Cheers,
>
> > >> Till
>
> > >>
>
> > >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
>
> > >> wrote:
>
> > >>
>
> > >> > Hi there,
>
> > >> >
>
> > >> > Since the voting time of FLIP-147[1] has passed, I'm
> closing the vote
>
> > >> now.
>
> > >> >
>
> > >> > There were seven +1 votes ( 6 / 7 are bindings) and no
> -1 votes:
>
> > >> >
>
> > >> > - Dawid Wysakowicz (binding)
>
> > >> > - Piotr Nowojski(binding)
>
> > >> > - Jiangang Liu (binding)
>
> > >> > - Arvid Heise (binding)
>
> > >> > - Jing Zhang (binding)
>
> > >> > - Leonard Xu (non-binding)
>
> > >> > - Guowei Ma (binding)
>
> > >> >
>
> > >> > Thus I'm happy to announce that the update to the
> FLIP-147 is accepted.
>
> > >> >
>
> > >> > Very thanks everyone!
>
> > >> >
>
> > >> > Best,
>
> > >> > Yun
>
> > >> >
>
> > >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>
> > >>
>
> > >>
>
> >
>
>
>
Re: Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi,
Very thanks Dawid for the thoughts!
Currently I also do not have different opinions regarding this part.
But I have one more issue to confirm: during the previous discussion we
have discussed that for the final checkpoint case, we might have an optmization
that if a task do not have operators using 2-pc, we might skip waiting for the
final checkpoint (but we could not skip the savepoint). To allow users to express
the logic, we have proposed to add one more method to StreamOperator & CheckpointListener:
interface StreamOperator {
default boolean requiresFinalCheckpoint() {
return true;
}
}
interface CheckpointListener {
default boolean requiresFinalCheckpoint() {
return true;
}
}
class AbstractUdfStreamOperator {
@Override
boolean requiresFinalCheckpoint() {
return userFunction instanceof CheckpointListener &&
((CheckpointListener) userFunction).requiresFinalCheckpoint();
}
}
I think we should still keep the change ?
Best,
Yun
------------------Original Mail ------------------
Sender:Dawid Wysakowicz <dw...@apache.org>
Send Date:Sun Jul 18 18:44:50 2021
Recipients:Flink Dev <de...@flink.apache.org>, Yun Gao <yu...@aliyun.com.INVALID>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
I think we're all really close to the same solution.
I second Yun's thoughts that MAX_WATERMARK works well for time based
buffering, but it does not solve flushing other operations such as e.g.
count windows or batching requests in Sinks. I'd prefer to treat the
finish() as a message for Operator to "flush all records". The
MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
don't think operators need to get a signal "stop-processing" if they
don't need to flush records. The "WHEN" records are emitted, should be
in control of the StreamTask, by firing timers or by processing a next
record from upstream.
The only difference of my previous proposal compared to Yun's is that I
did not want to send the EndOfUserRecords event in case of stop w/o
drain. My thinking was that we could directly go from RUNNING to
WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
EndOfUserRecordsEvent with an additional flag and e.g. stop firing
timers and processing events (without calling finish() on Operator). In
my initial suggestion I though we don't care about some events
potentially being emitted after the savepoint was taken, as they would
anyway belong to the next after FINAL, which would be discarded. I think
though the proposal to suspend records processing and timers is a
sensible thing to do and would go with the version that Yun put into the
FLIP Wiki.
What do you think Till?
Best,
Dawid
On 16/07/2021 10:03, Yun Gao wrote:
> Hi Till, Piotr
>
> Very thanks for the comments!
>
>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> I also agree with Piotr that currently they are independent mechanisms, and they are basically the same
> for the event time.
>
> For more details, first there are some difference among the three scenarios regarding the finish:
> For normal finish and stop-with-savepoint --drain, the job would not be expected to be restarted,
> and for stop-with-savepoint the job would be expected restart later.
>
> Then for finish / stop-with-savepoint --drain, currently Flink would emit MAX_WATERMARK before the
> EndOfPartition. Besides, as we have discussed before [1], endOfInput / finish() should also only be called
> for finish / stop-with-savepoint --drain. Thus currently they always occurs at the same time. After the change,
> we could emit MAX_WATERMARK before endOfInput event for the finish / stop-with-savepoint --drain cases.
>
>> 2) StreamOperator.finish says to flush all buffered events. Would a
>> WindowOperator close all windows and emit the results upon calling
>> finish, for example?
> As discussed above for stop-with-savepoint, we would always keep the window as is, and restore them after restart.
> Then for the finish / stop-with-savepoint --drain, I think perhaps it depends on the Triggers. For
> event-time triggers / process time triggers, it would be reasonable to flush all the windows since logically
> the time would always elapse and the window would always get triggered in a logical future. But for triggers
> like CountTrigger, no matter how much time pass logically, the windows would not trigger, thus we may not
> flush these windows. If there are requirements we may provide additional triggers.
>
>> It's a bit messy and I'm not sure if this should be strengthened out? Each one of those has a little bit different semantic/meaning,
>> but at the same time they are very similar. For single input operators `endInput()` and `finish()` are actually the very same thing.
> Currently MAX_WATERMARK / endInput / finish indeed always happen at the same time, and for single input operators `endInput()` and `finish()`
> are indeed the same thing. During the last discussion we ever mentioned this issue and at then we thought that we might deprecate `endInput()`
> in the future, then we would only have endInput(int input) and finish().
>
> Best,
> Yun
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-21132
>
>
>
> ------------------------------------------------------------------
> From:Piotr Nowojski
> Send Time:2021 Jul. 16 (Fri.) 13:48
> To:dev
> Cc:Yun Gao
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
>
> Hi Till,
>
>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>
>> 2) StreamOperator.finish says to flush all buffered events. Would a> WindowOperator close all windows and emit the results upon calling
>> finish, for example?
> 1) currently they are independent but parallel mechanisms. With event time, they are basically the same.
> 2) it probably should for the sake of processing time windows.
>
> Here you are touching the bit of the current design that I like the least. We basically have now three different ways of conveying very similar things:
> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about processing time?)
> b) endInput(), used for example by AsyncWaitOperator to flush it's internal state
> c) finish(), used for example by ContinuousFileReaderOperator
>
> It's a bit messy and I'm not sure if this should be strengthened out? Each one of those has a little bit different semantic/meaning, but at the same time they are very similar. For single input operators `endInput()` and `finish()` are actually the very same thing.
>
> Piotrek
> czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a):
> Thanks for updating the FLIP. Based on the new section about
> stop-with-savepoint [--drain] I got two other questions:
>
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
>
> Cheers,
> Till
>
> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote:
>
> > Thanks a lot for your answers and clarifications Yun.
> >
> > 1+2) Agreed, this can be a future improvement if this becomes a problem.
> >
> > 3) Great, this will help a lot with understanding the FLIP.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao
> > wrote:
> >
> >> Hi Till,
> >>
> >> Very thanks for the review and comments!
> >>
> >> 1) First I think in fact we could be able to do the computation outside
> >> of the main thread,
> >> and the current implementation mainly due to the computation is in
> >> general fast and we
> >> initially want to have a simplified first version.
> >>
> >> The main requirement here is to have a constant view of the state of the
> >> tasks, otherwise
> >> for example if we have A -> B, if A is running when we check if we need
> >> to trigger A, we will
> >> mark A as have to trigger, but if A gets to finished when we check B, we
> >> will also mark B as
> >> have to trigger, then B will receive both rpc trigger and checkpoint
> >> barrier, which would break
> >> some assumption on the task side and complicate the implementation.
> >>
> >> But to cope this issue, we in fact could first have a snapshot of the
> >> tasks' state and then do the
> >> computation, both the two step do not need to be in the main thread.
> >>
> >> 2) For the computation logic, in fact currently we benefit a lot from
> >> some shortcuts on all-to-all
> >> edges and job vertex with all tasks running, these shortcuts could do
> >> checks on the job vertex level
> >> first and skip some job vertices as a whole. With this optimization we
> >> have a O(V) algorithm, and the
> >> current running time of the worst case for a job with 320,000 tasks is
> >> less than 100ms. For
> >> daily graph sizes the time would be further reduced linearly.
> >>
> >> If we do the computation based on the last triggered tasks, we may not
> >> easily encode this information
> >> into the shortcuts on the job vertex level. And since the time seems to
> >> be short, perhaps it is enough
> >> to do re-computation from the scratch in consideration of the tradeoff
> >> between the performance and
> >> the complexity ?
> >>
> >> 3) We are going to emit the EndOfInput event exactly after the finish()
> >> method and before the last
> >> snapshotState() method so that we could shut down the whole topology with
> >> a single final checkpoint.
> >> Very sorry for not include enough details for this part and I'll
> >> complement the FLIP with the details on
> >> the process of the final checkpoint / savepoint.
> >>
> >> Best,
> >> Yun
> >>
> >>
> >>
> >> ------------------------------------------------------------------
> >> From:Till Rohrmann
> >> Send Time:2021 Jul. 14 (Wed.) 22:05
> >> To:dev
> >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >> Finished
> >>
> >> Hi everyone,
> >>
> >> I am a bit late to the voting party but let me ask three questions:
> >>
> >> 1) Why do we execute the trigger plan computation in the main thread if we
> >> cannot guarantee that all tasks are still running when triggering the
> >> checkpoint? Couldn't we do the computation in a different thread in order
> >> to relieve the main thread a bit.
> >>
> >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
> >> over the whole topology for every calculation. Wouldn't it be more
> >> efficient to maintain the set of current tasks to trigger and check
> >> whether
> >> anything has changed and if so check the succeeding tasks until we have
> >> found the current checkpoint trigger frontier?
> >>
> >> 3) When are we going to send the endOfInput events to a downstream task?
> >> If
> >> this happens after we call finish on the upstream operator but before
> >> snapshotState then it would be possible to shut down the whole topology
> >> with a single final checkpoint. I think this part could benefit from a bit
> >> more detailed description in the FLIP.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao
> >> wrote:
> >>
> >> > Hi there,
> >> >
> >> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
> >> now.
> >> >
> >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >> >
> >> > - Dawid Wysakowicz (binding)
> >> > - Piotr Nowojski(binding)
> >> > - Jiangang Liu (binding)
> >> > - Arvid Heise (binding)
> >> > - Jing Zhang (binding)
> >> > - Leonard Xu (non-binding)
> >> > - Guowei Ma (binding)
> >> >
> >> > Thus I'm happy to announce that the update to the FLIP-147 is accepted.
> >> >
> >> > Very thanks everyone!
> >> >
> >> > Best,
> >> > Yun
> >> >
> >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>
> >>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
I think we're all really close to the same solution.
I second Yun's thoughts that MAX_WATERMARK works well for time based
buffering, but it does not solve flushing other operations such as e.g.
count windows or batching requests in Sinks. I'd prefer to treat the
finish() as a message for Operator to "flush all records". The
MAX_WATERMARK in my opinion is mostly for backwards compatibility imo. I
don't think operators need to get a signal "stop-processing" if they
don't need to flush records. The "WHEN" records are emitted, should be
in control of the StreamTask, by firing timers or by processing a next
record from upstream.
The only difference of my previous proposal compared to Yun's is that I
did not want to send the EndOfUserRecords event in case of stop w/o
drain. My thinking was that we could directly go from RUNNING to
WAITING_FOR_FINAL_CP on EndOfPartitionEvent. I agree we could emit
EndOfUserRecordsEvent with an additional flag and e.g. stop firing
timers and processing events (without calling finish() on Operator). In
my initial suggestion I though we don't care about some events
potentially being emitted after the savepoint was taken, as they would
anyway belong to the next after FINAL, which would be discarded. I think
though the proposal to suspend records processing and timers is a
sensible thing to do and would go with the version that Yun put into the
FLIP Wiki.
What do you think Till?
Best,
Dawid
On 16/07/2021 10:03, Yun Gao wrote:
> Hi Till, Piotr
>
> Very thanks for the comments!
>
>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
> I also agree with Piotr that currently they are independent mechanisms, and they are basically the same
> for the event time.
>
> For more details, first there are some difference among the three scenarios regarding the finish:
> For normal finish and stop-with-savepoint --drain, the job would not be expected to be restarted,
> and for stop-with-savepoint the job would be expected restart later.
>
> Then for finish / stop-with-savepoint --drain, currently Flink would emit MAX_WATERMARK before the
> EndOfPartition. Besides, as we have discussed before [1], endOfInput / finish() should also only be called
> for finish / stop-with-savepoint --drain. Thus currently they always occurs at the same time. After the change,
> we could emit MAX_WATERMARK before endOfInput event for the finish / stop-with-savepoint --drain cases.
>
>> 2) StreamOperator.finish says to flush all buffered events. Would a
>> WindowOperator close all windows and emit the results upon calling
>> finish, for example?
> As discussed above for stop-with-savepoint, we would always keep the window as is, and restore them after restart.
> Then for the finish / stop-with-savepoint --drain, I think perhaps it depends on the Triggers. For
> event-time triggers / process time triggers, it would be reasonable to flush all the windows since logically
> the time would always elapse and the window would always get triggered in a logical future. But for triggers
> like CountTrigger, no matter how much time pass logically, the windows would not trigger, thus we may not
> flush these windows. If there are requirements we may provide additional triggers.
>
>> It's a bit messy and I'm not sure if this should be strengthened out? Each one of those has a little bit different semantic/meaning,
>> but at the same time they are very similar. For single input operators `endInput()` and `finish()` are actually the very same thing.
> Currently MAX_WATERMARK / endInput / finish indeed always happen at the same time, and for single input operators `endInput()` and `finish()`
> are indeed the same thing. During the last discussion we ever mentioned this issue and at then we thought that we might deprecate `endInput()`
> in the future, then we would only have endInput(int input) and finish().
>
> Best,
> Yun
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-21132
>
>
>
> ------------------------------------------------------------------
> From:Piotr Nowojski <pn...@apache.org>
> Send Time:2021 Jul. 16 (Fri.) 13:48
> To:dev <de...@flink.apache.org>
> Cc:Yun Gao <yu...@aliyun.com>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
>
> Hi Till,
>
>> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>>
>> 2) StreamOperator.finish says to flush all buffered events. Would a> WindowOperator close all windows and emit the results upon calling
>> finish, for example?
> 1) currently they are independent but parallel mechanisms. With event time, they are basically the same.
> 2) it probably should for the sake of processing time windows.
>
> Here you are touching the bit of the current design that I like the least. We basically have now three different ways of conveying very similar things:
> a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about processing time?)
> b) endInput(), used for example by AsyncWaitOperator to flush it's internal state
> c) finish(), used for example by ContinuousFileReaderOperator
>
> It's a bit messy and I'm not sure if this should be strengthened out? Each one of those has a little bit different semantic/meaning, but at the same time they are very similar. For single input operators `endInput()` and `finish()` are actually the very same thing.
>
> Piotrek
> czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
> Thanks for updating the FLIP. Based on the new section about
> stop-with-savepoint [--drain] I got two other questions:
>
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
>
> Cheers,
> Till
>
> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org> wrote:
>
> > Thanks a lot for your answers and clarifications Yun.
> >
> > 1+2) Agreed, this can be a future improvement if this becomes a problem.
> >
> > 3) Great, this will help a lot with understanding the FLIP.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > wrote:
> >
> >> Hi Till,
> >>
> >> Very thanks for the review and comments!
> >>
> >> 1) First I think in fact we could be able to do the computation outside
> >> of the main thread,
> >> and the current implementation mainly due to the computation is in
> >> general fast and we
> >> initially want to have a simplified first version.
> >>
> >> The main requirement here is to have a constant view of the state of the
> >> tasks, otherwise
> >> for example if we have A -> B, if A is running when we check if we need
> >> to trigger A, we will
> >> mark A as have to trigger, but if A gets to finished when we check B, we
> >> will also mark B as
> >> have to trigger, then B will receive both rpc trigger and checkpoint
> >> barrier, which would break
> >> some assumption on the task side and complicate the implementation.
> >>
> >> But to cope this issue, we in fact could first have a snapshot of the
> >> tasks' state and then do the
> >> computation, both the two step do not need to be in the main thread.
> >>
> >> 2) For the computation logic, in fact currently we benefit a lot from
> >> some shortcuts on all-to-all
> >> edges and job vertex with all tasks running, these shortcuts could do
> >> checks on the job vertex level
> >> first and skip some job vertices as a whole. With this optimization we
> >> have a O(V) algorithm, and the
> >> current running time of the worst case for a job with 320,000 tasks is
> >> less than 100ms. For
> >> daily graph sizes the time would be further reduced linearly.
> >>
> >> If we do the computation based on the last triggered tasks, we may not
> >> easily encode this information
> >> into the shortcuts on the job vertex level. And since the time seems to
> >> be short, perhaps it is enough
> >> to do re-computation from the scratch in consideration of the tradeoff
> >> between the performance and
> >> the complexity ?
> >>
> >> 3) We are going to emit the EndOfInput event exactly after the finish()
> >> method and before the last
> >> snapshotState() method so that we could shut down the whole topology with
> >> a single final checkpoint.
> >> Very sorry for not include enough details for this part and I'll
> >> complement the FLIP with the details on
> >> the process of the final checkpoint / savepoint.
> >>
> >> Best,
> >> Yun
> >>
> >>
> >>
> >> ------------------------------------------------------------------
> >> From:Till Rohrmann <tr...@apache.org>
> >> Send Time:2021 Jul. 14 (Wed.) 22:05
> >> To:dev <de...@flink.apache.org>
> >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >> Finished
> >>
> >> Hi everyone,
> >>
> >> I am a bit late to the voting party but let me ask three questions:
> >>
> >> 1) Why do we execute the trigger plan computation in the main thread if we
> >> cannot guarantee that all tasks are still running when triggering the
> >> checkpoint? Couldn't we do the computation in a different thread in order
> >> to relieve the main thread a bit.
> >>
> >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
> >> over the whole topology for every calculation. Wouldn't it be more
> >> efficient to maintain the set of current tasks to trigger and check
> >> whether
> >> anything has changed and if so check the succeeding tasks until we have
> >> found the current checkpoint trigger frontier?
> >>
> >> 3) When are we going to send the endOfInput events to a downstream task?
> >> If
> >> this happens after we call finish on the upstream operator but before
> >> snapshotState then it would be possible to shut down the whole topology
> >> with a single final checkpoint. I think this part could benefit from a bit
> >> more detailed description in the FLIP.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> >> wrote:
> >>
> >> > Hi there,
> >> >
> >> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
> >> now.
> >> >
> >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >> >
> >> > - Dawid Wysakowicz (binding)
> >> > - Piotr Nowojski(binding)
> >> > - Jiangang Liu (binding)
> >> > - Arvid Heise (binding)
> >> > - Jing Zhang (binding)
> >> > - Leonard Xu (non-binding)
> >> > - Guowei Ma (binding)
> >> >
> >> > Thus I'm happy to announce that the update to the FLIP-147 is accepted.
> >> >
> >> > Very thanks everyone!
> >> >
> >> > Best,
> >> > Yun
> >> >
> >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>
> >>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Till, Piotr
Very thanks for the comments!
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
I also agree with Piotr that currently they are independent mechanisms, and they are basically the same
for the event time.
For more details, first there are some difference among the three scenarios regarding the finish:
For normal finish and stop-with-savepoint --drain, the job would not be expected to be restarted,
and for stop-with-savepoint the job would be expected restart later.
Then for finish / stop-with-savepoint --drain, currently Flink would emit MAX_WATERMARK before the
EndOfPartition. Besides, as we have discussed before [1], endOfInput / finish() should also only be called
for finish / stop-with-savepoint --drain. Thus currently they always occurs at the same time. After the change,
we could emit MAX_WATERMARK before endOfInput event for the finish / stop-with-savepoint --drain cases.
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
As discussed above for stop-with-savepoint, we would always keep the window as is, and restore them after restart.
Then for the finish / stop-with-savepoint --drain, I think perhaps it depends on the Triggers. For
event-time triggers / process time triggers, it would be reasonable to flush all the windows since logically
the time would always elapse and the window would always get triggered in a logical future. But for triggers
like CountTrigger, no matter how much time pass logically, the windows would not trigger, thus we may not
flush these windows. If there are requirements we may provide additional triggers.
> It's a bit messy and I'm not sure if this should be strengthened out? Each one of those has a little bit different semantic/meaning,
> but at the same time they are very similar. For single input operators `endInput()` and `finish()` are actually the very same thing.
Currently MAX_WATERMARK / endInput / finish indeed always happen at the same time, and for single input operators `endInput()` and `finish()`
are indeed the same thing. During the last discussion we ever mentioned this issue and at then we thought that we might deprecate `endInput()`
in the future, then we would only have endInput(int input) and finish().
Best,
Yun
[1] https://issues.apache.org/jira/browse/FLINK-21132
------------------------------------------------------------------
From:Piotr Nowojski <pn...@apache.org>
Send Time:2021 Jul. 16 (Fri.) 13:48
To:dev <de...@flink.apache.org>
Cc:Yun Gao <yu...@aliyun.com>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Till,
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a> WindowOperator close all windows and emit the results upon calling
> finish, for example?
1) currently they are independent but parallel mechanisms. With event time, they are basically the same.
2) it probably should for the sake of processing time windows.
Here you are touching the bit of the current design that I like the least. We basically have now three different ways of conveying very similar things:
a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about processing time?)
b) endInput(), used for example by AsyncWaitOperator to flush it's internal state
c) finish(), used for example by ContinuousFileReaderOperator
It's a bit messy and I'm not sure if this should be strengthened out? Each one of those has a little bit different semantic/meaning, but at the same time they are very similar. For single input operators `endInput()` and `finish()` are actually the very same thing.
Piotrek
czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
Thanks for updating the FLIP. Based on the new section about
stop-with-savepoint [--drain] I got two other questions:
1) Does endOfInput entail sending of the MAX_WATERMARK?
2) StreamOperator.finish says to flush all buffered events. Would a
WindowOperator close all windows and emit the results upon calling
finish, for example?
Cheers,
Till
On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org> wrote:
> Thanks a lot for your answers and clarifications Yun.
>
> 1+2) Agreed, this can be a future improvement if this becomes a problem.
>
> 3) Great, this will help a lot with understanding the FLIP.
>
> Cheers,
> Till
>
> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> wrote:
>
>> Hi Till,
>>
>> Very thanks for the review and comments!
>>
>> 1) First I think in fact we could be able to do the computation outside
>> of the main thread,
>> and the current implementation mainly due to the computation is in
>> general fast and we
>> initially want to have a simplified first version.
>>
>> The main requirement here is to have a constant view of the state of the
>> tasks, otherwise
>> for example if we have A -> B, if A is running when we check if we need
>> to trigger A, we will
>> mark A as have to trigger, but if A gets to finished when we check B, we
>> will also mark B as
>> have to trigger, then B will receive both rpc trigger and checkpoint
>> barrier, which would break
>> some assumption on the task side and complicate the implementation.
>>
>> But to cope this issue, we in fact could first have a snapshot of the
>> tasks' state and then do the
>> computation, both the two step do not need to be in the main thread.
>>
>> 2) For the computation logic, in fact currently we benefit a lot from
>> some shortcuts on all-to-all
>> edges and job vertex with all tasks running, these shortcuts could do
>> checks on the job vertex level
>> first and skip some job vertices as a whole. With this optimization we
>> have a O(V) algorithm, and the
>> current running time of the worst case for a job with 320,000 tasks is
>> less than 100ms. For
>> daily graph sizes the time would be further reduced linearly.
>>
>> If we do the computation based on the last triggered tasks, we may not
>> easily encode this information
>> into the shortcuts on the job vertex level. And since the time seems to
>> be short, perhaps it is enough
>> to do re-computation from the scratch in consideration of the tradeoff
>> between the performance and
>> the complexity ?
>>
>> 3) We are going to emit the EndOfInput event exactly after the finish()
>> method and before the last
>> snapshotState() method so that we could shut down the whole topology with
>> a single final checkpoint.
>> Very sorry for not include enough details for this part and I'll
>> complement the FLIP with the details on
>> the process of the final checkpoint / savepoint.
>>
>> Best,
>> Yun
>>
>>
>>
>> ------------------------------------------------------------------
>> From:Till Rohrmann <tr...@apache.org>
>> Send Time:2021 Jul. 14 (Wed.) 22:05
>> To:dev <de...@flink.apache.org>
>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>> Finished
>>
>> Hi everyone,
>>
>> I am a bit late to the voting party but let me ask three questions:
>>
>> 1) Why do we execute the trigger plan computation in the main thread if we
>> cannot guarantee that all tasks are still running when triggering the
>> checkpoint? Couldn't we do the computation in a different thread in order
>> to relieve the main thread a bit.
>>
>> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
>> over the whole topology for every calculation. Wouldn't it be more
>> efficient to maintain the set of current tasks to trigger and check
>> whether
>> anything has changed and if so check the succeeding tasks until we have
>> found the current checkpoint trigger frontier?
>>
>> 3) When are we going to send the endOfInput events to a downstream task?
>> If
>> this happens after we call finish on the upstream operator but before
>> snapshotState then it would be possible to shut down the whole topology
>> with a single final checkpoint. I think this part could benefit from a bit
>> more detailed description in the FLIP.
>>
>> Cheers,
>> Till
>>
>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
>> wrote:
>>
>> > Hi there,
>> >
>> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
>> now.
>> >
>> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>> >
>> > - Dawid Wysakowicz (binding)
>> > - Piotr Nowojski(binding)
>> > - Jiangang Liu (binding)
>> > - Arvid Heise (binding)
>> > - Jing Zhang (binding)
>> > - Leonard Xu (non-binding)
>> > - Guowei Ma (binding)
>> >
>> > Thus I'm happy to announce that the update to the FLIP-147 is accepted.
>> >
>> > Very thanks everyone!
>> >
>> > Best,
>> > Yun
>> >
>> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>
>>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Piotr Nowojski <pn...@apache.org>.
Hi Till,
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
1) currently they are independent but parallel mechanisms. With event time,
they are basically the same.
2) it probably should for the sake of processing time windows.
Here you are touching the bit of the current design that I like the least.
We basically have now three different ways of conveying very similar things:
a) sending `MAX_WATERMARK`, used by event time WindowOperator (what about
processing time?)
b) endInput(), used for example by AsyncWaitOperator to flush it's internal
state
c) finish(), used for example by ContinuousFileReaderOperator
It's a bit messy and I'm not sure if this should be strengthened out? Each
one of those has a little bit different semantic/meaning, but at the same
time they are very similar. For single input operators `endInput()` and
`finish()` are actually the very same thing.
Piotrek
czw., 15 lip 2021 o 16:47 Till Rohrmann <tr...@apache.org> napisał(a):
> Thanks for updating the FLIP. Based on the new section about
> stop-with-savepoint [--drain] I got two other questions:
>
> 1) Does endOfInput entail sending of the MAX_WATERMARK?
>
> 2) StreamOperator.finish says to flush all buffered events. Would a
> WindowOperator close all windows and emit the results upon calling
> finish, for example?
>
> Cheers,
> Till
>
> On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Thanks a lot for your answers and clarifications Yun.
> >
> > 1+2) Agreed, this can be a future improvement if this becomes a problem.
> >
> > 3) Great, this will help a lot with understanding the FLIP.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> > wrote:
> >
> >> Hi Till,
> >>
> >> Very thanks for the review and comments!
> >>
> >> 1) First I think in fact we could be able to do the computation outside
> >> of the main thread,
> >> and the current implementation mainly due to the computation is in
> >> general fast and we
> >> initially want to have a simplified first version.
> >>
> >> The main requirement here is to have a constant view of the state of the
> >> tasks, otherwise
> >> for example if we have A -> B, if A is running when we check if we need
> >> to trigger A, we will
> >> mark A as have to trigger, but if A gets to finished when we check B, we
> >> will also mark B as
> >> have to trigger, then B will receive both rpc trigger and checkpoint
> >> barrier, which would break
> >> some assumption on the task side and complicate the implementation.
> >>
> >> But to cope this issue, we in fact could first have a snapshot of the
> >> tasks' state and then do the
> >> computation, both the two step do not need to be in the main thread.
> >>
> >> 2) For the computation logic, in fact currently we benefit a lot from
> >> some shortcuts on all-to-all
> >> edges and job vertex with all tasks running, these shortcuts could do
> >> checks on the job vertex level
> >> first and skip some job vertices as a whole. With this optimization we
> >> have a O(V) algorithm, and the
> >> current running time of the worst case for a job with 320,000 tasks is
> >> less than 100ms. For
> >> daily graph sizes the time would be further reduced linearly.
> >>
> >> If we do the computation based on the last triggered tasks, we may not
> >> easily encode this information
> >> into the shortcuts on the job vertex level. And since the time seems to
> >> be short, perhaps it is enough
> >> to do re-computation from the scratch in consideration of the tradeoff
> >> between the performance and
> >> the complexity ?
> >>
> >> 3) We are going to emit the EndOfInput event exactly after the finish()
> >> method and before the last
> >> snapshotState() method so that we could shut down the whole topology
> with
> >> a single final checkpoint.
> >> Very sorry for not include enough details for this part and I'll
> >> complement the FLIP with the details on
> >> the process of the final checkpoint / savepoint.
> >>
> >> Best,
> >> Yun
> >>
> >>
> >>
> >> ------------------------------------------------------------------
> >> From:Till Rohrmann <tr...@apache.org>
> >> Send Time:2021 Jul. 14 (Wed.) 22:05
> >> To:dev <de...@flink.apache.org>
> >> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> >> Finished
> >>
> >> Hi everyone,
> >>
> >> I am a bit late to the voting party but let me ask three questions:
> >>
> >> 1) Why do we execute the trigger plan computation in the main thread if
> we
> >> cannot guarantee that all tasks are still running when triggering the
> >> checkpoint? Couldn't we do the computation in a different thread in
> order
> >> to relieve the main thread a bit.
> >>
> >> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
> >> over the whole topology for every calculation. Wouldn't it be more
> >> efficient to maintain the set of current tasks to trigger and check
> >> whether
> >> anything has changed and if so check the succeeding tasks until we have
> >> found the current checkpoint trigger frontier?
> >>
> >> 3) When are we going to send the endOfInput events to a downstream task?
> >> If
> >> this happens after we call finish on the upstream operator but before
> >> snapshotState then it would be possible to shut down the whole topology
> >> with a single final checkpoint. I think this part could benefit from a
> bit
> >> more detailed description in the FLIP.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> >> wrote:
> >>
> >> > Hi there,
> >> >
> >> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
> >> now.
> >> >
> >> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >> >
> >> > - Dawid Wysakowicz (binding)
> >> > - Piotr Nowojski(binding)
> >> > - Jiangang Liu (binding)
> >> > - Arvid Heise (binding)
> >> > - Jing Zhang (binding)
> >> > - Leonard Xu (non-binding)
> >> > - Guowei Ma (binding)
> >> >
> >> > Thus I'm happy to announce that the update to the FLIP-147 is
> accepted.
> >> >
> >> > Very thanks everyone!
> >> >
> >> > Best,
> >> > Yun
> >> >
> >> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>
> >>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
Thanks for updating the FLIP. Based on the new section about
stop-with-savepoint [--drain] I got two other questions:
1) Does endOfInput entail sending of the MAX_WATERMARK?
2) StreamOperator.finish says to flush all buffered events. Would a
WindowOperator close all windows and emit the results upon calling
finish, for example?
Cheers,
Till
On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann <tr...@apache.org> wrote:
> Thanks a lot for your answers and clarifications Yun.
>
> 1+2) Agreed, this can be a future improvement if this becomes a problem.
>
> 3) Great, this will help a lot with understanding the FLIP.
>
> Cheers,
> Till
>
> On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
> wrote:
>
>> Hi Till,
>>
>> Very thanks for the review and comments!
>>
>> 1) First I think in fact we could be able to do the computation outside
>> of the main thread,
>> and the current implementation mainly due to the computation is in
>> general fast and we
>> initially want to have a simplified first version.
>>
>> The main requirement here is to have a constant view of the state of the
>> tasks, otherwise
>> for example if we have A -> B, if A is running when we check if we need
>> to trigger A, we will
>> mark A as have to trigger, but if A gets to finished when we check B, we
>> will also mark B as
>> have to trigger, then B will receive both rpc trigger and checkpoint
>> barrier, which would break
>> some assumption on the task side and complicate the implementation.
>>
>> But to cope this issue, we in fact could first have a snapshot of the
>> tasks' state and then do the
>> computation, both the two step do not need to be in the main thread.
>>
>> 2) For the computation logic, in fact currently we benefit a lot from
>> some shortcuts on all-to-all
>> edges and job vertex with all tasks running, these shortcuts could do
>> checks on the job vertex level
>> first and skip some job vertices as a whole. With this optimization we
>> have a O(V) algorithm, and the
>> current running time of the worst case for a job with 320,000 tasks is
>> less than 100ms. For
>> daily graph sizes the time would be further reduced linearly.
>>
>> If we do the computation based on the last triggered tasks, we may not
>> easily encode this information
>> into the shortcuts on the job vertex level. And since the time seems to
>> be short, perhaps it is enough
>> to do re-computation from the scratch in consideration of the tradeoff
>> between the performance and
>> the complexity ?
>>
>> 3) We are going to emit the EndOfInput event exactly after the finish()
>> method and before the last
>> snapshotState() method so that we could shut down the whole topology with
>> a single final checkpoint.
>> Very sorry for not include enough details for this part and I'll
>> complement the FLIP with the details on
>> the process of the final checkpoint / savepoint.
>>
>> Best,
>> Yun
>>
>>
>>
>> ------------------------------------------------------------------
>> From:Till Rohrmann <tr...@apache.org>
>> Send Time:2021 Jul. 14 (Wed.) 22:05
>> To:dev <de...@flink.apache.org>
>> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
>> Finished
>>
>> Hi everyone,
>>
>> I am a bit late to the voting party but let me ask three questions:
>>
>> 1) Why do we execute the trigger plan computation in the main thread if we
>> cannot guarantee that all tasks are still running when triggering the
>> checkpoint? Couldn't we do the computation in a different thread in order
>> to relieve the main thread a bit.
>>
>> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
>> over the whole topology for every calculation. Wouldn't it be more
>> efficient to maintain the set of current tasks to trigger and check
>> whether
>> anything has changed and if so check the succeeding tasks until we have
>> found the current checkpoint trigger frontier?
>>
>> 3) When are we going to send the endOfInput events to a downstream task?
>> If
>> this happens after we call finish on the upstream operator but before
>> snapshotState then it would be possible to shut down the whole topology
>> with a single final checkpoint. I think this part could benefit from a bit
>> more detailed description in the FLIP.
>>
>> Cheers,
>> Till
>>
>> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
>> wrote:
>>
>> > Hi there,
>> >
>> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
>> now.
>> >
>> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>> >
>> > - Dawid Wysakowicz (binding)
>> > - Piotr Nowojski(binding)
>> > - Jiangang Liu (binding)
>> > - Arvid Heise (binding)
>> > - Jing Zhang (binding)
>> > - Leonard Xu (non-binding)
>> > - Guowei Ma (binding)
>> >
>> > Thus I'm happy to announce that the update to the FLIP-147 is accepted.
>> >
>> > Very thanks everyone!
>> >
>> > Best,
>> > Yun
>> >
>> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>
>>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
Thanks a lot for your answers and clarifications Yun.
1+2) Agreed, this can be a future improvement if this becomes a problem.
3) Great, this will help a lot with understanding the FLIP.
Cheers,
Till
On Wed, Jul 14, 2021 at 5:41 PM Yun Gao <yu...@aliyun.com.invalid>
wrote:
> Hi Till,
>
> Very thanks for the review and comments!
>
> 1) First I think in fact we could be able to do the computation outside of
> the main thread,
> and the current implementation mainly due to the computation is in general
> fast and we
> initially want to have a simplified first version.
>
> The main requirement here is to have a constant view of the state of the
> tasks, otherwise
> for example if we have A -> B, if A is running when we check if we need to
> trigger A, we will
> mark A as have to trigger, but if A gets to finished when we check B, we
> will also mark B as
> have to trigger, then B will receive both rpc trigger and checkpoint
> barrier, which would break
> some assumption on the task side and complicate the implementation.
>
> But to cope this issue, we in fact could first have a snapshot of the
> tasks' state and then do the
> computation, both the two step do not need to be in the main thread.
>
> 2) For the computation logic, in fact currently we benefit a lot from some
> shortcuts on all-to-all
> edges and job vertex with all tasks running, these shortcuts could do
> checks on the job vertex level
> first and skip some job vertices as a whole. With this optimization we
> have a O(V) algorithm, and the
> current running time of the worst case for a job with 320,000 tasks is
> less than 100ms. For
> daily graph sizes the time would be further reduced linearly.
>
> If we do the computation based on the last triggered tasks, we may not
> easily encode this information
> into the shortcuts on the job vertex level. And since the time seems to be
> short, perhaps it is enough
> to do re-computation from the scratch in consideration of the tradeoff
> between the performance and
> the complexity ?
>
> 3) We are going to emit the EndOfInput event exactly after the finish()
> method and before the last
> snapshotState() method so that we could shut down the whole topology with
> a single final checkpoint.
> Very sorry for not include enough details for this part and I'll
> complement the FLIP with the details on
> the process of the final checkpoint / savepoint.
>
> Best,
> Yun
>
>
>
> ------------------------------------------------------------------
> From:Till Rohrmann <tr...@apache.org>
> Send Time:2021 Jul. 14 (Wed.) 22:05
> To:dev <de...@flink.apache.org>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
>
> Hi everyone,
>
> I am a bit late to the voting party but let me ask three questions:
>
> 1) Why do we execute the trigger plan computation in the main thread if we
> cannot guarantee that all tasks are still running when triggering the
> checkpoint? Couldn't we do the computation in a different thread in order
> to relieve the main thread a bit.
>
> 2) The implementation of the DefaultCheckpointPlanCalculator seems to go
> over the whole topology for every calculation. Wouldn't it be more
> efficient to maintain the set of current tasks to trigger and check whether
> anything has changed and if so check the succeeding tasks until we have
> found the current checkpoint trigger frontier?
>
> 3) When are we going to send the endOfInput events to a downstream task? If
> this happens after we call finish on the upstream operator but before
> snapshotState then it would be possible to shut down the whole topology
> with a single final checkpoint. I think this part could benefit from a bit
> more detailed description in the FLIP.
>
> Cheers,
> Till
>
> On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid>
> wrote:
>
> > Hi there,
> >
> > Since the voting time of FLIP-147[1] has passed, I'm closing the vote
> now.
> >
> > There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
> >
> > - Dawid Wysakowicz (binding)
> > - Piotr Nowojski(binding)
> > - Jiangang Liu (binding)
> > - Arvid Heise (binding)
> > - Jing Zhang (binding)
> > - Leonard Xu (non-binding)
> > - Guowei Ma (binding)
> >
> > Thus I'm happy to announce that the update to the FLIP-147 is accepted.
> >
> > Very thanks everyone!
> >
> > Best,
> > Yun
> >
> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>
>
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Till,
Very thanks for the review and comments!
1) First I think in fact we could be able to do the computation outside of the main thread,
and the current implementation mainly due to the computation is in general fast and we
initially want to have a simplified first version.
The main requirement here is to have a constant view of the state of the tasks, otherwise
for example if we have A -> B, if A is running when we check if we need to trigger A, we will
mark A as have to trigger, but if A gets to finished when we check B, we will also mark B as
have to trigger, then B will receive both rpc trigger and checkpoint barrier, which would break
some assumption on the task side and complicate the implementation.
But to cope this issue, we in fact could first have a snapshot of the tasks' state and then do the
computation, both the two step do not need to be in the main thread.
2) For the computation logic, in fact currently we benefit a lot from some shortcuts on all-to-all
edges and job vertex with all tasks running, these shortcuts could do checks on the job vertex level
first and skip some job vertices as a whole. With this optimization we have a O(V) algorithm, and the
current running time of the worst case for a job with 320,000 tasks is less than 100ms. For
daily graph sizes the time would be further reduced linearly.
If we do the computation based on the last triggered tasks, we may not easily encode this information
into the shortcuts on the job vertex level. And since the time seems to be short, perhaps it is enough
to do re-computation from the scratch in consideration of the tradeoff between the performance and
the complexity ?
3) We are going to emit the EndOfInput event exactly after the finish() method and before the last
snapshotState() method so that we could shut down the whole topology with a single final checkpoint.
Very sorry for not include enough details for this part and I'll complement the FLIP with the details on
the process of the final checkpoint / savepoint.
Best,
Yun
------------------------------------------------------------------
From:Till Rohrmann <tr...@apache.org>
Send Time:2021 Jul. 14 (Wed.) 22:05
To:dev <de...@flink.apache.org>
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi everyone,
I am a bit late to the voting party but let me ask three questions:
1) Why do we execute the trigger plan computation in the main thread if we
cannot guarantee that all tasks are still running when triggering the
checkpoint? Couldn't we do the computation in a different thread in order
to relieve the main thread a bit.
2) The implementation of the DefaultCheckpointPlanCalculator seems to go
over the whole topology for every calculation. Wouldn't it be more
efficient to maintain the set of current tasks to trigger and check whether
anything has changed and if so check the succeeding tasks until we have
found the current checkpoint trigger frontier?
3) When are we going to send the endOfInput events to a downstream task? If
this happens after we call finish on the upstream operator but before
snapshotState then it would be possible to shut down the whole topology
with a single final checkpoint. I think this part could benefit from a bit
more detailed description in the FLIP.
Cheers,
Till
On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid> wrote:
> Hi there,
>
> Since the voting time of FLIP-147[1] has passed, I'm closing the vote now.
>
> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>
> - Dawid Wysakowicz (binding)
> - Piotr Nowojski(binding)
> - Jiangang Liu (binding)
> - Arvid Heise (binding)
> - Jing Zhang (binding)
> - Leonard Xu (non-binding)
> - Guowei Ma (binding)
>
> Thus I'm happy to announce that the update to the FLIP-147 is accepted.
>
> Very thanks everyone!
>
> Best,
> Yun
>
> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Till Rohrmann <tr...@apache.org>.
Hi everyone,
I am a bit late to the voting party but let me ask three questions:
1) Why do we execute the trigger plan computation in the main thread if we
cannot guarantee that all tasks are still running when triggering the
checkpoint? Couldn't we do the computation in a different thread in order
to relieve the main thread a bit.
2) The implementation of the DefaultCheckpointPlanCalculator seems to go
over the whole topology for every calculation. Wouldn't it be more
efficient to maintain the set of current tasks to trigger and check whether
anything has changed and if so check the succeeding tasks until we have
found the current checkpoint trigger frontier?
3) When are we going to send the endOfInput events to a downstream task? If
this happens after we call finish on the upstream operator but before
snapshotState then it would be possible to shut down the whole topology
with a single final checkpoint. I think this part could benefit from a bit
more detailed description in the FLIP.
Cheers,
Till
On Fri, Jul 2, 2021 at 8:36 AM Yun Gao <yu...@aliyun.com.invalid> wrote:
> Hi there,
>
> Since the voting time of FLIP-147[1] has passed, I'm closing the vote now.
>
> There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
>
> - Dawid Wysakowicz (binding)
> - Piotr Nowojski(binding)
> - Jiangang Liu (binding)
> - Arvid Heise (binding)
> - Jing Zhang (binding)
> - Leonard Xu (non-binding)
> - Guowei Ma (binding)
>
> Thus I'm happy to announce that the update to the FLIP-147 is accepted.
>
> Very thanks everyone!
>
> Best,
> Yun
>
> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
[RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi there,
Since the voting time of FLIP-147[1] has passed, I'm closing the vote now.
There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
- Dawid Wysakowicz (binding)
- Piotr Nowojski(binding)
- Jiangang Liu (binding)
- Arvid Heise (binding)
- Jing Zhang (binding)
- Leonard Xu (non-binding)
- Guowei Ma (binding)
Thus I'm happy to announce that the update to the FLIP-147 is accepted.
Very thanks everyone!
Best,
Yun
[1] https://cwiki.apache.org/confluence/x/mw-ZCQ
Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Guowei Ma <gu...@gmail.com>.
+1 (binding)
Best,
Guowei
On Fri, Jul 2, 2021 at 11:56 AM Leonard Xu <xb...@gmail.com> wrote:
> +1,
>
> This may help https://issues.apache.org/jira/browse/FLINK-22764 <
> https://issues.apache.org/jira/browse/FLINK-22764>
>
> Best,
> Leonard
>
> > 在 2021年7月2日，10:17，JING ZHANG <be...@gmail.com> 写道：
> >
> > +1 (binding)
> >
> >
> > Arvid Heise <ar...@apache.org> 于2021年7月1日周四 下午3:10写道：
> >
> >> Looks good: +1 (binding)
> >>
> >> On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 <li...@gmail.com> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> Best
> >>> liujiangang
> >>>
> >>> Piotr Nowojski <pn...@apache.org> 于2021年6月29日周二 上午2:05写道：
> >>>
> >>>> +1 (binding)
> >>>>
> >>>> Piotrek
> >>>>
> >>>> pon., 28 cze 2021 o 12:48 Dawid Wysakowicz <dw...@apache.org>
> >>>> napisał(a):
> >>>>
> >>>>> +1 (binding)
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Dawid
> >>>>>
> >>>>> On 28/06/2021 10:45, Yun Gao wrote:
> >>>>>> Hi all,
> >>>>>>
> >>>>>> For FLIP-147[1] which targets at supports checkpoints after tasks
> >>>>> finished and modify operator
> >>>>>> API and implementation to ensures the commit of last piece of data,
> >>>>> since after the last vote
> >>>>>> we have more discussions[2][3] and a few updates, including changes
> >>> to
> >>>>> PublicEvolving API,
> >>>>>> I'd like to have another VOTE on the current state of the FLIP.
> >>>>>>
> >>>>>> The vote will last at least 72 hours (Jul 1st), following the
> >>> consensus
> >>>>>> voting process.
> >>>>>>
> >>>>>> thanks,
> >>>>>> Yun
> >>>>>>
> >>>>>>
> >>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >>>>>> [2]
> >>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
> >>>>>> [3]
> >>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Leonard Xu <xb...@gmail.com>.
+1,
This may help https://issues.apache.org/jira/browse/FLINK-22764 <https://issues.apache.org/jira/browse/FLINK-22764>
Best,
Leonard
> 在 2021年7月2日，10:17，JING ZHANG <be...@gmail.com> 写道：
>
> +1 (binding)
>
>
> Arvid Heise <ar...@apache.org> 于2021年7月1日周四 下午3:10写道：
>
>> Looks good: +1 (binding)
>>
>> On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 <li...@gmail.com> wrote:
>>
>>> +1 (binding)
>>>
>>> Best
>>> liujiangang
>>>
>>> Piotr Nowojski <pn...@apache.org> 于2021年6月29日周二 上午2:05写道：
>>>
>>>> +1 (binding)
>>>>
>>>> Piotrek
>>>>
>>>> pon., 28 cze 2021 o 12:48 Dawid Wysakowicz <dw...@apache.org>
>>>> napisał(a):
>>>>
>>>>> +1 (binding)
>>>>>
>>>>> Best,
>>>>>
>>>>> Dawid
>>>>>
>>>>> On 28/06/2021 10:45, Yun Gao wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> For FLIP-147[1] which targets at supports checkpoints after tasks
>>>>> finished and modify operator
>>>>>> API and implementation to ensures the commit of last piece of data,
>>>>> since after the last vote
>>>>>> we have more discussions[2][3] and a few updates, including changes
>>> to
>>>>> PublicEvolving API,
>>>>>> I'd like to have another VOTE on the current state of the FLIP.
>>>>>>
>>>>>> The vote will last at least 72 hours (Jul 1st), following the
>>> consensus
>>>>>> voting process.
>>>>>>
>>>>>> thanks,
>>>>>> Yun
>>>>>>
>>>>>>
>>>>>> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
>>>>>> [2]
>>>>>
>>>>
>>>
>> https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
>>>>>> [3]
>>>>>
>>>>
>>>
>> https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
>>>>>
>>>>>
>>>>
>>>
>>
Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by JING ZHANG <be...@gmail.com>.
+1 (binding)
Arvid Heise <ar...@apache.org> 于2021年7月1日周四 下午3:10写道：
> Looks good: +1 (binding)
>
> On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 <li...@gmail.com> wrote:
>
> > +1 (binding)
> >
> > Best
> > liujiangang
> >
> > Piotr Nowojski <pn...@apache.org> 于2021年6月29日周二 上午2:05写道：
> >
> > > +1 (binding)
> > >
> > > Piotrek
> > >
> > > pon., 28 cze 2021 o 12:48 Dawid Wysakowicz <dw...@apache.org>
> > > napisał(a):
> > >
> > > > +1 (binding)
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 28/06/2021 10:45, Yun Gao wrote:
> > > > > Hi all,
> > > > >
> > > > > For FLIP-147[1] which targets at supports checkpoints after tasks
> > > > finished and modify operator
> > > > > API and implementation to ensures the commit of last piece of data,
> > > > since after the last vote
> > > > > we have more discussions[2][3] and a few updates, including changes
> > to
> > > > PublicEvolving API,
> > > > > I'd like to have another VOTE on the current state of the FLIP.
> > > > >
> > > > > The vote will last at least 72 hours (Jul 1st), following the
> > consensus
> > > > > voting process.
> > > > >
> > > > > thanks,
> > > > > Yun
> > > > >
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > > > [2]
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
> > > > > [3]
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
> > > >
> > > >
> > >
> >
>
Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Arvid Heise <ar...@apache.org>.
Looks good: +1 (binding)
On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 <li...@gmail.com> wrote:
> +1 (binding)
>
> Best
> liujiangang
>
> Piotr Nowojski <pn...@apache.org> 于2021年6月29日周二 上午2:05写道：
>
> > +1 (binding)
> >
> > Piotrek
> >
> > pon., 28 cze 2021 o 12:48 Dawid Wysakowicz <dw...@apache.org>
> > napisał(a):
> >
> > > +1 (binding)
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 28/06/2021 10:45, Yun Gao wrote:
> > > > Hi all,
> > > >
> > > > For FLIP-147[1] which targets at supports checkpoints after tasks
> > > finished and modify operator
> > > > API and implementation to ensures the commit of last piece of data,
> > > since after the last vote
> > > > we have more discussions[2][3] and a few updates, including changes
> to
> > > PublicEvolving API,
> > > > I'd like to have another VOTE on the current state of the FLIP.
> > > >
> > > > The vote will last at least 72 hours (Jul 1st), following the
> consensus
> > > > voting process.
> > > >
> > > > thanks,
> > > > Yun
> > > >
> > > >
> > > > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > > [2]
> > >
> >
> https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
> > > > [3]
> > >
> >
> https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
> > >
> > >
> >
>
Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by 刘建刚 <li...@gmail.com>.
+1 (binding)
Best
liujiangang
Piotr Nowojski <pn...@apache.org> 于2021年6月29日周二 上午2:05写道：
> +1 (binding)
>
> Piotrek
>
> pon., 28 cze 2021 o 12:48 Dawid Wysakowicz <dw...@apache.org>
> napisał(a):
>
> > +1 (binding)
> >
> > Best,
> >
> > Dawid
> >
> > On 28/06/2021 10:45, Yun Gao wrote:
> > > Hi all,
> > >
> > > For FLIP-147[1] which targets at supports checkpoints after tasks
> > finished and modify operator
> > > API and implementation to ensures the commit of last piece of data,
> > since after the last vote
> > > we have more discussions[2][3] and a few updates, including changes to
> > PublicEvolving API,
> > > I'd like to have another VOTE on the current state of the FLIP.
> > >
> > > The vote will last at least 72 hours (Jul 1st), following the consensus
> > > voting process.
> > >
> > > thanks,
> > > Yun
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > > [2]
> >
> https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
> > > [3]
> >
> https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
> >
> >
>
Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Piotr Nowojski <pn...@apache.org>.
+1 (binding)
Piotrek
pon., 28 cze 2021 o 12:48 Dawid Wysakowicz <dw...@apache.org>
napisał(a):
> +1 (binding)
>
> Best,
>
> Dawid
>
> On 28/06/2021 10:45, Yun Gao wrote:
> > Hi all,
> >
> > For FLIP-147[1] which targets at supports checkpoints after tasks
> finished and modify operator
> > API and implementation to ensures the commit of last piece of data,
> since after the last vote
> > we have more discussions[2][3] and a few updates, including changes to
> PublicEvolving API,
> > I'd like to have another VOTE on the current state of the FLIP.
> >
> > The vote will last at least 72 hours (Jul 1st), following the consensus
> > voting process.
> >
> > thanks,
> > Yun
> >
> >
> > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> > [2]
> https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
> > [3]
> https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
>
>
Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Posted by Dawid Wysakowicz <dw...@apache.org>.
+1 (binding)
Best,
Dawid
On 28/06/2021 10:45, Yun Gao wrote:
> Hi all,
>
> For FLIP-147[1] which targets at supports checkpoints after tasks finished and modify operator
> API and implementation to ensures the commit of last piece of data, since after the last vote
> we have more discussions[2][3] and a few updates, including changes to PublicEvolving API,
> I'd like to have another VOTE on the current state of the FLIP.
>
> The vote will last at least 72 hours (Jul 1st), following the consensus
> voting process.
>
> thanks,
> Yun
>
>
> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> [2] https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
> [3] https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E