You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ga...@apache.org on 2022/07/11 12:24:32 UTC
[flink-web] branch asf-site updated (0223e1748 -> ec54aa1f2)
This is an automated email from the ASF dual-hosted git repository.
gaoyunhaii pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
from 0223e1748 Release Flink ML 2.1.0 (#556)
new 313d98723 Add blogs for FLIP-147 support checkpoints after tasks finished
new ec54aa1f2 Rebuild website
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_posts/2022-07-11-final-checkpoint-part1.md | 176 ++++++
_posts/2022-07-11-final-checkpoint-part2.md | 244 ++++++++
.../07/11/final-checkpoint-part1.html} | 302 ++++-----
content/2022/07/11/final-checkpoint-part2.html | 510 +++++++++++++++
content/blog/feed.xml | 695 ++++++++++++---------
content/blog/index.html | 75 ++-
content/blog/page10/index.html | 76 ++-
content/blog/page11/index.html | 74 ++-
content/blog/page12/index.html | 76 ++-
content/blog/page13/index.html | 80 ++-
content/blog/page14/index.html | 76 ++-
content/blog/page15/index.html | 73 ++-
content/blog/page16/index.html | 76 ++-
content/blog/page17/index.html | 76 ++-
content/blog/page18/index.html | 81 ++-
content/blog/page19/index.html | 84 ++-
content/blog/page2/index.html | 75 ++-
content/blog/page20/index.html | 50 ++
content/blog/page3/index.html | 74 ++-
content/blog/page4/index.html | 76 ++-
content/blog/page5/index.html | 78 ++-
content/blog/page6/index.html | 76 ++-
content/blog/page7/index.html | 72 ++-
content/blog/page8/index.html | 74 ++-
content/blog/page9/index.html | 76 ++-
.../checkpoint_format.png | Bin 0 -> 96886 bytes
.../checkpoint_trigger.png | Bin 0 -> 146793 bytes
.../2022-07-11-final-checkpoint/example_job.png | Bin 0 -> 26922 bytes
.../example_job_finish.png | Bin 0 -> 42969 bytes
.../2022-07-11-final-checkpoint/finish_cmp.png | Bin 0 -> 26074 bytes
.../stream_batch_cmp.png | Bin 0 -> 44100 bytes
content/index.html | 15 +-
content/zh/index.html | 15 +-
.../checkpoint_format.png | Bin 0 -> 96886 bytes
.../checkpoint_trigger.png | Bin 0 -> 146793 bytes
.../2022-07-11-final-checkpoint/example_job.png | Bin 0 -> 26922 bytes
.../example_job_finish.png | Bin 0 -> 42969 bytes
.../2022-07-11-final-checkpoint/finish_cmp.png | Bin 0 -> 26074 bytes
.../stream_batch_cmp.png | Bin 0 -> 44100 bytes
39 files changed, 2446 insertions(+), 1009 deletions(-)
create mode 100644 _posts/2022-07-11-final-checkpoint-part1.md
create mode 100644 _posts/2022-07-11-final-checkpoint-part2.md
copy content/{2020/08/19/statefun.html => 2022/07/11/final-checkpoint-part1.html} (50%)
create mode 100644 content/2022/07/11/final-checkpoint-part2.html
create mode 100644 content/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png
create mode 100644 content/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png
create mode 100644 content/img/blog/2022-07-11-final-checkpoint/example_job.png
create mode 100644 content/img/blog/2022-07-11-final-checkpoint/example_job_finish.png
create mode 100644 content/img/blog/2022-07-11-final-checkpoint/finish_cmp.png
create mode 100644 content/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png
create mode 100644 img/blog/2022-07-11-final-checkpoint/checkpoint_format.png
create mode 100644 img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png
create mode 100644 img/blog/2022-07-11-final-checkpoint/example_job.png
create mode 100644 img/blog/2022-07-11-final-checkpoint/example_job_finish.png
create mode 100644 img/blog/2022-07-11-final-checkpoint/finish_cmp.png
create mode 100644 img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png
[flink-web] 01/02: Add blogs for FLIP-147 support checkpoints after tasks finished
Posted by ga...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gaoyunhaii pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 313d98723676c865c3057145319854af3864a538
Author: Yun Gao <ga...@gmail.com>
AuthorDate: Mon Jun 6 16:01:48 2022 +0800
Add blogs for FLIP-147 support checkpoints after tasks finished
---
_posts/2022-07-11-final-checkpoint-part1.md | 176 +++++++++++++++
_posts/2022-07-11-final-checkpoint-part2.md | 244 +++++++++++++++++++++
.../checkpoint_format.png | Bin 0 -> 96886 bytes
.../checkpoint_trigger.png | Bin 0 -> 146793 bytes
.../2022-07-11-final-checkpoint/example_job.png | Bin 0 -> 26922 bytes
.../example_job_finish.png | Bin 0 -> 42969 bytes
.../2022-07-11-final-checkpoint/finish_cmp.png | Bin 0 -> 26074 bytes
.../stream_batch_cmp.png | Bin 0 -> 44100 bytes
8 files changed, 420 insertions(+)
diff --git a/_posts/2022-07-11-final-checkpoint-part1.md b/_posts/2022-07-11-final-checkpoint-part1.md
new file mode 100644
index 000000000..5eff0740e
--- /dev/null
+++ b/_posts/2022-07-11-final-checkpoint-part1.md
@@ -0,0 +1,176 @@
+---
+layout: post
+title: "FLIP-147: Support Checkpoints After Tasks Finished - Part One"
+date: 2022-07-11T00:00:00.000Z
+authors:
+- Yun Gao:
+ name: "Yun Gao"
+- Dawid Wysakowicz:
+ name: "Dawid Wysakowicz"
+- Daisy Tsang:
+ name: "Daisy Tsang"
+excerpt: This post briefly describes the motivation and changes made by the final checkpoint mechanism, including the changes to the checkpoint procedure and how tasks finish.
+
+---
+
+# Motivation
+
+Flink is a distributed processing engine for both unbounded and bounded streams of data. In recent versions,
+Flink has unified the DataStream API and the Table / SQL API to support both streaming and batch cases.
+Since most users require both types of data processing pipelines, the unification helps reduce the complexity of developing,
+operating, and maintaining consistency between streaming and batch backfilling jobs, like
+[the case for Alibaba](https://www.ververica.com/blog/apache-flinks-stream-batch-unification-powers-alibabas-11.11-in-2020).
+
+Flink provides two execution modes under the unified programming API: the streaming mode and the batch mode.
+The streaming mode processes records incrementally based on the states, thus it supports both bounded and unbounded sources.
+The batch mode works with bounded sources and usually has a better performance for bounded jobs because it executes all the
+tasks in topological order and avoids random state access by pre-sorting the input records. Although batch mode is often the
+preferred mode to process bounded jobs, streaming mode is also required for various reasons. For example, users may want to deal
+with records containing retraction or exploit the property that data is roughly sorted by event times in streaming mode
+(like the case in [Kappa+ Architecture](https://www.youtube.com/watch?t=666&v=4qSlsYogALo&feature=youtu.be)). Moreover,
+users often have mixed jobs involving both unbounded streams and bounded side-inputs, which also require streaming execution mode.
+
+<center>
+<img vspace="20" style="width:70%" src="{{site.baseurl}}/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png" />
+<p style="font-size: 0.6em;text-align:left;margin-top:-1em;margin-bottom: 4em">
+ Figure 1. A comparison of the Streaming mode and Batch mode for the example Count operator. For streaming mode, the arrived
+ elements are not sorted, the operator would read / write the state corresponding to the element for computation.
+ For batch mode, the arrived elements are first sorted as a whole and then processed.
+</p>
+</center>
+
+In streaming mode, [checkpointing](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/checkpointing/)
+is the vital mechanism in supporting exactly-once guarantees. By periodically snapshotting the
+aligned states of operators, Flink can recover from the latest checkpoint and continue execution when failover happens. However,
+previously Flink could not take checkpoints if any task gets finished. This would cause problems for jobs with both bounded and unbounded
+sources: if there are no checkpoints after the bounded part finished, the unbounded part might need to reprocess a large amount of
+records in case of a failure.
+
+Furthermore, being unable to take checkpoints with finished tasks is a problem for jobs using two-phase-commit sinks to achieve
+[end-to-end exactly-once processing](https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html).
+The two-phase-commit sinks first write data to temporary files or external transactions,
+and commit the data only after a checkpoint completes to ensure the data would not be replayed on failure. However, if a job
+contains bounded sources, committing the results would not be possible after the bounded sources finish. Also because of that,
+for bounded jobs we have no way to commit the last piece of data after the first source task finished, and previously the bounded
+jobs just ignore the uncommitted data when finishing. These behaviors caused a lot of confusion and are always asked in the user
+mailing list.
+
+Therefore, to complete the support of streaming mode for jobs using bounded sources, it is important for us to
+
+1. Support taking checkpoints with finished tasks.
+2. Furthermore, revise the process of finishing so that all the data could always be committed.
+
+The remaining blog briefly describes the changes we made to achieve the above goals. In the next blog,
+we’ll share more details on how they are implemented.
+
+# Support Checkpointing with Finished Tasks
+
+The core idea of supporting checkpoints with finished tasks is to mark the finished operators in checkpoints and skip
+executing these operators after recovery. As illustrated in Figure 2, a checkpoint is composed of the states of all
+the operators. If all the subtasks of an operator have finished, we could mark it as fully finished and skip the
+execution of this operator on startup. For other operators, their states are composed of the states of all the
+running subtasks. The states will be repartitioned on restarting and all the new subtasks restarted with the assigned states.
+
+<center>
+<img vspace="20" style="width:50%" src="{{site.baseurl}}/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png" />
+<p style="font-size: 0.6em;text-align:center;margin-top:-1em;margin-bottom: 4em">
+ Figure 2. An illustration of the extended checkpoint format.
+</p>
+</center>
+
+To support creating such a checkpoint for jobs with finished tasks, we extended the checkpoint procedure.
+Previously the checkpoint coordinator inside the JobManager first notifies all the sources to report snapshots,
+then all the sources further notify their descendants via broadcasting barrier events. Since now the sources might
+have already finished, the checkpoint coordinator would instead treat the running tasks who also do not have running
+precedent tasks as "new sources", and it notifies these tasks to initiate the checkpoints. Finally, if the subtasks of
+an operator are either finished on triggering checkpoint or have finished processing all the data on snapshotting states,
+the operator would be marked as fully finished.
+
+The changes of the checkpoint procedure are transparent to users except that for checkpoints indeed containing
+finished tasks, we disallowed adding new operators as precedents of the fully finished ones, since it would make the fully
+finished operators have running precedents after restarting, which conflicts with the design that tasks finished
+in topological order.
+
+# Revise the Process of Finishing
+
+Based on the ability to take checkpoints with finished tasks, we could then solve the issue that two-phase-commit
+operators could not commit all the data when running in streaming mode. As the background, Flink jobs
+have two ways to finish:
+
+1. All sources are bound and they processed all the input records. The job will finish after all the
+input records are processed and all the result are committed to external systems.
+2. Users execute `stop-with-savepoint [--drain]`. The job will take a savepoint and then finish. With `–-drain`, the job
+will be stopped permanently and is also required to commit all the data. However, without `--drain` the job might
+be resumed from the savepoint later, thus not all data are required to be committed, as long as the state of the data could be
+recovered from the savepoint.
+
+Let's first have a look at the case of bounded sources. To achieve end-to-end exactly-once,
+two-phase-commit operators only commit data after a checkpoint following this piece of data succeeded.
+However, previously there is no such an opportunity for the data between the last periodic checkpoint and job getting finished,
+and the data finally gets lost. Note that it is also not correct if we directly commit the data on job finished, since
+if there are failovers after that (like due to other unfinished tasks getting failed), the data will be replayed and cause duplication.
+
+The case of `stop-with-savepoint --drain` also has problems. The previous implementation first stalls the execution and
+takes a savepoint. After the savepoint succeeds, all the source tasks would stop actively. Although the savepoint seems to
+provide the opportunity to commit all the data, some processing logic is in fact executed during the job getting stopped,
+and the records produced would be discarded by mistake. For example, calling `endInput()` method for operators happens during
+the stopping phase, some operators like the async operator might still emit new records in this method.
+
+At last, although `stop-with-savepoint` without draining is not required to commit all the data, we hope the job finish process could
+be unified for all the cases to keep the code clean.
+
+To fix the remaining issues, we need to modify the process of finishing to ensure all the data getting committed for the required cases.
+An intuitive idea is to directly insert a step to the tasks’ lifecycle to wait for the next checkpoint, as shown in the left part
+of Figure 3. However, it could not solve all the issues.
+
+<center>
+<img vspace="20" style="width:90%" src="{{site.baseurl}}/img/blog/2022-07-11-final-checkpoint/finish_cmp.png" />
+<p style="font-size: 0.6em;text-align:left;margin-top:-1em;margin-bottom: 4em">
+ Figure 3. A comparison of the two options to ensure tasks committed all the data before getting finished. The first
+ option directly inserts a step in the tasks’ lifecycle to wait for the next checkpoint, which disallows the tasks to wait
+ for the same checkpoint / savepoint. The second option decouples the notification of finishing operator logic and finishing tasks,
+ thus it allows all the tasks to first process all records, then they have the chance to wait for the same checkpoint / savepoint.
+</p>
+</center>
+
+For the case of bounded sources, the intuitive idea works, but it might have performance issues in some cases:
+as exemplified in Figure 4, If there are multiple cascading tasks containing two-phase commit sinks, each task would
+wait for the next checkpoint separately, thus the job needs to wait for three more checkpoints during finishing,
+which might prolong the total execution time for a long time.
+
+<center>
+<img vspace="20" style="width:90%" src="{{site.baseurl}}/img/blog/2022-07-11-final-checkpoint/example_job.png" />
+<p style="font-size: 0.6em;text-align:center;margin-top:-1em;margin-bottom: 4em">
+ Figure 4. An example job that contains a chain of tasks containing two-phase-commit operators.
+</p>
+</center>
+
+For the case of `stop-with-savepoint [--drain]`, the intuitive idea does not work since different tasks have to
+wait for different checkpoints / savepoints, thus we could not finish the job with a specific savepoint.
+
+Therefore, we do not take the intuitive option. Instead, we decoupled *"finishing operator logic"* and *"finishing tasks"*:
+all the tasks would first finish their execution logic as a whole, including calling lifecycle methods like `endInput()`,
+then each task could wait for the next checkpoint concurrently. Besides, for stop-with-savepoint we also reverted the current
+implementation similarly: all the tasks will first finish executing the operators' logic, then they simply wait for the next savepoint
+to happen before finish. Therefore, in this way the finishing processes are unified and the data could be fully committed for all the cases.
+
+Based on this thought, as shown in the right part of Figure 3, to decoupled the process of "finishing operator logic"
+and "finishing tasks", we introduced a new `EndOfData` event. For each task, after executing all the operator logic it would first notify
+the descendants with an `EndOfData` event so that the descendants also have chances to finish executing the operator logic. Then all
+the tasks could wait for the next checkpoint or the specified savepoint concurrently to commit all the remaining data before getting finished.
+
+At last, it is also worthy to mention we have clarified and renamed the `close()` and `dispose()` methods in the operators’ lifecycle.
+The two methods are in fact different since `close()` is only called when the task finishes normally and dispose() is called in both
+cases of normal finishing and failover. However, this was not clear from their names. Therefore, we rename the two methods to `finish()` and `close()`:
+
+- `finish()` marks the termination of the operator and no more records are allowed after `finish()` is called. It should
+ only be called when sources are finished or when the `-–drain` parameter is specified.
+- `close()` is used to do cleanup and release all the held resources.
+
+# Conclusion
+
+By supporting the checkpoints after tasks finished and revising the process of finishing, we can support checkpoints for jobs with
+both bounded and unbounded sources, and ensure the bounded job gets all output records committed before it finishes. The motivation
+behind this change is to ensure data consistency, results completeness, and failure recovery if there are bounded sources in the pipeline.
+The final checkpoint mechanism was first implemented in Flink 1.14 and enabled by default in Flink 1.15. If you have any questions,
+please feel free to start a discussion or report an issue in the dev or user mailing list.
diff --git a/_posts/2022-07-11-final-checkpoint-part2.md b/_posts/2022-07-11-final-checkpoint-part2.md
new file mode 100644
index 000000000..c6eda82e5
--- /dev/null
+++ b/_posts/2022-07-11-final-checkpoint-part2.md
@@ -0,0 +1,244 @@
+---
+layout: post
+title: "FLIP-147: Support Checkpoints After Tasks Finished - Part Two"
+date: 2022-07-11T00:00:00.000Z
+authors:
+- Yun Gao:
+ name: "Yun Gao"
+- Dawid Wysakowicz:
+ name: "Dawid Wysakowicz"
+- Daisy Tsang:
+ name: "Daisy Tsang"
+excerpt: This post presents more details on the changes on the checkpoint procedure and task finish process made by the final checkpoint mechanism.
+
+---
+
+In the [first part]({{site.baseurl}}/2022/06/01/final-checkpoint-part1.html) of this blog,
+we have briefly introduced the work to support checkpoints after tasks get
+finished and revised the process of finishing. In this part we will present more details on the implementation,
+including how we support checkpoints with finished tasks and the revised protocol of the finish process.
+
+# Implementation of support Checkpointing with Finished Tasks
+
+As described in part one,
+to support checkpoints after some tasks are finished, the core idea is to mark
+the finished operators in checkpoints and skip executing these operators after recovery. To implement this idea,
+we enhanced the checkpointing procedure to generate the flag and use the flag on recovery. This section presents
+more details on the process of taking checkpoints with finished tasks and recovery from such checkpoints.
+
+Previously, checkpointing only worked when all tasks were running. As shown in the Figure 1, in this case the
+checkpoint coordinator first notify all the source tasks, and then the source tasks further notify the
+downstream tasks to take snapshots via barrier events. Similarly, if there are finished tasks, we need to
+find the new "source" tasks to initiate the checkpoint, namely those tasks that are still running but have
+no running precedent tasks. CheckpointCoordinator does the computation atomically at the JobManager side
+based on the latest states recorded in the execution graph.
+
+There might be race conditions when triggering tasks: when the checkpoint coordinator
+decides to trigger one task and starts emitting the RPC, it is possible that the task is just finished and
+reporting the FINISHED status to JobManager. In this case, the RPC message would fail and the checkpoint would be aborted.
+
+<center>
+<img vspace="20" style="width:50%" src="{{site.baseurl}}/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png" />
+<p style="font-size: 0.6em">
+ Figure 1. The tasks chosen as the new sources when taking checkpoint with finished tasks. The principle is to
+ choose the running tasks whose precedent tasks are all finished.
+</p>
+</center>
+
+In order to keep track of the finish status of each operator, we need to extend the checkpoint format.
+A checkpoint consists of the states of all the stateful operators, and the state of one operator consists of the
+entries from all its parallel instances. Note that the concept of Task is not reflected in the checkpoint. Task
+is more of a physical execution container that drives the behavior of operators. It is not well-defined across
+multiple executions of the same job since job upgrades might modify the operators contained in one task.
+Therefore, the finished status should also be attached to the operators.
+
+As shown in the Figure 2, operators could be classified into three types according to their finished status:
+
+1. Fully finished: If all the instances of an operator are finished, we could view the logic of the operators as
+fully executed and we should skip the execution of the operator after recovery. We need to store a special flag for this
+kind of operator.
+2. Partially finished: If only some instances of an operator are finished, then we still need to continue executing the
+remaining logic of this operator. As a whole we could view the state of the operator as the set of entries collected from all the
+running instances, which represents the remaining workload for this operator.
+3. No finished instances: In this case, the state of the operator is the same as the one taken when no tasks are finished.
+
+<center>
+<img vspace="20" style="width:50%" src="{{site.baseurl}}/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png" />
+<p style="font-size: 0.6em">
+ Figure 2. An illustration of the extended checkpoint format.
+</p>
+</center>
+
+If the job is later restored from a checkpoint taken with finished tasks, we would skip executing all the logic for fully
+finished operators, and execute normally for the operators with no finished instances.
+
+However, this would be a bit complex for the partially finished operators. The state of partially finished operators would be
+redistributed to all the instances, similar to rescaling when the parallelism is changed. Among all the types of states that
+Flink offers, the [keyed state](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-keyed-state)
+and [operator state](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-operator-state)
+with even-split redistribution would work normally, but the
+[broadcast state](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#broadcast-state) and
+[operator state with union redistribution](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-operator-state)
+would be affected for the following reasons:
+
+1. The broadcast state always replicates the state of the first subtask to the other subtasks. If the first subtask is finished,
+an empty state would be distributed and the operator would run from scratch, which is not correct.
+2. The operator state with union distribution merges the states of all the subtasks and then sends the merged state to all the
+subtasks. Based on this behavior, some operators may choose one subtask to store a shared value and after restarting this value will
+be distributed to all the subtasks. However, if this chosen task is finished, the state would be lost.
+
+These two issues would not occur when rescaling since there would be no finished tasks in that scenario. To address
+these issues, we chose one of the running subtasks instead to acquire the current state for the broadcast state. For the operator
+state with union redistribution, we have to collect the states of all the subtasks to maintain the semantics. Thus, currently we
+abort the checkpoint if parts of subtasks finished for operators using this kind of state.
+
+In principle, you should be able to modify your job (which changes the dataflow graph) and restore from a previous checkpoint. That said,
+there are certain graph modifications that are not supported. These kinds of changes include adding a new operator as the precedent of a fully finished
+one. Flink would check for such modifications and throw exceptions while restoring.
+
+# The Revised Process of Finishing
+
+As described in the part one, based on the ability to take checkpoints with finished tasks, we revised the process of finishing
+so that we could always commit all the data for two-phase-commit sinks. We’ll show the detailed protocol of the finished process in this
+section.
+
+## How did Jobs in Flink Finish Before?
+
+A job might finish in two ways: all sources finish or users execute
+[`stop-with-savepoint [--drain]`](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint).
+Let’s first have a look at the detailed process of finishing before FLIP-147.
+
+### When sources finish
+
+If all the sources are bounded, The job will finish after all the input records are processed and all the result are
+committed to external systems. In this case, the sources would first
+emit a `MAX_WATERMARK` (`Long.MAX_VALUE`) and then start to terminate the task. On termination, a task would call `endOfInput()`,
+`close()` and `dispose()` for all the operators, then emit an `EndOfPartitionEvent` to the downstream tasks. The intermediate tasks
+would start terminating after receiving an `EndOfPartitionEvent` from all the input channels, and this process will continue
+until the last task is finished.
+
+```
+1. Source operators emit MAX_WATERMARK
+2. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event-time timers
+ b. Emit MAX_WATERMARK
+3. Source tasks finished
+ a. endInput(inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+4. On received EndOfPartitionEvent for non-source tasks
+ a. endInput(int inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+```
+
+### When users execute stop-with-savepoint [--drain]
+
+Users could execute the command stop-with-savepoint [--drain] for both bounded and unbounded jobs to trigger jobs to finish.
+In this case, Flink first triggers a synchronous savepoint and all the tasks would stall after seeing the synchronous
+savepoint. If the savepoint succeeds, all the source operators would finish actively and the job would finish the same as the above scenario.
+
+```
+1. Trigger a savepoint
+2. Sources received savepoint trigger RPC
+ a. If with –-drain
+ i. source operators emit MAX_WATERMARK
+ b. Source emits savepoint barrier
+3. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event times
+ b. Emit MAX_WATERMARK
+4. On received savepoint barrier for non-source operators
+ a. The task blocks till the savepoint succeed
+5. Finish the source tasks actively
+ a. If with –-drain
+ ii. endInput(inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+6. On received EndOfPartitionEvent for non-source tasks
+ a. If with –-drain
+ i. endInput(int inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+```
+
+A parameter `–-drain` is supported with `stop-with-savepoint`: if not specified, the job is expected to resume from this savepoint,
+otherwise the job is expected to terminate permanently. Thus we only emit `MAX_WATERMARK` to trigger all the event timers and call
+`endInput()` in the latter case.
+
+## Revise the Finishing Steps
+
+As described in part one, after revising the process of finishing, we have decoupled the process of "finishing operator logic"
+and "finishing task" by introducing a new `EndOfData` event. After the revision each task will first
+notify the descendants with an `EndOfData` event after executing all the logic
+so that the descendants also have chances to finish executing the operator logic, then
+all the tasks could wait for the next checkpoint or the specified savepoint concurrently to commit all the remaining data.
+This section will present the detailed protocol of the revised process. Since we have renamed
+`close()` /`dispose()` to `finish()` / `close()`, we’ll stick to the new terminologies in the following description.
+
+The revised process of finishing is shown as follows:
+
+```
+1. Source tasks finished due to no more records or stop-with-savepoint.
+ a. if no more records or stop-with-savepoint –-drain
+ i. source operators emit MAX_WATERMARK
+ ii. endInput(inputId) for all the operators
+ iii. finish() for all the operators
+ iv. emit EndOfData[isDrain = true] event
+ b. else if stop-with-savepoint
+ i. emit EndOfData[isDrain = false] event
+ c. Wait for the next checkpoint / the savepoint after operator finished complete
+ d. close() for all the operators
+ e. Emit EndOfPartitionEvent
+ f. Task cleanup
+2. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event times
+ b. Emit MAX_WATERMARK
+3. On received EndOfData for non-source tasks
+ a. If isDrain
+ i. endInput(int inputId) for all the operators
+ ii. finish() for all the operators
+ b. Emit EndOfData[isDrain = the flag value of the received event]
+4. On received EndOfPartitionEvent for non-source tasks
+ a. Wait for the next checkpoint / the savepoint after operator finished complete
+ b. close() for all the operators
+ c. Emit EndOfPartitionEvent
+ d. Task cleanup
+```
+
+<center>
+<img vspace="20" style="width:60%" src="{{site.baseurl}}/img/blog/2022-07-11-final-checkpoint/example_job_finish.png" />
+<p style="font-size: 0.6em">
+ Figure 3. An example job of the revised process of finishing.
+</p>
+</center>
+
+An example of the process of job finishing is shown in Figure 3.
+
+Let's first have a look at the example that all the source tasks are bounded.
+If Task `C` finishes after processing all the records, it first emits the max-watermark, then finishes the operators and emits
+the `EndOfData` event. After that, it waits for the next checkpoint to complete and then emits the `EndOfPartitionEvent`.
+
+Task `D` finishes all the operators right after receiving the `EndOfData` event. Since any checkpoints taken after operators finish
+can commit all the pending records and be the final checkpoint, Task `D`’s final checkpoint would be the same as Task `C`’s since
+the barrier must be emitted after the `EndOfData` event.
+
+Task `E` is a bit different in that it has two inputs. Task `A` might continue to run for a while and, thus, Task `E` needs to wait
+until it receives an `EndOfData` event also from the other input before finishing operators and its final checkpoint might be different.
+
+On the other hand, when using `stop-with-savepoint [--drain]`, the process is similar except that all the tasks need to wait for the exact
+savepoint before finishing instead of just any checkpoints. Moreover, since both Task `C` and Task `A` would finish at the same time,
+Task `E` would also be able to wait for this particular savepoint before finishing.
+
+# Conclusion
+
+In this part we have presented more details of how the checkpoints are taken with finished tasks and the revised process
+of finishing. We hope the details could provide more insights of the thoughts and implementations for this part of work. Still, if you
+have any questions, please feel free to start a discussion or report an issue in the dev or user mailing list.
\ No newline at end of file
diff --git a/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png b/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png
new file mode 100644
index 000000000..98cb22ca3
Binary files /dev/null and b/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png differ
diff --git a/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png b/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png
new file mode 100644
index 000000000..4e2012524
Binary files /dev/null and b/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png differ
diff --git a/img/blog/2022-07-11-final-checkpoint/example_job.png b/img/blog/2022-07-11-final-checkpoint/example_job.png
new file mode 100644
index 000000000..4996050b5
Binary files /dev/null and b/img/blog/2022-07-11-final-checkpoint/example_job.png differ
diff --git a/img/blog/2022-07-11-final-checkpoint/example_job_finish.png b/img/blog/2022-07-11-final-checkpoint/example_job_finish.png
new file mode 100644
index 000000000..730a918f9
Binary files /dev/null and b/img/blog/2022-07-11-final-checkpoint/example_job_finish.png differ
diff --git a/img/blog/2022-07-11-final-checkpoint/finish_cmp.png b/img/blog/2022-07-11-final-checkpoint/finish_cmp.png
new file mode 100644
index 000000000..2b634594e
Binary files /dev/null and b/img/blog/2022-07-11-final-checkpoint/finish_cmp.png differ
diff --git a/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png b/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png
new file mode 100644
index 000000000..6c50e6938
Binary files /dev/null and b/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png differ
[flink-web] 02/02: Rebuild website
Posted by ga...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gaoyunhaii pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit ec54aa1f210d131f82396daae36a5f015698d656
Author: Yun Gao <ga...@gmail.com>
AuthorDate: Mon Jul 11 20:21:07 2022 +0800
Rebuild website
---
content/2022/07/11/final-checkpoint-part1.html | 447 +++++++++++++
content/2022/07/11/final-checkpoint-part2.html | 510 +++++++++++++++
content/blog/feed.xml | 695 ++++++++++++---------
content/blog/index.html | 75 ++-
content/blog/page10/index.html | 76 ++-
content/blog/page11/index.html | 74 ++-
content/blog/page12/index.html | 76 ++-
content/blog/page13/index.html | 80 ++-
content/blog/page14/index.html | 76 ++-
content/blog/page15/index.html | 73 ++-
content/blog/page16/index.html | 76 ++-
content/blog/page17/index.html | 76 ++-
content/blog/page18/index.html | 81 ++-
content/blog/page19/index.html | 84 ++-
content/blog/page2/index.html | 75 ++-
content/blog/page20/index.html | 50 ++
content/blog/page3/index.html | 74 ++-
content/blog/page4/index.html | 76 ++-
content/blog/page5/index.html | 78 ++-
content/blog/page6/index.html | 76 ++-
content/blog/page7/index.html | 72 ++-
content/blog/page8/index.html | 74 ++-
content/blog/page9/index.html | 76 ++-
.../checkpoint_format.png | Bin 0 -> 96886 bytes
.../checkpoint_trigger.png | Bin 0 -> 146793 bytes
.../2022-07-11-final-checkpoint/example_job.png | Bin 0 -> 26922 bytes
.../example_job_finish.png | Bin 0 -> 42969 bytes
.../2022-07-11-final-checkpoint/finish_cmp.png | Bin 0 -> 26074 bytes
.../stream_batch_cmp.png | Bin 0 -> 44100 bytes
content/index.html | 15 +-
content/zh/index.html | 15 +-
31 files changed, 2344 insertions(+), 836 deletions(-)
diff --git a/content/2022/07/11/final-checkpoint-part1.html b/content/2022/07/11/final-checkpoint-part1.html
new file mode 100644
index 000000000..1601d7967
--- /dev/null
+++ b/content/2022/07/11/final-checkpoint-part1.html
@@ -0,0 +1,447 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: FLIP-147: Support Checkpoints After Tasks Finished - Part One</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <!-- Matomo -->
+ <script>
+ var _paq = window._paq = window._paq || [];
+ /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
+ /* We explicitly disable cookie tracking to avoid privacy issues */
+ _paq.push(['disableCookies']);
+ /* Measure a visit to flink.apache.org and nightlies.apache.org/flink as the same visit */
+ _paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]);
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//matomo.privacy.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '1']);
+ var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+ <!-- End Matomo Code -->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+
+
+ <!-- Stateful Functions? -->
+
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/">What is Stateful Functions?</a></li>
+
+ <!-- Flink ML? -->
+
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-stable/">What is Flink ML?</a></li>
+
+ <!-- Flink Kubernetes Operator? -->
+
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/">What is the Flink Kubernetes Operator?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Getting Started -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Getting Started<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.15//docs/try-flink/local_installation/" target="_blank">With Flink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/getting-started/project-setup.html" target="_blank">With Flink Stateful Functions <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.1/try-flink-ml/quick-start.html" target="_blank">With Flink ML <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.0/try-flink-kubernetes-operator/quick-start.html" target="_blank">With Flink Kubernetes Operator <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/try-table-store/quick-start.html" target="_blank">With Flink Table Store <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="/training.html">Training Course</a></li>
+ </ul>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.15" target="_blank">Flink 1.15 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-docs-master" target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2" target="_blank">Flink Stateful Functions 3.2 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-master" target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.1" target="_blank">Flink ML 2.1 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-master" target="_blank">Flink ML Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.0" target="_blank">Flink Kubernetes Operator 1.0 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main" target="_blank">Flink Kubernetes Operator Main (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1" target="_blank">Flink Table Store 0.1 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-table-store-docs-master" target="_blank">Flink Table Store Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+ <!-- Flink-packages -->
+ <li>
+ <a href="https://flink-packages.org" target="_blank">flink-packages.org <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Roadmap -->
+ <li><a href="/roadmap.html">Roadmap</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/contributing/how-to-contribute.html">How to Contribute</a></li>
+
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <a href="/zh/2022/07/11/final-checkpoint-part1.html">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <style>
+ .smalllinks:link {
+ display: inline-block !important; background: none; padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px;
+ }
+ </style>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li >
+ <a href="/security.html">Flink Security</a>
+ </li>
+
+ <hr />
+
+ <li><a href="https://apache.org" target="_blank">Apache Software Foundation <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li>
+
+ <a class="smalllinks" href="https://www.apache.org/licenses/" target="_blank">License</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/security/" target="_blank">Security</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Donate</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+ </li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>FLIP-147: Support Checkpoints After Tasks Finished - Part One</h1>
+ <p><i></i></p>
+
+ <article>
+ <p>11 Jul 2022 Yun Gao , Dawid Wysakowicz , & Daisy Tsang </p>
+
+<h1 id="motivation">Motivation</h1>
+
+<p>Flink is a distributed processing engine for both unbounded and bounded streams of data. In recent versions,
+Flink has unified the DataStream API and the Table / SQL API to support both streaming and batch cases.
+Since most users require both types of data processing pipelines, the unification helps reduce the complexity of developing,
+operating, and maintaining consistency between streaming and batch backfilling jobs, like
+<a href="https://www.ververica.com/blog/apache-flinks-stream-batch-unification-powers-alibabas-11.11-in-2020">the case for Alibaba</a>.</p>
+
+<p>Flink provides two execution modes under the unified programming API: the streaming mode and the batch mode.
+The streaming mode processes records incrementally based on the states, thus it supports both bounded and unbounded sources.
+The batch mode works with bounded sources and usually has a better performance for bounded jobs because it executes all the
+tasks in topological order and avoids random state access by pre-sorting the input records. Although batch mode is often the
+preferred mode to process bounded jobs, streaming mode is also required for various reasons. For example, users may want to deal
+with records containing retraction or exploit the property that data is roughly sorted by event times in streaming mode
+(like the case in <a href="https://www.youtube.com/watch?t=666&v=4qSlsYogALo&feature=youtu.be">Kappa+ Architecture</a>). Moreover,
+users often have mixed jobs involving both unbounded streams and bounded side-inputs, which also require streaming execution mode.</p>
+
+<center>
+<img vspace="20" style="width:70%" src="/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png" />
+<p style="font-size: 0.6em;text-align:left;margin-top:-1em;margin-bottom: 4em">
+ Figure 1. A comparison of the Streaming mode and Batch mode for the example Count operator. For streaming mode, the arrived
+ elements are not sorted, the operator would read / write the state corresponding to the element for computation.
+ For batch mode, the arrived elements are first sorted as a whole and then processed.
+</p>
+</center>
+
+<p>In streaming mode, <a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/checkpointing/">checkpointing</a>
+is the vital mechanism in supporting exactly-once guarantees. By periodically snapshotting the
+aligned states of operators, Flink can recover from the latest checkpoint and continue execution when failover happens. However,
+previously Flink could not take checkpoints if any task gets finished. This would cause problems for jobs with both bounded and unbounded
+sources: if there are no checkpoints after the bounded part finished, the unbounded part might need to reprocess a large amount of
+records in case of a failure.</p>
+
+<p>Furthermore, being unable to take checkpoints with finished tasks is a problem for jobs using two-phase-commit sinks to achieve
+<a href="https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">end-to-end exactly-once processing</a>.
+The two-phase-commit sinks first write data to temporary files or external transactions,
+and commit the data only after a checkpoint completes to ensure the data would not be replayed on failure. However, if a job
+contains bounded sources, committing the results would not be possible after the bounded sources finish. Also because of that,
+for bounded jobs we have no way to commit the last piece of data after the first source task finished, and previously the bounded
+jobs just ignore the uncommitted data when finishing. These behaviors caused a lot of confusion and are always asked in the user
+mailing list.</p>
+
+<p>Therefore, to complete the support of streaming mode for jobs using bounded sources, it is important for us to</p>
+
+<ol>
+ <li>Support taking checkpoints with finished tasks.</li>
+ <li>Furthermore, revise the process of finishing so that all the data could always be committed.</li>
+</ol>
+
+<p>The remaining blog briefly describes the changes we made to achieve the above goals. In the next blog,
+we’ll share more details on how they are implemented.</p>
+
+<h1 id="support-checkpointing-with-finished-tasks">Support Checkpointing with Finished Tasks</h1>
+
+<p>The core idea of supporting checkpoints with finished tasks is to mark the finished operators in checkpoints and skip
+executing these operators after recovery. As illustrated in Figure 2, a checkpoint is composed of the states of all
+the operators. If all the subtasks of an operator have finished, we could mark it as fully finished and skip the
+execution of this operator on startup. For other operators, their states are composed of the states of all the
+running subtasks. The states will be repartitioned on restarting and all the new subtasks restarted with the assigned states.</p>
+
+<center>
+<img vspace="20" style="width:50%" src="/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png" />
+<p style="font-size: 0.6em;text-align:center;margin-top:-1em;margin-bottom: 4em">
+ Figure 2. An illustration of the extended checkpoint format.
+</p>
+</center>
+
+<p>To support creating such a checkpoint for jobs with finished tasks, we extended the checkpoint procedure.
+Previously the checkpoint coordinator inside the JobManager first notifies all the sources to report snapshots,
+then all the sources further notify their descendants via broadcasting barrier events. Since now the sources might
+have already finished, the checkpoint coordinator would instead treat the running tasks who also do not have running
+precedent tasks as “new sources”, and it notifies these tasks to initiate the checkpoints. Finally, if the subtasks of
+an operator are either finished on triggering checkpoint or have finished processing all the data on snapshotting states,
+the operator would be marked as fully finished.</p>
+
+<p>The changes of the checkpoint procedure are transparent to users except that for checkpoints indeed containing
+finished tasks, we disallowed adding new operators as precedents of the fully finished ones, since it would make the fully
+finished operators have running precedents after restarting, which conflicts with the design that tasks finished
+in topological order.</p>
+
+<h1 id="revise-the-process-of-finishing">Revise the Process of Finishing</h1>
+
+<p>Based on the ability to take checkpoints with finished tasks, we could then solve the issue that two-phase-commit
+operators could not commit all the data when running in streaming mode. As the background, Flink jobs
+have two ways to finish:</p>
+
+<ol>
+ <li>All sources are bound and they processed all the input records. The job will finish after all the
+input records are processed and all the result are committed to external systems.</li>
+ <li>Users execute <code>stop-with-savepoint [--drain]</code>. The job will take a savepoint and then finish. With <code>–-drain</code>, the job
+will be stopped permanently and is also required to commit all the data. However, without <code>--drain</code> the job might
+be resumed from the savepoint later, thus not all data are required to be committed, as long as the state of the data could be
+recovered from the savepoint.</li>
+</ol>
+
+<p>Let’s first have a look at the case of bounded sources. To achieve end-to-end exactly-once,
+two-phase-commit operators only commit data after a checkpoint following this piece of data succeeded.
+However, previously there is no such an opportunity for the data between the last periodic checkpoint and job getting finished,
+and the data finally gets lost. Note that it is also not correct if we directly commit the data on job finished, since
+if there are failovers after that (like due to other unfinished tasks getting failed), the data will be replayed and cause duplication.</p>
+
+<p>The case of <code>stop-with-savepoint --drain</code> also has problems. The previous implementation first stalls the execution and
+takes a savepoint. After the savepoint succeeds, all the source tasks would stop actively. Although the savepoint seems to
+provide the opportunity to commit all the data, some processing logic is in fact executed during the job getting stopped,
+and the records produced would be discarded by mistake. For example, calling <code>endInput()</code> method for operators happens during
+the stopping phase, some operators like the async operator might still emit new records in this method.</p>
+
+<p>At last, although <code>stop-with-savepoint</code> without draining is not required to commit all the data, we hope the job finish process could
+be unified for all the cases to keep the code clean.</p>
+
+<p>To fix the remaining issues, we need to modify the process of finishing to ensure all the data getting committed for the required cases.
+An intuitive idea is to directly insert a step to the tasks’ lifecycle to wait for the next checkpoint, as shown in the left part
+of Figure 3. However, it could not solve all the issues.</p>
+
+<center>
+<img vspace="20" style="width:90%" src="/img/blog/2022-07-11-final-checkpoint/finish_cmp.png" />
+<p style="font-size: 0.6em;text-align:left;margin-top:-1em;margin-bottom: 4em">
+ Figure 3. A comparison of the two options to ensure tasks committed all the data before getting finished. The first
+ option directly inserts a step in the tasks’ lifecycle to wait for the next checkpoint, which disallows the tasks to wait
+ for the same checkpoint / savepoint. The second option decouples the notification of finishing operator logic and finishing tasks,
+ thus it allows all the tasks to first process all records, then they have the chance to wait for the same checkpoint / savepoint.
+</p>
+</center>
+
+<p>For the case of bounded sources, the intuitive idea works, but it might have performance issues in some cases:
+as exemplified in Figure 4, If there are multiple cascading tasks containing two-phase commit sinks, each task would
+wait for the next checkpoint separately, thus the job needs to wait for three more checkpoints during finishing,
+which might prolong the total execution time for a long time.</p>
+
+<center>
+<img vspace="20" style="width:90%" src="/img/blog/2022-07-11-final-checkpoint/example_job.png" />
+<p style="font-size: 0.6em;text-align:center;margin-top:-1em;margin-bottom: 4em">
+ Figure 4. An example job that contains a chain of tasks containing two-phase-commit operators.
+</p>
+</center>
+
+<p>For the case of <code>stop-with-savepoint [--drain]</code>, the intuitive idea does not work since different tasks have to
+wait for different checkpoints / savepoints, thus we could not finish the job with a specific savepoint.</p>
+
+<p>Therefore, we do not take the intuitive option. Instead, we decoupled <em>“finishing operator logic”</em> and <em>“finishing tasks”</em>:
+all the tasks would first finish their execution logic as a whole, including calling lifecycle methods like <code>endInput()</code>,
+then each task could wait for the next checkpoint concurrently. Besides, for stop-with-savepoint we also reverted the current
+implementation similarly: all the tasks will first finish executing the operators’ logic, then they simply wait for the next savepoint
+to happen before finish. Therefore, in this way the finishing processes are unified and the data could be fully committed for all the cases.</p>
+
+<p>Based on this thought, as shown in the right part of Figure 3, to decoupled the process of “finishing operator logic”
+and “finishing tasks”, we introduced a new <code>EndOfData</code> event. For each task, after executing all the operator logic it would first notify
+the descendants with an <code>EndOfData</code> event so that the descendants also have chances to finish executing the operator logic. Then all
+the tasks could wait for the next checkpoint or the specified savepoint concurrently to commit all the remaining data before getting finished.</p>
+
+<p>At last, it is also worthy to mention we have clarified and renamed the <code>close()</code> and <code>dispose()</code> methods in the operators’ lifecycle.
+The two methods are in fact different since <code>close()</code> is only called when the task finishes normally and dispose() is called in both
+cases of normal finishing and failover. However, this was not clear from their names. Therefore, we rename the two methods to <code>finish()</code> and <code>close()</code>:</p>
+
+<ul>
+ <li><code>finish()</code> marks the termination of the operator and no more records are allowed after <code>finish()</code> is called. It should
+only be called when sources are finished or when the <code>-–drain</code> parameter is specified.</li>
+ <li><code>close()</code> is used to do cleanup and release all the held resources.</li>
+</ul>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>By supporting the checkpoints after tasks finished and revising the process of finishing, we can support checkpoints for jobs with
+both bounded and unbounded sources, and ensure the bounded job gets all output records committed before it finishes. The motivation
+behind this change is to ensure data consistency, results completeness, and failure recovery if there are bounded sources in the pipeline.
+The final checkpoint mechanism was first implemented in Flink 1.14 and enabled by default in Flink 1.15. If you have any questions,
+please feel free to start a discussion or report an issue in the dev or user mailing list.</p>
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2022 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="/js/jquery.matchHeight-min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+ </body>
+</html>
diff --git a/content/2022/07/11/final-checkpoint-part2.html b/content/2022/07/11/final-checkpoint-part2.html
new file mode 100644
index 000000000..847bebd98
--- /dev/null
+++ b/content/2022/07/11/final-checkpoint-part2.html
@@ -0,0 +1,510 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: FLIP-147: Support Checkpoints After Tasks Finished - Part Two</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <!-- Matomo -->
+ <script>
+ var _paq = window._paq = window._paq || [];
+ /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
+ /* We explicitly disable cookie tracking to avoid privacy issues */
+ _paq.push(['disableCookies']);
+ /* Measure a visit to flink.apache.org and nightlies.apache.org/flink as the same visit */
+ _paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]);
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//matomo.privacy.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '1']);
+ var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+ <!-- End Matomo Code -->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+
+
+ <!-- Stateful Functions? -->
+
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/">What is Stateful Functions?</a></li>
+
+ <!-- Flink ML? -->
+
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-stable/">What is Flink ML?</a></li>
+
+ <!-- Flink Kubernetes Operator? -->
+
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/">What is the Flink Kubernetes Operator?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Getting Started -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Getting Started<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.15//docs/try-flink/local_installation/" target="_blank">With Flink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/getting-started/project-setup.html" target="_blank">With Flink Stateful Functions <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.1/try-flink-ml/quick-start.html" target="_blank">With Flink ML <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.0/try-flink-kubernetes-operator/quick-start.html" target="_blank">With Flink Kubernetes Operator <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/try-table-store/quick-start.html" target="_blank">With Flink Table Store <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="/training.html">Training Course</a></li>
+ </ul>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.15" target="_blank">Flink 1.15 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-docs-master" target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2" target="_blank">Flink Stateful Functions 3.2 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-statefun-docs-master" target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.1" target="_blank">Flink ML 2.1 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-ml-docs-master" target="_blank">Flink ML Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.0" target="_blank">Flink Kubernetes Operator 1.0 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main" target="_blank">Flink Kubernetes Operator Main (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1" target="_blank">Flink Table Store 0.1 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://nightlies.apache.org/flink/flink-table-store-docs-master" target="_blank">Flink Table Store Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+ <!-- Flink-packages -->
+ <li>
+ <a href="https://flink-packages.org" target="_blank">flink-packages.org <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Roadmap -->
+ <li><a href="/roadmap.html">Roadmap</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/contributing/how-to-contribute.html">How to Contribute</a></li>
+
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <a href="/zh/2022/07/11/final-checkpoint-part2.html">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <style>
+ .smalllinks:link {
+ display: inline-block !important; background: none; padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px;
+ }
+ </style>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li >
+ <a href="/security.html">Flink Security</a>
+ </li>
+
+ <hr />
+
+ <li><a href="https://apache.org" target="_blank">Apache Software Foundation <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li>
+
+ <a class="smalllinks" href="https://www.apache.org/licenses/" target="_blank">License</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/security/" target="_blank">Security</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Donate</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+ </li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>FLIP-147: Support Checkpoints After Tasks Finished - Part Two</h1>
+ <p><i></i></p>
+
+ <article>
+ <p>11 Jul 2022 Yun Gao , Dawid Wysakowicz , & Daisy Tsang </p>
+
+<p>In the <a href="/2022/06/01/final-checkpoint-part1.html">first part</a> of this blog,
+we have briefly introduced the work to support checkpoints after tasks get
+finished and revised the process of finishing. In this part we will present more details on the implementation,
+including how we support checkpoints with finished tasks and the revised protocol of the finish process.</p>
+
+<h1 id="implementation-of-support-checkpointing-with-finished-tasks">Implementation of support Checkpointing with Finished Tasks</h1>
+
+<p>As described in part one,
+to support checkpoints after some tasks are finished, the core idea is to mark
+the finished operators in checkpoints and skip executing these operators after recovery. To implement this idea,
+we enhanced the checkpointing procedure to generate the flag and use the flag on recovery. This section presents
+more details on the process of taking checkpoints with finished tasks and recovery from such checkpoints.</p>
+
+<p>Previously, checkpointing only worked when all tasks were running. As shown in the Figure 1, in this case the
+checkpoint coordinator first notify all the source tasks, and then the source tasks further notify the
+downstream tasks to take snapshots via barrier events. Similarly, if there are finished tasks, we need to
+find the new “source” tasks to initiate the checkpoint, namely those tasks that are still running but have
+no running precedent tasks. CheckpointCoordinator does the computation atomically at the JobManager side
+based on the latest states recorded in the execution graph.</p>
+
+<p>There might be race conditions when triggering tasks: when the checkpoint coordinator
+decides to trigger one task and starts emitting the RPC, it is possible that the task is just finished and
+reporting the FINISHED status to JobManager. In this case, the RPC message would fail and the checkpoint would be aborted.</p>
+
+<center>
+<img vspace="20" style="width:50%" src="/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png" />
+<p style="font-size: 0.6em">
+ Figure 1. The tasks chosen as the new sources when taking checkpoint with finished tasks. The principle is to
+ choose the running tasks whose precedent tasks are all finished.
+</p>
+</center>
+
+<p>In order to keep track of the finish status of each operator, we need to extend the checkpoint format.
+A checkpoint consists of the states of all the stateful operators, and the state of one operator consists of the
+entries from all its parallel instances. Note that the concept of Task is not reflected in the checkpoint. Task
+is more of a physical execution container that drives the behavior of operators. It is not well-defined across
+multiple executions of the same job since job upgrades might modify the operators contained in one task.
+Therefore, the finished status should also be attached to the operators.</p>
+
+<p>As shown in the Figure 2, operators could be classified into three types according to their finished status:</p>
+
+<ol>
+ <li>Fully finished: If all the instances of an operator are finished, we could view the logic of the operators as
+fully executed and we should skip the execution of the operator after recovery. We need to store a special flag for this
+kind of operator.</li>
+ <li>Partially finished: If only some instances of an operator are finished, then we still need to continue executing the
+remaining logic of this operator. As a whole we could view the state of the operator as the set of entries collected from all the
+running instances, which represents the remaining workload for this operator.</li>
+ <li>No finished instances: In this case, the state of the operator is the same as the one taken when no tasks are finished.</li>
+</ol>
+
+<center>
+<img vspace="20" style="width:50%" src="/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png" />
+<p style="font-size: 0.6em">
+ Figure 2. An illustration of the extended checkpoint format.
+</p>
+</center>
+
+<p>If the job is later restored from a checkpoint taken with finished tasks, we would skip executing all the logic for fully
+finished operators, and execute normally for the operators with no finished instances.</p>
+
+<p>However, this would be a bit complex for the partially finished operators. The state of partially finished operators would be
+redistributed to all the instances, similar to rescaling when the parallelism is changed. Among all the types of states that
+Flink offers, the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-keyed-state">keyed state</a>
+and <a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-operator-state">operator state</a>
+with even-split redistribution would work normally, but the
+<a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#broadcast-state">broadcast state</a> and
+<a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-operator-state">operator state with union redistribution</a>
+would be affected for the following reasons:</p>
+
+<ol>
+ <li>The broadcast state always replicates the state of the first subtask to the other subtasks. If the first subtask is finished,
+an empty state would be distributed and the operator would run from scratch, which is not correct.</li>
+ <li>The operator state with union distribution merges the states of all the subtasks and then sends the merged state to all the
+subtasks. Based on this behavior, some operators may choose one subtask to store a shared value and after restarting this value will
+be distributed to all the subtasks. However, if this chosen task is finished, the state would be lost.</li>
+</ol>
+
+<p>These two issues would not occur when rescaling since there would be no finished tasks in that scenario. To address
+these issues, we chose one of the running subtasks instead to acquire the current state for the broadcast state. For the operator
+state with union redistribution, we have to collect the states of all the subtasks to maintain the semantics. Thus, currently we
+abort the checkpoint if parts of subtasks finished for operators using this kind of state.</p>
+
+<p>In principle, you should be able to modify your job (which changes the dataflow graph) and restore from a previous checkpoint. That said,
+there are certain graph modifications that are not supported. These kinds of changes include adding a new operator as the precedent of a fully finished
+one. Flink would check for such modifications and throw exceptions while restoring.</p>
+
+<h1 id="the-revised-process-of-finishing">The Revised Process of Finishing</h1>
+
+<p>As described in the part one, based on the ability to take checkpoints with finished tasks, we revised the process of finishing
+so that we could always commit all the data for two-phase-commit sinks. We’ll show the detailed protocol of the finished process in this
+section.</p>
+
+<h2 id="how-did-jobs-in-flink-finish-before">How did Jobs in Flink Finish Before?</h2>
+
+<p>A job might finish in two ways: all sources finish or users execute
+<a href="https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint"><code>stop-with-savepoint [--drain]</code></a>.
+Let’s first have a look at the detailed process of finishing before FLIP-147.</p>
+
+<h3 id="when-sources-finish">When sources finish</h3>
+
+<p>If all the sources are bounded, The job will finish after all the input records are processed and all the result are
+committed to external systems. In this case, the sources would first
+emit a <code>MAX_WATERMARK</code> (<code>Long.MAX_VALUE</code>) and then start to terminate the task. On termination, a task would call <code>endOfInput()</code>,
+<code>close()</code> and <code>dispose()</code> for all the operators, then emit an <code>EndOfPartitionEvent</code> to the downstream tasks. The intermediate tasks
+would start terminating after receiving an <code>EndOfPartitionEvent</code> from all the input channels, and this process will continue
+until the last task is finished.</p>
+
+<div class="highlight"><pre><code>1. Source operators emit MAX_WATERMARK
+2. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event-time timers
+ b. Emit MAX_WATERMARK
+3. Source tasks finished
+ a. endInput(inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+4. On received EndOfPartitionEvent for non-source tasks
+ a. endInput(int inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+</code></pre></div>
+
+<h3 id="when-users-execute-stop-with-savepoint---drain">When users execute stop-with-savepoint [–drain]</h3>
+
+<p>Users could execute the command stop-with-savepoint [–drain] for both bounded and unbounded jobs to trigger jobs to finish.
+In this case, Flink first triggers a synchronous savepoint and all the tasks would stall after seeing the synchronous
+savepoint. If the savepoint succeeds, all the source operators would finish actively and the job would finish the same as the above scenario.</p>
+
+<div class="highlight"><pre><code>1. Trigger a savepoint
+2. Sources received savepoint trigger RPC
+ a. If with –-drain
+ i. source operators emit MAX_WATERMARK
+ b. Source emits savepoint barrier
+3. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event times
+ b. Emit MAX_WATERMARK
+4. On received savepoint barrier for non-source operators
+ a. The task blocks till the savepoint succeed
+5. Finish the source tasks actively
+ a. If with –-drain
+ ii. endInput(inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+6. On received EndOfPartitionEvent for non-source tasks
+ a. If with –-drain
+ i. endInput(int inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+</code></pre></div>
+
+<p>A parameter <code>–-drain</code> is supported with <code>stop-with-savepoint</code>: if not specified, the job is expected to resume from this savepoint,
+otherwise the job is expected to terminate permanently. Thus we only emit <code>MAX_WATERMARK</code> to trigger all the event timers and call
+<code>endInput()</code> in the latter case.</p>
+
+<h2 id="revise-the-finishing-steps">Revise the Finishing Steps</h2>
+
+<p>As described in part one, after revising the process of finishing, we have decoupled the process of “finishing operator logic”
+and “finishing task” by introducing a new <code>EndOfData</code> event. After the revision each task will first
+notify the descendants with an <code>EndOfData</code> event after executing all the logic
+so that the descendants also have chances to finish executing the operator logic, then
+all the tasks could wait for the next checkpoint or the specified savepoint concurrently to commit all the remaining data.
+This section will present the detailed protocol of the revised process. Since we have renamed
+<code>close()</code> /<code>dispose()</code> to <code>finish()</code> / <code>close()</code>, we’ll stick to the new terminologies in the following description.</p>
+
+<p>The revised process of finishing is shown as follows:</p>
+
+<div class="highlight"><pre><code>1. Source tasks finished due to no more records or stop-with-savepoint.
+ a. if no more records or stop-with-savepoint –-drain
+ i. source operators emit MAX_WATERMARK
+ ii. endInput(inputId) for all the operators
+ iii. finish() for all the operators
+ iv. emit EndOfData[isDrain = true] event
+ b. else if stop-with-savepoint
+ i. emit EndOfData[isDrain = false] event
+ c. Wait for the next checkpoint / the savepoint after operator finished complete
+ d. close() for all the operators
+ e. Emit EndOfPartitionEvent
+ f. Task cleanup
+2. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event times
+ b. Emit MAX_WATERMARK
+3. On received EndOfData for non-source tasks
+ a. If isDrain
+ i. endInput(int inputId) for all the operators
+ ii. finish() for all the operators
+ b. Emit EndOfData[isDrain = the flag value of the received event]
+4. On received EndOfPartitionEvent for non-source tasks
+ a. Wait for the next checkpoint / the savepoint after operator finished complete
+ b. close() for all the operators
+ c. Emit EndOfPartitionEvent
+ d. Task cleanup
+</code></pre></div>
+
+<center>
+<img vspace="20" style="width:60%" src="/img/blog/2022-07-11-final-checkpoint/example_job_finish.png" />
+<p style="font-size: 0.6em">
+ Figure 3. An example job of the revised process of finishing.
+</p>
+</center>
+
+<p>An example of the process of job finishing is shown in Figure 3.</p>
+
+<p>Let’s first have a look at the example that all the source tasks are bounded.
+If Task <code>C</code> finishes after processing all the records, it first emits the max-watermark, then finishes the operators and emits
+the <code>EndOfData</code> event. After that, it waits for the next checkpoint to complete and then emits the <code>EndOfPartitionEvent</code>.</p>
+
+<p>Task <code>D</code> finishes all the operators right after receiving the <code>EndOfData</code> event. Since any checkpoints taken after operators finish
+can commit all the pending records and be the final checkpoint, Task <code>D</code>’s final checkpoint would be the same as Task <code>C</code>’s since
+the barrier must be emitted after the <code>EndOfData</code> event.</p>
+
+<p>Task <code>E</code> is a bit different in that it has two inputs. Task <code>A</code> might continue to run for a while and, thus, Task <code>E</code> needs to wait
+until it receives an <code>EndOfData</code> event also from the other input before finishing operators and its final checkpoint might be different.</p>
+
+<p>On the other hand, when using <code>stop-with-savepoint [--drain]</code>, the process is similar except that all the tasks need to wait for the exact
+savepoint before finishing instead of just any checkpoints. Moreover, since both Task <code>C</code> and Task <code>A</code> would finish at the same time,
+Task <code>E</code> would also be able to wait for this particular savepoint before finishing.</p>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>In this part we have presented more details of how the checkpoints are taken with finished tasks and the revised process
+of finishing. We hope the details could provide more insights of the thoughts and implementations for this part of work. Still, if you
+have any questions, please feel free to start a discussion or report an issue in the dev or user mailing list.</p>
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2022 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="/js/jquery.matchHeight-min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+ </body>
+</html>
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index 0e9c23adf..1b5e30ed6 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -6,6 +6,419 @@
<link>https://flink.apache.org/blog</link>
<atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" />
+<item>
+<title>FLIP-147: Support Checkpoints After Tasks Finished - Part Two</title>
+<description><p>In the <a href="/2022/06/01/final-checkpoint-part1.html">first part</a> of this blog,
+we have briefly introduced the work to support checkpoints after tasks get
+finished and revised the process of finishing. In this part we will present more details on the implementation,
+including how we support checkpoints with finished tasks and the revised protocol of the finish process.</p>
+
+<h1 id="implementation-of-support-checkpointing-with-finished-tasks">Implementation of support Checkpointing with Finished Tasks</h1>
+
+<p>As described in part one,
+to support checkpoints after some tasks are finished, the core idea is to mark
+the finished operators in checkpoints and skip executing these operators after recovery. To implement this idea,
+we enhanced the checkpointing procedure to generate the flag and use the flag on recovery. This section presents
+more details on the process of taking checkpoints with finished tasks and recovery from such checkpoints.</p>
+
+<p>Previously, checkpointing only worked when all tasks were running. As shown in the Figure 1, in this case the
+checkpoint coordinator first notify all the source tasks, and then the source tasks further notify the
+downstream tasks to take snapshots via barrier events. Similarly, if there are finished tasks, we need to
+find the new “source” tasks to initiate the checkpoint, namely those tasks that are still running but have
+no running precedent tasks. CheckpointCoordinator does the computation atomically at the JobManager side
+based on the latest states recorded in the execution graph.</p>
+
+<p>There might be race conditions when triggering tasks: when the checkpoint coordinator
+decides to trigger one task and starts emitting the RPC, it is possible that the task is just finished and
+reporting the FINISHED status to JobManager. In this case, the RPC message would fail and the checkpoint would be aborted.</p>
+
+<center>
+<img vspace="20" style="width:50%" src="/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png" />
+<p style="font-size: 0.6em">
+ Figure 1. The tasks chosen as the new sources when taking checkpoint with finished tasks. The principle is to
+ choose the running tasks whose precedent tasks are all finished.
+</p>
+</center>
+
+<p>In order to keep track of the finish status of each operator, we need to extend the checkpoint format.
+A checkpoint consists of the states of all the stateful operators, and the state of one operator consists of the
+entries from all its parallel instances. Note that the concept of Task is not reflected in the checkpoint. Task
+is more of a physical execution container that drives the behavior of operators. It is not well-defined across
+multiple executions of the same job since job upgrades might modify the operators contained in one task.
+Therefore, the finished status should also be attached to the operators.</p>
+
+<p>As shown in the Figure 2, operators could be classified into three types according to their finished status:</p>
+
+<ol>
+ <li>Fully finished: If all the instances of an operator are finished, we could view the logic of the operators as
+fully executed and we should skip the execution of the operator after recovery. We need to store a special flag for this
+kind of operator.</li>
+ <li>Partially finished: If only some instances of an operator are finished, then we still need to continue executing the
+remaining logic of this operator. As a whole we could view the state of the operator as the set of entries collected from all the
+running instances, which represents the remaining workload for this operator.</li>
+ <li>No finished instances: In this case, the state of the operator is the same as the one taken when no tasks are finished.</li>
+</ol>
+
+<center>
+<img vspace="20" style="width:50%" src="/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png" />
+<p style="font-size: 0.6em">
+ Figure 2. An illustration of the extended checkpoint format.
+</p>
+</center>
+
+<p>If the job is later restored from a checkpoint taken with finished tasks, we would skip executing all the logic for fully
+finished operators, and execute normally for the operators with no finished instances.</p>
+
+<p>However, this would be a bit complex for the partially finished operators. The state of partially finished operators would be
+redistributed to all the instances, similar to rescaling when the parallelism is changed. Among all the types of states that
+Flink offers, the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-keyed-state">keyed state</a>
+and <a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-operator-state">operator state</a>
+with even-split redistribution would work normally, but the
+<a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#broadcast-state">broadcast state</a> and
+<a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/#using-operator-state">operator state with union redistribution</a>
+would be affected for the following reasons:</p>
+
+<ol>
+ <li>The broadcast state always replicates the state of the first subtask to the other subtasks. If the first subtask is finished,
+an empty state would be distributed and the operator would run from scratch, which is not correct.</li>
+ <li>The operator state with union distribution merges the states of all the subtasks and then sends the merged state to all the
+subtasks. Based on this behavior, some operators may choose one subtask to store a shared value and after restarting this value will
+be distributed to all the subtasks. However, if this chosen task is finished, the state would be lost.</li>
+</ol>
+
+<p>These two issues would not occur when rescaling since there would be no finished tasks in that scenario. To address
+these issues, we chose one of the running subtasks instead to acquire the current state for the broadcast state. For the operator
+state with union redistribution, we have to collect the states of all the subtasks to maintain the semantics. Thus, currently we
+abort the checkpoint if parts of subtasks finished for operators using this kind of state.</p>
+
+<p>In principle, you should be able to modify your job (which changes the dataflow graph) and restore from a previous checkpoint. That said,
+there are certain graph modifications that are not supported. These kinds of changes include adding a new operator as the precedent of a fully finished
+one. Flink would check for such modifications and throw exceptions while restoring.</p>
+
+<h1 id="the-revised-process-of-finishing">The Revised Process of Finishing</h1>
+
+<p>As described in the part one, based on the ability to take checkpoints with finished tasks, we revised the process of finishing
+so that we could always commit all the data for two-phase-commit sinks. We’ll show the detailed protocol of the finished process in this
+section.</p>
+
+<h2 id="how-did-jobs-in-flink-finish-before">How did Jobs in Flink Finish Before?</h2>
+
+<p>A job might finish in two ways: all sources finish or users execute
+<a href="https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint"><code>stop-with-savepoint [--drain]</code></a>.
+Let’s first have a look at the detailed process of finishing before FLIP-147.</p>
+
+<h3 id="when-sources-finish">When sources finish</h3>
+
+<p>If all the sources are bounded, The job will finish after all the input records are processed and all the result are
+committed to external systems. In this case, the sources would first
+emit a <code>MAX_WATERMARK</code> (<code>Long.MAX_VALUE</code>) and then start to terminate the task. On termination, a task would call <code>endOfInput()</code>,
+<code>close()</code> and <code>dispose()</code> for all the operators, then emit an <code>EndOfPartitionEvent</code> to the downstream tasks. The intermediate tasks
+would start terminating after receiving an <code>EndOfPartitionEvent</code> from all the input channels, and this process will continue
+until the last task is finished.</p>
+
+<div class="highlight"><pre><code>1. Source operators emit MAX_WATERMARK
+2. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event-time timers
+ b. Emit MAX_WATERMARK
+3. Source tasks finished
+ a. endInput(inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+4. On received EndOfPartitionEvent for non-source tasks
+ a. endInput(int inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+</code></pre></div>
+
+<h3 id="when-users-execute-stop-with-savepoint---drain">When users execute stop-with-savepoint [–drain]</h3>
+
+<p>Users could execute the command stop-with-savepoint [–drain] for both bounded and unbounded jobs to trigger jobs to finish.
+In this case, Flink first triggers a synchronous savepoint and all the tasks would stall after seeing the synchronous
+savepoint. If the savepoint succeeds, all the source operators would finish actively and the job would finish the same as the above scenario.</p>
+
+<div class="highlight"><pre><code>1. Trigger a savepoint
+2. Sources received savepoint trigger RPC
+ a. If with –-drain
+ i. source operators emit MAX_WATERMARK
+ b. Source emits savepoint barrier
+3. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event times
+ b. Emit MAX_WATERMARK
+4. On received savepoint barrier for non-source operators
+ a. The task blocks till the savepoint succeed
+5. Finish the source tasks actively
+ a. If with –-drain
+ ii. endInput(inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+6. On received EndOfPartitionEvent for non-source tasks
+ a. If with –-drain
+ i. endInput(int inputId) for all the operators
+ b. close() for all the operators
+ c. dispose() for all the operators
+ d. Emit EndOfPartitionEvent
+ e. Task cleanup
+</code></pre></div>
+
+<p>A parameter <code>–-drain</code> is supported with <code>stop-with-savepoint</code>: if not specified, the job is expected to resume from this savepoint,
+otherwise the job is expected to terminate permanently. Thus we only emit <code>MAX_WATERMARK</code> to trigger all the event timers and call
+<code>endInput()</code> in the latter case.</p>
+
+<h2 id="revise-the-finishing-steps">Revise the Finishing Steps</h2>
+
+<p>As described in part one, after revising the process of finishing, we have decoupled the process of “finishing operator logic”
+and “finishing task” by introducing a new <code>EndOfData</code> event. After the revision each task will first
+notify the descendants with an <code>EndOfData</code> event after executing all the logic
+so that the descendants also have chances to finish executing the operator logic, then
+all the tasks could wait for the next checkpoint or the specified savepoint concurrently to commit all the remaining data.
+This section will present the detailed protocol of the revised process. Since we have renamed
+<code>close()</code> /<code>dispose()</code> to <code>finish()</code> / <code>close()</code>, we’ll stick to the new terminologies in the following description.</p>
+
+<p>The revised process of finishing is shown as follows:</p>
+
+<div class="highlight"><pre><code>1. Source tasks finished due to no more records or stop-with-savepoint.
+ a. if no more records or stop-with-savepoint –-drain
+ i. source operators emit MAX_WATERMARK
+ ii. endInput(inputId) for all the operators
+ iii. finish() for all the operators
+ iv. emit EndOfData[isDrain = true] event
+ b. else if stop-with-savepoint
+ i. emit EndOfData[isDrain = false] event
+ c. Wait for the next checkpoint / the savepoint after operator finished complete
+ d. close() for all the operators
+ e. Emit EndOfPartitionEvent
+ f. Task cleanup
+2. On received MAX_WATERMARK for non-source operators
+ a. Trigger all the event times
+ b. Emit MAX_WATERMARK
+3. On received EndOfData for non-source tasks
+ a. If isDrain
+ i. endInput(int inputId) for all the operators
+ ii. finish() for all the operators
+ b. Emit EndOfData[isDrain = the flag value of the received event]
+4. On received EndOfPartitionEvent for non-source tasks
+ a. Wait for the next checkpoint / the savepoint after operator finished complete
+ b. close() for all the operators
+ c. Emit EndOfPartitionEvent
+ d. Task cleanup
+</code></pre></div>
+
+<center>
+<img vspace="20" style="width:60%" src="/img/blog/2022-07-11-final-checkpoint/example_job_finish.png" />
+<p style="font-size: 0.6em">
+ Figure 3. An example job of the revised process of finishing.
+</p>
+</center>
+
+<p>An example of the process of job finishing is shown in Figure 3.</p>
+
+<p>Let’s first have a look at the example that all the source tasks are bounded.
+If Task <code>C</code> finishes after processing all the records, it first emits the max-watermark, then finishes the operators and emits
+the <code>EndOfData</code> event. After that, it waits for the next checkpoint to complete and then emits the <code>EndOfPartitionEvent</code>.</p>
+
+<p>Task <code>D</code> finishes all the operators right after receiving the <code>EndOfData</code> event. Since any checkpoints taken after operators finish
+can commit all the pending records and be the final checkpoint, Task <code>D</code>’s final checkpoint would be the same as Task <code>C</code>’s since
+the barrier must be emitted after the <code>EndOfData</code> event.</p>
+
+<p>Task <code>E</code> is a bit different in that it has two inputs. Task <code>A</code> might continue to run for a while and, thus, Task <code>E</code> needs to wait
+until it receives an <code>EndOfData</code> event also from the other input before finishing operators and its final checkpoint might be different.</p>
+
+<p>On the other hand, when using <code>stop-with-savepoint [--drain]</code>, the process is similar except that all the tasks need to wait for the exact
+savepoint before finishing instead of just any checkpoints. Moreover, since both Task <code>C</code> and Task <code>A</code> would finish at the same time,
+Task <code>E</code> would also be able to wait for this particular savepoint before finishing.</p>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>In this part we have presented more details of how the checkpoints are taken with finished tasks and the revised process
+of finishing. We hope the details could provide more insights of the thoughts and implementations for this part of work. Still, if you
+have any questions, please feel free to start a discussion or report an issue in the dev or user mailing list.</p>
+</description>
+<pubDate>Mon, 11 Jul 2022 02:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2022/07/11/final-checkpoint-part2.html</link>
+<guid isPermaLink="true">/2022/07/11/final-checkpoint-part2.html</guid>
+</item>
+
+<item>
+<title>FLIP-147: Support Checkpoints After Tasks Finished - Part One</title>
+<description><h1 id="motivation">Motivation</h1>
+
+<p>Flink is a distributed processing engine for both unbounded and bounded streams of data. In recent versions,
+Flink has unified the DataStream API and the Table / SQL API to support both streaming and batch cases.
+Since most users require both types of data processing pipelines, the unification helps reduce the complexity of developing,
+operating, and maintaining consistency between streaming and batch backfilling jobs, like
+<a href="https://www.ververica.com/blog/apache-flinks-stream-batch-unification-powers-alibabas-11.11-in-2020">the case for Alibaba</a>.</p>
+
+<p>Flink provides two execution modes under the unified programming API: the streaming mode and the batch mode.
+The streaming mode processes records incrementally based on the states, thus it supports both bounded and unbounded sources.
+The batch mode works with bounded sources and usually has a better performance for bounded jobs because it executes all the
+tasks in topological order and avoids random state access by pre-sorting the input records. Although batch mode is often the
+preferred mode to process bounded jobs, streaming mode is also required for various reasons. For example, users may want to deal
+with records containing retraction or exploit the property that data is roughly sorted by event times in streaming mode
+(like the case in <a href="https://www.youtube.com/watch?t=666&amp;v=4qSlsYogALo&amp;feature=youtu.be">Kappa+ Architecture</a>). Moreover,
+users often have mixed jobs involving both unbounded streams and bounded side-inputs, which also require streaming execution mode.</p>
+
+<center>
+<img vspace="20" style="width:70%" src="/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png" />
+<p style="font-size: 0.6em;text-align:left;margin-top:-1em;margin-bottom: 4em">
+ Figure 1. A comparison of the Streaming mode and Batch mode for the example Count operator. For streaming mode, the arrived
+ elements are not sorted, the operator would read / write the state corresponding to the element for computation.
+ For batch mode, the arrived elements are first sorted as a whole and then processed.
+</p>
+</center>
+
+<p>In streaming mode, <a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/checkpointing/">checkpointing</a>
+is the vital mechanism in supporting exactly-once guarantees. By periodically snapshotting the
+aligned states of operators, Flink can recover from the latest checkpoint and continue execution when failover happens. However,
+previously Flink could not take checkpoints if any task gets finished. This would cause problems for jobs with both bounded and unbounded
+sources: if there are no checkpoints after the bounded part finished, the unbounded part might need to reprocess a large amount of
+records in case of a failure.</p>
+
+<p>Furthermore, being unable to take checkpoints with finished tasks is a problem for jobs using two-phase-commit sinks to achieve
+<a href="https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">end-to-end exactly-once processing</a>.
+The two-phase-commit sinks first write data to temporary files or external transactions,
+and commit the data only after a checkpoint completes to ensure the data would not be replayed on failure. However, if a job
+contains bounded sources, committing the results would not be possible after the bounded sources finish. Also because of that,
+for bounded jobs we have no way to commit the last piece of data after the first source task finished, and previously the bounded
+jobs just ignore the uncommitted data when finishing. These behaviors caused a lot of confusion and are always asked in the user
+mailing list.</p>
+
+<p>Therefore, to complete the support of streaming mode for jobs using bounded sources, it is important for us to</p>
+
+<ol>
+ <li>Support taking checkpoints with finished tasks.</li>
+ <li>Furthermore, revise the process of finishing so that all the data could always be committed.</li>
+</ol>
+
+<p>The remaining blog briefly describes the changes we made to achieve the above goals. In the next blog,
+we’ll share more details on how they are implemented.</p>
+
+<h1 id="support-checkpointing-with-finished-tasks">Support Checkpointing with Finished Tasks</h1>
+
+<p>The core idea of supporting checkpoints with finished tasks is to mark the finished operators in checkpoints and skip
+executing these operators after recovery. As illustrated in Figure 2, a checkpoint is composed of the states of all
+the operators. If all the subtasks of an operator have finished, we could mark it as fully finished and skip the
+execution of this operator on startup. For other operators, their states are composed of the states of all the
+running subtasks. The states will be repartitioned on restarting and all the new subtasks restarted with the assigned states.</p>
+
+<center>
+<img vspace="20" style="width:50%" src="/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png" />
+<p style="font-size: 0.6em;text-align:center;margin-top:-1em;margin-bottom: 4em">
+ Figure 2. An illustration of the extended checkpoint format.
+</p>
+</center>
+
+<p>To support creating such a checkpoint for jobs with finished tasks, we extended the checkpoint procedure.
+Previously the checkpoint coordinator inside the JobManager first notifies all the sources to report snapshots,
+then all the sources further notify their descendants via broadcasting barrier events. Since now the sources might
+have already finished, the checkpoint coordinator would instead treat the running tasks who also do not have running
+precedent tasks as “new sources”, and it notifies these tasks to initiate the checkpoints. Finally, if the subtasks of
+an operator are either finished on triggering checkpoint or have finished processing all the data on snapshotting states,
+the operator would be marked as fully finished.</p>
+
+<p>The changes of the checkpoint procedure are transparent to users except that for checkpoints indeed containing
+finished tasks, we disallowed adding new operators as precedents of the fully finished ones, since it would make the fully
+finished operators have running precedents after restarting, which conflicts with the design that tasks finished
+in topological order.</p>
+
+<h1 id="revise-the-process-of-finishing">Revise the Process of Finishing</h1>
+
+<p>Based on the ability to take checkpoints with finished tasks, we could then solve the issue that two-phase-commit
+operators could not commit all the data when running in streaming mode. As the background, Flink jobs
+have two ways to finish:</p>
+
+<ol>
+ <li>All sources are bound and they processed all the input records. The job will finish after all the
+input records are processed and all the result are committed to external systems.</li>
+ <li>Users execute <code>stop-with-savepoint [--drain]</code>. The job will take a savepoint and then finish. With <code>–-drain</code>, the job
+will be stopped permanently and is also required to commit all the data. However, without <code>--drain</code> the job might
+be resumed from the savepoint later, thus not all data are required to be committed, as long as the state of the data could be
+recovered from the savepoint.</li>
+</ol>
+
+<p>Let’s first have a look at the case of bounded sources. To achieve end-to-end exactly-once,
+two-phase-commit operators only commit data after a checkpoint following this piece of data succeeded.
+However, previously there is no such an opportunity for the data between the last periodic checkpoint and job getting finished,
+and the data finally gets lost. Note that it is also not correct if we directly commit the data on job finished, since
+if there are failovers after that (like due to other unfinished tasks getting failed), the data will be replayed and cause duplication.</p>
+
+<p>The case of <code>stop-with-savepoint --drain</code> also has problems. The previous implementation first stalls the execution and
+takes a savepoint. After the savepoint succeeds, all the source tasks would stop actively. Although the savepoint seems to
+provide the opportunity to commit all the data, some processing logic is in fact executed during the job getting stopped,
+and the records produced would be discarded by mistake. For example, calling <code>endInput()</code> method for operators happens during
+the stopping phase, some operators like the async operator might still emit new records in this method.</p>
+
+<p>At last, although <code>stop-with-savepoint</code> without draining is not required to commit all the data, we hope the job finish process could
+be unified for all the cases to keep the code clean.</p>
+
+<p>To fix the remaining issues, we need to modify the process of finishing to ensure all the data getting committed for the required cases.
+An intuitive idea is to directly insert a step to the tasks’ lifecycle to wait for the next checkpoint, as shown in the left part
+of Figure 3. However, it could not solve all the issues.</p>
+
+<center>
+<img vspace="20" style="width:90%" src="/img/blog/2022-07-11-final-checkpoint/finish_cmp.png" />
+<p style="font-size: 0.6em;text-align:left;margin-top:-1em;margin-bottom: 4em">
+ Figure 3. A comparison of the two options to ensure tasks committed all the data before getting finished. The first
+ option directly inserts a step in the tasks’ lifecycle to wait for the next checkpoint, which disallows the tasks to wait
+ for the same checkpoint / savepoint. The second option decouples the notification of finishing operator logic and finishing tasks,
+ thus it allows all the tasks to first process all records, then they have the chance to wait for the same checkpoint / savepoint.
+</p>
+</center>
+
+<p>For the case of bounded sources, the intuitive idea works, but it might have performance issues in some cases:
+as exemplified in Figure 4, If there are multiple cascading tasks containing two-phase commit sinks, each task would
+wait for the next checkpoint separately, thus the job needs to wait for three more checkpoints during finishing,
+which might prolong the total execution time for a long time.</p>
+
+<center>
+<img vspace="20" style="width:90%" src="/img/blog/2022-07-11-final-checkpoint/example_job.png" />
+<p style="font-size: 0.6em;text-align:center;margin-top:-1em;margin-bottom: 4em">
+ Figure 4. An example job that contains a chain of tasks containing two-phase-commit operators.
+</p>
+</center>
+
+<p>For the case of <code>stop-with-savepoint [--drain]</code>, the intuitive idea does not work since different tasks have to
+wait for different checkpoints / savepoints, thus we could not finish the job with a specific savepoint.</p>
+
+<p>Therefore, we do not take the intuitive option. Instead, we decoupled <em>“finishing operator logic”</em> and <em>“finishing tasks”</em>:
+all the tasks would first finish their execution logic as a whole, including calling lifecycle methods like <code>endInput()</code>,
+then each task could wait for the next checkpoint concurrently. Besides, for stop-with-savepoint we also reverted the current
+implementation similarly: all the tasks will first finish executing the operators’ logic, then they simply wait for the next savepoint
+to happen before finish. Therefore, in this way the finishing processes are unified and the data could be fully committed for all the cases.</p>
+
+<p>Based on this thought, as shown in the right part of Figure 3, to decoupled the process of “finishing operator logic”
+and “finishing tasks”, we introduced a new <code>EndOfData</code> event. For each task, after executing all the operator logic it would first notify
+the descendants with an <code>EndOfData</code> event so that the descendants also have chances to finish executing the operator logic. Then all
+the tasks could wait for the next checkpoint or the specified savepoint concurrently to commit all the remaining data before getting finished.</p>
+
+<p>At last, it is also worthy to mention we have clarified and renamed the <code>close()</code> and <code>dispose()</code> methods in the operators’ lifecycle.
+The two methods are in fact different since <code>close()</code> is only called when the task finishes normally and dispose() is called in both
+cases of normal finishing and failover. However, this was not clear from their names. Therefore, we rename the two methods to <code>finish()</code> and <code>close()</code>:</p>
+
+<ul>
+ <li><code>finish()</code> marks the termination of the operator and no more records are allowed after <code>finish()</code> is called. It should
+only be called when sources are finished or when the <code>-–drain</code> parameter is specified.</li>
+ <li><code>close()</code> is used to do cleanup and release all the held resources.</li>
+</ul>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>By supporting the checkpoints after tasks finished and revising the process of finishing, we can support checkpoints for jobs with
+both bounded and unbounded sources, and ensure the bounded job gets all output records committed before it finishes. The motivation
+behind this change is to ensure data consistency, results completeness, and failure recovery if there are bounded sources in the pipeline.
+The final checkpoint mechanism was first implemented in Flink 1.14 and enabled by default in Flink 1.15. If you have any questions,
+please feel free to start a discussion or report an issue in the dev or user mailing list.</p>
+</description>
+<pubDate>Mon, 11 Jul 2022 02:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2022/07/11/final-checkpoint-part1.html</link>
+<guid isPermaLink="true">/2022/07/11/final-checkpoint-part1.html</guid>
+</item>
+
<item>
<title>Apache Flink ML 2.1.0 Release Announcement</title>
<description><p>The Apache Flink community is excited to announce the release of Flink ML 2.1.0!
@@ -19953,287 +20366,5 @@ To understand why this is the case, let us start with articulating a realistic s
<guid isPermaLink="true">/news/2020/01/15/demo-fraud-detection.html</guid>
</item>
-<item>
-<title>Apache Flink 1.8.3 Released</title>
-<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p>
-
-<p>This release includes 45 fixes and minor improvements for Flink 1.8.2. The list below includes a detailed list of all fixes and improvements.</p>
-
-<p>We highly recommend all users to upgrade to Flink 1.8.3.</p>
-
-<p>Updated Maven dependencies:</p>
-
-<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
- <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
- <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
- <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
-<span class="nt">&lt;/dependency&gt;</span>
-<span class="nt">&lt;dependency&gt;</span>
- <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
- <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
- <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
-<span class="nt">&lt;/dependency&gt;</span>
-<span class="nt">&lt;dependency&gt;</span>
- <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
- <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
- <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
-<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-
-<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-
-<p>List of resolved issues:</p>
-
-<h2> Sub-task
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by
-</li>
-</ul>
-
-<h2> Bug
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13728">FLINK-13728</a>] - Fix wrong closing tag order in sidenav
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServerTest.testHistoryServerIntegration failed on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread
-</li>
-</ul>
-
-<h2> Improvement
-</h2>
-<ul>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13991">FLINK-13991</a>] - Add git exclusion for 1.9+ features to 1.8
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Fix the document of Metrics that has an error for `User Scope`
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
-</li>
-<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect
-</li>
-</ul>
-</description>
-<pubDate>Wed, 11 Dec 2019 13:00:00 +0100</pubDate>
-<link>https://flink.apache.org/news/2019/12/11/release-1.8.3.html</link>
-<guid isPermaLink="true">/news/2019/12/11/release-1.8.3.html</guid>
-</item>
-
-<item>
-<title>Running Apache Flink on Kubernetes with KUDO</title>
-<description><p>A common use case for Apache Flink is streaming data analytics together with Apache Kafka, which provides a pub/sub model and durability for data streams. To achieve elastic scalability, both are typically deployed in clustered environments, and increasingly on top of container orchestration platforms like Kubernetes. The <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Operator pattern</a> provides an extension mechani [...]
-
-<p>In this blog post we demonstrate how to orchestrate a streaming data analytics application based on Flink and Kafka with KUDO. It consists of a Flink job that checks financial transactions for fraud, and two microservices that generate and display the transactions. You can find more details about this demo in the <a href="https://github.com/kudobuilder/operators/tree/master/repository/flink/docs/demo/financial-fraud">KUDO Operators repository</a>, including [...]
-
-<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
- <img src="/img/blog/2019-11-06-flink-kubernetes-kudo/flink-kudo-architecture.png" width="600px" alt="Application: My App" />
-</p>
-
-<h2 id="prerequisites">Prerequisites</h2>
-
-<p>You can run this demo on your local machine using <a href="https://github.com/kubernetes/minikube">minikube</a>. The instructions below were tested with minikube v1.5.1 and Kubernetes v1.16.2 but should work on any Kubernetes version above v1.15.0. First, start a minikube cluster with enough capacity:</p>
-
-<p><code>minikube start --cpus=6 --memory=9216 --disk-size=10g</code></p>
-
-<p>If you’re using a different way to provision Kubernetes, make sure you have at least 6 CPU Cores, 9 GB of RAM and 10 GB of disk space available.</p>
-
-<p>Install the <code>kubectl</code> CLI tool. The KUDO CLI is a plugin for the Kubernetes CLI. The official instructions for installing and setting up kubectl are <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">here</a>.</p>
-
-<p>Next, let’s install the KUDO CLI. At the time of this writing, the latest KUDO version is v0.10.0. You can find the CLI binaries for download <a href="https://github.com/kudobuilder/kudo/releases">here</a>. Download the <code>kubectl-kudo</code> binary for your OS and architecture.</p>
-
-<p>If you’re using Homebrew on MacOS, you can install the CLI via:</p>
-
-<div class="highlight"><pre><code>$ brew tap kudobuilder/tap
-$ brew install kudo-cli
-</code></pre></div>
-
-<p>Now, let’s initialize KUDO on our Kubernetes cluster:</p>
-
-<div class="highlight"><pre><code>$ kubectl kudo init
-$KUDO_HOME has been configured at /Users/gerred/.kudo
-</code></pre></div>
-
-<p>This will create several resources. First, it will create the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Custom Resource Definitions</a>, <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/">service account</a>, and <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">role bindings</a> necessary for KUD [...]
-
-<p>The KUDO CLI leverages the kubectl plugin system, which gives you all its functionality under <code>kubectl kudo</code>. This is a convenient way to install and deal with your KUDO Operators. For our demo, we use Kafka and Flink which depend on ZooKeeper. To make the ZooKeeper Operator available on the cluster, run:</p>
-
-<div class="highlight"><pre><code>$ kubectl kudo install zookeeper --version=0.3.0 --skip-instance
-</code></pre></div>
-
-<p>The –skip-instance flag skips the creation of a ZooKeeper instance. The flink-demo Operator that we’re going to install below will create it as a dependency instead. Now let’s make the Kafka and Flink Operators available the same way:</p>
-
-<div class="highlight"><pre><code>$ kubectl kudo install kafka --version=1.2.0 --skip-instance
-</code></pre></div>
-
-<div class="highlight"><pre><code>$ kubectl kudo install flink --version=0.2.1 --skip-instance
-</code></pre></div>
-
-<p>This installs all the Operator versions needed for our demo.</p>
-
-<h2 id="financial-fraud-demo">Financial Fraud Demo</h2>
-
-<p>In our financial fraud demo we have two microservices, called “generator” and “actor”. The generator produces transactions with random amounts and writes them into a Kafka topic. Occasionally, the value will be over 10,000 which is considered fraud for the purpose of this demo. The Flink job subscribes to the Kafka topic and detects fraudulent transactions. When it does, it submits them to another Kafka topic which the actor consumes. The actor simply displays each fraudulent tr [...]
-
-<p>The KUDO CLI by default installs Operators from the <a href="https://github.com/kudobuilder/operators/">official repository</a>, but it also supports installation from your local filesystem. This is useful if you want to develop your own Operator, or modify this demo for your own purposes.</p>
-
-<p>First, clone the “kudobuilder/operators” repository via:</p>
-
-<div class="highlight"><pre><code>$ git clone https://github.com/kudobuilder/operators.git
-</code></pre></div>
-
-<p>Next, change into the “operators” directory and install the demo-operator from your local filesystem:</p>
-
-<div class="highlight"><pre><code>$ cd operators
-$ kubectl kudo install repository/flink/docs/demo/financial-fraud/demo-operator --instance flink-demo
-instance.kudo.dev/v1beta1/flink-demo created
-</code></pre></div>
-
-<p>This time we didn’t include the –skip-instance flag, so KUDO will actually deploy all the components, including Flink, Kafka, and ZooKeeper. KUDO orchestrates deployments and other lifecycle operations using <a href="https://kudo.dev/docs/concepts.html#plan">plans</a> that were defined by the Operator developer. Plans are similar to <a href="https://en.wikipedia.org/wiki/Runbook">runbooks</a> and encapsulate all the procedures required [...]
-
-<div class="highlight"><pre><code>$ kubectl kudo plan status --instance flink-demo
-Plan(s) for "flink-demo" in namespace "default":
-.
-└── flink-demo (Operator-Version: "flink-demo-0.1.4" Active-Plan: "deploy")
- └── Plan deploy (serial strategy) [IN_PROGRESS]
- ├── Phase dependencies [IN_PROGRESS]
- │ ├── Step zookeeper (COMPLETE)
- │ └── Step kafka (IN_PROGRESS)
- ├── Phase flink-cluster [PENDING]
- │ └── Step flink (PENDING)
- ├── Phase demo [PENDING]
- │ ├── Step gen (PENDING)
- │ └── Step act (PENDING)
- └── Phase flink-job [PENDING]
- └── Step submit (PENDING)
-</code></pre></div>
-
-<p>The output shows that the “deploy” plan is in progress and that it consists of 4 phases: “dependencies”, “flink-cluster”, “demo” and “flink-job”. The “dependencies” phase includes steps for “zookeeper” and “kafka”. This is where both dependencies get installed, before KUDO continues to install the Flink cluster and the demo itself. We also see that ZooKeeper installation completed, and that Kafka installation is currently in progress. We can view details about Kafka’s deployment [...]
-
-<div class="highlight"><pre><code>$ kubectl kudo plan status --instance flink-demo-kafka
-Plan(s) for "flink-demo-kafka" in namespace "default":
-.
-└── flink-demo-kafka (Operator-Version: "kafka-1.2.0" Active-Plan: "deploy")
- ├── Plan deploy (serial strategy) [IN_PROGRESS]
- │ └── Phase deploy-kafka [IN_PROGRESS]
- │ └── Step deploy (IN_PROGRESS)
- └── Plan not-allowed (serial strategy) [NOT ACTIVE]
- └── Phase not-allowed (serial strategy) [NOT ACTIVE]
- └── Step not-allowed (serial strategy) [NOT ACTIVE]
- └── not-allowed [NOT ACTIVE]
-</code></pre></div>
-
-<p>After Kafka was successfully installed the next phase “flink-cluster” will start and bring up, you guessed it, your flink-cluster. After this is done, the demo phase creates the generator and actor pods that generate and display transactions for this demo. Lastly, we have the flink-job phase in which we submit the actual FinancialFraudJob to the Flink cluster. Once the flink job is submitted, we will be able to see fraud logs in our actor pod shortly after.</p>
-
-<p>After a while, the state of all plans, phases and steps will change to “COMPLETE”. Now we can view the Flink dashboard to verify that our job is running. To access it from outside the Kubernetes cluster, first start the client proxy, then open the URL below in your browser:</p>
-
-<div class="highlight"><pre><code>$ kubectl proxy
-</code></pre></div>
-
-<p><a href="http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview">http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview</a></p>
-
-<p>It should look similar to this, depending on your local machine and how many cores you have available:</p>
-
-<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
- <img src="/img/blog/2019-11-06-flink-kubernetes-kudo/flink-dashboard-ui.png" width="600px" alt="Application: My App" />
-</p>
-
-<p>The job is up and running and we should now be able to see fraudulent transaction in the logs of the actor pod:</p>
-
-<div class="highlight"><pre><code>$ kubectl logs $(kubectl get pod -l actor=flink-demo -o jsonpath="{.items[0].metadata.name}")
-Broker: flink-demo-kafka-kafka-0.flink-demo-kafka-svc:9093
-Topic: fraud
-
-Detected Fraud: TransactionAggregate {startTimestamp=0, endTimestamp=1563395831000, totalAmount=19895:
-Transaction{timestamp=1563395778000, origin=1, target='3', amount=8341}
-Transaction{timestamp=1563395813000, origin=1, target='3', amount=8592}
-Transaction{timestamp=1563395817000, origin=1, target='3', amount=2802}
-Transaction{timestamp=1563395831000, origin=1, target='3', amount=160}}
-</code></pre></div>
-
-<p>If you add the “-f” flag to the previous command, you can follow along as more transactions are streaming in and are evaluated by our Flink job.</p>
-
-<h2 id="conclusion">Conclusion</h2>
-
-<p>In this blog post we demonstrated how to easily deploy an end-to-end streaming data application on Kubernetes using KUDO. We deployed a Flink job and two microservices, as well as all the required infrastructure - Flink, Kafka, and ZooKeeper using just a few kubectl commands. To find out more about KUDO, visit the <a href="https://kudo.dev">project website</a> or join the community on <a href="https://kubernetes.slack.com/messages/kudo/">Sla [...]
-</description>
-<pubDate>Mon, 09 Dec 2019 13:00:00 +0100</pubDate>
-<link>https://flink.apache.org/news/2019/12/09/flink-kubernetes-kudo.html</link>
-<guid isPermaLink="true">/news/2019/12/09/flink-kubernetes-kudo.html</guid>
-</item>
-
</channel>
</rss>
diff --git a/content/blog/index.html b/content/blog/index.html
index a93601a0a..c6b88d0dd 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -236,6 +236,32 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></h2>
+
+ <p>11 Jul 2022
+ Yun Gao , Dawid Wysakowicz , & Daisy Tsang </p>
+
+ <p>This post presents more details on the changes on the checkpoint procedure and task finish process made by the final checkpoint mechanism.</p>
+
+ <p><a href="/2022/07/11/final-checkpoint-part2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></h2>
+
+ <p>11 Jul 2022
+ Yun Gao , Dawid Wysakowicz , & Daisy Tsang </p>
+
+ <p>This post briefly describes the motivation and changes made by the final checkpoint mechanism, including the changes to the checkpoint procedure and how tasks finish.</p>
+
+ <p><a href="/2022/07/11/final-checkpoint-part1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></h2>
@@ -343,35 +369,6 @@ We are now proud to announce the first production ready release of the operator
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2022/05/11/release-table-store-0.1.0.html">Apache Flink Table Store 0.1.0 Release Announcement</a></h2>
-
- <p>11 May 2022
- Jingsong Lee & Jiangjie (Becket) Qin </p>
-
- <p><p>The Apache Flink community is pleased to announce the preview release of the
-<a href="https://github.com/apache/flink-table-store">Apache Flink Table Store</a> (0.1.0).</p>
-
-</p>
-
- <p><a href="/news/2022/05/11/release-table-store-0.1.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/2022/05/06/async-sink-base.html">The Generic Asynchronous Base Sink</a></h2>
-
- <p>06 May 2022
- Zichen Liu </p>
-
- <p>An overview of the new AsyncBaseSink and how to use it for building your own concrete sink</p>
-
- <p><a href="/2022/05/06/async-sink-base.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -404,6 +401,26 @@ We are now proud to announce the first production ready release of the operator
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html
index 0a71370dc..0b7fad9b6 100644
--- a/content/blog/page10/index.html
+++ b/content/blog/page10/index.html
@@ -236,6 +236,34 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2020/04/01/community-update.html">Flink Community Update - April'20</a></h2>
+
+ <p>01 Apr 2020
+ Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
+
+ <p>While things slow down around us, the Apache Flink community is privileged to remain as active as ever. This blogpost combs through the past few months to give you an update on the state of things in Flink — from core releases to Stateful Functions; from some good old community stats to a new development blog.</p>
+
+ <p><a href="/news/2020/04/01/community-update.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/features/2020/03/27/flink-for-data-warehouse.html">Flink as Unified Engine for Modern Data Warehousing: Production-Ready Hive Integration</a></h2>
+
+ <p>27 Mar 2020
+ Bowen Li (<a href="https://twitter.com/Bowen__Li">@Bowen__Li</a>)</p>
+
+ <p><p>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.</p>
+
+</p>
+
+ <p><a href="/features/2020/03/27/flink-for-data-warehouse.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2020/03/24/demo-fraud-detection-2.html">Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</a></h2>
@@ -344,34 +372,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2019/12/11/release-1.8.3.html">Apache Flink 1.8.3 Released</a></h2>
-
- <p>11 Dec 2019
- Hequn Cheng </p>
-
- <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p>
-
-</p>
-
- <p><a href="/news/2019/12/11/release-1.8.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2019/12/09/flink-kubernetes-kudo.html">Running Apache Flink on Kubernetes with KUDO</a></h2>
-
- <p>09 Dec 2019
- Gerred Dillon </p>
-
- <p>A common use case for Apache Flink is streaming data analytics together with Apache Kafka, which provides a pub/sub model and durability for data streams. In this post, we demonstrate how to orchestrate Flink and Kafka with KUDO.</p>
-
- <p><a href="/news/2019/12/09/flink-kubernetes-kudo.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -404,6 +404,26 @@
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html
index e04b9b74e..ff5a1fe91 100644
--- a/content/blog/page11/index.html
+++ b/content/blog/page11/index.html
@@ -236,6 +236,34 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2019/12/11/release-1.8.3.html">Apache Flink 1.8.3 Released</a></h2>
+
+ <p>11 Dec 2019
+ Hequn Cheng </p>
+
+ <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p>
+
+</p>
+
+ <p><a href="/news/2019/12/11/release-1.8.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2019/12/09/flink-kubernetes-kudo.html">Running Apache Flink on Kubernetes with KUDO</a></h2>
+
+ <p>09 Dec 2019
+ Gerred Dillon </p>
+
+ <p>A common use case for Apache Flink is streaming data analytics together with Apache Kafka, which provides a pub/sub model and durability for data streams. In this post, we demonstrate how to orchestrate Flink and Kafka with KUDO.</p>
+
+ <p><a href="/news/2019/12/09/flink-kubernetes-kudo.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2019/11/25/query-pulsar-streams-using-apache-flink.html">How to query Pulsar Streams using Apache Flink</a></h2>
@@ -349,32 +377,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2019/06/26/broadcast-state.html">A Practical Guide to Broadcast State in Apache Flink</a></h2>
-
- <p>26 Jun 2019
- Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p>
-
- <p>Apache Flink has multiple types of operator state, one of which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream.</p>
-
- <p><a href="/2019/06/26/broadcast-state.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/2019/06/05/flink-network-stack.html">A Deep-Dive into Flink's Network Stack</a></h2>
-
- <p>05 Jun 2019
- Nico Kruber </p>
-
- <p>Flink’s network stack is one of the core components that make up Apache Flink's runtime module sitting at the core of every Flink job. In this post, which is the first in a series of posts about the network stack, we look at the abstractions exposed to the stream operators and detail their physical implementation and various optimisations in Apache Flink.</p>
-
- <p><a href="/2019/06/05/flink-network-stack.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -407,6 +409,26 @@
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html
index a46832226..e46a5b5a6 100644
--- a/content/blog/page12/index.html
+++ b/content/blog/page12/index.html
@@ -236,6 +236,32 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2019/06/26/broadcast-state.html">A Practical Guide to Broadcast State in Apache Flink</a></h2>
+
+ <p>26 Jun 2019
+ Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p>
+
+ <p>Apache Flink has multiple types of operator state, one of which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream.</p>
+
+ <p><a href="/2019/06/26/broadcast-state.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/2019/06/05/flink-network-stack.html">A Deep-Dive into Flink's Network Stack</a></h2>
+
+ <p>05 Jun 2019
+ Nico Kruber </p>
+
+ <p>Flink’s network stack is one of the core components that make up Apache Flink's runtime module sitting at the core of every Flink job. In this post, which is the first in a series of posts about the network stack, we look at the abstractions exposed to the stream operators and detail their physical implementation and various optimisations in Apache Flink.</p>
+
+ <p><a href="/2019/06/05/flink-network-stack.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></h2>
@@ -348,36 +374,6 @@ for more details.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2019/02/25/release-1.6.4.html">Apache Flink 1.6.4 Released</a></h2>
-
- <p>25 Feb 2019
- </p>
-
- <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.6 series.</p>
-
-</p>
-
- <p><a href="/news/2019/02/25/release-1.6.4.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2019/02/15/release-1.7.2.html">Apache Flink 1.7.2 Released</a></h2>
-
- <p>15 Feb 2019
- </p>
-
- <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.7 series.</p>
-
-</p>
-
- <p><a href="/news/2019/02/15/release-1.7.2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -410,6 +406,26 @@ for more details.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page13/index.html b/content/blog/page13/index.html
index 8f4d94f7b..b2a19978f 100644
--- a/content/blog/page13/index.html
+++ b/content/blog/page13/index.html
@@ -236,6 +236,36 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2019/02/25/release-1.6.4.html">Apache Flink 1.6.4 Released</a></h2>
+
+ <p>25 Feb 2019
+ </p>
+
+ <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.6 series.</p>
+
+</p>
+
+ <p><a href="/news/2019/02/25/release-1.6.4.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2019/02/15/release-1.7.2.html">Apache Flink 1.7.2 Released</a></h2>
+
+ <p>15 Feb 2019
+ </p>
+
+ <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.7 series.</p>
+
+</p>
+
+ <p><a href="/news/2019/02/15/release-1.7.2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></h2>
@@ -356,36 +386,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/09/20/release-1.5.4.html">Apache Flink 1.5.4 Released</a></h2>
-
- <p>20 Sep 2018
- </p>
-
- <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.5 series.</p>
-
-</p>
-
- <p><a href="/news/2018/09/20/release-1.5.4.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2018/08/21/release-1.5.3.html">Apache Flink 1.5.3 Released</a></h2>
-
- <p>21 Aug 2018
- </p>
-
- <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.5 series.</p>
-
-</p>
-
- <p><a href="/news/2018/08/21/release-1.5.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -418,6 +418,26 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page14/index.html b/content/blog/page14/index.html
index 66ae5990d..c2ae97020 100644
--- a/content/blog/page14/index.html
+++ b/content/blog/page14/index.html
@@ -236,6 +236,36 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2018/09/20/release-1.5.4.html">Apache Flink 1.5.4 Released</a></h2>
+
+ <p>20 Sep 2018
+ </p>
+
+ <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.5 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/09/20/release-1.5.4.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2018/08/21/release-1.5.3.html">Apache Flink 1.5.3 Released</a></h2>
+
+ <p>21 Aug 2018
+ </p>
+
+ <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.5 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/08/21/release-1.5.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2018/08/09/release-1.6.0.html">Apache Flink 1.6.0 Release Announcement</a></h2>
@@ -354,32 +384,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></h2>
-
- <p>30 Jan 2018
- Stefan Ricther (<a href="https://twitter.com/StefanRRicther">@StefanRRicther</a>) & Chris Ward (<a href="https://twitter.com/chrischinch">@chrischinch</a>)</p>
-
- <p>Flink 1.3.0 introduced incremental checkpointing, making it possible for applications with large state to generate checkpoints more efficiently.</p>
-
- <p><a href="/features/2018/01/30/incremental-checkpointing.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></h2>
-
- <p>21 Dec 2017
- Chris Ward (<a href="https://twitter.com/chrischinch">@chrischinch</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
-
- <p>As 2017 comes to a close, let's take a moment to look back on the Flink community's great work during the past year.</p>
-
- <p><a href="/news/2017/12/21/2017-year-in-review.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -412,6 +416,26 @@
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page15/index.html b/content/blog/page15/index.html
index 4ff91238a..6c8caa92c 100644
--- a/content/blog/page15/index.html
+++ b/content/blog/page15/index.html
@@ -236,6 +236,32 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></h2>
+
+ <p>30 Jan 2018
+ Stefan Ricther (<a href="https://twitter.com/StefanRRicther">@StefanRRicther</a>) & Chris Ward (<a href="https://twitter.com/chrischinch">@chrischinch</a>)</p>
+
+ <p>Flink 1.3.0 introduced incremental checkpointing, making it possible for applications with large state to generate checkpoints more efficiently.</p>
+
+ <p><a href="/features/2018/01/30/incremental-checkpointing.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></h2>
+
+ <p>21 Dec 2017
+ Chris Ward (<a href="https://twitter.com/chrischinch">@chrischinch</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
+
+ <p>As 2017 comes to a close, let's take a moment to look back on the Flink community's great work during the past year.</p>
+
+ <p><a href="/news/2017/12/21/2017-year-in-review.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2017/12/12/release-1.4.0.html">Apache Flink 1.4.0 Release Announcement</a></h2>
@@ -359,33 +385,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/04/04/dynamic-tables.html">Continuous Queries on Dynamic Tables</a></h2>
-
- <p>04 Apr 2017 by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang
- </p>
-
- <p><p>Flink's relational APIs, the Table API and SQL, are unified APIs for stream and batch processing, meaning that a query produces the same result when being evaluated on streaming or static data.</p>
-<p>In this blog post we discuss the future of these APIs and introduce the concept of Dynamic Tables. Dynamic tables will significantly expand the scope of the Table API and SQL on streams and enable many more advanced use cases. We discuss how streams and dynamic tables relate to each other and explain the semantics of continuously evaluating queries on dynamic tables.</p></p>
-
- <p><a href="/news/2017/04/04/dynamic-tables.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></h2>
-
- <p>29 Mar 2017 by Timo Walther (<a href="https://twitter.com/">@twalthr</a>)
- </p>
-
- <p><p>Broadening the user base and unifying batch & streaming with relational APIs</p></p>
-
- <p><a href="/news/2017/03/29/table-sql-api-update.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -418,6 +417,26 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page16/index.html b/content/blog/page16/index.html
index 1d1414cfc..f81ee8ea4 100644
--- a/content/blog/page16/index.html
+++ b/content/blog/page16/index.html
@@ -236,6 +236,33 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2017/04/04/dynamic-tables.html">Continuous Queries on Dynamic Tables</a></h2>
+
+ <p>04 Apr 2017 by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang
+ </p>
+
+ <p><p>Flink's relational APIs, the Table API and SQL, are unified APIs for stream and batch processing, meaning that a query produces the same result when being evaluated on streaming or static data.</p>
+<p>In this blog post we discuss the future of these APIs and introduce the concept of Dynamic Tables. Dynamic tables will significantly expand the scope of the Table API and SQL on streams and enable many more advanced use cases. We discuss how streams and dynamic tables relate to each other and explain the semantics of continuously evaluating queries on dynamic tables.</p></p>
+
+ <p><a href="/news/2017/04/04/dynamic-tables.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></h2>
+
+ <p>29 Mar 2017 by Timo Walther (<a href="https://twitter.com/">@twalthr</a>)
+ </p>
+
+ <p><p>Broadening the user base and unifying batch & streaming with relational APIs</p></p>
+
+ <p><a href="/news/2017/03/29/table-sql-api-update.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2>
@@ -352,35 +379,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/08/08/release-1.1.0.html">Announcing Apache Flink 1.1.0</a></h2>
-
- <p>08 Aug 2016
- </p>
-
- <p><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div>
-
-</p>
-
- <p><a href="/news/2016/08/08/release-1.1.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2016/05/24/stream-sql.html">Stream Processing for Everyone with SQL and Apache Flink</a></h2>
-
- <p>24 May 2016 by Fabian Hueske (<a href="https://twitter.com/">@fhueske</a>)
- </p>
-
- <p><p>About six months ago, the Apache Flink community started an effort to add a SQL interface for stream data analysis. SQL is <i>the</i> standard language to access and process data. Everybody who occasionally analyzes data is familiar with SQL. Consequently, a SQL interface for stream data processing will make this technology accessible to a much wider audience. Moreover, SQL support for streaming data will also enable new use cases such as interactive and ad-hoc stream analysi [...]
-<p>In this blog post, we report on the current status, architectural design, and future plans of the Apache Flink community to implement support for SQL as a language for analyzing data streams.</p></p>
-
- <p><a href="/news/2016/05/24/stream-sql.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -413,6 +411,26 @@
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page17/index.html b/content/blog/page17/index.html
index c446a633e..cb193be00 100644
--- a/content/blog/page17/index.html
+++ b/content/blog/page17/index.html
@@ -236,6 +236,35 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2016/08/08/release-1.1.0.html">Announcing Apache Flink 1.1.0</a></h2>
+
+ <p>08 Aug 2016
+ </p>
+
+ <p><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div>
+
+</p>
+
+ <p><a href="/news/2016/08/08/release-1.1.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2016/05/24/stream-sql.html">Stream Processing for Everyone with SQL and Apache Flink</a></h2>
+
+ <p>24 May 2016 by Fabian Hueske (<a href="https://twitter.com/">@fhueske</a>)
+ </p>
+
+ <p><p>About six months ago, the Apache Flink community started an effort to add a SQL interface for stream data analysis. SQL is <i>the</i> standard language to access and process data. Everybody who occasionally analyzes data is familiar with SQL. Consequently, a SQL interface for stream data processing will make this technology accessible to a much wider audience. Moreover, SQL support for streaming data will also enable new use cases such as interactive and ad-hoc stream analysi [...]
+<p>In this blog post, we report on the current status, architectural design, and future plans of the Apache Flink community to implement support for SQL as a language for analyzing data streams.</p></p>
+
+ <p><a href="/news/2016/05/24/stream-sql.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2>
@@ -352,33 +381,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/12/11/storm-compatibility.html">Storm Compatibility in Apache Flink: How to run existing Storm topologies on Flink</a></h2>
-
- <p>11 Dec 2015 by Matthias J. Sax (<a href="https://twitter.com/">@MatthiasJSax</a>)
- </p>
-
- <p>In this blog post, we describe Flink's compatibility package for <a href="https://storm.apache.org">Apache Storm</a> that allows to embed Spouts (sources) and Bolts (operators) in a regular Flink streaming job. Furthermore, the compatibility package provides a Storm compatible API in order to execute whole Storm topologies with (almost) no code adaption.</p>
-
- <p><a href="/news/2015/12/11/storm-compatibility.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2015/12/04/Introducing-windows.html">Introducing Stream Windows in Apache Flink</a></h2>
-
- <p>04 Dec 2015 by Fabian Hueske (<a href="https://twitter.com/">@fhueske</a>)
- </p>
-
- <p><p>The data analysis space is witnessing an evolution from batch to stream processing for many use cases. Although batch can be handled as a special case of stream processing, analyzing never-ending streaming data often requires a shift in the mindset and comes with its own terminology (for example, “windowing” and “at-least-once”/”exactly-once” processing). This shift and the new terminology can be quite confusing for people being new to the space of stream processing. Apache F [...]
-<p>In this blog post, we discuss the concept of windows for stream processing, present Flink's built-in windows, and explain its support for custom windowing semantics.</p></p>
-
- <p><a href="/news/2015/12/04/Introducing-windows.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -411,6 +413,26 @@
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page18/index.html b/content/blog/page18/index.html
index b9bc3da09..0652cd41a 100644
--- a/content/blog/page18/index.html
+++ b/content/blog/page18/index.html
@@ -236,6 +236,33 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2015/12/11/storm-compatibility.html">Storm Compatibility in Apache Flink: How to run existing Storm topologies on Flink</a></h2>
+
+ <p>11 Dec 2015 by Matthias J. Sax (<a href="https://twitter.com/">@MatthiasJSax</a>)
+ </p>
+
+ <p>In this blog post, we describe Flink's compatibility package for <a href="https://storm.apache.org">Apache Storm</a> that allows to embed Spouts (sources) and Bolts (operators) in a regular Flink streaming job. Furthermore, the compatibility package provides a Storm compatible API in order to execute whole Storm topologies with (almost) no code adaption.</p>
+
+ <p><a href="/news/2015/12/11/storm-compatibility.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2015/12/04/Introducing-windows.html">Introducing Stream Windows in Apache Flink</a></h2>
+
+ <p>04 Dec 2015 by Fabian Hueske (<a href="https://twitter.com/">@fhueske</a>)
+ </p>
+
+ <p><p>The data analysis space is witnessing an evolution from batch to stream processing for many use cases. Although batch can be handled as a special case of stream processing, analyzing never-ending streaming data often requires a shift in the mindset and comes with its own terminology (for example, “windowing” and “at-least-once”/”exactly-once” processing). This shift and the new terminology can be quite confusing for people being new to the space of stream processing. Apache F [...]
+<p>In this blog post, we discuss the concept of windows for stream processing, present Flink's built-in windows, and explain its support for custom windowing semantics.</p></p>
+
+ <p><a href="/news/2015/12/04/Introducing-windows.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2>
@@ -360,40 +387,6 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Juggling with Bits and Bytes</a></h2>
-
- <p>11 May 2015 by Fabian Hüske (<a href="https://twitter.com/">@fhueske</a>)
- </p>
-
- <p><p>Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the [...]
-<p>In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.</p></p>
-
- <p><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Announcing Flink 0.9.0-milestone1 preview release</a></h2>
-
- <p>13 Apr 2015
- </p>
-
- <p><p>The Apache Flink community is pleased to announce the availability of
-the 0.9.0-milestone-1 release. The release is a preview of the
-upcoming 0.9.0 release. It contains many new features which will be
-available in the upcoming 0.9 release. Interested users are encouraged
-to try it out and give feedback. As the version number indicates, this
-release is a preview release that contains known issues.</p>
-
-</p>
-
- <p><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -426,6 +419,26 @@ release is a preview release that contains known issues.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page19/index.html b/content/blog/page19/index.html
index 295a05a11..b262e784b 100644
--- a/content/blog/page19/index.html
+++ b/content/blog/page19/index.html
@@ -236,6 +236,40 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Juggling with Bits and Bytes</a></h2>
+
+ <p>11 May 2015 by Fabian Hüske (<a href="https://twitter.com/">@fhueske</a>)
+ </p>
+
+ <p><p>Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the [...]
+<p>In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.</p></p>
+
+ <p><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Announcing Flink 0.9.0-milestone1 preview release</a></h2>
+
+ <p>13 Apr 2015
+ </p>
+
+ <p><p>The Apache Flink community is pleased to announce the availability of
+the 0.9.0-milestone-1 release. The release is a preview of the
+upcoming 0.9.0 release. It contains many new features which will be
+available in the upcoming 0.9 release. Interested users are encouraged
+to try it out and give feedback. As the version number indicates, this
+release is a preview release that contains known issues.</p>
+
+</p>
+
+ <p><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2>
@@ -361,36 +395,6 @@ and offers a new API including definition of flexible windows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2014/11/04/release-0.7.0.html">Apache Flink 0.7.0 available</a></h2>
-
- <p>04 Nov 2014
- </p>
-
- <p><p>We are pleased to announce the availability of Flink 0.7.0. This release includes new user-facing features as well as performance and bug fixes, brings the Scala and Java APIs in sync, and introduces Flink Streaming. A total of 34 people have contributed to this release, a big thanks to all of them!</p>
-
-</p>
-
- <p><a href="/news/2014/11/04/release-0.7.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2014/10/03/upcoming_events.html">Upcoming Events</a></h2>
-
- <p>03 Oct 2014
- </p>
-
- <p><p>We are happy to announce several upcoming Flink events both in Europe and the US. Starting with a <strong>Flink hackathon in Stockholm</strong> (Oct 8-9) and a talk about Flink at the <strong>Stockholm Hadoop User Group</strong> (Oct 8). This is followed by the very first <strong>Flink Meetup in Berlin</strong> (Oct 15). In the US, there will be two Flink Meetup talks: the first one at the <strong>Pasadena Big Data User Group</strong> (Oct 29) and the second one at <strong>Si [...]
-
-</p>
-
- <p><a href="/news/2014/10/03/upcoming_events.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -423,6 +427,26 @@ and offers a new API including definition of flexible windows.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index e9cfbcf36..9f4a3b952 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -236,6 +236,35 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2022/05/11/release-table-store-0.1.0.html">Apache Flink Table Store 0.1.0 Release Announcement</a></h2>
+
+ <p>11 May 2022
+ Jingsong Lee & Jiangjie (Becket) Qin </p>
+
+ <p><p>The Apache Flink community is pleased to announce the preview release of the
+<a href="https://github.com/apache/flink-table-store">Apache Flink Table Store</a> (0.1.0).</p>
+
+</p>
+
+ <p><a href="/news/2022/05/11/release-table-store-0.1.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/2022/05/06/async-sink-base.html">The Generic Asynchronous Base Sink</a></h2>
+
+ <p>06 May 2022
+ Zichen Liu </p>
+
+ <p>An overview of the new AsyncBaseSink and how to use it for building your own concrete sink</p>
+
+ <p><a href="/2022/05/06/async-sink-base.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/2022/05/06/pyflink-1.15-thread-mode.html">Exploring the thread mode in PyFlink</a></h2>
@@ -351,32 +380,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<hr>
- <article>
- <h2 class="blog-title"><a href="/2022/01/20/pravega-connector-101.html">Pravega Flink Connector 101</a></h2>
-
- <p>20 Jan 2022
- Yumin Zhou (Brian) (<a href="https://twitter.com/crazy__zhou">@crazy__zhou</a>)</p>
-
- <p>A brief introduction to the Pravega Flink Connector</p>
-
- <p><a href="/2022/01/20/pravega-connector-101.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2022/01/17/release-1.14.3.html">Apache Flink 1.14.3 Release Announcement</a></h2>
-
- <p>17 Jan 2022
- Thomas Weise (<a href="https://twitter.com/thweise">@thweise</a>) & Martijn Visser (<a href="https://twitter.com/martijnvisser82">@martijnvisser82</a>)</p>
-
- <p>The Apache Flink community released the second bugfix version of the Apache Flink 1.14 series.</p>
-
- <p><a href="/news/2022/01/17/release-1.14.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -409,6 +412,26 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page20/index.html b/content/blog/page20/index.html
index 9d42d6614..ed49b86de 100644
--- a/content/blog/page20/index.html
+++ b/content/blog/page20/index.html
@@ -236,6 +236,36 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2014/11/04/release-0.7.0.html">Apache Flink 0.7.0 available</a></h2>
+
+ <p>04 Nov 2014
+ </p>
+
+ <p><p>We are pleased to announce the availability of Flink 0.7.0. This release includes new user-facing features as well as performance and bug fixes, brings the Scala and Java APIs in sync, and introduces Flink Streaming. A total of 34 people have contributed to this release, a big thanks to all of them!</p>
+
+</p>
+
+ <p><a href="/news/2014/11/04/release-0.7.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2014/10/03/upcoming_events.html">Upcoming Events</a></h2>
+
+ <p>03 Oct 2014
+ </p>
+
+ <p><p>We are happy to announce several upcoming Flink events both in Europe and the US. Starting with a <strong>Flink hackathon in Stockholm</strong> (Oct 8-9) and a talk about Flink at the <strong>Stockholm Hadoop User Group</strong> (Oct 8). This is followed by the very first <strong>Flink Meetup in Berlin</strong> (Oct 15). In the US, there will be two Flink Meetup talks: the first one at the <strong>Pasadena Big Data User Group</strong> (Oct 29) and the second one at <strong>Si [...]
+
+</p>
+
+ <p><a href="/news/2014/10/03/upcoming_events.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2014/09/26/release-0.6.1.html">Apache Flink 0.6.1 available</a></h2>
@@ -301,6 +331,26 @@ academic and open source project that Flink originates from.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index 84c61d125..a147677ee 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -236,6 +236,32 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2022/01/20/pravega-connector-101.html">Pravega Flink Connector 101</a></h2>
+
+ <p>20 Jan 2022
+ Yumin Zhou (Brian) (<a href="https://twitter.com/crazy__zhou">@crazy__zhou</a>)</p>
+
+ <p>A brief introduction to the Pravega Flink Connector</p>
+
+ <p><a href="/2022/01/20/pravega-connector-101.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2022/01/17/release-1.14.3.html">Apache Flink 1.14.3 Release Announcement</a></h2>
+
+ <p>17 Jan 2022
+ Thomas Weise (<a href="https://twitter.com/thweise">@thweise</a>) & Martijn Visser (<a href="https://twitter.com/martijnvisser82">@martijnvisser82</a>)</p>
+
+ <p>The Apache Flink community released the second bugfix version of the Apache Flink 1.14 series.</p>
+
+ <p><a href="/news/2022/01/17/release-1.14.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2022/01/07/release-ml-2.0.0.html">Apache Flink ML 2.0.0 Release Announcement</a></h2>
@@ -344,34 +370,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2021/10/26/sort-shuffle-part1.html">Sort-Based Blocking Shuffle Implementation in Flink - Part One</a></h2>
-
- <p>26 Oct 2021
- Yingjie Cao (Kevin) & Daisy Tsang </p>
-
- <p>Flink has implemented the sort-based blocking shuffle (FLIP-148) for batch data processing. In this blog post, we will take a close look at the design & implementation details and see what we can gain from it.</p>
-
- <p><a href="/2021/10/26/sort-shuffle-part1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2021/10/19/release-1.13.3.html">Apache Flink 1.13.3 Released</a></h2>
-
- <p>19 Oct 2021
- Chesnay Schepler </p>
-
- <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.13 series.</p>
-
-</p>
-
- <p><a href="/news/2021/10/19/release-1.13.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -404,6 +402,26 @@
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index 2e28654f7..c1856ea09 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -236,6 +236,34 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2021/10/26/sort-shuffle-part1.html">Sort-Based Blocking Shuffle Implementation in Flink - Part One</a></h2>
+
+ <p>26 Oct 2021
+ Yingjie Cao (Kevin) & Daisy Tsang </p>
+
+ <p>Flink has implemented the sort-based blocking shuffle (FLIP-148) for batch data processing. In this blog post, we will take a close look at the design & implementation details and see what we can gain from it.</p>
+
+ <p><a href="/2021/10/26/sort-shuffle-part1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2021/10/19/release-1.13.3.html">Apache Flink 1.13.3 Released</a></h2>
+
+ <p>19 Oct 2021
+ Chesnay Schepler </p>
+
+ <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.13 series.</p>
+
+</p>
+
+ <p><a href="/news/2021/10/19/release-1.13.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2021/09/29/release-1.14.0.html">Apache Flink 1.14.0 Release Announcement</a></h2>
@@ -362,34 +390,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<hr>
- <article>
- <h2 class="blog-title"><a href="/2021/07/07/backpressure.html">How to identify the source of backpressure?</a></h2>
-
- <p>07 Jul 2021
- Piotr Nowojski (<a href="https://twitter.com/PiotrNowojski">@PiotrNowojski</a>)</p>
-
- <p>Apache Flink 1.13 introduced a couple of important changes in the area of backpressure monitoring and performance analysis of Flink Jobs. This blog post aims to introduce those changes and explain how to use them.</p>
-
- <p><a href="/2021/07/07/backpressure.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2021/05/28/release-1.13.1.html">Apache Flink 1.13.1 Released</a></h2>
-
- <p>28 May 2021
- Dawid Wysakowicz (<a href="https://twitter.com/dwysakowicz">@dwysakowicz</a>)</p>
-
- <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.13 series.</p>
-
-</p>
-
- <p><a href="/news/2021/05/28/release-1.13.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -422,6 +422,26 @@ This new release brings various improvements to the StateFun runtime, a leaner w
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index 652dc6cd9..10b406ea2 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -236,6 +236,34 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2021/07/07/backpressure.html">How to identify the source of backpressure?</a></h2>
+
+ <p>07 Jul 2021
+ Piotr Nowojski (<a href="https://twitter.com/PiotrNowojski">@PiotrNowojski</a>)</p>
+
+ <p>Apache Flink 1.13 introduced a couple of important changes in the area of backpressure monitoring and performance analysis of Flink Jobs. This blog post aims to introduce those changes and explain how to use them.</p>
+
+ <p><a href="/2021/07/07/backpressure.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2021/05/28/release-1.13.1.html">Apache Flink 1.13.1 Released</a></h2>
+
+ <p>28 May 2021
+ Dawid Wysakowicz (<a href="https://twitter.com/dwysakowicz">@dwysakowicz</a>)</p>
+
+ <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.13 series.</p>
+
+</p>
+
+ <p><a href="/news/2021/05/28/release-1.13.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2021/05/21/release-1.12.4.html">Apache Flink 1.12.4 Released</a></h2>
@@ -350,36 +378,6 @@ to develop scalable, consistent, and elastic distributed applications.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2021/01/29/release-1.10.3.html">Apache Flink 1.10.3 Released</a></h2>
-
- <p>29 Jan 2021
- Xintong Song </p>
-
- <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.10 series.</p>
-
-</p>
-
- <p><a href="/news/2021/01/29/release-1.10.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2021/01/19/release-1.12.1.html">Apache Flink 1.12.1 Released</a></h2>
-
- <p>19 Jan 2021
- Xintong Song </p>
-
- <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.12 series.</p>
-
-</p>
-
- <p><a href="/news/2021/01/19/release-1.12.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -412,6 +410,26 @@ to develop scalable, consistent, and elastic distributed applications.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 1284553a7..c8e9b3707 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -236,6 +236,36 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2021/01/29/release-1.10.3.html">Apache Flink 1.10.3 Released</a></h2>
+
+ <p>29 Jan 2021
+ Xintong Song </p>
+
+ <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.10 series.</p>
+
+</p>
+
+ <p><a href="/news/2021/01/29/release-1.10.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2021/01/19/release-1.12.1.html">Apache Flink 1.12.1 Released</a></h2>
+
+ <p>19 Jan 2021
+ Xintong Song </p>
+
+ <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.12 series.</p>
+
+</p>
+
+ <p><a href="/news/2021/01/19/release-1.12.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></h2>
@@ -346,32 +376,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">From Aligned to Unaligned Checkpoints - Part 1: Checkpoints, Alignment, and Backpressure</a></h2>
-
- <p>15 Oct 2020
- Arvid Heise & Stephan Ewen </p>
-
- <p>Apache Flink’s checkpoint-based fault tolerance mechanism is one of its defining features. Because of that design, Flink unifies batch and stream processing, can easily scale to both very small and extremely large scenarios and provides support for many operational features. In this post we recap the original checkpointing process in Flink, its core properties and issues under backpressure.</p>
-
- <p><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2020/10/13/stateful-serverless-internals.html">Stateful Functions Internals: Behind the scenes of Stateful Serverless</a></h2>
-
- <p>13 Oct 2020
- Tzu-Li (Gordon) Tai (<a href="https://twitter.com/tzulitai">@tzulitai</a>)</p>
-
- <p>This blog post dives deep into the internals of the StateFun runtime, taking a look at how it enables consistent and fault-tolerant stateful serverless applications.</p>
-
- <p><a href="/news/2020/10/13/stateful-serverless-internals.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -404,6 +408,26 @@
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index 427f8abf6..934fcc4e1 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -236,6 +236,32 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">From Aligned to Unaligned Checkpoints - Part 1: Checkpoints, Alignment, and Backpressure</a></h2>
+
+ <p>15 Oct 2020
+ Arvid Heise & Stephan Ewen </p>
+
+ <p>Apache Flink’s checkpoint-based fault tolerance mechanism is one of its defining features. Because of that design, Flink unifies batch and stream processing, can easily scale to both very small and extremely large scenarios and provides support for many operational features. In this post we recap the original checkpointing process in Flink, its core properties and issues under backpressure.</p>
+
+ <p><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2020/10/13/stateful-serverless-internals.html">Stateful Functions Internals: Behind the scenes of Stateful Serverless</a></h2>
+
+ <p>13 Oct 2020
+ Tzu-Li (Gordon) Tai (<a href="https://twitter.com/tzulitai">@tzulitai</a>)</p>
+
+ <p>This blog post dives deep into the internals of the StateFun runtime, taking a look at how it enables consistent and fault-tolerant stateful serverless applications.</p>
+
+ <p><a href="/news/2020/10/13/stateful-serverless-internals.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2020/09/28/release-statefun-2.2.0.html">Stateful Functions 2.2.0 Release Announcement</a></h2>
@@ -352,32 +378,6 @@ as well as increased observability for operational purposes.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink: The integration of Pandas into PyFlink</a></h2>
-
- <p>04 Aug 2020
- Jincheng Sun (<a href="https://twitter.com/sunjincheng121">@sunjincheng121</a>) & Markos Sfikas (<a href="https://twitter.com/MarkSfik">@MarkSfik</a>)</p>
-
- <p>The Apache Flink community put some great effort into integrating Pandas with PyFlink in the latest Flink version 1.11. Some of the added features include support for Pandas UDF and the conversion between Pandas DataFrame and Table. In this article, we will introduce how these functionalities work and how to use them with a step-by-step example.</p>
-
- <p><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2020/07/30/demo-fraud-detection-3.html">Advanced Flink Application Patterns Vol.3: Custom Window Processing</a></h2>
-
- <p>30 Jul 2020
- Alexander Fedulov (<a href="https://twitter.com/alex_fedulov">@alex_fedulov</a>)</p>
-
- <p>In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.</p>
-
- <p><a href="/news/2020/07/30/demo-fraud-detection-3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -410,6 +410,26 @@ as well as increased observability for operational purposes.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index 83c70d905..8911ada41 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -236,6 +236,32 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink: The integration of Pandas into PyFlink</a></h2>
+
+ <p>04 Aug 2020
+ Jincheng Sun (<a href="https://twitter.com/sunjincheng121">@sunjincheng121</a>) & Markos Sfikas (<a href="https://twitter.com/MarkSfik">@MarkSfik</a>)</p>
+
+ <p>The Apache Flink community put some great effort into integrating Pandas with PyFlink in the latest Flink version 1.11. Some of the added features include support for Pandas UDF and the conversion between Pandas DataFrame and Table. In this article, we will introduce how these functionalities work and how to use them with a step-by-step example.</p>
+
+ <p><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2020/07/30/demo-fraud-detection-3.html">Advanced Flink Application Patterns Vol.3: Custom Window Processing</a></h2>
+
+ <p>30 Jul 2020
+ Alexander Fedulov (<a href="https://twitter.com/alex_fedulov">@alex_fedulov</a>)</p>
+
+ <p>In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.</p>
+
+ <p><a href="/news/2020/07/30/demo-fraud-detection-3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/2020/07/28/flink-sql-demo-building-e2e-streaming-application.html">Flink SQL Demo: Building an End-to-End Streaming Application</a></h2>
@@ -358,34 +384,6 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/06/11/community-update.html">Flink Community Update - June'20</a></h2>
-
- <p>11 Jun 2020
- Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
-
- <p>And suddenly it’s June. The previous month has been calm on the surface, but quite hectic underneath — the final testing phase for Flink 1.11 is moving at full speed, Stateful Functions 2.1 is out in the wild and Flink has made it into Google Season of Docs 2020.</p>
-
- <p><a href="/news/2020/06/11/community-update.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/news/2020/06/09/release-statefun-2.1.0.html">Stateful Functions 2.1.0 Release Announcement</a></h2>
-
- <p>09 Jun 2020
- Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
-
- <p><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for fast [...]
-
-</p>
-
- <p><a href="/news/2020/06/09/release-statefun-2.1.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -418,6 +416,26 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html
index c8f6b4d68..c7d0abe66 100644
--- a/content/blog/page9/index.html
+++ b/content/blog/page9/index.html
@@ -236,6 +236,34 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2020/06/11/community-update.html">Flink Community Update - June'20</a></h2>
+
+ <p>11 Jun 2020
+ Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
+
+ <p>And suddenly it’s June. The previous month has been calm on the surface, but quite hectic underneath — the final testing phase for Flink 1.11 is moving at full speed, Stateful Functions 2.1 is out in the wild and Flink has made it into Google Season of Docs 2020.</p>
+
+ <p><a href="/news/2020/06/11/community-update.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
+ <h2 class="blog-title"><a href="/news/2020/06/09/release-statefun-2.1.0.html">Stateful Functions 2.1.0 Release Announcement</a></h2>
+
+ <p>09 Jun 2020
+ Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
+
+ <p><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for fast [...]
+
+</p>
+
+ <p><a href="/news/2020/06/09/release-statefun-2.1.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/news/2020/05/12/release-1.10.1.html">Apache Flink 1.10.1 Released</a></h2>
@@ -347,34 +375,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/04/01/community-update.html">Flink Community Update - April'20</a></h2>
-
- <p>01 Apr 2020
- Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p>
-
- <p>While things slow down around us, the Apache Flink community is privileged to remain as active as ever. This blogpost combs through the past few months to give you an update on the state of things in Flink — from core releases to Stateful Functions; from some good old community stats to a new development blog.</p>
-
- <p><a href="/news/2020/04/01/community-update.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
- <article>
- <h2 class="blog-title"><a href="/features/2020/03/27/flink-for-data-warehouse.html">Flink as Unified Engine for Modern Data Warehousing: Production-Ready Hive Integration</a></h2>
-
- <p>27 Mar 2020
- Bowen Li (<a href="https://twitter.com/Bowen__Li">@Bowen__Li</a>)</p>
-
- <p><p>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.</p>
-
-</p>
-
- <p><a href="/features/2020/03/27/flink-for-data-warehouse.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -407,6 +407,26 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up
<ul id="markdown-toc">
+ <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li>
+
+
+
+
+
+
+
+
+
+ <li><a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li>
diff --git a/content/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png b/content/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png
new file mode 100644
index 000000000..98cb22ca3
Binary files /dev/null and b/content/img/blog/2022-07-11-final-checkpoint/checkpoint_format.png differ
diff --git a/content/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png b/content/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png
new file mode 100644
index 000000000..4e2012524
Binary files /dev/null and b/content/img/blog/2022-07-11-final-checkpoint/checkpoint_trigger.png differ
diff --git a/content/img/blog/2022-07-11-final-checkpoint/example_job.png b/content/img/blog/2022-07-11-final-checkpoint/example_job.png
new file mode 100644
index 000000000..4996050b5
Binary files /dev/null and b/content/img/blog/2022-07-11-final-checkpoint/example_job.png differ
diff --git a/content/img/blog/2022-07-11-final-checkpoint/example_job_finish.png b/content/img/blog/2022-07-11-final-checkpoint/example_job_finish.png
new file mode 100644
index 000000000..730a918f9
Binary files /dev/null and b/content/img/blog/2022-07-11-final-checkpoint/example_job_finish.png differ
diff --git a/content/img/blog/2022-07-11-final-checkpoint/finish_cmp.png b/content/img/blog/2022-07-11-final-checkpoint/finish_cmp.png
new file mode 100644
index 000000000..2b634594e
Binary files /dev/null and b/content/img/blog/2022-07-11-final-checkpoint/finish_cmp.png differ
diff --git a/content/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png b/content/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png
new file mode 100644
index 000000000..6c50e6938
Binary files /dev/null and b/content/img/blog/2022-07-11-final-checkpoint/stream_batch_cmp.png differ
diff --git a/content/index.html b/content/index.html
index 57965edd1..3c22bf8c2 100644
--- a/content/index.html
+++ b/content/index.html
@@ -401,6 +401,12 @@
<dl>
+ <dt> <a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></dt>
+ <dd>This post presents more details on the changes on the checkpoint procedure and task finish process made by the final checkpoint mechanism.</dd>
+
+ <dt> <a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></dt>
+ <dd>This post briefly describes the motivation and changes made by the final checkpoint mechanism, including the changes to the checkpoint procedure and how tasks finish.</dd>
+
<dt> <a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></dt>
<dd>The Apache Flink community is excited to announce the release of Flink ML 2.1.0! This release focuses on improving Flink ML's infrastructure, such as Python SDK, memory management, and benchmark framework, to facilitate the development of performant, memory-safe, and easy-to-use algorithm libraries. We validated the enhanced infrastructure via benchmarks and confirmed that Flink ML can meet or exceed the performance of selected algorithms from alternative popular ML libraries [...]
@@ -409,15 +415,6 @@
<dt> <a href="/news/2022/06/22/release-1.14.5.html">Apache Flink 1.14.5 Release Announcement</a></dt>
<dd>The Apache Flink Community is pleased to announce another bug fix release for Flink 1.14.</dd>
-
- <dt> <a href="/2022/06/17/adaptive-batch-scheduler.html">Adaptive Batch Scheduler: Automatically Decide Parallelism of Flink Batch Jobs</a></dt>
- <dd>To automatically decide parallelism of Flink batch jobs, we introduced adaptive batch scheduler in Flink 1.15. In this post, we'll take a close look at the design & implementation details.</dd>
-
- <dt> <a href="/news/2022/06/05/release-kubernetes-operator-1.0.0.html">Apache Flink Kubernetes Operator 1.0.0 Release Announcement</a></dt>
- <dd><p>In the last two months since our <a href="https://flink.apache.org/news/2022/04/03/release-kubernetes-operator-0.1.0.html">initial preview release</a> the community has been hard at work to stabilize and improve the core Flink Kubernetes Operator logic.
-We are now proud to announce the first production ready release of the operator project.</p>
-
-</dd>
</dl>
diff --git a/content/zh/index.html b/content/zh/index.html
index 1c82f5d34..b19b3096c 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -398,6 +398,12 @@
<dl>
+ <dt> <a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></dt>
+ <dd>This post presents more details on the changes on the checkpoint procedure and task finish process made by the final checkpoint mechanism.</dd>
+
+ <dt> <a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></dt>
+ <dd>This post briefly describes the motivation and changes made by the final checkpoint mechanism, including the changes to the checkpoint procedure and how tasks finish.</dd>
+
<dt> <a href="/news/2022/07/07/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></dt>
<dd>The Apache Flink community is excited to announce the release of Flink ML 2.1.0! This release focuses on improving Flink ML's infrastructure, such as Python SDK, memory management, and benchmark framework, to facilitate the development of performant, memory-safe, and easy-to-use algorithm libraries. We validated the enhanced infrastructure via benchmarks and confirmed that Flink ML can meet or exceed the performance of selected algorithms from alternative popular ML libraries [...]
@@ -406,15 +412,6 @@
<dt> <a href="/news/2022/06/22/release-1.14.5.html">Apache Flink 1.14.5 Release Announcement</a></dt>
<dd>The Apache Flink Community is pleased to announce another bug fix release for Flink 1.14.</dd>
-
- <dt> <a href="/2022/06/17/adaptive-batch-scheduler.html">Adaptive Batch Scheduler: Automatically Decide Parallelism of Flink Batch Jobs</a></dt>
- <dd>To automatically decide parallelism of Flink batch jobs, we introduced adaptive batch scheduler in Flink 1.15. In this post, we'll take a close look at the design & implementation details.</dd>
-
- <dt> <a href="/news/2022/06/05/release-kubernetes-operator-1.0.0.html">Apache Flink Kubernetes Operator 1.0.0 Release Announcement</a></dt>
- <dd><p>In the last two months since our <a href="https://flink.apache.org/news/2022/04/03/release-kubernetes-operator-0.1.0.html">initial preview release</a> the community has been hard at work to stabilize and improve the core Flink Kubernetes Operator logic.
-We are now proud to announce the first production ready release of the operator project.</p>
-
-</dd>
</dl>