You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/09/03 10:07:03 UTC

[GitHub] [flink] dawidwys opened a new pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

dawidwys opened a new pull request #17135:
URL: https://github.com/apache/flink/pull/17135


   ## What is the purpose of the change
   
   The PR documents how to enable the feature introduced in FLIP-147 and also describes things to consider before doing so.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**yes** / no)
     - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9c71ff03ec389f946c3a8b603a2a5abd7647cca7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715) 
   * 6976c1e0ff29913256425672b5173806e3ce691e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9c71ff03ec389f946c3a8b603a2a5abd7647cca7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 65e3e34b2a805678d692766c747149182843c225 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502) 
   * 9c71ff03ec389f946c3a8b603a2a5abd7647cca7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23752",
       "triggerID" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23776",
       "triggerID" : "3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6976c1e0ff29913256425672b5173806e3ce691e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23752) 
   * 3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23776) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704346332



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,41 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, because it was a bounded source. The feature must be enabled

Review comment:
       First of all, I agree, this sentence is not very clear. I would not use `all` / `only`, because this way the reader might be confused that it works iff the job graph contains exclusively bounded sources.
   
   I will rephrase it to: 
   > Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job
   graph have finished processing all data, which might happen if it contains bounded sources. The
   WDYT?

##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,41 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, because it was a bounded source. The feature must be enabled

Review comment:
       First of all, I agree, this sentence is not very clear. I would not use `all` / `only`, because this way the reader might be confused that it works iff the job graph contains exclusively bounded sources.
   
   I will rephrase it to: 
   > Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job
   graph have finished processing all data, which might happen if it contains bounded sources. The...
   
   WDYT?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23752",
       "triggerID" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6976c1e0ff29913256425672b5173806e3ce691e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23752) 
   * 3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912436745


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 65e3e34b2a805678d692766c747149182843c225 (Fri Sep 03 10:31:30 UTC 2021)
   
   **Warnings:**
    * Documentation files were touched, but no `docs/content.zh/` files: Update Chinese documentation or file Jira ticket.
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] infoverload commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r707108617



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -83,6 +83,8 @@ Other parameters for checkpointing include:
 
   - *unaligned checkpoints*: You can enable [unaligned checkpoints]({{< ref "docs/ops/state/unaligned_checkpoints" >}}) to greatly reduce checkpointing times under backpressure. Only works for exactly-once checkpoints and with number of concurrent checkpoints of 1.
 
+  - *checkpoints with finished tasks*: You can enable an experimental feature to continue performing checkpoints even if parts of the DAG have finished processing all of their records. Before doing so please read through the [important considerations](#checkpointing-with-parts-of-the-graph-finished).
+

Review comment:
       ```suggestion
     - *unaligned checkpoints*: You can enable [unaligned checkpoints]({{< ref "docs/ops/state/unaligned_checkpoints" >}}) to greatly reduce checkpointing times under backpressure. This only works for exactly-once checkpoints and with a number of concurrent checkpoints of 1.
   
     - *checkpoints with finished tasks*: You can enable an experimental feature to continue performing checkpoints even if parts of the DAG have finished processing all of their records. Before doing so, please read through some [important considerations](#checkpointing-with-parts-of-the-graph-finished).
   
   ```

##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,46 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job
+graph have finished processing all data, which might happen if it contains bounded sources. The
+feature must be enabled via a feature flag:
+
+```java
+Configuration config = new Configuration();
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+```
+
+Once tasks/subtasks are finished they don't contribute to the checkpoints any longer. It is an
+important observation that puts certain requirements on the implementation of any custom operators
+or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
+and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
+method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
+All checkpoints taken after the `finish` method has been called should be in most cases empty and
+shouldn't contain any buffered data, as there will be no way to emit this data. One notable
+exception is if your operator has some pointers to transactions in external systems, for example in
+order to implement the exactly-once semantic. In such a case, checkpoints taken after invoking `finish()`
+method should keep a pointer to the last transaction(s) that will be committed in the final checkpoint
+before the operator is closed. A good built-in example of this are exactly-once sinks and the 
+`TwoPhaseCommitSinkFunction`. What does it mean in more details?

Review comment:
       ```suggestion
   ## Checkpointing with parts of the job graph finished *(BETA)*
   
   Starting from Flink 1.14, it is possible to continue performing checkpoints even if parts of the job
   graph have finished processing all the data. This might happen if there are bounded sources. This
   feature must be enabled via a feature flag:
   
   ```java
   Configuration config = new Configuration();
   config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
   StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
   ```
   
   Once the tasks/subtasks are finished, they do not contribute to the checkpoints any longer. This is an
   an important consideration when implementing any custom operators or UDFs (User-Defined Functions). 
   
   In order to support checkpointing with tasks that finished, we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}}) and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}} method. This method is expected to be a clear cutoff point for flushing any remaining buffered state. All checkpoints taken after the `finish` method has been called should be empty (in most cases) and should not contain any buffered data since there will be no way to emit this data. One notable exception is if your operator has some pointers to transactions in external systems (i.e. order to implement the exactly-once semantic). In such a case, checkpoints taken after invoking the `finish()` method should keep a pointer to the last transaction(s) that will be committed in the final checkpoint before the operator is closed. A good built-in example of this are exactly-once sinks and the `TwoPhaseCommitSinkFu
 nction`. 
   ```

##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,46 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job
+graph have finished processing all data, which might happen if it contains bounded sources. The
+feature must be enabled via a feature flag:
+
+```java
+Configuration config = new Configuration();
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+```
+
+Once tasks/subtasks are finished they don't contribute to the checkpoints any longer. It is an
+important observation that puts certain requirements on the implementation of any custom operators
+or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
+and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
+method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
+All checkpoints taken after the `finish` method has been called should be in most cases empty and
+shouldn't contain any buffered data, as there will be no way to emit this data. One notable
+exception is if your operator has some pointers to transactions in external systems, for example in
+order to implement the exactly-once semantic. In such a case, checkpoints taken after invoking `finish()`
+method should keep a pointer to the last transaction(s) that will be committed in the final checkpoint
+before the operator is closed. A good built-in example of this are exactly-once sinks and the 
+`TwoPhaseCommitSinkFunction`. What does it mean in more details?
+
+### Operator state
+
+The aforementioned requirement implies certain contract especially on the operator state. First and
+foremost it forced us to implement a special handling for `UnionListState`. Often a `UnionListState`
+has been used to implement a global view over offsets in external system (e.g. we use it to store
+current offsets of Kafka partitions). If we had discarded a state for just a single subtask that had
+it's `finish` method called, we would've lost offsets for partitions it had been assigned. In order
+to work this problem around, we let checkpoints succeed, only if either none or all subtasks
+finished that use `UnionListState`.
+
+We have not seen a `ListState` used in a similar way, but you must be aware that any state checkpointed
+after the `finish` method will be discarded and not available after a restore (if the corresponding
+checkpoint succeeds). We truly believe that any operator that is prepared to be rescaled should work
+well with tasks that partially finish (only a subset of it tasks finish). Restoring from a checkpoint
+which has some subtasks finished is equivalent to restoring such a task with the number of new
+subtasks equal to the number of finished tasks.

Review comment:
       ```suggestion
   ### How does this impact the operator state?
   
   There is a special handling for `UnionListState`, which has often been used to implement a global view over offsets in an external system (i.e. storing current offsets of Kafka partitions). If we had discarded a state for a single subtask that had its `finish` method called, we would have lost the offsets for partitions that it had been assigned. In order to work around this problem, we let checkpoints succeed only if none or all subtasks that use `UnionListState` are finished.
   
   We have not seen `ListState` used in a similar way, but you should be aware that any state checkpointed after the `finish` method will be discarded and not be available after a restore. 
   
   Any operator that is prepared to be rescaled should work well with tasks that partially finish. Restoring from a checkpoint where only a subset of tasks finished is equivalent to restoring such a task with the number of new subtasks equal to the number of finished tasks.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704089770



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -147,6 +154,11 @@ env.getCheckpointConfig.enableUnalignedCheckpoints()
 
 // sets the checkpoint storage where checkpoint snapshots will be written
 env.getCheckpointConfig.setCheckpointStorage("hdfs:///my/checkpoint/dir")
+
+// enable checkpointing with finished tasks
+val config = new Configuration()
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true)
+env.configure(config)

Review comment:
       I encourage you to try add it yourself, sometimes it's not that difficult:
   https://github.com/apache/flink/pull/17100/commits/e83081e3d734448f232e6f676d71c46a44c4081e
   
   Here it might be a bit more tricky with using {{Configuration}}, but maybe there is already an example how to do it that you can just copy/paste/rename.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912426762


   No problem, I'll have a look~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704116077



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -83,6 +83,8 @@ Other parameters for checkpointing include:
 
   - *unaligned checkpoints*: You can enable [unaligned checkpoints]({{< ref "docs/ops/state/unaligned_checkpoints" >}}) to greatly reduce checkpointing times under backpressure. Only works for exactly-once checkpoints and with number of concurrent checkpoints of 1.
 
+  - *checkpoints with finished tasks*: You can enable an experimental feature to continue performing checkpoints even if parts of the DAG have finished processing all of their records. Before doing so please read through the [important considerations](#checkpointing-with-parts-of-the-graph-finished)

Review comment:
       dot missed at the end of line.

##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,41 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, because it was a bounded source. The feature must be enabled

Review comment:
       Perhaps `it was a bounded source` -> `all the sources are bounded`  or `it used only bounded sources` ? 

##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,45 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, because it was a bounded source. The feature must be enabled
+via a feature flag:
+
+```java
+Configuration config = new Configuration();
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+```
+
+Once tasks/subtasks are finished they don't contribute to the checkpoints any longer. It is an
+important observation that puts certain requirements on the implementation of any custom operators
+or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
+and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
+method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
+All checkpoints taken after the `finish` method has been called should be in most cases empty and
+shouldn't contain any buffered data, as there will be no way to emit this data. One notable
+exception is if your operator has some pointers to transactions in external systems, for example in
+order to implement the exactly-once semantic. In such a case, checkpoints taken after invoking `finish()`
+method should keep a pointer to the last transaction(s) that will be committed in the final checkpoint
+before the operator is closed. A good built-in example of this are exactly-once sinks and the 
+`TwoPhaseCommitSinkFunction`. What does it mean in more details?

Review comment:
       Do we need to remove the "What does it mean in more details?"  ? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23752",
       "triggerID" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9c71ff03ec389f946c3a8b603a2a5abd7647cca7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715) 
   * 6976c1e0ff29913256425672b5173806e3ce691e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23752) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704130575



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -147,6 +154,11 @@ env.getCheckpointConfig.enableUnalignedCheckpoints()
 
 // sets the checkpoint storage where checkpoint snapshots will be written
 env.getCheckpointConfig.setCheckpointStorage("hdfs:///my/checkpoint/dir")
+
+// enable checkpointing with finished tasks
+val config = new Configuration()
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true)
+env.configure(config)

Review comment:
       I did add methods to the Python API in the past on multiple occasions. Indeed, if it is just a setter, the issue is simple. Here, I'd really prefer not to deal with it, as it deals with `Configuration`. 
   
   1. First of all I am not able to run the python tests. Configuring the IDE to make them run, would take me quite some time.
   2. In Java/Scala we use a type safe version of configuration `ReadableConfig`. I don't know how we should solve that in python.
   
   I can prepare a very naive PR, that could be a starter, but it would require a lot of input from python maintainers anyway.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912423667


   Do you mind taking a look? @pnowojski @gaoyunhaii @infoverload 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704343221



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,45 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, because it was a bounded source. The feature must be enabled
+via a feature flag:
+
+```java
+Configuration config = new Configuration();
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+```
+
+Once tasks/subtasks are finished they don't contribute to the checkpoints any longer. It is an
+important observation that puts certain requirements on the implementation of any custom operators
+or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
+and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
+method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
+All checkpoints taken after the `finish` method has been called should be in most cases empty and
+shouldn't contain any buffered data, as there will be no way to emit this data. One notable
+exception is if your operator has some pointers to transactions in external systems, for example in
+order to implement the exactly-once semantic. In such a case, checkpoints taken after invoking `finish()`
+method should keep a pointer to the last transaction(s) that will be committed in the final checkpoint
+before the operator is closed. A good built-in example of this are exactly-once sinks and the 
+`TwoPhaseCommitSinkFunction`. What does it mean in more details?

Review comment:
       My idea is to have it there to make it less dull and catch the readers attention again for another important section. I don't have strong attachment to the sentence though ;)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 65e3e34b2a805678d692766c747149182843c225 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704143588



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -147,6 +154,11 @@ env.getCheckpointConfig.enableUnalignedCheckpoints()
 
 // sets the checkpoint storage where checkpoint snapshots will be written
 env.getCheckpointConfig.setCheckpointStorage("hdfs:///my/checkpoint/dir")
+
+// enable checkpointing with finished tasks
+val config = new Configuration()
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true)
+env.configure(config)

Review comment:
       Ok, I get it. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704130575



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -147,6 +154,11 @@ env.getCheckpointConfig.enableUnalignedCheckpoints()
 
 // sets the checkpoint storage where checkpoint snapshots will be written
 env.getCheckpointConfig.setCheckpointStorage("hdfs:///my/checkpoint/dir")
+
+// enable checkpointing with finished tasks
+val config = new Configuration()
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true)
+env.configure(config)

Review comment:
       I did add methods to the Python API in the past on multiple occasions. Here, I'd really prefer not to deal with it, as it deals with `Configuration`. 
   
   1. First of all I am not able to run the python tests. Configuring the IDE to make them run, would take me quite some time.
   2. In Java/Scala we use a type safe version of configuration `ReadableConfig`. I don't know how we should solve that in python.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r703684747



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,40 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, because it was a bounded source. The feature must be enabled
+via a feature flag:
+
+```java
+Configuration config = new Configuration();
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+```
+
+Once tasks/subtasks are finished they don't contribute to the checkpoints any longer. It is an
+important observation that puts certain requirements on the implementation of any custom operators
+or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
+and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
+method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
+All checkpoints taken after the `finish` method has been called should not contain any significant state that is required
+after a restore. What does it mean in details?

Review comment:
       I tried rephrasing it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r703681012



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -147,6 +154,11 @@ env.getCheckpointConfig.enableUnalignedCheckpoints()
 
 // sets the checkpoint storage where checkpoint snapshots will be written
 env.getCheckpointConfig.setCheckpointStorage("hdfs:///my/checkpoint/dir")
+
+// enable checkpointing with finished tasks
+val config = new Configuration()
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true)
+env.configure(config)

Review comment:
       https://issues.apache.org/jira/browse/FLINK-24199




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 65e3e34b2a805678d692766c747149182843c225 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704130575



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -147,6 +154,11 @@ env.getCheckpointConfig.enableUnalignedCheckpoints()
 
 // sets the checkpoint storage where checkpoint snapshots will be written
 env.getCheckpointConfig.setCheckpointStorage("hdfs:///my/checkpoint/dir")
+
+// enable checkpointing with finished tasks
+val config = new Configuration()
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true)
+env.configure(config)

Review comment:
       I did add methods to the Python API in the past on multiple occasions. Here, I'd really prefer not to deal with it, as it deals with `Configuration`. 
   
   1. First of all I am not able to run the python tests. Configuring the IDE to make them run, would take me quite some time.
   2. In Java/Scala we use a type safe version of configuration `ReadableConfig`. I don't know how we should solve that in python.
   
   I can prepare a very naive PR, that could be a starter, but it would require a lot of input from python maintainers anyway.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 65e3e34b2a805678d692766c747149182843c225 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502) 
   * 9c71ff03ec389f946c3a8b603a2a5abd7647cca7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 65e3e34b2a805678d692766c747149182843c225 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r702671777



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -147,6 +154,11 @@ env.getCheckpointConfig.enableUnalignedCheckpoints()
 
 // sets the checkpoint storage where checkpoint snapshots will be written
 env.getCheckpointConfig.setCheckpointStorage("hdfs:///my/checkpoint/dir")
+
+// enable checkpointing with finished tasks
+val config = new Configuration()
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true)
+env.configure(config)

Review comment:
       What is the python way to enable this feature?

##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -211,4 +223,40 @@ Flink currently only provides processing guarantees for jobs without iterations.
 
 Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
 
+## Checkpointing with parts of the graph finished *(BETA)*
+
+Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, because it was a bounded source. The feature must be enabled
+via a feature flag:
+
+```java
+Configuration config = new Configuration();
+config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+```
+
+Once tasks/subtasks are finished they don't contribute to the checkpoints any longer. It is an
+important observation that puts certain requirements on the implementation of any custom operators
+or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
+and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
+method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
+All checkpoints taken after the `finish` method has been called should not contain any significant state that is required
+after a restore. What does it mean in details?

Review comment:
       This and later `discarded` is a bit confusing, as the finished subtasks still can/should have a state pointing to side effects (2pc). It should be phrased that after calling `finish()` state should be empty except of pointers to commit side effects, like external transactions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17135:
URL: https://github.com/apache/flink/pull/17135#issuecomment-912443975


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "65e3e34b2a805678d692766c747149182843c225",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23502",
       "triggerID" : "65e3e34b2a805678d692766c747149182843c225",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23715",
       "triggerID" : "9c71ff03ec389f946c3a8b603a2a5abd7647cca7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23752",
       "triggerID" : "6976c1e0ff29913256425672b5173806e3ce691e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23776",
       "triggerID" : "3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3d3f9ac9493f48a68a4aafaf3f13c44241fa3b06 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23776) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #17135:
URL: https://github.com/apache/flink/pull/17135#discussion_r704100504



##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -239,8 +239,9 @@ important observation that puts certain requirements on the implementation of an
 or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
 and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
 method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
-All checkpoints taken after the `finish` method has been called should not contain any significant state that is required
-after a restore. What does it mean in details?
+All checkpoints taken after the `finish` method has been called can contain only pointers to the last transactions
+that will be closed in the final checkpoint or in case of a failure should be closed when restoring.
+Apart from those pointers it should be empty. What does it mean in details?

Review comment:
       ```suggestion
   All checkpoints taken after the `finish` method has been called should be in most cases empty and
   shouldn't contain any buffered data, as there will be no way to emit this data. One noteable 
   exception is if your operator has some pointers to transactions in external systems, for example in 
   order to implement exactly-once semantic. In such case checkpoints taken after invoking `finish()`
   method should keep pointer to last transaction(s) that will be committed in the final checkpoint
   before operator is closed. Good built-in example of this are exactly-once sinks and {{TwoPhaseCommitSinkFunction}}.
   What does it mean in more details?
   ```

##########
File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md
##########
@@ -239,8 +239,9 @@ important observation that puts certain requirements on the implementation of an
 or UDFs. In order to support checkpointing with tasks that finish we adjusted the [task lifecycle]({{< ref "docs/internals/task_lifecycle" >}})
 and introduced the {{< javadoc file="org/apache/flink/streaming/api/operators/StreamOperator.html#finish--" name="StreamOperator#finish" >}}
 method. The method is expected to be a clear cutoff point for flushing any remaining buffered state.
-All checkpoints taken after the `finish` method has been called should not contain any significant state that is required
-after a restore. What does it mean in details?
+All checkpoints taken after the `finish` method has been called can contain only pointers to the last transactions
+that will be closed in the final checkpoint or in case of a failure should be closed when restoring.
+Apart from those pointers it should be empty. What does it mean in details?

Review comment:
       
   And what do you mean by:
   > or in case of a failure should be closed when restoring.
   
   ? close -> committed? or aborted? I guess you mean committing when failure happens before `notifyCheckpointComplete()` reached the task? In such case I would skip this as too much details.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys closed pull request #17135: [FLINK-24098] Document FLIP-147 capabiliites and limitations

Posted by GitBox <gi...@apache.org>.
dawidwys closed pull request #17135:
URL: https://github.com/apache/flink/pull/17135


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org