You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by sihuazhou <gi...@git.apache.org> on 2018/04/27 09:50:03 UTC

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

GitHub user sihuazhou opened a pull request:

    https://github.com/apache/flink/pull/5928

    [hotfix][doc] fix doc of externalized checkpoint

    ## What is the purpose of the change
    
    This PR intend to fix the incorrect doc of externalized checkpoint.
    
    ## Brief change log
    
    - fix doc of externalized checkpoint
    
    ## Verifying this change
    
    This change is a trivial rework / code cleanup without any test coverage.
    
    ## Does this pull request potentially affect one of the following parts:
    
    no
    
    ## Documentation
    
    no

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sihuazhou/flink fix_doc_of_externalized_checkpoint

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5928.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5928
    
----
commit e2d02c06433243987561fdf9da95d11a61059c51
Author: sihuazhou <su...@...>
Date:   2018-04-27T08:15:04Z

    [hotfix] fix doc of externalized checkpoint.

----


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r188548955
  
    --- Diff: docs/ops/state/checkpoints.md ---
    @@ -35,60 +35,62 @@ the same semantics as a failure-free execution.
     See [Checkpointing]({{ site.baseurl }}/dev/stream/state/checkpointing.html) for how to enable and
     configure checkpoints for your program.
     
    -## Externalized Checkpoints
    +## Retain The Checkpoints
    --- End diff --
    
    "Retained Checkpoints"


---

[GitHub] flink issue #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/5928
  
    The configuration (`config.md`)should be generated from the config options by now, so not be manually edited.
    
    (@zentol could you chime in here?)


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5928


---

[GitHub] flink issue #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/5928
  
    In Flink 1.5, all checkpoints are externalized. There is no notion of externalized checkpoints any more, just a setting to configure the retention (retain on cancel, retain on fail, never retain).
    
    The `state.checkpoints.dir` is the recommended option for the directory where checkpoints are stored (and `state.backend.fs.checkpointdir` is a deprecated key for the same setting).


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r188549219
  
    --- Diff: docs/ops/state/checkpoints.md ---
    @@ -35,60 +35,62 @@ the same semantics as a failure-free execution.
     See [Checkpointing]({{ site.baseurl }}/dev/stream/state/checkpointing.html) for how to enable and
     configure checkpoints for your program.
     
    -## Externalized Checkpoints
    +## Retain The Checkpoints
     
     Checkpoints are by default not persisted externally and are only used to
     resume a job from failures. They are deleted when a program is cancelled.
     You can, however, configure periodic checkpoints to be persisted externally
    -similarly to [savepoints](savepoints.html). These *externalized checkpoints*
    -write their meta data out to persistent storage and are *not* automatically
    -cleaned up when the job fails. This way, you will have a checkpoint around
    -to resume from if your job fails.
    +similarly to [savepoints](savepoints.html). This way, you will have a persisted 
    +checkpoint around to resume from if your job fails.
     
     {% highlight java %}
     CheckpointConfig config = env.getCheckpointConfig();
     config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
     {% endhighlight %}
     
    -The `ExternalizedCheckpointCleanup` mode configures what happens with externalized checkpoints when you cancel the job:
    +The `ExternalizedCheckpointCleanup` mode configures what happens with checkpoints when you cancel the job:
     
    -- **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**: Retain the externalized checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
    +- **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**: Retain the checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
     
    -- **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: Delete the externalized checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
    +- **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: Delete the checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
     
     ### Directory Structure
     
    -Similarly to [savepoints](savepoints.html), an externalized checkpoint consists
    -of a meta data file and, depending on the state back-end, some additional data
    -files. The **target directory** for the externalized checkpoint's meta data is
    -determined from the configuration key `state.checkpoints.dir` which, currently,
    -can only be set via the configuration files.
    +Similarly to [savepoints](savepoints.html), an checkpoint consists
    --- End diff --
    
    typo: "a checkpoint"


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r185021293
  
    --- Diff: docs/dev/stream/state/checkpointing.md ---
    @@ -137,11 +137,9 @@ Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see
        -  `jobmanager`: In-memory state, backup to JobManager's/ZooKeeper's memory. Should be used only for minimal state (Kafka offsets) or testing and local debugging.
        -  `filesystem`: State is in-memory on the TaskManagers, and state snapshots are stored in a file system. Supported are all filesystems supported by Flink, for example HDFS, S3, ...
     
    -- `state.backend.fs.checkpointdir`: Directory for storing checkpoints in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
    +- `state.checkpoints.dir`: The target directory for storing checkpoints data files and meta data of [externalized checkpoints]({{ site.baseurl }}/ops/state/checkpoints.html#externalized-checkpoints) in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
    --- End diff --
    
    Yes, `file:///` is what you use for many NAS style storage systems, so it is not local-only. Let's change this to say that the storage path must be accessible from all participating processes/nodes, i.e., all TaskManagers and JobManagers


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r185569523
  
    --- Diff: docs/dev/stream/state/checkpointing.md ---
    @@ -137,11 +137,9 @@ Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see
        -  `jobmanager`: In-memory state, backup to JobManager's/ZooKeeper's memory. Should be used only for minimal state (Kafka offsets) or testing and local debugging.
        -  `filesystem`: State is in-memory on the TaskManagers, and state snapshots are stored in a file system. Supported are all filesystems supported by Flink, for example HDFS, S3, ...
     
    -- `state.backend.fs.checkpointdir`: Directory for storing checkpoints in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
    +- `state.checkpoints.dir`: The target directory for storing checkpoints data files and meta data of [externalized checkpoints]({{ site.baseurl }}/ops/state/checkpoints.html#externalized-checkpoints) in a Flink supported filesystem. Note: the storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers).
    --- End diff --
    
    please sync these entries with the descriptions in `CheckpointingOptions`, so that they are identical with the one seen on the configuration page.


---

[GitHub] flink issue #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by sihuazhou <gi...@git.apache.org>.
Github user sihuazhou commented on the issue:

    https://github.com/apache/flink/pull/5928
  
    Travis failed is unrelated, it cause by the concurrency problem in `DefaultOperatorStateBackend` when performing checkpoint, which this [PR](https://github.com/apache/flink/pull/5930) try to fix.


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r185569814
  
    --- Diff: docs/dev/stream/state/checkpointing.md ---
    @@ -137,11 +137,9 @@ Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see
        -  `jobmanager`: In-memory state, backup to JobManager's/ZooKeeper's memory. Should be used only for minimal state (Kafka offsets) or testing and local debugging.
        -  `filesystem`: State is in-memory on the TaskManagers, and state snapshots are stored in a file system. Supported are all filesystems supported by Flink, for example HDFS, S3, ...
     
    -- `state.backend.fs.checkpointdir`: Directory for storing checkpoints in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
    +- `state.checkpoints.dir`: The target directory for storing checkpoints data files and meta data of [externalized checkpoints]({{ site.baseurl }}/ops/state/checkpoints.html#externalized-checkpoints) in a Flink supported filesystem. Note: the storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers).
    --- End diff --
    
    in fact, you could replace this entire section with `{% include generated/checkpointing_configuration.html %}`


---

[GitHub] flink issue #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by sihuazhou <gi...@git.apache.org>.
Github user sihuazhou commented on the issue:

    https://github.com/apache/flink/pull/5928
  
    cc @StefanRRichter @StephanEwen 


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r185568867
  
    --- Diff: docs/ops/state/checkpoints.md ---
    @@ -60,25 +60,29 @@ The `ExternalizedCheckpointCleanup` mode configures what happens with externaliz
     
     Similarly to [savepoints](savepoints.html), an externalized checkpoint consists
     of a meta data file and, depending on the state back-end, some additional data
    -files. The **target directory** for the externalized checkpoint's meta data is
    -determined from the configuration key `state.checkpoints.dir` which, currently,
    -can only be set via the configuration files.
    +files. The externalized checkpoint's meta data is stored in the same directory 
    +as data files. So the **target directory** can be set via configuration key 
    +`state.checkpoints.dir` in the configuration files, and also can be specified 
    +for per job in the code.
     
    +- Configure globally via configuration files
     ```
     state.checkpoints.dir: hdfs:///checkpoints/
     ```
     
    +- Configure for per job via code 
    +```java
    --- End diff --
    
    please use `{% highlight java %}` syntax instead


---

[GitHub] flink issue #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by sihuazhou <gi...@git.apache.org>.
Github user sihuazhou commented on the issue:

    https://github.com/apache/flink/pull/5928
  
    @StephanEwen @alpinegizmo Thanks for your comments, I will revert the changes of `config.md` and address the comments concerning to the ``` `file://` only for local setups```.


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by alpinegizmo <gi...@git.apache.org>.
Github user alpinegizmo commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r184959202
  
    --- Diff: docs/dev/stream/state/checkpointing.md ---
    @@ -137,11 +137,9 @@ Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see
        -  `jobmanager`: In-memory state, backup to JobManager's/ZooKeeper's memory. Should be used only for minimal state (Kafka offsets) or testing and local debugging.
        -  `filesystem`: State is in-memory on the TaskManagers, and state snapshots are stored in a file system. Supported are all filesystems supported by Flink, for example HDFS, S3, ...
     
    -- `state.backend.fs.checkpointdir`: Directory for storing checkpoints in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
    +- `state.checkpoints.dir`: The target directory for storing checkpoints data files and meta data of [externalized checkpoints]({{ site.baseurl }}/ops/state/checkpoints.html#externalized-checkpoints) in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
    --- End diff --
    
    This seems potentially misleading -- isn't it okay to use a `file://` URI in the case of a distributed filesystem that is mounted at the same mount point across the cluster?


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r188548729
  
    --- Diff: docs/_includes/generated/checkpointing_configuration.html ---
    @@ -40,7 +40,7 @@
             <tr>
                 <td><h5>state.checkpoints.dir</h5></td>
                 <td style="word-wrap: break-word;">(none)</td>
    -            <td>The default directory used for checkpoints. Used by the state backends that write checkpoints to file systems (MemoryStateBackend, FsStateBackend, RocksDBStateBackend).</td>
    +            <td>The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. Note: the storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers).</td>
    --- End diff --
    
    typo: "Note: The"


---

[GitHub] flink issue #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by sihuazhou <gi...@git.apache.org>.
Github user sihuazhou commented on the issue:

    https://github.com/apache/flink/pull/5928
  
    Hi guys, I think maybe the doc that related to the `Externalized Checkpoint` should be updated for 1.5. Are there any other suggestions? And if there is anyone who is willing to take over this plz feel free to do that, I would glad to close this PR if someone willing to take over this...


---

[GitHub] flink pull request #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5928#discussion_r185737131
  
    --- Diff: docs/ops/state/state_backends.md ---
    @@ -152,7 +152,7 @@ Possible values for the config entry are *jobmanager* (MemoryStateBackend), *fil
     name of the class that implements the state backend factory [FsStateBackendFactory](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackendFactory.java),
     such as `org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory` for RocksDBStateBackend.
     
    -In the case where the default state backend is set to *filesystem*, the entry `state.backend.fs.checkpointdir` defines the directory where the checkpoint data will be stored.
    +In the case where the default state backend is set to *filesystem*, the entry `state.checkpoints.dir` defines the directory where the checkpoint data will be stored.
    --- End diff --
    
    The option is used by all backends that eventually store data to file system, including RocksDBStateBackend and MemoryStateBackend (the MemoryStateBackend writes its single checkpoint metadata file there, optionally)


---

[GitHub] flink issue #5928: [hotfix][doc] fix doc of externalized checkpoint

Posted by sihuazhou <gi...@git.apache.org>.
Github user sihuazhou commented on the issue:

    https://github.com/apache/flink/pull/5928
  
    @zentol I have addressed your comments.


---