You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/02/22 09:52:46 UTC

[GitHub] [flink] dawidwys opened a new pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

dawidwys opened a new pull request #14981:
URL: https://github.com/apache/flink/pull/14981


   ## What is the purpose of the change
   Updates the documentation about the possibility to migrate from one state backend to another.
   
   ## Verifying this change
   
   The documentation can be compiled.
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / **no**)
     - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] sjwiesman commented on a change in pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #14981:
URL: https://github.com/apache/flink/pull/14981#discussion_r580593585



##########
File path: docs/content/docs/ops/state/savepoints.md
##########
@@ -46,9 +46,9 @@ In contrast to all this, Savepoints are created, owned, and deleted by the user.
 changing parallelism, forking a second job like for a red/blue deployment, and so on. Of course, Savepoints must survive job termination. Conceptually, Savepoints can be a bit more expensive to produce and restore and focus
 more on portability and support for the previously mentioned changes to the job.
 
-Those conceptual differences aside, the current implementations of Checkpoints and Savepoints are basically using the same code and produce the same format. However, there is currently one exception from this, and we might
-introduce more differences in the future. The exception is incremental checkpoints with the RocksDB state backend. They are using some RocksDB internal format instead of Flink’s native savepoint format. This makes them the
-first instance of a more lightweight checkpointing mechanism, compared to Savepoints.
+In Flink 1.13 we unified the binary format of savepoints across all state backends. That means you can take a savepoint and then restore from it using a different state backend.
+All the state backends produce a common format only starting from the version 1.13. You cannot do that with checkpoints. It means the savepoint binary format might be different from the one used for checkpoints. This is the case e.g. for Heap state backend or
+if you use incremental checkpoints with the RocksDB state backend. The RocksDB full snapshot share the format with unified savepoints, but it is not guaranteed, and we might introduce further modifications in the future.

Review comment:
       ```suggestion
   Flink's savepoint binary format is unified across all state backends. That means you can take a savepoint with one state backend and then restore it using another.
   
   {{< hint info >}}
   State backends did not start producing a common format until version 1.13. Therefore, if you want to switch the state backend you should first upgrade your Flink version than
   take a savepoint with the new version, and only after that, you can restore it with a different state backend.
   {{< /hint >}}
   ```
   
   

##########
File path: docs/content/docs/ops/state/state_backends.md
##########
@@ -136,15 +136,16 @@ The total memory amount of RocksDB instance(s) per slot can also be bounded, ple
 
 # Choose The Right State Backend
 
-Currently, Flink's savepoint binary format is state backend specific.
-A savepoint taken with one state backend cannot be restored using another, and you should carefully consider which backend you use before going to production.
-
 In general, we recommend avoiding `MemoryStateBackend` in production because it stores its snapshots inside the JobManager as opposed to persistent disk.
 When deciding between `FsStateBackend` and `RocksDB`, it is a choice between performance and scalability.
 `FsStateBackend` is very fast as each state access and update operates on objects on the Java heap; however, state size is limited by available memory within the cluster.
 On the other hand, `RocksDB` can scale based on available disk space and is the only state backend to support incremental snapshots.
 However, each state access and update requires (de-)serialization and potentially reading from disk which leads to average performance that is an order of magnitude slower than the memory state backends.
 
+In Flink 1.13 we unified the binary format of Flink's savepoints. That means you can take a savepoint and then restore from it using a different state backend.
+All the state backends produce a common format only starting from the version 1.13. Therefore, if you want to switch the state backend you should first upgrade your Flink version than
+take a savepoint with the new version and only after that you can restore it with a different state backend.
+

Review comment:
       ```suggestion
   {{< hint info >}}
   In Flink 1.13 we unified the binary format of Flink's savepoints. That means you can take a savepoint and then restore from it using a different state backend.
   All the state backends produce a common format only starting from version 1.13. Therefore, if you want to switch the state backend you should first upgrade your Flink version than
   take a savepoint with the new version, and only after that you can restore it with a different state backend.
   {{< /hint >}}
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14981:
URL: https://github.com/apache/flink/pull/14981#issuecomment-783266867


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * effa665a8f5549c858354f093a85fdfb1b439a1e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14981:
URL: https://github.com/apache/flink/pull/14981#issuecomment-783248913


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit effa665a8f5549c858354f093a85fdfb1b439a1e (Mon Feb 22 09:55:19 UTC 2021)
   
    ✅no warnings
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14981:
URL: https://github.com/apache/flink/pull/14981#issuecomment-783266867


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13576",
       "triggerID" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1774e0ac0afb93f529b6831b74f773312685ca6c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1774e0ac0afb93f529b6831b74f773312685ca6c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * effa665a8f5549c858354f093a85fdfb1b439a1e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13576) 
   * 1774e0ac0afb93f529b6831b74f773312685ca6c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14981:
URL: https://github.com/apache/flink/pull/14981#issuecomment-783266867


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13576",
       "triggerID" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * effa665a8f5549c858354f093a85fdfb1b439a1e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13576) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14981:
URL: https://github.com/apache/flink/pull/14981#issuecomment-783266867


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13576",
       "triggerID" : "effa665a8f5549c858354f093a85fdfb1b439a1e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * effa665a8f5549c858354f093a85fdfb1b439a1e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13576) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] dawidwys closed pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
dawidwys closed pull request #14981:
URL: https://github.com/apache/flink/pull/14981


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #14981: [FLINK-21343] Update documentation regarding unified savepoint format

Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #14981:
URL: https://github.com/apache/flink/pull/14981#issuecomment-785732676


   Thanks for the review @sjwiesman !


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org