You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "masteryhx (via GitHub)" <gi...@apache.org> on 2024/02/28 02:06:18 UTC

[PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

masteryhx opened a new pull request, #24396:
URL: https://github.com/apache/flink/pull/24396

   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   Current documentation of allowNonRestoredState is not clear, we should clarify:
   
   - It can lead to serious issues with correctness if it's used incorrectly.
   - The correctness is related to the topological order and the logic of job when removing operator by default.
   - For DataStream Job, the operator uid could be assigned explicitly to avoid the reassignment of operator uid.
   
   
   ## Brief change log
   
     - Add Warning of allowNonRestoredState in the doc
   
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / no / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / **no**)
     - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #24396:
URL: https://github.com/apache/flink/pull/24396#discussion_r1505277616


##########
docs/content/docs/ops/state/savepoints.md:
##########
@@ -208,7 +208,13 @@ This submits a job and specifies a savepoint to resume from. You may give a path
 
 #### Allowing Non-Restored State
 
-By default, the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option:
+By default, the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option.
+
+{{< hint warning  >}}
+Improper usage of this feature could result in significant issues with the correctness of the application. It is crucial to verify that any remaining states can be accurately mapped to the appropriate operators.
+It is worth noting that operator UIDs are reassigned based on topological order by default, which may lead to incorrect associations between states and operators. 
+To prevent such mismatches, it is advisable to explicitly [assign UIDs] ({{< ref "docs/ops/production_ready" >}}/#set-uuids-for-all-operators) to all operators in a DataStream job. In other scenarios involving mismatches, it is necessary to start the new job discarding all states.

Review Comment:
   what do you mean by 'In other scenarios involving mismatches, it is necessary to start the new job discarding all states.'



##########
docs/content/docs/ops/state/savepoints.md:
##########
@@ -208,7 +208,13 @@ This submits a job and specifies a savepoint to resume from. You may give a path
 
 #### Allowing Non-Restored State
 
-By default, the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option:
+By default, the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option.
+
+{{< hint warning  >}}
+Improper usage of this feature could result in significant issues with the correctness of the application. It is crucial to verify that any remaining states can be accurately mapped to the appropriate operators.
+It is worth noting that operator UIDs are reassigned based on topological order by default, which may lead to incorrect associations between states and operators. 

Review Comment:
   More description for consequence? Such as
   ```suggestion
   It is worth noting that operator UIDs are reassigned based on topological order by default, which may lead to incorrect associations between states and operators, thus consequently the state is not correctly restored as wished.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "masteryhx (via GitHub)" <gi...@apache.org>.
masteryhx commented on PR #24396:
URL: https://github.com/apache/flink/pull/24396#issuecomment-1976157579

   > Hi @masteryhx , I suggest minor adjustments in Chinese. PTAL, thanks!
   
   Thanks for the suggestions, I have addressed and rebased commits. PTAL again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "masteryhx (via GitHub)" <gi...@apache.org>.
masteryhx commented on code in PR #24396:
URL: https://github.com/apache/flink/pull/24396#discussion_r1505327190


##########
docs/content/docs/ops/state/savepoints.md:
##########
@@ -208,7 +208,13 @@ This submits a job and specifies a savepoint to resume from. You may give a path
 
 #### Allowing Non-Restored State
 
-By default, the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option:
+By default, the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option.
+
+{{< hint warning  >}}
+Improper usage of this feature could result in significant issues with the correctness of the application. It is crucial to verify that any remaining states can be accurately mapped to the appropriate operators.
+It is worth noting that operator UIDs are reassigned based on topological order by default, which may lead to incorrect associations between states and operators. 
+To prevent such mismatches, it is advisable to explicitly [assign UIDs] ({{< ref "docs/ops/production_ready" >}}/#set-uuids-for-all-operators) to all operators in a DataStream job. In other scenarios involving mismatches, it is necessary to start the new job discarding all states.

Review Comment:
   The description is somewhat confusing and redundant. I just removed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #24396:
URL: https://github.com/apache/flink/pull/24396#issuecomment-1968059291

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "c2eb42b0454045275b3a27c781437365afa0494a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c2eb42b0454045275b3a27c781437365afa0494a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c2eb42b0454045275b3a27c781437365afa0494a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "masteryhx (via GitHub)" <gi...@apache.org>.
masteryhx closed pull request #24396: [FLINK-34352][doc] Improve the documentation of allowNonRestoredState
URL: https://github.com/apache/flink/pull/24396


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #24396:
URL: https://github.com/apache/flink/pull/24396#discussion_r1507343507


##########
docs/content.zh/docs/ops/state/savepoints.md:
##########
@@ -180,7 +180,15 @@ $ bin/flink run -s :savepointPath [:runArgs]
 
 #### 跳过无法映射的状态恢复
 
-默认情况下,resume 操作将尝试将 Savepoint 的所有状态映射回你要还原的程序。 如果删除了运算符,则可以通过 `--allowNonRestoredState`(short:`-n`)选项跳过无法映射到新程序的状态:
+默认情况下,resume 操作将尝试将 Savepoint 的所有状态映射回你要还原的程序。 如果删除了运算符,则可以通过 `--allowNonRestoredState`(short:`-n`)选项跳过无法映射到新程序的状态。
+
+{{< hint warning  >}}
+错误使用这个功能可能会导致应用程序出现严重的正确性问题。其中最重要的是确认剩余的状态是否能够准确地映射到相应的运算符上。
+
+值得注意的是,默认情况下,操作符的 UID 是根据拓扑顺序重新分配的,这可能会导致状态和运算符之间的关联不正确,最终造成状态未能按照预期正确恢复。

Review Comment:
   ```suggestion
   需要注意在默认情况下,运算符的 UID 是根据拓扑顺序重新分配的,这会导致状态与运算符无法正确关联,最终造成状态未能按照预期正确恢复。
   ```



##########
docs/content.zh/docs/ops/state/savepoints.md:
##########
@@ -180,7 +180,15 @@ $ bin/flink run -s :savepointPath [:runArgs]
 
 #### 跳过无法映射的状态恢复
 
-默认情况下,resume 操作将尝试将 Savepoint 的所有状态映射回你要还原的程序。 如果删除了运算符,则可以通过 `--allowNonRestoredState`(short:`-n`)选项跳过无法映射到新程序的状态:
+默认情况下,resume 操作将尝试将 Savepoint 的所有状态映射回你要还原的程序。 如果删除了运算符,则可以通过 `--allowNonRestoredState`(short:`-n`)选项跳过无法映射到新程序的状态。
+
+{{< hint warning  >}}
+错误使用这个功能可能会导致应用程序出现严重的正确性问题。其中最重要的是确认剩余的状态是否能够准确地映射到相应的运算符上。
+
+值得注意的是,默认情况下,操作符的 UID 是根据拓扑顺序重新分配的,这可能会导致状态和运算符之间的关联不正确,最终造成状态未能按照预期正确恢复。
+
+为了防止这种不匹配,建议在 DataStream 作业中显式地为所有操作符[指定 UID ]({{< ref "docs/ops/production_ready" >}}/#set-uuids-for-all-operators)。对其他不匹配的场景,需要舍弃所有状态并启动新作业。

Review Comment:
   ```suggestion
   为了防止这种不匹配,建议在 DataStream 作业中显式地为所有运算符[指定 UID ]({{< ref "docs/ops/production_ready" >}}/#set-uuids-for-all-operators)。
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "masteryhx (via GitHub)" <gi...@apache.org>.
masteryhx commented on PR #24396:
URL: https://github.com/apache/flink/pull/24396#issuecomment-1968060127

   @Zakelly @wuchong @lincoln-lil Could you help to review this ? Thanks a lot!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34352][doc] Improve the documentation of allowNonRestoredState [flink]

Posted by "masteryhx (via GitHub)" <gi...@apache.org>.
masteryhx commented on PR #24396:
URL: https://github.com/apache/flink/pull/24396#issuecomment-1968514402

   @Zakelly Thanks for the review. I have addressed your comments. PTAL again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org