[jira] [Comment Edited] (FLINK-28863) Snapshot result of RocksDB native savepoint should have empty shared-state

["Jinzhong Li (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-28863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627484#comment-17627484 ] 

Jinzhong Li edited comment on FLINK-28863 at 11/2/22 6:45 AM:
--------------------------------------------------------------

[~yunta] 

1. I found that the changes I metioned above would change native savepoint's behavior in CLAIM mode.

Now, when job restore from native savepoint in CLAIM mode, the first checkpoint will be incremental based on savepoint sstFiles.
This means although native savepoint sstFiles stay in the exclusive scope folder, the checkpoints of the job which restore from the savepoint can still share sstFiles with it.
If we put [sstFiles|https://github.com/apache/flink/blob/bb9f2525e6e16d00ef2f0739d9cb96c2e47e35e7/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L302] into privateState Map of IncrementalRemoteKeyedStateHandle for native savepoint, the above behavior will be changed.

2. Back to the issue itself, i think, if the semantic of "[sharedState|https://github.com/apache/flink/blob/bb9f2525e6e16d00ef2f0739d9cb96c2e47e35e7/flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalRemoteKeyedStateHandle.java#L80]" in IncrementalRemoteKeyedStateHandle is that state files can be shared by other checkpoints (include checkpoints after restore), *current behavior is by design.*   In other words,  the snapshot artifacts in the exclusive scope folder can still share sstFiles with checkpoints after restore in CLAIM mode.

[~yunta]  WDYT? 


was (Author: lijinzhong):
[~yunta] 

1. I found that the changes I metioned above would change native savepoint's behavior in CLAIM mode.

Now, when job restore from native savepoint in CLAIM mode, the first checkpoint will be incremental based on savepoint sstFiles.
This means although native savepoint sstFiles stay in the exclusive scope folder, the checkpoints of the job which restore from the savepoint can still share sstFiles with it.
If we put sstFiles into privateState Map of IncrementalRemoteKeyedStateHandle for native savepoint, the above behavior will be changed.

2. Back to the issue itself, i think, if the semantic of "sharedState" in IncrementalRemoteKeyedStateHandle is that state files can be shared by other checkpoints (include checkpoints after restore), *current behavior is by design.*   In other words,  the snapshot artifacts in the exclusive scope folder can still share sstFiles with checkpoints after restore in CLAIM mode.

[~yunta]  WDYT? 

> Snapshot result of RocksDB native savepoint should have empty shared-state
> --------------------------------------------------------------------------
>
>                 Key: FLINK-28863
>                 URL: https://issues.apache.org/jira/browse/FLINK-28863
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Checkpointing, Runtime / State Backends
>            Reporter: Yun Tang
>            Assignee: Jinzhong Li
>            Priority: Major
>             Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> The current snapshot result of RocksDB native savepoint has non-empty shared state, which is obviously not correct as all snapshot artifacts already stay in the exclusive checkpoint scope folder.
> This does not bring real harmful result due to we would not register the snapshot results of RocksDB native savepoint.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)