[GitHub] [flink] StephanEwen commented on issue #11722: [FLINK-5763][state backends] Make savepoints self-contained and relocatable

[GitBox <gi...@apache.org>]

StephanEwen commented on issue #11722: [FLINK-5763][state backends] Make savepoints self-contained and relocatable
URL: https://github.com/apache/flink/pull/11722#issuecomment-614927614
 
 
   I dug a bit through the code to try and understand the design here.
   
   *(A high level description of the approach would really help here. Because the way to use this is not described in the PR, I need to re-engineer what is the intended way of using this (and I am partially guessing).*
   
   It looks like the workflow is the following:
     - The path is split up into a base and a relative path before metadata serialization.
     - Then the savepoint is moved, then the user needs to resume the savepoint and specify the savepoint, plus the new "relocated path".
     - That relocated path is injected during metadata deserialization.
   
   I would be good if the solution would work without needing to specify the "relocate path" at all. It makes it harder to use for users and it also should not be necessary. That means dropping the config option, the CLI parameter, the REST parameter. That "relocate path" is also a bit redundant, because the "pointer" used to locate the savepoint is already that path (or it can be derived from that).
   
   What do you think about the following changes?
   
     - The `RelativeFileStateHandle` extends the `FileStateHandle`, for transparency. It has the full path, but also a the relative part of the path.
     - The `FsCheckpointStreamFactory` directly creates `RelativeFileStateHandle` if the state is created in the exclusive directory. That way we work around the "prefix matching" of the paths when writing the metadata out.
     - During metadata serialization, the only `RelativeFileStateHandle`'s relative part is written.
     - The `MetadataSerializer` takes the "external pointer" as an additional argument in the "deserialize" method.
     - In the "deserialize" method, when encountering a `RelativeFileStateHandle` for the first time, it constructs the base path, using `((FsCompletedCheckpointStorageLocation) (AbstractFsCheckpointStorage.resolveCheckpointPointer(pointer))).exclusiveCheckpointDir()`
     - When it reads a `RelativeFileStateHandle`, it reads only the relative part and constructs the full path from the base path and relative path.
   
     - This solution is slightly tight-coupled to the Fs Storage, but we have an easy way to decouple that in a follow-up step.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services