You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2023/03/09 00:31:00 UTC

[jira] [Created] (HUDI-5906) Consolidate checkpointing logic with deltastreamer into CheckpointStrategy model

Raymond Xu created HUDI-5906:
--------------------------------

             Summary: Consolidate checkpointing logic with deltastreamer into CheckpointStrategy model
                 Key: HUDI-5906
                 URL: https://issues.apache.org/jira/browse/HUDI-5906
             Project: Apache Hudi
          Issue Type: Improvement
          Components: deltastreamer
            Reporter: Raymond Xu
             Fix For: 0.14.0


checkpointing logic with deltastreamer is unorganized. there are logic spread in different places

- HoodieDeltaStreamer#CHECKPOINT_KEY
- HoodieDeltaStreamer#CHECKPOINT_RESET_KEY
- HoodieDeltaStreamer#CHECKPOINT_FORCE_SKIP_PROP
- org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config#checkpoint
- org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config#initialCheckpointProvider
- org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config#allowCommitOnNoCheckpointChange

- MUTLI_WRITER_SOURCE_CHECKPOINT_ID
- org.apache.hudi.utilities.checkpointing.InitialCheckPointProvider
- org.apache.hudi.utilities.sources.HoodieIncrSource.Config#MISSING_CHECKPOINT_STRATEGY
- org.apache.hudi.utilities.sources.helpers.IncrSourceHelper.MissingCheckpointStrategy


There should be a consolidated model say HoodieCheckpointStrategy to manage all these logic with pluggable APIs.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)