You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/05/31 22:12:27 UTC

[GitHub] [hudi] n3nash commented on a change in pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp

n3nash commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r642690647



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -326,6 +328,16 @@ private boolean onDeltaSyncShutdown(boolean error) {
     @Parameter(names = {"--checkpoint"}, description = "Resume Delta Streamer from this checkpoint.")
     public String checkpoint = null;
 
+    /**
+     * 1. string: topicName,partition number 0:offset value,partition number 1:offset value
+     * 2. timestamp: kafka offset timestamp
+     * example
+     * 1. hudi_topic,0:100,1:101,2:201
+     * 2. 1621947081
+     */
+    @Parameter(names = {"--checkpoint-type"}, description = "Checkpoint type, divided into timestamp or string offset")
+    public String checkpointType = "string";

Review comment:
       @nsivabalan Do we need to introduce something explicitly here ? Can we just introduce another property like below `hoodie.deltastreamer.source.kafka.checkpoint.type` and not have this change present as a top level option ? This checkpoint type seems very specific to a use-case in kafka and would like to reduce the confusions at the top level configs for users who want to use other source types.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org