[GitHub] [hudi] bvaradar commented on a diff in pull request #8570: [HUDI-6136] Adding idempotent writes to Hudi

["bvaradar (via GitHub)" <gi...@apache.org>]

bvaradar commented on code in PR #8570:
URL: https://github.com/apache/hudi/pull/8570#discussion_r1285308295


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -698,6 +698,29 @@ public class HoodieWriteConfig extends HoodieConfig {
           + "will not print any configuration which contains the configured filter. For example with "
           + "a configured filter `ssl`, value for config `ssl.trustore.location` would be masked.");
 
+  public static final ConfigProperty<String> WRITER_IDENTIFIER = ConfigProperty
+      .key("hoodie.datasource.write.writer.identifier")
+      .defaultValue("default_writer")
+      .markAdvanced()
+      .sinceVersion("0.14.0")
+      .withDocumentation("An identifier to unqiuely identify a writer. This config needs to be used along with "
+          + "hoodie.datasource.write.batch.identifier and is used to ensure idempotency of writes. For a given writer (identified by "
+          + "the writer identifier), hudi will ensure idempotency if same batch of data(identified by batch identifier) is ingested again. "
+          + "Users should ensure to set monotonically increasing batch identifier for subsequent batches of ingest for a given writer."
+          + " If there are multiple writers, each writer is expected to set a unique value for this config. ");
+
+  public static final ConfigProperty<String> WRITE_BATCH_IDENTIFIER = ConfigProperty
+      .key("hoodie.datasource.write.batch.identifier")
+      .noDefaultValue()
+      .markAdvanced()
+      .sinceVersion("0.14.0")
+      .withDocumentation("A identifier to uniquely identify a batch of data being ingested. This config needs to be set along with "
+          + "hoodie.datasource.write.writer.identifier. These two configs ensure idempotency is same batch of data is ingested again by the same"
+          + " writer to a given hudi table.");
+
+  // This constant serves as the checkpoint key for hudi writes to assist in ensuring idempotency
+  public static final String WRITES_CHECKPOINT_KEY = "_hudi_writes_checkpoint";

Review Comment:
   nit: writes -> writer



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org