[GitHub] [hudi] nsivabalan commented on a diff in pull request #5543: [HUDI-4072] Fixing NULL schema for empty batches in deltastreamer

[GitBox <gi...@apache.org>]

nsivabalan commented on code in PR #5543:
URL: https://github.com/apache/hudi/pull/5543#discussion_r871257552


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -829,6 +831,25 @@ private HoodieWriteConfig getHoodieClientConfig(Schema schema) {
     return config;
   }
 
+  private Schema getSchemaForWriteConfig(Schema targetSchema) {
+    Schema newWriteSchema = targetSchema;
+    try {
+      if (targetSchema != null) {
+        // check if targetSchema is equal to NULL schema
+        if (SchemaCompatibility.checkReaderWriterCompatibility(targetSchema, InputBatch.NULL_SCHEMA).getType() == SchemaCompatibility.SchemaCompatibilityType.COMPATIBLE
+            && SchemaCompatibility.checkReaderWriterCompatibility(InputBatch.NULL_SCHEMA, targetSchema).getType() == SchemaCompatibility.SchemaCompatibilityType.COMPATIBLE) {
+          // target schema is null. fetch schema from commit metadata and use it
+          HoodieTableMetaClient meta = HoodieTableMetaClient.builder().setConf(new Configuration(fs.getConf())).setBasePath(cfg.targetBasePath).setPayloadClassName(cfg.payloadClassName).build();

Review Comment:
   yeah. I thought about that. this happens only incase of empty batches from the source, on which case anyways there is not much load on the system in general. So, resolved to using a local one. If the scope of usage of metaclient increases in future, we can add it as instance variable. wdyt. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org