[GitHub] [hudi] nsivabalan commented on a change in pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.

[GitBox <gi...@apache.org>]

nsivabalan commented on a change in pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#discussion_r789367919



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
##########
@@ -378,16 +383,23 @@ private static String resetTarget(Config configuration, String database, String
   /**
    * Creates actual HoodieDeltaStreamer objects for every table/topic and does incremental sync.
    */
-  public void sync() {
-    for (TableExecutionContext context : tableExecutionContexts) {
-      try {
-        new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
-        successTables.add(Helpers.getTableWithDatabase(context));
-      } catch (Exception e) {
-        logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e);
-        failedTables.add(Helpers.getTableWithDatabase(context));
-      }
-    }
+  public void sync() throws InterruptedException {
+    ExecutorService executorService = Executors.newFixedThreadPool(tableExecutionContexts.size());
+    tableExecutionContexts.forEach(context -> {
+      executorService.execute(new Runnable() {
+        @Override
+        public void run() {
+          try {
+            new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
+            successTables.add(Helpers.getTableWithDatabase(context));
+          } catch (Exception e) {
+            logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e);
+            failedTables.add(Helpers.getTableWithDatabase(context));
+          }
+        }
+      });
+    });
+    executorService.shutdown();

Review comment:
       should we add awaitTermination here ? we can't proceed w/ next batch until all tables in current batch is completed right? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org