Re: [PR] [HUDI-1881]: draft implementation for trigger based on data availability [hudi]

["Sarfaraz-214 (via GitHub)" <gi...@apache.org>]

Sarfaraz-214 commented on code in PR #5071:
URL: https://github.com/apache/hudi/pull/5071#discussion_r1584145214


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java:
##########
@@ -425,15 +432,29 @@ private static String resetTarget(Config configuration, String database, String
    * Creates actual HoodieDeltaStreamer objects for every table/topic and does incremental sync.
    */
   public void sync() {
-    for (TableExecutionContext context : tableExecutionContexts) {
-      try {
-        new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
-        successTables.add(Helpers.getTableWithDatabase(context));
-      } catch (Exception e) {
-        logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e);
-        failedTables.add(Helpers.getTableWithDatabase(context));
-      }
-    }
+    ExecutorService executorService = Executors.newFixedThreadPool(tableExecutionContexts.size());
+    tableExecutionContexts.forEach(context -> {
+      executorService.execute(new Runnable() {
+        @Override
+        public void run() {
+          try {
+            HoodieDeltaStreamer streamerInstance = new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties()));
+            int iteration = 1;
+            while (iteration <= context.getConfig().continuousIterations) {
+              if (streamerInstance.getDeltaSyncService().getDeltaSync().isDataAvailableForIngestion()) {

Review Comment:
   Hi @pratyakshsharma 
   What is the implication of not having the **if** condition?
   
   I removed it locally because of this use-case: If I kill an existing MultiTableStreamer job and relaunch it again and upon relaunch the jobs doesn't see any new messages in Kafka to consume then it would fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org