Re: [PR] AddFiles: extract bounded async task plumbing and Parquet footer reads [beam]

["stankiewicz (via GitHub)" <gi...@apache.org>]

stankiewicz commented on code in PR #39896:
URL: https://github.com/apache/beam/pull/39896#discussion_r3914384157


##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:
##########
@@ -329,50 +314,33 @@ private static class ProcessResult {
 
     @Setup
     public void setup() {
-      executor = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
+      tasks = new BoundedAsyncTasks<>(THREAD_POOL_SIZE, MAX_IN_FLIGHT_TASKS);
+    }
+
+    /** Clears anything left behind if the runner reuses this instance after a failed bundle. */
+    @StartBundle
+    public void startBundle() {
+
+      checkStateNotNull(tasks).cancelAll();
     }
 
     @Teardown
     public void teardown() {
-      if (executor != null) {
-        executor.shutdownNow();
+      if (tasks != null) {
+        tasks.shutdown();
       }
     }
 
-    @StartBundle
-    public void startBundle() {
-      activeTasks = Lists.newLinkedList();
-    }
-
     @ProcessElement
     public void process(
         @Element String filePath,
         @Timestamp Instant timestamp,
         BoundedWindow window,
         PaneInfo paneInfo,
         MultiOutputReceiver output)
-        throws IOException, InterruptedException, ExecutionException {
-      LinkedList<Future<ProcessResult>> activeTasks = checkStateNotNull(this.activeTasks);
-
-      // start draining finished tasks, but don't block
-      Iterator<Future<ProcessResult>> iterator = activeTasks.iterator();
-      while (iterator.hasNext()) {
-        Future<ProcessResult> future = iterator.next();
-        if (future.isDone()) {
-          outputResult(future.get(), output);
-          iterator.remove();
-        }
-      }
-
-      // if we have too many active tasks, wait until some finish
-      while (activeTasks.size() >= MAX_IN_FLIGHT_TASKS) {
-        Future<ProcessResult> oldestTask = activeTasks.removeFirst();
-        outputResult(oldestTask.get(), output); // .get() blocks until the task completes
-      }
-
-      // create a new task for the current element and add to queue
+        throws Exception {
       Callable<ProcessResult> task = createProcessTask(filePath, timestamp, window, paneInfo);
-      activeTasks.add(checkStateNotNull(executor).submit(task));
+      checkStateNotNull(tasks).submit(task, result -> outputResult(result, output));

Review Comment:
   @claudevdm you shouldn't output async here. OutputReceiver and ProcessContext are not thread safe. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org