[GitHub] [beam] scwhittle commented on a diff in pull request #22645: [WIP][BEAM-12776, fixes #21095] Limit parallel closes from the prior element to the next element.

[GitBox <gi...@apache.org>]

scwhittle commented on code in PR #22645:
URL: https://github.com/apache/beam/pull/22645#discussion_r941711400


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -954,7 +949,17 @@ public void processElement(ProcessContext c, BoundedWindow window) throws Except
         writeOrClose(writer, getDynamicDestinations().formatRecord(input));
       }
 
-      // Close all writers.
+      // Clean-up any prior writers that were being closed as part of this bundle before

Review Comment:
   If we have many windows, we'll have many calls to ProcessElement and block sequentially, only overlapping closes with single window processing instead of multiple.
   
   Could we instead block if the # of closing but not yet closed writers exceeds some amount (which could be controlled by an option)?



##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -996,10 +1001,18 @@ public void finishBundle(FinishBundleContext c) throws Exception {
           c.output(result.getValue(), result.getKey(), result.getValue().getWindow());
         }
       } finally {
-        deferredOutput = null;
-        closeFutures = null;
+        deferredOutput.clear();
+        closeFutures.clear();
       }
     }
+
+    // Ensure that transient fields are initialized.

Review Comment:
   I haven't seen this before, is this preferrable to just initializing these in class constructor?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org