[GitHub] [beam] lukecwik commented on a change in pull request #11811: [BEAM-10075] Add a flag to tune the size of the grouping tables used by the dataflow workers

[GitBox <gi...@apache.org>]

lukecwik commented on a change in pull request #11811:
URL: https://github.com/apache/beam/pull/11811#discussion_r430831035



##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java
##########
@@ -177,6 +177,20 @@ public Dataflow create(PipelineOptions options) {
 
   void setNumberOfWorkerHarnessThreads(int value);
 
+  /**
+   * Size (in MB) of each grouping table used to pre-combine elements. If unset, defaults to 100 MB.
+   *
+   * <p>CAUTION: If set too large, workers may run into OOM conditions more easily, each worker may
+   * have many grouping tables in-memory concurrently.
+   */
+  @Description(
+      "The size (in MB) of the grouping tables used to pre-combine elements before "
+          + "shuffling.  Larger values may reduce the amount of data shuffled.")
+  @Default.Long(100)
+  Long getGroupingTableMaxSizeMb();
+
+  void setGroupingTableMaxSizeMb(Long value);

Review comment:
       Please add this option to https://github.com/apache/beam/blob/32c657955d05cb5c558def91c2d1e7edb80fb123/sdks/java/core/src/main/java/org/apache/beam/sdk/options/SdkHarnessOptions.java#L32 instead of here.
   
   Please also plumb the value to https://github.com/apache/beam/blob/32c657955d05cb5c558def91c2d1e7edb80fb123/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/PrecombineGroupingTable.java#L44
   
   We should be able to use a primitive type (also long seems really large for specifying amount of MBs):
   ```suggestion
     int getGroupingTableMaxSizeMb();
   
     void setGroupingTableMaxSizeMb(int value);
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org