[GitHub] [iceberg] xloya commented on a change in pull request #3977: [Core][Spark][Flink] Change partitioned fanout/delta writers map to caffine cache

[GitBox <gi...@apache.org>]

xloya commented on a change in pull request #3977:
URL: https://github.com/apache/iceberg/pull/3977#discussion_r801204803



##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -210,6 +211,20 @@ private TableProperties() {
   public static final String SPARK_WRITE_PARTITIONED_FANOUT_ENABLED = "write.spark.fanout.enabled";
   public static final boolean SPARK_WRITE_PARTITIONED_FANOUT_ENABLED_DEFAULT = false;
 
+  public static final String PARTITIONED_FANOUT_WRITERS_CACHE_SIZE = "write.partition.fanout.writers-cache-size";
+  public static final int PARTITIONED_FANOUT_WRITERS_CACHE_SIZE_DEFAULT = Integer.MAX_VALUE;
+
+  public static final String PARTITIONED_FANOUT_WRITERS_CACHE_EVICT_MS =
+      "write.partition.fanout.writers-cache-eviction-ms";
+  public static final long PARTITIONED_FANOUT_WRITERS_CACHE_EVICT_MS_DEFAULT = TimeUnit.MINUTES.toMillis(5);

Review comment:
       @rdblue The reason for adding the timeout option is that it is often difficult to determine the memory consumption based on the number of writers




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org