[GitHub] [beam] anthonyqzhu commented on a change in pull request #15300: [WIP] [BEAM-10212] State cache memory limit

[GitBox <gi...@apache.org>]

anthonyqzhu commented on a change in pull request #15300:
URL: https://github.com/apache/beam/pull/15300#discussion_r686967100



##########
File path: sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/CachingBeamFnStateClient.java
##########
@@ -175,4 +179,71 @@ static StateCacheKey create(ByteString cacheToken, ByteString continuationToken)
       return new AutoValue_CachingBeamFnStateClient_StateCacheKey(cacheToken, continuationToken);
     }
   }
+
+  /**
+   * Entry for a key in the state cache that adds a weight counter onto the map of cache keys to
+   * state responses.
+   */
+  public static class StateCacheEntry {
+    // Initial size of hash tables per entry.
+    private static final int INITIAL_HASH_MAP_CAPACITY = 4;
+    // Overhead of each hash map entry.
+    private static final int HASH_MAP_ENTRY_OVERHEAD = 16;
+    // Overhead of each StateCacheEntry. One long, plus a concurrent hash map
+    private static final int PER_CACHE_ENTRY_OVERHEAD =
+        8 + HASH_MAP_ENTRY_OVERHEAD * INITIAL_HASH_MAP_CAPACITY;

Review comment:
       These are based off of WindmillStateCache (https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateCache.java) - can someone let me know if these numbers are reasonable? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org