[GitHub] [beam] kileys commented on a change in pull request #16092: [BEAM-13354, BEAM-13015, BEAM-12802, BEAM-12588] Support prefetch for multimap and set state making loading keys and values truly lazy

[GitBox <gi...@apache.org>]

kileys commented on a change in pull request #16092:
URL: https://github.com/apache/beam/pull/16092#discussion_r761589631



##########
File path: sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java
##########
@@ -151,19 +150,89 @@ public void clear() {
   /*
    * Returns an iterables containing all distinct keys in this multimap.
    */
-  public Iterable<K> keys() {
+  public PrefetchableIterable<K> keys() {
     checkState(
         !isClosed,
         "Multimap user state is no longer usable because it is closed for %s",
         keysStateRequest.getStateKey());
     if (isCleared) {
-      return Collections.unmodifiableCollection(Lists.newArrayList(pendingAdds.keySet()));
+      List<K> keys = new ArrayList<>(pendingAdds.size());
+      for (Map.Entry<?, KV<K, List<V>>> entry : pendingAdds.entrySet()) {
+        keys.add(entry.getValue().getKey());
+      }
+      return PrefetchableIterables.concat(keys);
+    }
+
+    PrefetchableIterable<K> persistedKeys = getPersistedKeys();
+    Map<Object, K> pendingRemovesNow = new HashMap<>(pendingRemoves);

Review comment:
       Could we just use a set here?

##########
File path: sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java
##########
@@ -65,13 +62,11 @@
   private boolean isClosed;
   private boolean isCleared;
   // Pending updates to persistent storage
-  private HashSet<K> pendingRemoves = Sets.newHashSet();
-  private HashMap<K, List<V>> pendingAdds = Maps.newHashMap();
-  // Map keys with no values in persistent storage
-  private HashSet<K> negativeCache = Sets.newHashSet();

Review comment:
       How come you removed this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org