[GitHub] [beam] mxm commented on a change in pull request #11533: [BEAM-9827] Ensure minimum watermark hold is computed across all keys

[GitBox <gi...@apache.org>]

mxm commented on a change in pull request #11533:
URL: https://github.com/apache/beam/pull/11533#discussion_r415802522



##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
##########
@@ -76,8 +76,8 @@
   private final KeyedStateBackend<ByteBuffer> flinkStateBackend;
   private Coder<K> keyCoder;
 
-  // Combined watermark holds for all keys of this partition
-  private final Map<String, Instant> watermarkHolds = new HashMap<>();
+  // Watermark holds for all keys/windows of this partition
+  private final PriorityQueue<Long> watermarkHolds = new PriorityQueue<>();

Review comment:
       Only keeping the minimum doesn't work. When the minimum hold is cleared, we will have to recompute the minimum by iterating over all the keys in the state backend which will defeat the purpose of having the cache.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org