[GitHub] [beam] robertwb commented on a diff in pull request #27461: Light cache weigher for StateFetchingIterators

["robertwb (via GitHub)" <gi...@apache.org>]

robertwb commented on code in PR #27461:
URL: https://github.com/apache/beam/pull/27461#discussion_r1283830743


##########
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java:
##########
@@ -185,15 +186,19 @@ public DataStreamDecoder(Coder<T> coder, PrefetchableIterator<ByteString> inputS
      * Skips any remaining bytes in the current {@link ByteString} moving to the next {@link
      * ByteString} in the underlying {@link ByteString} {@link Iterator iterator} and decoding
      * elements till at the next boundary.
+     *
+     * @param weightEstimatorFn Pass in what is the function to be used for the weight.
      */
-    public List<T> decodeFromChunkBoundaryToChunkBoundary() {
+    public WeightedList<T> decodeFromChunkBoundaryToChunkBoundary(
+        Function<T, Long> weightEstimatorFn) {
       inbound.currentStream = inputByteStrings.next().newInput();

Review Comment:
   Currently we're only keeping the total weight anyhow. (Keeping per-element weights seems to be a large expense, especially as the goal with these weights is to try to manage memory.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org