You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Robert Bradshaw (Jira)" <ji...@apache.org> on 2021/04/21 16:38:00 UTC

[jira] [Created] (BEAM-12204) Portable Java caches entirety of iterable side inputs.

Robert Bradshaw created BEAM-12204:
--------------------------------------

             Summary: Portable Java caches entirety of iterable side inputs.
                 Key: BEAM-12204
                 URL: https://issues.apache.org/jira/browse/BEAM-12204
             Project: Beam
          Issue Type: Bug
          Components: java-fn-execution
            Reporter: Robert Bradshaw
            Assignee: Robert Bradshaw
             Fix For: 2.30.0


Though the data is read over the state channel in a paginated way, it is wrapped in LazyCachingIteratorToIterable which forces the entire iterable to be cached. On the other hand, not caching at all for re-iteration could be expensive. We should do what Python does and cache a bit (e.g. the first page) and still support re-iteration by re-requesting the data over the state channel. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)