[GitHub] [beam] damccorm opened a new issue, #21129: [Go SDK] Add support for setting upper memory limit for side input cache

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #21129:
URL: https://github.com/apache/beam/issues/21129

   This is implementing https://issues.apache.org/jira/browse/BEAM-8297 for the Go SDK's side input cache. The current implementation reads the entire side input into memory and keeps track of how many distinct inputs to hold; however, each side input is not guaranteed to occupy the same amount of memory. An extra parameter should be created to be passed in to the SideInputCache ([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go)](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go)) to set a maximum memory limit in place of a maximum element capacity, then add a hook (see https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/cache_hooks.go) and corresponding function to the harnessopts package ([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go)](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/h
 arnessopts/cache.go)) so users can set this value themselves. 
   
   Imported from Jira [BEAM-13212](https://issues.apache.org/jira/browse/BEAM-13212). Original Jira may contain additional context.
   Reported by: jrmccluskey.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org