[GitHub] [beam] scwhittle commented on pull request #15300: [BEAM-10212] State cache memory limit

[GitBox <gi...@apache.org>]

scwhittle commented on pull request #15300:
URL: https://github.com/apache/beam/pull/15300#issuecomment-907203685


   One benefit of the static limit versus percentage is that if OOMs are caused by caching too much (or cache memory tracking not working as well as expected), the static limit allows for the machine size to be increased to avoid the OOMs as the cache will use the same amount of memory.  If the max cache size is automatically increased proportional to the worker shape, the OOMs will just continue possibly negating VM right-sizing or manual size tuning.
   
   Perhaps instead of a static value or a static percentage we could do a combination:
   static_value + % of machine size
   Then we don't scale entirely with the machine size but also will use a larger cache by default for larger machines.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org