[GitHub] [kafka] vvcephei opened a new pull request #9214: [DO NOT MERGE] POC: unify all jvm cache pools

[GitBox <gi...@apache.org>]

vvcephei opened a new pull request #9214:
URL: https://github.com/apache/kafka/pull/9214


   Currently, users of Suppress in strict mode must either not configure a memory bound or consider a per-operator, per-partition bound. The former would result in the application crashing ungracefully if it needs too much memory, which is sub-optimal for some deployment strategies. The latter is nice for determinism, but is difficult to configure in practice.
   
   In addition to suppress buffers, Streams has a record cache configuration. Currently, we make the assumption that all threads would probably use a uniform amount of cache space, but this assumption is clearly wrong in some cases.
   
   Finally, there are some applications that want to run multiple Streams instances in the same JVM, probably for running different Streams topologies.
   
   In aggregate, there are quite a few "pools" of heap space that users need to configure if they want to avoid an OOME, and the more threads, applications, and Suppress operators there are, the more granular these pools become. Of course, the more granular the pools are, the lower utilization of the available memory we will see. Plus, especially for Suppression, very granular pool configuration means a higher likelihood that the operator will run out of space and shut the app down.
   
   This POC demonstrates the feasibility of unifying all these pools with one logical bound on total memory usage for all caches and suppression buffers, across all operators/tasks and all threads, and even across all Streams instances in the JVM.
   
   Most of the tests pass right now, but not all of them. I also need to clean up a few more things before really starting a discussion.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org