[GitHub] [incubator-druid] jon-wei opened a new pull request #8201: Simple memory allocation for CliIndexer tasks

[GitBox <gi...@apache.org>]

jon-wei opened a new pull request #8201: Simple memory allocation for CliIndexer tasks
URL: https://github.com/apache/incubator-druid/pull/8201
 
 
   Part of #7900 series of patches
   
   ### Description
   
   This is a small patch that makes the new CliIndexer incrementally more usable for experimental purposes while the more sophisticated memory management features described in proposal #7900 are still being developed.
   
   This patch adds the following:
   - A new WorkerConfig property `globalIngestionHeapLimitBytes`, which specifies a total limit for `maxBytesInMemory` across all the tasks being executed by a CliIndexer process. In this incremental patch, this limit is divided evenly across the configured worker capacity. The default is 60% of the JVM heap.
   - To help reduce overall memory usage while using the existing IndexMerger, the CliIndexer will limit merge/persist calls made by appenderators using a shared executor, with number of threads equal to half of the configured worked capacity.
   - WorkerConfig is moved into `server` package for accessibility
   
   I am currently testing the patch on a test cluster.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
      - [x] using the [concurrency checklist](https://github.com/apache/incubator-druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] been tested in a test Druid cluster.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org