[GitHub] [incubator-druid] jon-wei opened a new pull request #8107: Add CliIndexer process type and initial task runner implementation

[GitBox <gi...@apache.org>]

jon-wei opened a new pull request #8107: Add CliIndexer process type and initial task runner implementation
URL: https://github.com/apache/incubator-druid/pull/8107
 
 
   First part of implementation for #7900 
   
   ### Description
   
   This PR adds the initial foundation for the work in #7900:
   - A new Indexer process type is added, running on port 8091 like MiddleManagers.
   - This process type is like MM but uses a new ThreadingTaskRunner, which runs Tasks using a thread pool instead of forking processes
   - A new interface, `AppenderatorsManager`, which creates and manages Appenderators for tasks, has been added. The AppenderatorsManager for the new Indexer process uses a shared query execution pool and per-datasource segment walkers/imelines for the Appenderators it creates.
   - The TaskReportWriter has been adjusted to support managing multiple tasks
   - The new Indexer process uses two ports, its configured port and (configured port + 1). The second port is used for ChatHandler requests, to give them a dedicated pool separate from the query processing connection pool.
   - Lookups are loaded once for the entire Indexer process
   
   No additional memory management for the tasks is present in this PR, they will use whatever has been configured in their task specs with no additional external control.
   
   This PR has:
   - [x] been self-reviewed.
      - [x] using the [concurrency checklist](https://github.com/apache/incubator-druid/blob/master/dev/code-review/concurrency.md)
   - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths.
   - [x] been tested in a test Druid cluster.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org