[GitHub] jon-wei commented on issue #6838: [Proposal] Introduce concept of master/data/query servers in docs and packaging

[GitBox <gi...@apache.org>]

jon-wei commented on issue #6838: [Proposal] Introduce concept of master/data/query servers in docs and packaging
URL: https://github.com/apache/incubator-druid/issues/6838#issuecomment-453338371
 
 
   > What is the plan for consolidating historical and MMs? The workloads for MMs and historicals are very different. Typically MMs don't need a lot of disk space, whereas Historicals do. Would this lead to overprovisioning of resources that are not fully used?
   
   For whether it makes sense to colocate MM and historical, it depends on load and access patterns; you could get more utilization of resources by colocating with the tradeoff of potential resource contention.
   
   For consolidating the historical and MM at the process level, these are just some thoughts and beyond the scope of this proposal, but I think that consolidation of processes in that area could potentially help avoid overprovisioning if combined with other potential changes that move Druid away from static provisioning of resources towards more dynamic resource allocation, e.g:
   - Suppose that instead of statically assigning per-worker resources (identical across all workers) in config on the MM, the MM had just a pool of compute/mem/network resources that it can freely assign to tasks, this could help avoid overprovisioning when not all tasks have consistent resource requirements
   - Similarly, global resource pool with dynamic allocation in a combined historical/MM process (replacing cases where it makes sense to colocate them as separate processes) could enable more efficient resource usage and simpler operation vs. the current model where the user has to manually calculate and provision fixed resource limits for individual pieces. You could also do some other optimizations like shortcuts for segment handoff (if the segment would go to the same machine), or sharing lookup mappings in memory.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org