[GitHub] jihoonson commented on issue #6629: Add support parallel combine in brokers

[GitBox <gi...@apache.org>]

jihoonson commented on issue #6629: Add support parallel combine in brokers
URL: https://github.com/apache/incubator-druid/pull/6629#issuecomment-446392815
 
 
   > My main question in the implementation is as follows: There is already a large body of effort and thought in the java.lang.Thread* and java.util.concurrent packages that seem to be purposefully avoided here. I haven't been able to fully dig into ParallelMergeCombineSequence yet, but it seems like a lot of the java concepts should be able to be used here. I'll try to give some examples when I can
   
   @drcrallen thank you for taking a look. I'll check your comments soon. For this question, if you talk about ForkJoinPool, I don't think it's appropriate for this kind of use case. The algorithm implemented in this PR is a sort of pipelining combining sorted data. Each thread combines data from its children, and the parent and its children threads can be run at the same time. I took this approach because the final results should be streamed to the query client.
   
   However, I think ForkJoinPool is appropriate for divide-and-conquer style algorithms. For example, it's useful when summing a large sequence of integers. It can automatically split the sequence into several small chunks, and sum them in parallel. The results of summing small chunks should be aggregated again. Here, summing small chunks and aggregating their results can't be done at the same time. What do you think?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org