[GitHub] [druid] yuanlihan commented on issue #9953: Broadcast join support based on broadcast rule

[GitBox <gi...@apache.org>]

yuanlihan commented on issue #9953:
URL: https://github.com/apache/druid/issues/9953#issuecomment-636584236


   Hi @jihoonson, thanks for your great ideas and plans to move forward from inline datasources joining.
   
   > The coordinator should assign all segments of a broadcast datasource to all historicals, realtime tasks, brokers, and indexers.
   
   
   Can we have a fine-grained segment assignment/loading plan with the help of `HashedPartitionsSpec`?
   
   As we know Druid already enables hash-based secondary partitioning for batch indexing tasks.
   Suppose that there are several datasources which are configured with same secondary hash-based `partitionsSpec` which contains `partitionDimensions` user intends to joining. Then we introduce a new LoadRule, like `ColocateDistributionRule` which colocates segments with same chunk id, to load segments among query processing services. And to work with realtime data, Druid has to enable `HashedPartitionsSpec` in realtime ingestion tasks while currently the historical segments with `hashed` `ShardSpec` seems not appendable by default. But still could it be a potential improvement that is compatible with this proposal?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org