[GitHub] [incubator-druid] quenlang commented on issue #8061: Native parallel batch indexing with shuffle

[GitBox <gi...@apache.org>]

quenlang commented on issue #8061: Native parallel batch indexing with shuffle
URL: https://github.com/apache/incubator-druid/issues/8061#issuecomment-543125241
 
 
   > Hi @quenlang, you should be able to do it. An example tuningConfig could be
   > 
   > ```json
   >     "tuningConfig" : {
   >       "type" : "index_parallel",
   >       "maxNumConcurrentSubTasks": 2, // max number of sub queries that can be run at the same time
   >       "forceGuaranteedRollup": true,
   >       "partitionsSpec": {
   >         "type": "hashed",
   >         "numShards": 3 // number of segments per time chunk after compaction
   >       }
   >     }
   > ```
   
   Hi @jihoonson, I split data into different segments by single dimension tenant_id for the multitenancy scene. In this way, I can get higher query performance that filters on the tenant_id dimension. But tenant data was skew, so the segment size was not perfect. For example, the max segment size was nearly 18GB but the min segment size was 5MB. Then perform queries on the 18GB segment ware more slower than that partitioned by all dimensions. 
   
   Assume there is a way(eg. maxRowsPerSegment=10000000) can split the max segment into some more smaller segments, the queries will be quick. But ```maxRowsPerSegment``` can not exist when specify ```numShards``` option.
   
   How do you think about this scene?
   Thanks a lot!
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org