[GitHub] jihoonson opened a new pull request #6203: Add support targetCompactionSizeBytes for compactionTask

[GitBox <gi...@apache.org>]

jihoonson opened a new pull request #6203: Add support targetCompactionSizeBytes for compactionTask
URL: https://github.com/apache/incubator-druid/pull/6203
 
 
   `targetCompactionSizeBytes` is similar to that of automatic compaction and means the size of the result segments after compaction. This is especially useful to compact a large interval partition into a few segments of the optimal size. A large interval partition means that the total size of segments in that partition is larger than `targetCompactionSizeBytes` no matter what the number of segments is.
   
   Since indexTask doesn't support a sort configuration for target segment size, compactionTask assumes that the segment size is proportional to the number of rows in the segment and automatically computes `targetPartitionSize` based on the given `targetCompactionSizeBytes`. This assumption is pretty naive, so we can improve it later.
   
   `targetCompactionSizeBytes` cannot be used together with `targetPartitionSize`, `maxTotalRows`, or `numShards` in `IndexTuningConfig`. If both are set, compactionTask will throw an exception and fail. If both are not set, `targetCompactionSizeBytes` is set to default (400MB).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org