[GitHub] jihoonson opened a new pull request #6095: Add support 'keepSegmentGranularity' for compactionTask

[GitBox <gi...@apache.org>]

jihoonson opened a new pull request #6095: Add support 'keepSegmentGranularity' for compactionTask
URL: https://github.com/apache/incubator-druid/pull/6095
 
 
   This PR introduces a new configuration, `keepSegmentGranularity`, to compactionTask. Once this is set to true, the compactionTask respects the existing segment boundaries and doesn't compact segments across those segment boundaries. 
   
   From the implementation side, the compactionTask generates multiple indexTaskSpecs per segment interval and runs them sequentially. This should be fine because there's no (or very little) performance penalty compared to running a single indexTaskSpec. 
   
   Another characteristics of this is, the compactionTask is always finished as `SUCCEEDED` no matter how many indexTaskSpecs succeed. This means, the application (the coordinator in automatic compaction) is responsible for checking which segments the compactionTask failed to compact and rerunning another compactionTask.
   
   This option is enabled by default in compactionTask, but automatic compaction currently disables this because it breaks the algorithm of `NewestSegmentFirstIterator`. I'll fix and enable in the follow-up pr.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org