[GitHub] [druid] kfaraz commented on pull request #13852: Continuous automatic compaction

["kfaraz (via GitHub)" <gi...@apache.org>]

kfaraz commented on PR #13852:
URL: https://github.com/apache/druid/pull/13852#issuecomment-1449814414

   Thanks a lot for taking this up, @suneet-s !
   
   I had a similar approach in mind to what @imply-cheddar has described. 
   @suneet-s , I think you too have mentioned this as a design that you considered, breaking up the `CompactSegments` duty into two.
   
   I feel it would be a cleaner approach and would avoid the need to add a new config. 
   - Keep the `CompactSegments` duty mostly unchanged
      - it would run at the custom period, if specified, otherwise, the default `indexingPeriod` (30 min)
      - it continues to call `policy.reset()` to create a new iterator
      - pass the `CompactionJobQueue` to this duty and update it with the new iterator
      - Optional: Rename this duty to reflect that it just identifies compactible intervals (maybe `GetIntervalsForCompaction` or something) and does not actually invoke compaction.
   - Add a new duty `QueueCompactionTasks`
      - This runs as frequently as other coordinator duties (maybe alongwith historical management duties, i.e. every 1 min or so).
      - Avoids the need of having a separate executor/thread and is frequent enough to make the best use of available compaction task slots.
      - Asks the `CompactionJobQueue` for the next compaction tasks to queue and sends them to the Overlord
      - For the most part, this translates to moving the `CompactSegments.doRun()` method to this new duty
   - Stats computation
      - It can remain in `CompactSegments` if it is a costly operation
      - If required, we could even choose to expand `CompactionJobQueue` to be a `ClusterCompactionState` which could accumulate stats and snapshots as and when they are built and report them when necessary.
   
   The other advantages of this approach are that users can benefit from it even if they don't specify a custom period for `CompactSegments`. (In the future, it might even help us do away with having `CompactSegments` as a custom duty, which is a little weird since compaction is a core feature of the ingestion system.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org