[GitHub] [druid] imply-cheddar commented on pull request #13852: Continuous automatic compaction

["imply-cheddar (via GitHub)" <gi...@apache.org>]

imply-cheddar commented on PR #13852:
URL: https://github.com/apache/druid/pull/13852#issuecomment-1449376685

   Another way you could achieve the intended outcome and, I think, have a bit more straightforward code (at least, you wouldn't need any new configs) would be to introduce a new thread.  Basically, continue to use the duty to figure out which time chunks should be compacted by looking at the segment list/whatever.  The duty populates the state of some object, let's call it a `CompactionJobQueue`.  The new thread that you create reads from the `CompactionJobQueue` and submits jobs until it fills up task slots.  If the next job would consume too many slots, it waits for jobs to finish before submitting the next one. 
   
   In this way, the Duty can change the state of CompactionJobQueue whenever it thinks that it should be changed and the extra thread can submit jobs and wait for their completion regardless of the duty's scheduled runtime.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org