[GitHub] [accumulo] keith-turner commented on pull request #2265: Fixes #2199 allow system compaction to preempt queued user compactions

[GitBox <gi...@apache.org>]

keith-turner commented on pull request #2265:
URL: https://github.com/apache/accumulo/pull/2265#issuecomment-916253828


   > With this change, is it the case that a user compaction could be delayed indefinitely if ingest is occurring into that tablet at such a rate that system compactions are always being run?
   
   I think this is possible. I suspect that as the configurable timeout is increased, that the probability of this happening approaches zero (but never gets to zero).  The configurable expiration time reserves files for a user compactions while its queued.  I set it to 1 minute (not sure what a good default is, still thinking about that).  So for 1 minute after a user compaction is queued, a system compaction can not take any of the files reserved for it.  After a user compaction starts running a system compaction can never take any of its files. 
   
   
   > I'm wondering if the user compaction should preempt the system compaction.
   
   One of the motivating cases for this change is a tablet with lots of files that is queued for really long periods for a user compaction.  For example if a tablet had 1000 files and a user compaction was initiated, then the tablet would reserve those files and queue a user compaction.  If the user compaction sat on the queue for 9 hours then during that time system compactions would not be able to work down the number of files (if system compactions had the capacity to do so).  I feel this potential problem is worth mitigating.  The question is, is the cure (and its potential to cause starvation) worse than the problem.    I feel that its not, but I don't have any numbers to back that up.  I will poke at the likelihood of starvation a bit more and see if I can roughly estimate it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org