[jira] [Commented] (KUDU-2726) Very large tablets defeat budgeted compaction

["wangningito (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247722#comment-17247722 ] 

wangningito commented on KUDU-2726:
-----------------------------------

Either modify tablet_compaction_budget_mb or KUDU-1400 could work around on this. But either of them has disadvantages, the first one may result in compaction time longer, and the second one is as mentioned by William.

To a CompactRowsetOp, it has a logic like this as my understanding,

picking rowsets, and calculate the quality for selection (stage 1) ->  amplify the quality with  FLAGS_maintenance_op_multiplier and '
{{kudu.table.maintenance_priority}}'  and set the amplified value as score of op (stage 2)
 
And to maintenance_manager, it iterate over over all registered op, and select one with higher op score.
 
But to a very big tablet, it may get very low solution value and below the FLAGS_compaction_minimum_improvement, so its' quality would be set to 0, and there is no amplify factor can be apply to compact ops of this tablet. It will be always 0, and always fail in 'stage 1'.
 
I saw now table maintenance strategy can be modified via tables' extra_config interface. So I come up with this: 
we can try pass the  '{{kudu.table.maintenance_priority}}' and multiple it in 'stage 1' or just move FLAGS_compaction_minimum_improvement detect to 'stage 2', then to a very big tablet, the compaction could also work. This also has advantage over the two work arounds mentioned before: its' scope is table, the adjustment can make compaction strategy more flexible. And also, all tablets in a table usually has similar size on disk.
 
How do you think about it? I can work on this.
 

> Very large tablets defeat budgeted compaction
> ---------------------------------------------
>
>                 Key: KUDU-2726
>                 URL: https://issues.apache.org/jira/browse/KUDU-2726
>             Project: Kudu
>          Issue Type: Improvement
>    Affects Versions: 1.9.0
>            Reporter: William Berkeley
>            Priority: Major
>              Labels: density, roadmap-candidate
>
> On very large tablets (50GB+), despite being very uncompacted with a large average rowset height, a default budget (128MB) worth of compaction may not reduce average rowset height enough to pass the minimum threshold. Thus the tablet stays uncompacted forever.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)