[jira] [Resolved] (HIVE-30026) (ACID Compaction) Extend the query for finding potential compactions with ordering

["Marta Kuczora (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-30026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marta Kuczora resolved HIVE-30026.
----------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed

> (ACID Compaction) Extend the query for finding potential compactions with ordering
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-30026
>                 URL: https://issues.apache.org/jira/browse/HIVE-30026
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>            Reporter: Marta Kuczora
>            Assignee: Marta Kuczora
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> After HIVE-27444, if the number of candidates is bigger than the compactor.fetch.size value, it can happen that the Initiator doesn't schedule all tables for compaction.
> For example: we have 4 tables, which are all eligible for compaction, but the fetch size is 2.
> In the first initiator run, it will pick two of these tables. It would be expected that during the second run of the initiator it would pick the other two tables, so all 4 tables have been scheduled for compaction.
> Since there is no ordering when selecting the tables for compaction, this was not always the case. It could happen that some tables never got compacted.
>  
> Also if the number of candidates is bigger than the compactor.fetch.size limit, the aborted txn candidates will never be picked up by the initiator. Because the "remaining budget" will always be 0 in the FindPotentialCompactionsFunction. 
> {code:java}
> Set<CompactionInfo> candidates = new HashSet<>(jdbcResource.execute(new CompactionCandidateHandler(lastChecked, fetchSize)));
> int remaining = fetchSize - candidates.size();
> if (collectAbortedTxns && remaining > 0) {
>   candidates.addAll(jdbcResource.execute(new AbortedTxnHandler(abortedTimeThreshold, abortedThreshold, remaining)));
> }{code}
> This affects only the cases when the aborted txn clean-up is done by the compaction cycle and not just the cleaner.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)