You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Zsolt Miskolczi (Jira)" <ji...@apache.org> on 2024/02/02 08:35:00 UTC

[jira] [Commented] (HIVE-27637) Compare highest write ID of compaction records when trying to perform abort cleanup

    [ https://issues.apache.org/jira/browse/HIVE-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813561#comment-17813561 ] 

Zsolt Miskolczi commented on HIVE-27637:
----------------------------------------

[~sbadhya] , thank you for the idea of improvement and a review. 

> Compare highest write ID of compaction records when trying to perform abort cleanup
> -----------------------------------------------------------------------------------
>
>                 Key: HIVE-27637
>                 URL: https://issues.apache.org/jira/browse/HIVE-27637
>             Project: Hive
>          Issue Type: Task
>          Components: Hive
>            Reporter: Zsolt Miskolczi
>            Assignee: Zsolt Miskolczi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>
> Compare highest write ID of compaction records when trying to get the potential table/partitions for abort cleanup.
> Idea: If there exists a highest write ID of a record in COMPACTION_QUEUE for a table/partition which is greater than the max(aborted write ID) for that table/partition, then we can potentially ignore abort cleanup for such tables/partitions. This is because compaction will perform cleanup of obsolete deltas and aborted deltas hence doing abort cleanup is redundant here.
> This is more of an optimisation since it can potentially save some filesystem operations (mainly file-listing during construction of Acid state).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)