You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Zsolt Miskolczi (Jira)" <ji...@apache.org> on 2024/02/02 08:35:00 UTC
[jira] [Commented] (HIVE-27637) Compare highest write ID of compaction records when trying to perform abort cleanup
[ https://issues.apache.org/jira/browse/HIVE-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813561#comment-17813561 ]
Zsolt Miskolczi commented on HIVE-27637:
----------------------------------------
[~sbadhya] , thank you for the idea of improvement and a review.
> Compare highest write ID of compaction records when trying to perform abort cleanup
> -----------------------------------------------------------------------------------
>
> Key: HIVE-27637
> URL: https://issues.apache.org/jira/browse/HIVE-27637
> Project: Hive
> Issue Type: Task
> Components: Hive
> Reporter: Zsolt Miskolczi
> Assignee: Zsolt Miskolczi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Compare highest write ID of compaction records when trying to get the potential table/partitions for abort cleanup.
> Idea: If there exists a highest write ID of a record in COMPACTION_QUEUE for a table/partition which is greater than the max(aborted write ID) for that table/partition, then we can potentially ignore abort cleanup for such tables/partitions. This is because compaction will perform cleanup of obsolete deltas and aborted deltas hence doing abort cleanup is redundant here.
> This is more of an optimisation since it can potentially save some filesystem operations (mainly file-listing during construction of Acid state).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)