You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Karen Coppage (Jira)" <ji...@apache.org> on 2020/11/09 16:00:00 UTC
[jira] [Resolved] (HIVE-24275) Configurations to delay the deletion
of obsolete files by the Cleaner
[ https://issues.apache.org/jira/browse/HIVE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karen Coppage resolved HIVE-24275.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Merged #1644 to master branch. Thanks [~pvargacl] for the patch!
> Configurations to delay the deletion of obsolete files by the Cleaner
> ---------------------------------------------------------------------
>
> Key: HIVE-24275
> URL: https://issues.apache.org/jira/browse/HIVE-24275
> Project: Hive
> Issue Type: New Feature
> Reporter: Kishen Das
> Assignee: Peter Varga
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Whenever compaction happens, the cleaner immediately deletes older obsolete files. In certain cases it would be beneficial to retain these for certain period. For example : if you are serving the file metadata from cache and don't want to invalidate the cache during compaction because of performance reasons.
> For this purpose we should introduce a configuration hive.compactor.delayed.cleanup.enabled, which if enabled will delay the cleaning up obsolete files. There should be a separate configuration CLEANER_RETENTION_TIME to specify the duration till which we should retain these older obsolete files.
> It might be beneficial to have one more configuration to decide whether to retain files involved in an aborted transaction hive.compactor.aborted.txn.delayed.cleanup.enabled .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)