[GitHub] [hive] deniskuzZ commented on a change in pull request #2651: HIVE-25535: Control cleaning obsolete directories/files of a table via property

[GitBox <gi...@apache.org>]

deniskuzZ commented on a change in pull request #2651:
URL: https://github.com/apache/hive/pull/2651#discussion_r712203180



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -179,6 +180,13 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB, boolean metricsEnabled
         txnHandler.markCleaned(ci);
         return;
       }
+      if (MetaStoreUtils.isNoCleanUpSet(t.getParameters())) {
+        // The table was marked no clean up true.
+        LOG.info("Skipping " + ci.getFullTableName() + " clean up, as NO_CLEANUP set to true");
+        txnHandler.markCleaned(ci);

Review comment:
       if we won't call ````markCleaned```` that would lead to the accumulation of COMPACTION_QUEUE entries in READY_FOR_CLEANING state (mil of duplicates) + infinite re-try every 5 sec (default). 
   However, if we do - some of the obsolete files could stay forever. 
   Would it be sufficient to have a config that disables the Cleaner completely?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org