[hbase] branch master updated: HBASE-27843 If moveAndClose fails HFileArchiver should delete any incomplete archive side changes (#5240)

[ap...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/master by this push:
     new 7510d061c39 HBASE-27843 If moveAndClose fails HFileArchiver should delete any incomplete archive side changes (#5240)
7510d061c39 is described below

commit 7510d061c39d59f2c83807a66865c896e2c00ba9
Author: Andrew Purtell <ap...@apache.org>
AuthorDate: Thu May 25 15:42:46 2023 -0700

    HBASE-27843 If moveAndClose fails HFileArchiver should delete any incomplete archive side changes (#5240)
    
    When HFiles are placed on a filesystem other than HDFS a rename operation can be
    a non-atomic file copy operation. It can take a long time to copy a large hfile
    and if interrupted there may be a partially copied file present at the destination.
    
    If we fail to “rename” the files into the archive we will continue to fail
    indefinitely. Before larger changes are considered, perhaps to StoreFileTracker, we
    should mitigate this problem.
    
    Signed-off-by: Duo Zhang <zh...@apache.org>
    Signed-off-by: Viraj Jasani <vj...@apache.org>
    Signed-off-by: Wellington Chevreuil <wc...@apache.org>
    Signed-off-by: Xiaolin Ha <ha...@apache.org>
---
 .../org/apache/hadoop/hbase/backup/HFileArchiver.java    | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
index 8615efe6a7e..b2ea9cd33a0 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
@@ -569,8 +569,22 @@ public class HFileArchiver {
           + " because it does not exist! Skipping and continuing on.", fnfe);
         success = true;
       } catch (IOException e) {
-        LOG.warn("Failed to archive " + currentFile + " on try #" + i, e);
         success = false;
+        // When HFiles are placed on a filesystem other than HDFS a rename operation can be a
+        // non-atomic file copy operation. It can take a long time to copy a large hfile and if
+        // interrupted there may be a partially copied file present at the destination. We must
+        // remove the partially copied file, if any, or otherwise the archive operation will fail
+        // indefinitely from this point.
+        LOG.warn("Failed to archive " + currentFile + " on try #" + i, e);
+        try {
+          fs.delete(archiveFile, false);
+        } catch (FileNotFoundException fnfe) {
+          // This case is fine.
+        } catch (IOException ee) {
+          // Complain about other IO exceptions
+          LOG.warn("Failed to clean up from failure to archive " + currentFile + " on try #" + i,
+            ee);
+        }
       }
     }