[hbase] branch branch-2.5 updated: HBASE-27843 If moveAndClose fails HFileArchiver should delete any incomplete archive side changes (#5240)

[ap...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/branch-2.5 by this push:
     new e43fdc75013 HBASE-27843 If moveAndClose fails HFileArchiver should delete any incomplete archive side changes (#5240)
e43fdc75013 is described below

commit e43fdc7501337f72625eb3221db703e801e119b4
Author: Andrew Purtell <ap...@apache.org>
AuthorDate: Thu May 25 15:42:46 2023 -0700

    HBASE-27843 If moveAndClose fails HFileArchiver should delete any incomplete archive side changes (#5240)
    
    When HFiles are placed on a filesystem other than HDFS a rename operation can be
    a non-atomic file copy operation. It can take a long time to copy a large hfile
    and if interrupted there may be a partially copied file present at the destination.
    
    If we fail to “rename” the files into the archive we will continue to fail
    indefinitely. Before larger changes are considered, perhaps to StoreFileTracker, we
    should mitigate this problem.
    
    Signed-off-by: Duo Zhang <zh...@apache.org>
    Signed-off-by: Viraj Jasani <vj...@apache.org>
    Signed-off-by: Wellington Chevreuil <wc...@apache.org>
    Signed-off-by: Xiaolin Ha <ha...@apache.org>
---
 .../org/apache/hadoop/hbase/backup/HFileArchiver.java    | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
index 8615efe6a7e..b2ea9cd33a0 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
@@ -569,8 +569,22 @@ public class HFileArchiver {
           + " because it does not exist! Skipping and continuing on.", fnfe);
         success = true;
       } catch (IOException e) {
-        LOG.warn("Failed to archive " + currentFile + " on try #" + i, e);
         success = false;
+        // When HFiles are placed on a filesystem other than HDFS a rename operation can be a
+        // non-atomic file copy operation. It can take a long time to copy a large hfile and if
+        // interrupted there may be a partially copied file present at the destination. We must
+        // remove the partially copied file, if any, or otherwise the archive operation will fail
+        // indefinitely from this point.
+        LOG.warn("Failed to archive " + currentFile + " on try #" + i, e);
+        try {
+          fs.delete(archiveFile, false);
+        } catch (FileNotFoundException fnfe) {
+          // This case is fine.
+        } catch (IOException ee) {
+          // Complain about other IO exceptions
+          LOG.warn("Failed to clean up from failure to archive " + currentFile + " on try #" + i,
+            ee);
+        }
       }
     }