[jira] [Created] (HBASE-28083) Failed backups block backup merges

["Dieter De Paepe (Jira)" <ji...@apache.org>]

Dieter De Paepe created HBASE-28083:
---------------------------------------

             Summary: Failed backups block backup merges
                 Key: HBASE-28083
                 URL: https://issues.apache.org/jira/browse/HBASE-28083
             Project: HBase
          Issue Type: Bug
          Components: backup&restore
         Environment: Encountered on HBase a2e7d2015e9f603e46339d0582e29a86843b9324 (branch-2), running in Kubernetes.
            Reporter: Dieter De Paepe


I was doing some experiments with HBase backup, and came to the following situation:

 
{noformat}
user@hadoop-client-769bc9946-g6p59:/$ hbase backup history
{ID=backup_1694682979468,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:16:20 UTC 2023,End time=Thu Sep 14 09:16:45 UTC 2023,Progress=100%}
{ID=backup_1694682942306,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:15:43 UTC 2023,End time=Thu Sep 14 09:16:11 UTC 2023,Progress=100%}
{ID=backup_1694682906267,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:15:08 UTC 2023,End time=Thu Sep 14 09:15:34 UTC 2023,Progress=100%}
{ID=backup_1694682869991,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:14:31 UTC 2023,End time=Thu Sep 14 09:14:58 UTC 2023,Progress=100%}
{ID=backup_1694681054166,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=FAILED,Start time=Thu Sep 14 08:44:16 UTC 2023,Failed message=REPAIR status: repaired after failure:
backup_1694681054166,Progress=0%}
{ID=backup_1694681006360,Type=FULL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 08:43:27 UTC 2023,End time=Thu Sep 14 08:43:56 UTC 2023,Progress=100%}
{noformat}
Even though I was able to create new incremental backups, I am not able to merge the old backups, due to the failed (& repaired) backup:

 

 
{code:java}
# Failed incremental + completed incremental
user@hadoop-client-769bc9946-g6p59:/$ hbase backup merge backup_1694681054166,backup_1694682869991
23/09/14 09:17:57 ERROR org.apache.hadoop.hbase.backup.BackupDriver: Error running command-line tool
java.io.IOException: Backup image backup_1694681054166 can not be merged becuase of its state: FAILED
    at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.checkIfValidForMerge(BackupAdminImpl.java:664)
    at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.mergeBackups(BackupAdminImpl.java:619)
    at org.apache.hadoop.hbase.backup.impl.BackupCommands$MergeCommand.execute(BackupCommands.java:834)
    at org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:134)
    at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:169)
    at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:199)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
    at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:177)
 {code}
 

 
{code:java}
# Completed full backup + failed incremental
user@hadoop-client-769bc9946-g6p59:/$ hbase backup merge backup_1694681006360,backup_1694681054166
23/09/14 09:19:40 ERROR org.apache.hadoop.hbase.backup.BackupDriver: Error running command-line tool
java.io.IOException: FULL backup image can not be merged for: 
backup_1694681006360
    at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.checkIfValidForMerge(BackupAdminImpl.java:659)
    at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.mergeBackups(BackupAdminImpl.java:619)
    at org.apache.hadoop.hbase.backup.impl.BackupCommands$MergeCommand.execute(BackupCommands.java:834)
    at org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:134)
    at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:169)
    at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:199)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
    at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:177)
 {code}
 

 

I'm not sure why merging a failed backup is not possible, given that it is possible to create new incremental backups on top of it. Either the data in that failed backup is reliable (meaning it should be merge-able), or not reliable (meaning it should not be possible to execute incremental backups on top of it).

If I were to delete the failed backup, all other incremental backups would also be deleted. This made me assume failed backups still keep track of reliable data.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)