[jira] [Created] (HBASE-28643) An unbounded backup failure message can cause an irrecoverable state for the given backup

["Ray Mattingly (Jira)" <ji...@apache.org>]

Ray Mattingly created HBASE-28643:
-------------------------------------

             Summary: An unbounded backup failure message can cause an irrecoverable state for the given backup
                 Key: HBASE-28643
                 URL: https://issues.apache.org/jira/browse/HBASE-28643
             Project: HBase
          Issue Type: Bug
    Affects Versions: 2.6.0
            Reporter: Ray Mattingly


The BackupInfo class has a failedMsg field which is a string of unbounded length. When a DistCp job fails then its failure message contains all of its source paths, and its failure message gets propagated to this failedMsg field on the given BackupInfo.

If a DistCp job has enough source paths, then this will result in backup status updates being rejected:
{noformat}
java.lang.IllegalArgumentException: KeyValue size too large
        at org.apache.hadoop.hbase.client.ConnectionUtils.validatePut(ConnectionUtils.java:513)
        at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1095)
        at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:564)
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:563)
        at org.apache.hadoop.hbase.backup.impl.BackupSystemTable.updateBackupInfo(BackupSystemTable.java:292)
        at org.apache.hadoop.hbase.backup.impl.BackupManager.updateBackupInfo(BackupManager.java:376)
        at org.apache.hadoop.hbase.backup.impl.TableBackupClient.failBackup(TableBackupClient.java:243)
        at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:317)
        at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:603)
        at com.hubspot.hbase.recovery.core.backup.BackupManager.lambda$runBackups$2(BackupManager.java:145){noformat}
Without the ability to update the backup's state, it will never be returned as a failed backup by the client. This means that any mechanisms designed for repairing or cleaning failed backups won't work properly.

I think that a simple fix here would be fine: we should truncate the failedMsg field to a reasonable maximum size.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)