[jira] [Created] (HBASE-27825) Store file archiving is not sufficiently indirected through the store file tracker

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

Andrew Kyle Purtell created HBASE-27825:
-------------------------------------------

             Summary: Store file archiving is not sufficiently indirected through the store file tracker
                 Key: HBASE-27825
                 URL: https://issues.apache.org/jira/browse/HBASE-27825
             Project: HBase
          Issue Type: Bug
    Affects Versions: 2.5.4
            Reporter: Andrew Kyle Purtell


GCRegionProcedure fails cleaning up old parents after splits. We time out “renaming” files into the archive. On S3, a rename operation is a whole file copy operation. The time required for the S3 operation to complete is O(n) with respect to the length of the file.

{noformat}
[PEWorker-21] backup.HFileArchiver: Failed to archive FileablePath, s3a://[...]
java.net.SocketTimeoutException: copyFile(data/default/cluster_test/[...], archive/data/default/cluster_test/[...]) on data/default/cluster_test/[...]: com.amazonaws.SdkClientException: Unable to execute HTTP request: Read timed out
{noformat}

Once we fail to “rename” the files into the archive we continue to fail because renames on S3 are not atomic. They are an object copy operation which is neither atomic nor automatically rolled back. The incomplete object remains present. The GCRegionProcedure can never complete successfully.

{noformat}
org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename s3a://[...] to s3a://[...]; destination file exists
{noformat}

{noformat}
org.apache.hadoop.hbase.backup.FailedArchiveException: Failed to archive/delete all the files for region:ddaf1fb41197254483dcfd1d63e869d0 into s3a://[...]. Something is probably awry on the filesystem.
{noformat}

Short term mitigations:

In HFileArchiver#resolveAndArchiveFile, if moveAndClose of the current file fails, attempt to delete the incomplete archive side file.

Also set the recommended default read timeout for S3A to a larger value.

Long term:

When the file based store file tracker is enabled, the archived files for a store should not longer be moved to a separate path from the live files in the store. Instead whether or not the file is archived or not should be a status bit maintained in the tracker manifest.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)