You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Viraj Jasani (Jira)" <ji...@apache.org> on 2023/08/23 21:36:00 UTC
[jira] [Created] (HBASE-28042) Snapshot corruptions due to non-atomic rename within same filesystem
Viraj Jasani created HBASE-28042:
------------------------------------
Summary: Snapshot corruptions due to non-atomic rename within same filesystem
Key: HBASE-28042
URL: https://issues.apache.org/jira/browse/HBASE-28042
Project: HBase
Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani
Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1
Sequence of events that can lead to snapshot corruptions:
# Create snapshot using admin command
# Active master triggers async snapshot creation
# If the snapshot operation doesn't complete within 5 min, client gets exception
{code:java}
org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Snapshot 'T1_1691888405683_1691888440827_1' wasn't completed in expectedTime:600000 ms {code}
# Client initiates snapshot deletion after this error
# In the snapshot completion/commit phase, the files are moved from tmp to final dir.
# Snapshot delete and snapshot commit operations can cause corruption by leaving incomplete metadata:
* [Snapshot commit] create "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo"
* [Snapshot delete from client] delete "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo"
* [Snapshot commit] create "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/data-manifest"
The changes introduced by HBASE-21098 performs atomic rename for hbase 1 but not for hbase 2
{code:java}
public static void completeSnapshot(Path snapshotDir, Path workingDir, FileSystem fs,
FileSystem workingDirFs, final Configuration conf)
throws SnapshotCreationException, IOException {
LOG.debug(
"Sentinel is done, just moving the snapshot from " + workingDir + " to " + snapshotDir);
URI workingURI = workingDirFs.getUri();
URI rootURI = fs.getUri();
if (
(!workingURI.getScheme().equals(rootURI.getScheme()) || workingURI.getAuthority() == null
|| !workingURI.getAuthority().equals(rootURI.getAuthority())
|| workingURI.getUserInfo() == null //always true for hdfs://{cluster}
|| !workingURI.getUserInfo().equals(rootURI.getUserInfo())
|| !fs.rename(workingDir, snapshotDir)) //this condition isn't even evaluated due to short circuit above
&& !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true, conf) // non-atomic rename operation
) {
throw new SnapshotCreationException("Failed to copy working directory(" + workingDir
+ ") to completed directory(" + snapshotDir + ").");
}
} {code}
whereas for hbase 1
{code:java}
// check UGI/userInfo
if (workingURI.getUserInfo() == null && rootURI.getUserInfo() != null) {
return true;
}
if (workingURI.getUserInfo() != null &&
!workingURI.getUserInfo().equals(rootURI.getUserInfo())) {
return true;
}
{code}
this causes shouldSkipRenameSnapshotDirectories() to return false if workingURI and rootURI share the same filesystem, which would always lead to atomic rename:
{code:java}
if ((shouldSkipRenameSnapshotDirectories(workingURI, rootURI)
|| !fs.rename(workingDir, snapshotDir))
&& !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true, conf)) {
throw new SnapshotCreationException("Failed to copy working directory(" + workingDir
+ ") to completed directory(" + snapshotDir + ").");
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)