You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Josh Elser (Jira)" <ji...@apache.org> on 2021/11/09 22:28:00 UTC
[jira] [Created] (HBASE-26437) [hboss] Rename does not clean up
znodes for src location
Josh Elser created HBASE-26437:
----------------------------------
Summary: [hboss] Rename does not clean up znodes for src location
Key: HBASE-26437
URL: https://issues.apache.org/jira/browse/HBASE-26437
Project: HBase
Issue Type: Bug
Components: hboss
Affects Versions: hbase-filesystem-1.0.0-alpha1
Reporter: Josh Elser
Assignee: Josh Elser
We ran into a fun situation where the partition hosting ZK data was repeatedly filling up while heavy ExportSnapshot+clone_snapshot operations were running (10's of TB). The cluster was previously working just fine.
Upon investigation of the ZK tree, we found a large number of znodes beneath /hboss, specifically many in the corresponding ZK HBOSS path for $hbase.rootdir/.tmp.
Tracing back from the code, we saw that the CloneSnapshotProcedure (like CreateTableProcedure) will create the table filesystem layout in $hbase.rootdir/.tmp and then rename it into $hbase.rootdir/data/<namespace>. However, it appears that, upon rename, HBOSS was not cleaning up the src path's znode. This is a bug as it allows ZK to grow unbounded (which explains why this problem slowly arose and not suddenly).
As a workaround, HBase can be stopped and the corresponding ZK path for $hbase.rootdir/.tmp can be cleaned up to reclaim 1/2 the space taken up by znodes for imported hbase tables (we would still have znodes for $hbase.rootdir/data/...)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)