You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hbase.apache.org by "Xiaolin Ha (Jira)" <ji...@apache.org> on 2021/10/09 05:35:00 UTC

[jira] [Created] (HBASE-26342) cleaner supports custom paths of independent configuration and pool

Xiaolin Ha created HBASE-26342:
----------------------------------

             Summary: cleaner supports custom paths of independent configuration and pool
                 Key: HBASE-26342
                 URL: https://issues.apache.org/jira/browse/HBASE-26342
             Project: HBase
          Issue Type: Improvement
          Components: master
    Affects Versions: 2.0.0, 3.0.0-alpha-2
            Reporter: Xiaolin Ha
            Assignee: Xiaolin Ha


With this, we can clean some paths more quickly.

We found in our cluster, when the very huge table with thousands of regions and high write throughputs and many snapshots tables on the same cluster, the speed of delete files in  archive path will lower than the speed of moved in files by compaction. Then archive may remains PB level data. 

The bottleneck is in cleaner but not in the thread pool size or queue size. It is because there is synchronized lock in SnapshotFileCache, and a batch of files need once SnapshotFileCache#refreshCache(), which look through all the snapshot dirs.

The speed of clear a path without the SnapshotHFileCleaner is thirty times faster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)