You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hbase.apache.org by "Lavinia-Stefania Sirbu (JIRA)" <ji...@apache.org> on 2018/10/10 11:04:00 UTC

[jira] [Created] (HBASE-21286) Parallelize computeHDFSBlocksDistribution when getting splits of a HBaseSnapshot

Lavinia-Stefania Sirbu created HBASE-21286:
----------------------------------------------

             Summary: Parallelize computeHDFSBlocksDistribution when getting splits of a HBaseSnapshot
                 Key: HBASE-21286
                 URL: https://issues.apache.org/jira/browse/HBASE-21286
             Project: HBase
          Issue Type: Improvement
          Components: snapshots
    Affects Versions: 1.4.0
            Reporter: Lavinia-Stefania Sirbu


Even if this step is called computeHDFSBlocksDistribution, this is executed no matter the file system of the snapshot. For example, we have observed an important slowness when we have a snapshot in s3 (~26k regions, 5column families, 2 files per column family) the getsplits time is ~40min due to the calls in s3 for listing the files to get the best locations.

Parallelizing this operation can reduce the overall setup time. The thread pool should be configurable and a good choice could be "hbase.snapshot.thread.pool.max" that is also used in RestoreSnapshotHelper.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)