You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "zhou wenjian (JIRA)" <ji...@apache.org> on 2012/07/26 10:58:34 UTC
[jira] [Created] (HBASE-6459) improve speed of create table
zhou wenjian created HBASE-6459:
-----------------------------------
Summary: improve speed of create table
Key: HBASE-6459
URL: https://issues.apache.org/jira/browse/HBASE-6459
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: zhou wenjian
Fix For: 0.94.1
in CreateTableHandler
for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
HRegionInfo newRegion = this.newRegions[regionIdx];
// 1. Create HRegion
HRegion region = HRegion.createHRegion(newRegion,
this.fileSystemManager.getRootDir(), this.conf,
this.hTableDescriptor, null, false, true);
regionInfos.add(region.getRegionInfo());
if (regionIdx % batchSize == 0) {
// 2. Insert into META
MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
regionInfos.clear();
}
// 3. Close the new region to flush to disk. Close log file too.
region.close();
}
All the region will be create serially.
If we have thousands of regions, that will be a huge cost.
We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of table creation
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480212#comment-13480212 ]
Hudson commented on HBASE-6459:
-------------------------------
Integrated in HBase-TRUNK #3460 (See [https://builds.apache.org/job/HBase-TRUNK/3460/])
HBASE-6459 improve speed of table creation (Zhou Wenjian) (Revision 1400152)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
> improve speed of table creation
> -------------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6459:
-------------------------
Attachment: 6459v3.txt
Updated to match trunk.
I don't think we should put these defines into HConstants. They are only used in CreateTableHandler.
Are these threads daemonized? I don't see it looking over in Threads. You think they should be?
In getRegionOpenAndInitializedThreadPool, why not just return result of Threads..getBoundedCachedThreadPool? Why have getBoundedCachedThreadPool at all? Just have it in getRegionOpenAndInitializedThreadPool.
Patch looks good otherwise.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426308#comment-13426308 ]
zhou wenjian commented on HBASE-6459:
-------------------------------------
I think failed testcase is invovled with HBASE-6478.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480027#comment-13480027 ]
Ted Yu commented on HBASE-6459:
-------------------------------
I ran the failed test locally and it passed.
{code}
Running org.apache.hadoop.hbase.client.TestFromClientSide
2012-10-19 06:55:19.937 java[39861:1903] Unable to load realm mapping info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.821 sec
{code}
Will integrate later today if there is no more review comment.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of table creation
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-6459:
--------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> improve speed of table creation
> -------------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423001#comment-13423001 ]
zhou wenjian commented on HBASE-6459:
-------------------------------------
I'll give a patch later
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of table creation
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480107#comment-13480107 ]
Ted Yu commented on HBASE-6459:
-------------------------------
Integrated to trunk.
Thanks for the patch, Wenjian.
Thanks for the review, Stack.
> improve speed of table creation
> -------------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of table creation
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-6459:
--------------------------
Summary: improve speed of table creation (was: improve speed of create table)
> improve speed of table creation
> -------------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Attachment: HBASE-6459-90.patch
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468223#comment-13468223 ]
Hadoop QA commented on HBASE-6459:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12547467/6459v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 83 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2990//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2990//console
This message is automatically generated.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-6459:
--------------------------
Assignee: Zhou wenjian
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Attachment: HBASE-6459-92.patch
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Attachment: HBASE-6459-94.patch
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Fix Version/s: (was: 0.94.1)
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424709#comment-13424709 ]
zhou wenjian commented on HBASE-6459:
-------------------------------------
patch for trunk
changed the name to "hbase.hregion.open.and.init.threads.max" as Ted suggests
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424705#comment-13424705 ]
Zhihong Ted Yu commented on HBASE-6459:
---------------------------------------
How about naming the new config param "hbase.hregion.open.and.init.threads.max" ?
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424675#comment-13424675 ]
zhou wenjian commented on HBASE-6459:
-------------------------------------
we have test the patch 90 in our test cluster (5rs,3u3)
create 2000 regions
before: 80s+
after: in 10s
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Attachment: HBASE-6459-trunk-v2.patch
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Fix Version/s: 0.96.0
Issue Type: Improvement (was: Bug)
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Status: Open (was: Patch Available)
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of table creation
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480606#comment-13480606 ]
Lars Hofhansl commented on HBASE-6459:
--------------------------------------
Do we want this in 0.94? Seems useful for presplit tables.
> improve speed of table creation
> -------------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479624#comment-13479624 ]
Hadoop QA commented on HBASE-6459:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12549802/6459.v4
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3086//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3086//console
This message is automatically generated.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Attachment: HBASE-6459-trunk.patch
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6459:
----------------------------------
Status: Patch Available (was: Open)
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424761#comment-13424761 ]
Hadoop QA commented on HBASE-6459:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538311/HBASE-6459-trunk-v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestClassLoading
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2455//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2455//console
This message is automatically generated.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhou wenjian updated HBASE-6459:
--------------------------------
Status: Patch Available (was: Open)
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of table creation
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480545#comment-13480545 ]
Hudson commented on HBASE-6459:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #228 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/228/])
HBASE-6459 improve speed of table creation (Zhou Wenjian) (Revision 1400152)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
> improve speed of table creation
> -------------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424681#comment-13424681 ]
Zhihong Ted Yu commented on HBASE-6459:
---------------------------------------
In your test, did you kill one of the region servers ?
{code}
+ public static final String HREGION_OPEN_AND_INITIALIZE_THREADS_MAX = "hbase.hregion.open.and.initialize.threads.max";
{code}
The above line is too long (> 100 characters). Can we find a shorter name for "hbase.hregion.open.and.initialize.threads.max" ?
I would suggest producing patch for trunk so that Hadoop QA can run tests over it.
After trunk patch passes review, patches for branches can be produced.
Since this is an improvement, I doubt it would go to other branches.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424700#comment-13424700 ]
zhou wenjian commented on HBASE-6459:
-------------------------------------
@Ted
the name i used is from the name below in HConstants.java
public static final String HSTORE_OPEN_AND_CLOSE_THREADS_MAX =
"hbase.hstore.open.and.close.threads.max";
would you please give a suitable name?
In your test, did you kill one of the region servers ?
No, I didn't. What i do is to create hregion in parallel (in master), I think it is not involved with rs.
I'll make a patch for trunk, sorry for that I am not quitely followed the rule with issue.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
Posted by "zhou wenjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426307#comment-13426307 ]
zhou wenjian commented on HBASE-6459:
-------------------------------------
I think failed testcase is invovled with HBASE-6459.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-6459:
--------------------------
Attachment: 6459.v4
Patch v4 moves the constants from HConstants to CreateTableHandler.java
Also removed the second getRegionOpenAndInitThreadPool() method.
Some typo is fixed.
> improve speed of create table
> -----------------------------
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: zhou wenjian
> Assignee: Zhou wenjian
> Fix For: 0.96.0
>
> Attachments: 6459v3.txt, 6459.v4, HBASE-6459-90.patch, HBASE-6459-92.patch, HBASE-6459-94.patch, HBASE-6459-trunk.patch, HBASE-6459-trunk-v2.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira