You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "chunhui shen (JIRA)" <ji...@apache.org> on 2012/05/02 07:39:02 UTC
[jira] [Created] (HBASE-5913) Speed up the full scan of META
chunhui shen created HBASE-5913:
-----------------------------------
Summary: Speed up the full scan of META
Key: HBASE-5913
URL: https://issues.apache.org/jira/browse/HBASE-5913
Project: HBase
Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-5913.patch
In the master, we will do the full scan of META in some situations
for example,
1.master start up
2.CatalogJanitor do the full scan per 5 mins
3.ServerShutdownHandler, getServerUserRegions for dead server.
For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
And it is caused by two reasons:
The first, we don't use cache and get one row per next() when fully scan .META.
The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266714#comment-13266714 ]
Hadoop QA commented on HBASE-5913:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525310/5913-v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestClassLoading
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1725//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1725//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1725//console
This message is automatically generated.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5913:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to 0.94 as well.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HBASE-5913) Speed up the full scan of META
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl closed HBASE-5913.
--------------------------------
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.94.1, 0.96.0
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267263#comment-13267263 ]
Hudson commented on HBASE-5913:
-------------------------------
Integrated in HBase-0.94 #174 (See [https://builds.apache.org/job/HBase-0.94/174/])
HBASE-5913 Speed up the full scan of META (Revision 1333315)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5913:
------------------------------
Attachment: (was: 5913-v2.txt)
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5913:
------------------------------
Attachment: 5913-v2.txt
Patch v2 plugs in HConstants.HBASE_META_SCANNER_CACHING
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5913:
------------------------------
Fix Version/s: 0.96.0
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5913:
------------------------------
Attachment: 5913-v2.txt
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266724#comment-13266724 ]
Zhihong Yu commented on HBASE-5913:
-----------------------------------
I looped TestClassLoading 5 times using patch v2 and they passed.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268836#comment-13268836 ]
Hudson commented on HBASE-5913:
-------------------------------
Integrated in HBase-0.94-security #26 (See [https://builds.apache.org/job/HBase-0.94-security/26/])
HBASE-5913 Speed up the full scan of META (Revision 1333315)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267179#comment-13267179 ]
Hudson commented on HBASE-5913:
-------------------------------
Integrated in HBase-TRUNK #2840 (See [https://builds.apache.org/job/HBase-TRUNK/2840/])
HBASE-5913 Speed up the full scan of META (Chunhui) (Revision 1333283)
Result = SUCCESS
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267008#comment-13267008 ]
Hadoop QA commented on HBASE-5913:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525353/5913-v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1729//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1729//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1729//console
This message is automatically generated.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267225#comment-13267225 ]
Hudson commented on HBASE-5913:
-------------------------------
Integrated in HBase-TRUNK-security #190 (See [https://builds.apache.org/job/HBase-TRUNK-security/190/])
HBASE-5913 Speed up the full scan of META (Chunhui) (Revision 1333283)
Result = SUCCESS
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "chunhui shen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-5913:
--------------------------------
Attachment: HBASE-5913.patch
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267162#comment-13267162 ]
Lars Hofhansl commented on HBASE-5913:
--------------------------------------
Will commit to 0.94 tonight if there are no objections.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267081#comment-13267081 ]
Zhihong Yu commented on HBASE-5913:
-----------------------------------
Integrated to trunk.
Thanks for the patch, Chunhui.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5913:
------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
Patch looks good.
Minor comment: can refer to HConstants.HBASE_META_SCANNER_CACHING
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266958#comment-13266958 ]
Zhihong Yu commented on HBASE-5913:
-----------------------------------
I will integrate this tonight if there are no further review comments.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5913) Speed up the full scan of META
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5913:
---------------------------------
Fix Version/s: 0.94.1
I'd like this in 0.94 as well.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266593#comment-13266593 ]
Hadoop QA commented on HBASE-5913:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525259/HBASE-5913.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1722//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1722//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1722//console
This message is automatically generated.
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5913) Speed up the full scan of META
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267178#comment-13267178 ]
stack commented on HBASE-5913:
------------------------------
+1 on patch
> Speed up the full scan of META
> ------------------------------
>
> Key: HBASE-5913
> URL: https://issues.apache.org/jira/browse/HBASE-5913
> Project: HBase
> Issue Type: Improvement
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5913-v2.txt, HBASE-5913.patch
>
>
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler in the situation 3.
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment it take 40ms for per next() (It is related to the length of row in the .META. , if someone also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira