You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (Created) (JIRA)" <ji...@apache.org> on 2011/12/09 18:36:40 UTC
[jira] [Created] (HBASE-4993) Performance regression in minicluster
creation
Performance regression in minicluster creation
----------------------------------------------
Key: HBASE-4993
URL: https://issues.apache.org/jira/browse/HBASE-4993
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169584#comment-13169584 ]
stack commented on HBASE-4993:
------------------------------
So yeah, HBaseConfiguration#create(Configuration) should be deprecated? And replaced by a clone method that does return new Configuration(conf)? Will a cloned Configuration have same identity -- connection key -- when it comes to connections (see HConnectionManager#getConnection)? Do we want that? Should it have a new identity? I opened hbase-5027 to address this.
Meantime let me apply this patch.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168374#comment-13168374 ]
nkeywal commented on HBASE-4993:
--------------------------------
Errors are hadoop-qa related, usual "Too many open files"
client.TestAdmin#testCheckHBaseAvailableClosesConnection: 559 threads (was 293), 1016 file descriptors (was 478).
java.io.FileNotFoundException: /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/41fdcdb6-e088-480d-b131-18b5ad56356b/dfscluster_822e8789-caad-4a9f-a090-44b5e9e28fc8/dfs/data/data5/blocksBeingWritten/blk_-3873967043651795350_1007.meta (Too many open files)
client.TestInstantSchemaChange#testInstantSchemaJanitor: 343 threads (was 120), 874 file descriptors (was 598).
Can be committed imho.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4993) Performance regression in minicluster
creation
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4993:
---------------------------
Attachment: 4993.patch
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170136#comment-13170136 ]
Hudson commented on HBASE-4993:
-------------------------------
Integrated in HBase-TRUNK-security #32 (See [https://builds.apache.org/job/HBase-TRUNK-security/32/])
HBASE-4993 Performance regression in minicluster creation
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
* /hbase/trunk/src/test/resources/hbase-site.xml
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169115#comment-13169115 ]
stack commented on HBASE-4993:
------------------------------
TestAdmin passes if you do something like this:
{code}
diff --git a/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java b/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
index 6bff130..1189696 100644
--- a/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
+++ b/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
@@ -1529,9 +1529,9 @@ public class HBaseAdmin implements Abortable, Closeable {
*/
public static void checkHBaseAvailable(Configuration conf)
throws MasterNotRunningException, ZooKeeperConnectionException {
- Configuration copyOfConf = HBaseConfiguration.create(conf);
- copyOfConf.setInt("hbase.client.retries.number", 1);
- HBaseAdmin admin = new HBaseAdmin(copyOfConf);
+ // Configuration copyOfConf = HBaseConfiguration.create(conf);
+ // copyOfConf.setInt("hbase.client.retries.number", 1);
+ HBaseAdmin admin = new HBaseAdmin(conf);
try {
admin.close();
} catch (IOException ioe) {
{code}
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4993) Performance regression in minicluster
creation
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4993:
-------------------------
Resolution: Fixed
Fix Version/s: 0.94.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to TRUNK. Thanks for patch Nicolas.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168802#comment-13168802 ]
Zhihong Yu commented on HBASE-4993:
-----------------------------------
Attempt of running TestAdmin timed out the second time:
{code}
[INFO] Total time: 15:12.890s
[INFO] Finished at: Tue Dec 13 14:00:06 PST 2011
[INFO] Final Memory: 22M/120M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.10:test (default-test) on project hbase: Failure or timeout -> [Help 1]
{code}
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237196#comment-13237196 ]
Jean-Daniel Cryans commented on HBASE-4993:
-------------------------------------------
@Nic
I think a bug was introduced here. Here's the new waiting logic in waitForRegionServers:
{code}
- the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
there have been no new region server in for
'hbase.master.wait.on.regionservers.interval' time
{code}
And the code that verifies that:
{code}
!(lastCountChange+interval > now && count >= minToStart)
{code}
If you have 0 region servers that checked in and you are under the interval, you wait: not (true and false) = true.
If you have 0 region servers but you are above the interval, you wait: not (false and false) = true.
If you have 1 or more region servers that checked in and you are under the interval, you continue: not (true and true) = false.
Here's an example:
{noformat}
2012-03-23 21:45:22,002 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2012-03-23 21:45:22,882 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r27s44,62023,1332539122398
2012-03-23 21:45:22,883 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r29s44,62023,1332539122438
2012-03-23 21:45:22,883 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r25s44,62023,1332539122404
2012-03-23 21:45:22,885 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r6s38,62023,1332539122354
2012-03-23 21:45:22,885 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r8s38,62023,1332539122396
2012-03-23 21:45:22,886 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r5s38,62023,1332539122427
2012-03-23 21:45:22,886 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r28s44,62023,1332539122402
2012-03-23 21:45:22,887 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r31s44,62023,1332539122387
2012-03-23 21:45:22,887 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=sv4r30s44,62023,1332539122392
2012-03-23 21:45:22,906 INFO org.apache.hadoop.hbase.master.ServerManager: Finished waiting for region servers count to settle; checked in 9, slept for 904 ms, expecting minimum of 1, maximum of 2147483647, master is running.
{noformat}
As you can see we haven't waited a second and the master is proceeding. This is here not too bad because in the cluster I have 9 servers, but the first time I ran 0.94 it proceeded with only 1 server. This could be disastrous at scale, we really need to wait more than that here. In fact I think I preferred the old way of doing it.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238962#comment-13238962 ]
Jean-Daniel Cryans commented on HBASE-4993:
-------------------------------------------
It seems this would work better, also reviewing the 0.90/0.92 code it think we should keep the new logic you introduced in this jira (with the fixed code). I opened HBASE-5639 and assigned it to you.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169106#comment-13169106 ]
stack commented on HBASE-4993:
------------------------------
Something wrong w/ that test. It loops 1000 times and each loop takes a couple of seconds. Either test was dumb in first place or something changed to slow it down (I was thinking it was the cloning of Configuration added below:
{code}
public static void checkHBaseAvailable(Configuration conf)
throws MasterNotRunningException, ZooKeeperConnectionException {
Configuration copyOfConf = HBaseConfiguration.create(conf);
copyOfConf.setInt("hbase.client.retries.number", 1);
HBaseAdmin admin = new HBaseAdmin(copyOfConf);
try {
admin.close();
} catch (IOException ioe) {
admin.LOG.info("Failed to close connection", ioe);
}
}
{code}
... but it doesn't seem so (I tried changing it). I think we should either figure what slowed this down or punt it over to IntegrationTests because it takes too long for unit tests.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4993) Performance regression in minicluster
creation
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4993:
---------------------------
Status: Patch Available (was: Open)
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166774#comment-13166774 ]
Hadoop QA commented on HBASE-4993:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506817/4993.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -160 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 75 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.client.TestAdmin
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/483//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/483//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/483//console
This message is automatically generated.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168882#comment-13168882 ]
Zhihong Yu commented on HBASE-4993:
-----------------------------------
testCheckHBaseAvailableClosesConnection hangs without this patch :-(
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169186#comment-13169186 ]
nkeywal commented on HBASE-4993:
--------------------------------
HBase#Configuration is expensive by nature, see the comment in @return.
As well the two comments are not saying the same thing: "Creates a clone" is not "reloading the xml file and adding the given conf".
{noformat}
/**
* Creates a clone of passed configuration.
* @param that Configuration to clone.
* @return a Configuration created with the hbase-*.xml files plus
* the given configuration.
*/
public static Configuration create(final Configuration that) {
Configuration conf = create(); // this loads the xml files
merge(conf, that); // for every entry in 'that', replace the content of 'conf' by the content of 'that'
return conf;
}
{noformat}
To be compared with a call to clone as implemented in Configuration
{noformat}
public static Configuration create(final Configuration that) {
new Configuration(that);
return conf;
{noformat}
When I replace the former by the later, the tests is 3 times faster.
We can either:
- remove the function
- replace the former implementation with the one mentioned above
- keep the function, add a performance warning, and replace the calls to this function by a call to the Configuration constructor.
I'm going to check the tests results on option 2.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168676#comment-13168676 ]
nkeywal commented on HBASE-4993:
--------------------------------
I've retested it. Executed 5 times here, worked 4 times, failed once.
I doubt it's related to this patch, the changes in 4993 should have no impact on it.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169929#comment-13169929 ]
Hudson commented on HBASE-4993:
-------------------------------
Integrated in HBase-TRUNK #2545 (See [https://builds.apache.org/job/HBase-TRUNK/2545/])
HBASE-4993 Performance regression in minicluster creation
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
* /hbase/trunk/src/test/resources/hbase-site.xml
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4993) Performance regression in minicluster
creation
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4993:
---------------------------
Attachment: 4993.v3.patch
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169526#comment-13169526 ]
nkeywal commented on HBASE-4993:
--------------------------------
@ted, stack;
Can be committed imho, it seems we all agree that the TestAdmin issues are not linked to this patch
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169585#comment-13169585 ]
nkeywal commented on HBASE-4993:
--------------------------------
Test done:
- it's much much faster if we remove the clone, even after hbase-5022 (3 times faster)
- the time is now constant, while I have a lot of variation when I keep the clone
- it still fails sometimes (1 out 5).
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237342#comment-13237342 ]
nkeywal commented on HBASE-4993:
--------------------------------
Hi JD,
You're right, there is a bug. The code does not do what the documentation
says it does.
It seems that changing the code to {code} (count < minToStart ||
lastCountChange+interval > now){code} would make the code works as
documented.
If you have 0 region servers that checked in and you are under the
interval, you wait: (true or true) = true.
If you have 0 region servers but you are above the interval, you wait:
(true or false) = true.
If you have 1 or more region servers that checked in and you are under the
interval, you wait: (false or true) = true.
Would that be ok for you, or do you want to change the documented behavior
as well?
Note that we could also change the default values for the min number of
servers or the interval.
On Fri, Mar 23, 2012 at 11:07 PM, Jean-Daniel Cryans (Commented) (JIRA) <
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4993) Performance regression in minicluster
creation
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4993:
---------------------------
Status: Open (was: Patch Available)
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169109#comment-13169109 ]
stack commented on HBASE-4993:
------------------------------
I take it back. It is the above change that causes this test to fail. If I do not clone, then the test runs to completion in a couple of seconds otherwise its a couple of seconds per time through the loop.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169565#comment-13169565 ]
Zhihong Yu commented on HBASE-4993:
-----------------------------------
What about Stack's proposed changes @ 14/Dec/11 06:42 ?
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4993) Performance regression in minicluster
creation
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4993:
---------------------------
Status: Patch Available (was: Open)
That's the v1. There is still the timeout issue in it. If I don't have an answer in hbase-4610 tomorrow, I will provide a v2 with a real timeout, but it will make the behavior very different from the current one.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168371#comment-13168371 ]
Hadoop QA commented on HBASE-4993:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507169/4993.v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -152 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 75 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.client.TestInstantSchemaChange
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/497//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/497//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/497//console
This message is automatically generated.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169572#comment-13169572 ]
nkeywal commented on HBASE-4993:
--------------------------------
It not really related to the patch, but to TestAdmin flakiness, so it can be done separately. Note that I still have random failures with 5022, I am not sure if removing the clone will change so much. I will try.
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in
minicluster creation
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168653#comment-13168653 ]
Zhihong Yu commented on HBASE-4993:
-----------------------------------
13 minutes after TestAdmin started to run, it seemed to hang:
{code}
"main" prio=5 tid=104000800 nid=0x100601000 in Object.wait() [1005fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1259)
- locked <78448ef88> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:998)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:229)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
- locked <78448ee80> (a org.apache.hadoop.hbase.MasterAddressTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
- locked <784467418> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:568)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:185)
- locked <790cd5d10> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:98)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1534)
at org.apache.hadoop.hbase.client.TestAdmin.testCheckHBaseAvailableClosesConnection(TestAdmin.java:1485)
{code}
I ran the test on MacBook:
{code}
Darwin LM-SJN-00713032 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64
{code}
> Performance regression in minicluster creation
> ----------------------------------------------
>
> Key: HBASE-4993
> URL: https://issues.apache.org/jira/browse/HBASE-4993
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 4993.patch, 4993.v3.patch
>
>
> Side effect of 4610: the mini cluster needs 4,5 seconds to start
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira