You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (Created) (JIRA)" <ji...@apache.org> on 2011/12/14 19:59:30 UTC
[jira] [Created] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-5027
URL: https://issues.apache.org/jira/browse/HBASE-5027
Project: HBase
Issue Type: Bug
Reporter: stack
Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170280#comment-13170280 ]
nkeywal commented on HBASE-5027:
--------------------------------
"Too many open files"
We're using quite a lot of threads here:
hbase.ResourceChecker(145): after client.TestInstantSchemaChange#testInstantSchemaJanitor: 343 threads (was 117), 863 file descriptors (was 600). 6 connections, -thread leak?- -file handle leak?-
The patch can be committed imho, but I wonder why we have so many threads & open files.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169715#comment-13169715 ]
nkeywal commented on HBASE-5027:
--------------------------------
Ok, I will do that and change the test then.
On Wed, Dec 14, 2011 at 10:27 PM, stack (Commented) (JIRA)
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5027:
---------------------------
Attachment: 5027.patch
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170348#comment-13170348 ]
nkeywal commented on HBASE-5027:
--------------------------------
Yes, i will work
On Thu, Dec 15, 2011 at 6:00 PM, stack (Commented) (JIRA)
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170775#comment-13170775 ]
Hudson commented on HBASE-5027:
-------------------------------
Integrated in HBase-TRUNK-security #33 (See [https://builds.apache.org/job/HBase-TRUNK-security/33/])
HBASE-5027 HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
stack :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170260#comment-13170260 ]
Hadoop QA commented on HBASE-5027:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507513/5027.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -152 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 75 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestInstantSchemaChange
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/513//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/513//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/513//console
This message is automatically generated.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170165#comment-13170165 ]
nkeywal commented on HBASE-5027:
--------------------------------
Patch submitted with the implementation proposed by Stack.
- I checked, if we leak a connection, the test fails
- I removed the clone anyway, it's more efficient
- I added a check on connection count in the ResourceChecker
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal reassigned HBASE-5027:
------------------------------
Assignee: nkeywal
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170346#comment-13170346 ]
stack commented on HBASE-5027:
------------------------------
Pardon me, you already made the cheaper clone fix over in hbase-5022. So, I should just apply this patch minus the change to src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java.... that should remove the regression?
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5027:
-------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed the patch minus TestAdmin change. Thanks Nicolas.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169685#comment-13169685 ]
Zhihong Yu commented on HBASE-5027:
-----------------------------------
I think we should reduce the number of iterations for this test to e.g. 200.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170347#comment-13170347 ]
nkeywal commented on HBASE-5027:
--------------------------------
The cheap clone was already committed by Ted in HBASE-5022. But even with
this, it's still expensive. As we now have only one check now instead of
1000 before, it won't really impact the test however.
On Thu, Dec 15, 2011 at 5:56 PM, stack (Commented) (JIRA)
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169711#comment-13169711 ]
stack commented on HBASE-5027:
------------------------------
bq. It it possible to count the number of connection open? Then we would just have to check that it does not increase after two calls.
If you added this:
{code}
diff --git a/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java b/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
index 17da42d..00f4541 100644
--- a/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
+++ b/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
@@ -85,4 +85,13 @@ public class HConnectionTestingUtility {
return connection;
}
}
-}
+
+ /**
+ * @return Count of extant connection instances
+ */
+ public static int getConnectionCount() {
+ synchronized (HConnectionManager.HBASE_INSTANCES) {
+ return HConnectionManager.HBASE_INSTANCES.size();
+ }
+ }
+}
{code}
... you could count connections. We shouldn't open access to HCM.HBASE_INSTANCES Map. Above does it in a way that allows tests count instances.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169693#comment-13169693 ]
nkeywal commented on HBASE-5027:
--------------------------------
It it possible to count the number of connection open? Then we would just have to check that it does not increase after two calls.
But may be there is a real issue behind this flakiness, even if it's not what the test is looking for.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169678#comment-13169678 ]
stack commented on HBASE-5027:
------------------------------
Is it same test that fails each time, this long-running one of 1000 cycles?
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169583#comment-13169583 ]
stack commented on HBASE-5027:
------------------------------
See tail of hbase-4993 for some discussion.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169590#comment-13169590 ]
nkeywal commented on HBASE-5027:
--------------------------------
In hbase-5022, I replaced the reload implementation with a simple clone. But the comparison with a full removal of the clone in HBaseAdmin shows that the clone is still an expensive function. I believe it triggers some GC as well.
TestAdmin fails sometimes even without the clone (may be less often ? I don't know. But still fails for sure).
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170342#comment-13170342 ]
nkeywal commented on HBASE-5027:
--------------------------------
Yes, I removed the retry counter because the clone is expensive. It's true that when HBase is not available it will be slower. Put it back if yoy think it was an error to remove it.
for lsof: will we have the access rights for this? I can give it a try. But with the few hundreds of open file descriptors, it can be hard to read. As the connection counter comes from HBase, we may get more info from HBase directly I think.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170343#comment-13170343 ]
stack commented on HBASE-5027:
------------------------------
You are right on the lsof; we're running as lowly user, not as root... though I suppose could run it on local machine as root and see?
Maybe I should make a cheaper clone first over in HBaseConfiguration, commit that, then commit above using the cheaper clone?
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169682#comment-13169682 ]
nkeywal commented on HBASE-5027:
--------------------------------
Yes, I executed it alone. On 5 times, it fails 1.
On Wed, Dec 14, 2011 at 9:49 PM, stack (Commented) (JIRA)
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169587#comment-13169587 ]
Zhihong Yu commented on HBASE-5027:
-----------------------------------
I think cloned Configuration should have the same identity with the original one in terms of ConnectionKey.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5027:
---------------------------
Fix Version/s: 0.94.0
Affects Version/s: 0.94.0
Status: Patch Available (was: Open)
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170330#comment-13170330 ]
stack commented on HBASE-5027:
------------------------------
First, counting connections addition is excellent.
Should we add an lsof dump when your checks notice leak? Could run it in hadoop qa only?
Is this a regression:
{code}
public static void checkHBaseAvailable(Configuration conf)
throws MasterNotRunningException, ZooKeeperConnectionException {
- Configuration copyOfConf = HBaseConfiguration.create(conf);
- copyOfConf.setInt("hbase.client.retries.number", 1);
- HBaseAdmin admin = new HBaseAdmin(copyOfConf);
+ HBaseAdmin admin = new HBaseAdmin(conf);
try {
{code}
You've dropped the retry once config?
I like your change to the testCheckHBaseAvailableClosesConnection test.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170350#comment-13170350 ]
nkeywal commented on HBASE-5027:
--------------------------------
't' is missing in my last comment => yes, "just apply this patch minus the change to src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java.... that should remove the regression" will work :-)
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: stack
> Assignee: nkeywal
> Fix For: 0.94.0
>
> Attachments: 5027.patch
>
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5027) HConnection.create(final Connection
conf) does not clone, it creates a new Configuration reading *.xmls and
then does a merge.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169687#comment-13169687 ]
stack commented on HBASE-5027:
------------------------------
I was thinking to whatever the zk client sessions max is + 1 or doubled.
> HConnection.create(final Connection conf) does not clone, it creates a new Configuration reading *.xmls and then does a merge.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5027
> URL: https://issues.apache.org/jira/browse/HBASE-5027
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Its more expensive that it should be; its causing TestAdmin to fail after HBASE-4417 went in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira