You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars Hofhansl (JIRA)" <ji...@apache.org> on 2012/08/14 07:05:38 UTC
[jira] [Created] (HBASE-6580) New HTable pools, based on
HBase(byte[], HConnection, ExecutorService) constructor
Lars Hofhansl created HBASE-6580:
------------------------------------
Summary: New HTable pools, based on HBase(byte[], HConnection, ExecutorService) constructor
Key: HBASE-6580
URL: https://issues.apache.org/jira/browse/HBASE-6580
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Priority: Minor
Fix For: 0.96.0, 0.94.2
Here I propose a very simple TablePool.
It could be called LightHTablePool (or something - if you have a better name).
Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510741#comment-13510741 ]
Adrian Muraru commented on HBASE-6580:
--------------------------------------
renaming the issue title would be good as well; to reflect the changes planned : something like "Remove HTablePool and replace it with lightweight HTable" would grab the necessary attn I guess :)
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508404#comment-13508404 ]
stack commented on HBASE-6580:
------------------------------
+1 on removing broke, misleading HTablePool in 0.96
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499973#comment-13499973 ]
Hadoop QA commented on HBASE-6580:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12554094/HBASE-6580_v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestCheckTestClasses
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3360//console
This message is automatically generated.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Muraru updated HBASE-6580:
---------------------------------
Attachment: (was: HBASE_v2.patch)
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507822#comment-13507822 ]
Ted Yu commented on HBASE-6580:
-------------------------------
For HConnection.java, please add javadoc for tableName parameter.
{code}
+ // thread executor shared by all HTableInterface instances created
+ // by this connection
{code}
nit: 'created by' -> 'created from'
{code}
+ } catch (InterruptedException e) {
+ this.hTableExecutor.shutdownNow();
+ }
{code}
Please restore interrupt status above.
{code}
+ if (hTableExecutor == null) {
+ // shared HTable thread executor not yet initialized
+ synchronized (this) {
+ if (hTableExecutor == null) {
{code}
Why use double checked locking ? Connection would be used to create (at least) one table, right ?
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434251#comment-13434251 ]
Andrew Purtell commented on HBASE-6580:
---------------------------------------
HConnectionPool ?
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6580:
---------------------------------
Summary: New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor (was: New HTable pools, based on HBase(byte[], HConnection, ExecutorService) constructor)
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501126#comment-13501126 ]
Adrian Muraru commented on HBASE-6580:
--------------------------------------
Lars, I agree the HTablePool is overkill here for these lightweight HTable instances but we'd still need a kind of borrow/release pooling mech for them to avoid creating a new instance every time one is needed.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435780#comment-13435780 ]
Lars Hofhansl commented on HBASE-6580:
--------------------------------------
I can even see replacing the current HTablePool with this.
It is simpler, should be faster, and definitely more lightweight.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507899#comment-13507899 ]
Lars Hofhansl commented on HBASE-6580:
--------------------------------------
Couple of things:
# Should allow passing an ExecutorService to HConnectionManager.createConnection(...). In fact I would require that now, and have that as the only option to setup the ExecutorService.
# getTable must fail if this is a "managed", i.e. not created by createConnection (check the managed flag for that). Otherwise the HTable and the HConnection will get *very* confused.
I love where this is going. I wanted to do this a while ago, but felt that we're not mentally ready for this :)
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Muraru updated HBASE-6580:
---------------------------------
Attachment: HBASE_v2.patch
Added HConnection#getTable()
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508112#comment-13508112 ]
Adrian Muraru commented on HBASE-6580:
--------------------------------------
[~tedyu@apache.org] Thanks for having a look
bq. Why use double checked locking ? Connection would be used to create (at least) one table, right ?
We want to have *multiple* HTable instances sharing the *same* HConnection (this) and the same *ExecutorService* so I ensure only one executor is ever instantiated
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436078#comment-13436078 ]
Andrew Purtell commented on HBASE-6580:
---------------------------------------
bq. I can even see replacing the current HTablePool with this.
I can see this. I believe we have a lot of users of the current HTablePool, so should be API compatible including constructors.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434794#comment-13434794 ]
Lars Hofhansl commented on HBASE-6580:
--------------------------------------
SingleConnectionPool? SharedConnectionPool?
Or just SharedConnection (because that is actually what it is)?
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6580:
---------------------------------
Fix Version/s: (was: 0.94.2)
(was: 0.96.0)
Issue Type: Improvement (was: Bug)
I likely won't get time to on this.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501343#comment-13501343 ]
Gary Helmling commented on HBASE-6580:
--------------------------------------
bq. Maybe HConnection should just finally get a getTable(...) method.
+1. Using HConnection to get an HTable seems more intuitive and more composable than our old monolithic HTable. Lightweight HTables that are easily GC'd would also be very nice. Agree that at that point we don't really need HTablePool.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508252#comment-13508252 ]
Adrian Muraru commented on HBASE-6580:
--------------------------------------
Lars, you're right, having a way to pass an already fine tuned executor is useful such cases. building one from configuration params might not be enough. Thinking of this, I'm more in favour of adding executor param to #getTable In order to Alloa mixt workloads to share the HConnection placement info
ExecutorService lowPriority=....
ExecutorService highPriority=....
HC conn= HCM.createConnection()
conn.getTable(table1, lowPriority)
conn.getTable(table2, highPriority)
+1 removing HTablePool
+1 removing managed HConnections - is this doable? Do we use it elsewhere?
+1 keep only lightweight HTable ctors
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499961#comment-13499961 ]
Lars Hofhansl commented on HBASE-6580:
--------------------------------------
That would work. Ideally I'd like to get HTablePool out of the mix. It's heavy weight and not needed (and in fact just adds overhead) if the both the connection and thread pool are shared between HTables.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Muraru updated HBASE-6580:
---------------------------------
Attachment: HBASE-6580_v1.patch
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508176#comment-13508176 ]
Lars Hofhansl commented on HBASE-6580:
--------------------------------------
Also as part of this, I would like to propose to remove HTablePool for good.
(You do not have to do this part Andrian, unless you like to :), I'll happily do an addon patch.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501302#comment-13501302 ]
Lars Hofhansl commented on HBASE-6580:
--------------------------------------
Not really. After HBASE-4805 you'd create a single HConnection (via HConnectionManager.createConnection(...)) and then use this for each HTable created afterwards. HTable creation is very cheap when both HConnection and ExecutorService are passed... Cheaper indeed than then the HTablePool protocol.
Maybe HConnection should just finally get a getTable(...) method.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Muraru updated HBASE-6580:
---------------------------------
Attachment: HBASE-6580_v2.patch
Added HConnection#getTable()
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508349#comment-13508349 ]
Ted Yu commented on HBASE-6580:
-------------------------------
w.r.t. removing HTablePool, is there discussion / notification on user mail list ?
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508172#comment-13508172 ]
Lars Hofhansl commented on HBASE-6580:
--------------------------------------
You will never be able to size the ExecutorService correctly without knowing the app. This pool is used by all HTable instances using this Connection to parallelize their requests to the RegionServers.
I'd rather have people think about it, rather than having a default that will likely not be right for their use case.
Starting with single core thread is not right. There needs to be at least an option to pass in a custom ExecutorService.
bq. I can do today: HTable t = new HTable(tableName, HConnectionManager.getConnection(conf), pool)
You can do that, and it might even work for a bit :)
HConnectionManager.getConnection(...) is meant for cases where the connection is created (and potentially cached) on behalf of an HTable.
I suppose that constructor should throw an exception is the connection is managed.
This stuff is very confusing about our client.
Like HTablePool, HConnectionManager.getConnection should just go away.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Muraru updated HBASE-6580:
---------------------------------
Affects Version/s: 0.92.2
0.94.2
Release Note: New HTableInterfaceFactory implementation using shared ExecutorService for all HTable instances managed by HTablePool
Status: Patch Available (was: Open)
This is something we need in production as well.
Digging deeper I see the underlying HConnection is already shared in the current HTablePool impl. (i.e there is a single, private Configuration instance passed when HTablePool is initialised and used for all HTable instances built by pool).
What is missing is a shared ExecutorService for all htable's so I called it SharedExecutorHTableFactory
Patch attached (it's on hbase-trunk but I can easily port to 0.92 and 0.94)
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.2, 0.92.2
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Adrian Muraru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508113#comment-13508113 ]
Adrian Muraru commented on HBASE-6580:
--------------------------------------
[~lhofhansl]
bq. Should allow passing an ExecutorService to HConnectionManager.createConnection(...). In fact I would require that now, and have that as the only option to setup the ExecutorService.
Managing the shared ExecutorService internally would be more usable for the user in my opinion, no need to create or shutdown - I look at this executor as an internal detail used to execute *some* table operations in parallel.
bq. getTable must fail if this is a "managed", i.e. not created by createConnection (check the managed flag for that). Otherwise the HTable and the HConnection will get very confused.
Can you elaborate? Why is not advisable to use a "managed" connection when creating a HTable.
I can do today :
HTable t = new HTable(tableName, HConnectionManager.getConnection(conf), pool)
and have an HTable using a "managed" connection
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.2, 0.94.2
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6580) New HTable pool, based on
HBase(byte[], HConnection, ExecutorService) constructor
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434251#comment-13434251 ]
Andrew Purtell edited comment on HBASE-6580 at 8/15/12 3:41 AM:
----------------------------------------------------------------
HConnectionPool ?
Edit: NM, that's not right either.
was (Author: apurtell):
HConnectionPool ?
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it.
> In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira