You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2009/04/23 19:00:31 UTC
[jira] Created: (HBASE-1341) Create HTable Pooler
Create HTable Pooler
--------------------
Key: HBASE-1341
URL: https://issues.apache.org/jira/browse/HBASE-1341
Project: Hadoop HBase
Issue Type: New Feature
Components: client, io
Reporter: Jonathan Gray
Priority: Minor
Fix For: 0.20.0
A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1341) Create HTable Pooler
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1341:
----------------------------------
Status: Patch Available (was: Open)
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1341) Create HTable Pooler
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1341:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Incorporated feedback and committed.
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341-3.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1341) Create HTable Pooler
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705340#action_12705340 ]
stack commented on HBASE-1341:
------------------------------
Patch looks good to me though I'd just synchronize put and get so following sections of code were executed by one thread at a time:
{code}
+ table = pool.remove();
+ currentSize--;
{code}
and
{code}
+ pool.add(table);
+ currentSize++;
{code}
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1341) Create HTable Pooler
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1341:
----------------------------------
Attachment: hbase-1341-3.patch
Patch -3 incorporates suggestions from comments.
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341-3.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1341) Create HTable Pooler
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705349#action_12705349 ]
stack commented on HBASE-1341:
------------------------------
A bit of fuzzyness on upper bounds should be fine I'd think. +1 (I would suggest you javadoc this on commit so no surprises when someone figures they have 12 HTables when they asked for 10 in the pool).
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1341) Create HTable Pooler
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1341:
----------------------------------
Attachment: hbase-1341.patch
Something like the attached?
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1341) Create HTable Pooler
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705772#action_12705772 ]
stack commented on HBASE-1341:
------------------------------
Minor, perhaps change when you commit:
Why not have getPool(byte []) call getPool(byte[], int)?
Otherwise, looks great.
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341-3.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1341) Create HTable Pooler
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705341#action_12705341 ]
stack commented on HBASE-1341:
------------------------------
That said, synchronizing undoes benefit of using ConcurrentLinkedQueue in first place -- but I don't see an alternative. Do you Andrew?
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1341) Create HTable Pooler
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1341:
----------------------------------
Attachment: hbase-1341-2.patch
Patch -2, a different direction.
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1341) Create HTable Pooler
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705348#action_12705348 ]
Andrew Purtell commented on HBASE-1341:
---------------------------------------
The patch could call size() on the CLQ instead, that would be no problem. "currentSize" was an attempt to avoid the overhead of that, which is llinear not constant time according to Sun's javadoc. A race would mean occasional extra pooled instances, some small factor. "currentSize" is declared as volatile, which would keep that window small, right?
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1341) Create HTable Pooler
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705756#action_12705756 ]
Jonathan Gray commented on HBASE-1341:
--------------------------------------
Looks great, Andrew. Patch looks good but did not test.
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341-3.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1341) Create HTable Pooler
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705456#action_12705456 ]
Jonathan Gray commented on HBASE-1341:
--------------------------------------
Sorry was not following the issue.
Looks good, Andrew. Very straightforward.
We have a much more complex class we use called HBaseRef. It pools HTables for all your tables, so you just pass a single object around in your code rather than one per table. Besides that it behaves very similarly.
One thing I used in a recent modification to HBaseRef was differentiating read/write HTable instances (in benchmarking, no performance hit sharing a single HTable for reads but there is for concurrent writes). Just returned the same one for reads but a growing list for writers. Not sure if this is necessary but it was good to have in some cases.
Is there a way to note in the javadoc what the default is if you don't set maxSize?
Regarding the locks, I'd prefer to synchronize the object and ensure correctness. If not, advertise the fact that there can be indeterminate behavior. I just don't like an optimization like that not done by or known by the user. A bunch of threads using the HTablePool at the same moment is probably the most common use case.
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1341) Create HTable Pooler
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reassigned HBASE-1341:
-------------------------------------
Assignee: Andrew Purtell
> Create HTable Pooler
> --------------------
>
> Key: HBASE-1341
> URL: https://issues.apache.org/jira/browse/HBASE-1341
> Project: Hadoop HBase
> Issue Type: New Feature
> Components: client, io
> Reporter: Jonathan Gray
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1341-2.patch, hbase-1341.patch
>
>
> A client class that takes care of properly pooling HTable references for use in multi-threaded, low-latency Java clients.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.