You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars Hofhansl (JIRA)" <ji...@apache.org> on 2011/08/17 06:06:27 UTC
[jira] [Created] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Allow coprocessor to interact with batches per region sent from a client(?)
---------------------------------------------------------------------------
Key: HBASE-4210
URL: https://issues.apache.org/jira/browse/HBASE-4210
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Priority: Minor
Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440570#comment-13440570 ]
Lars Hofhansl commented on HBASE-4210:
--------------------------------------
An example is to keep a 2ndary index up to date. In that case we definitely want to make use of the batching. (and we cannot collect in the single hooks, since we have no indication about when we're done).
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---------------------------------------------------------------------------
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440593#comment-13440593 ]
Andrew Purtell commented on HBASE-4210:
---------------------------------------
+1
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---------------------------------------------------------------------------
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086112#comment-13086112 ]
Lars Hofhansl commented on HBASE-4210:
--------------------------------------
The options for that might be {pre|post}Batch hook, or an RpcObserver and start- and end-type hooks.
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---------------------------------------------------------------------------
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Posted by "Asaf Mesika (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499868#comment-13499868 ]
Asaf Mesika commented on HBASE-4210:
------------------------------------
One perfect use case for this is [https://issues.apache.org/jira/browse/HBASE-3434].
You can have a co-processor hook on preIncrementValue which converts a list of Increment to a list of Put, and then have a co-processor hook on preGet which aggregates them and serve them as one result. On the preCompact hook you can aggregate them and increment the original KV.
Without the preMulti you are forced to write each Put to the WAL, thus not enjoying the write once to WAL on a list of Increment. All caused by the fact that you are only seeing one Increment at a time
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---------------------------------------------------------------------------
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440832#comment-13440832 ]
Lars Hofhansl commented on HBASE-4210:
--------------------------------------
Then the question: Where to hook this up. Looking at HRegion.doMiniBatchMutation, we'd leak a lot of internals into the coprocessor hooks, which is bad.
We also do not want rebuild in array or List or Puts or Deletes again just because there might be a coprocessor hook.
I guess we have the following options:
# Pass BatchOperationInProgress to these coprocessor hooks (in HRegion.doMiniBatchMutation)
# Pass the original set of Mutations to the batch pre hook and the same set and their results (OperationStatus[]) to post hook. This means the post hook will have to do an extra loop now, to filter out the failed mutations)
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---------------------------------------------------------------------------
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086155#comment-13086155 ]
Andrew Purtell commented on HBASE-4210:
---------------------------------------
@Lars: Can you talk a bit about the use case you are considering here? What is the motivation?
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---------------------------------------------------------------------------
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with
batches per region sent from a client(?)
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088079#comment-13088079 ]
Lars Hofhansl commented on HBASE-4210:
--------------------------------------
The motivation is generally the same as for batching on the client. There's cost that can be amortized over a set of KVs.
Another reason is that inside a region we can do atomic operations (i.e. we can actually use the rowlocks here), whereas on the client rowlocks are not reliable.
I have no specific use cases at this point, though.
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---------------------------------------------------------------------------
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira