You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Mingjie Lai <mj...@gmail.com> on 2010/10/01 04:18:18 UTC
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
(Updated 2010-09-30 19:18:18.297399)
Review request for hbase.
Changes
-------
Fixes by review board comments
Summary
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
src/main/resources/hbase-default.xml
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
* CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
CommandTarget:
Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs (updated)
-----
conf/hbase-policy.xml PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 365e4b9
src/main/java/org/apache/hadoop/hbase/HServerAddress.java 3859968
src/main/java/org/apache/hadoop/hbase/HServerInfo.java aaf4835
src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java fbdec0b
src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 3a429c0
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 19dbf2b
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bacf8b1
src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 71a0447
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 0a4fbce
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e80ece8
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
src/main/resources/hbase-default.xml 5fafe65
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java e5b1a30
src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 63a6956
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 377e6b1
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java dc55407
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Gary Helmling <gh...@gmail.com>.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > Sorry for all the whitespace comments :)
> > There are a bunch more in the test classes.
Since some of this covers the included patch from HBASE-2321/HBASE-2002, I'll reply to those issues.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/client/Action.java, line 91
> > <http://review.cloudera.org/r/876/diff/5/?file=13544#file13544line91>
> >
> > The ternary operator does not need braces.
Sure, I just find the parens add clarity. I can remove.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1142
> > <http://review.cloudera.org/r/876/diff/5/?file=13549#file13549line1142>
> >
> > Should be of Type List<R> not ArrayList<R>
Seems less important to me for a local variable, but I'll change.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1143
> > <http://review.cloudera.org/r/876/diff/5/?file=13549#file13549line1143>
> >
> > Why is this necessary? You already set the size by using the correct constructor.
The constructor argument sets initial capacity, not size. In this case we need to set size, which means prefilling the list, since null entries are how we indicate failure for individual rows.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java, line 353
> > <http://review.cloudera.org/r/876/diff/5/?file=13551#file13551line353>
> >
> > Remove the "public", interfaces don't need that.
Sure, copy-n-paste error, will remove.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java, line 398
> > <http://review.cloudera.org/r/876/diff/5/?file=13551#file13551line398>
> >
> > Remove the "public", interfaces don't need that.
Will remove.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java, line 36
> > <http://review.cloudera.org/r/876/diff/5/?file=13565#file13565line36>
> >
> > public Log?
Odd, it's never even used. Removing.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 348
> > <http://review.cloudera.org/r/876/diff/5/?file=13578#file13578line348>
> >
> > Wrong formatting
Should not be included in this patch. Will correct.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 807
> > <http://review.cloudera.org/r/876/diff/5/?file=13580#file13580line807>
> >
> > Whitespace stuff
Should not be included in this patch. Will correct.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java, line 62
> > <http://review.cloudera.org/r/876/diff/5/?file=13592#file13592line62>
> >
> > Whitespace stuff
Should not be included in this patch. Will correct.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java, line 115
> > <http://review.cloudera.org/r/876/diff/5/?file=13592#file13592line115>
> >
> > Whitespace stuff
Should not be included in this patch. Will correct.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java, line 128
> > <http://review.cloudera.org/r/876/diff/5/?file=13593#file13593line128>
> >
> > Whitespace stuff
Should not be included in this patch. Will correct.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java, line 39
> > <http://review.cloudera.org/r/876/diff/5/?file=13567#file13567line39>
> >
> > private static final
Will fix.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java, line 92
> > <http://review.cloudera.org/r/876/diff/5/?file=13570#file13570line92>
> >
> > Whitespace stuff
Should not be included in this patch. Fixing.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java, line 372
> > <http://review.cloudera.org/r/876/diff/5/?file=13551#file13551line372>
> >
> > Remove the "public", interfaces don't need that.
> >
> > Also byte[] key in Map so every implementor has to make sure to use a Map that does this correctly.
Will remove public.
Not sure I understand the Map keyed by byte[] comment. That's part of the contract for clients. So all implementors must implement it correctly or indicate they don't support it through something like an UnsupportedOperationException.
I suppose I could narrow the type to SortedMap, since that indicates use of a comparator for byte[] keys. What were you suggesting?
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/HServerInfo.java, line 99
> > <http://review.cloudera.org/r/876/diff/5/?file=13543#file13543line99>
> >
> > The ternary operator does not need braces.
Should not actually be included in this patch. Fixing.
> On 2010-10-03 02:28:20, Lars Francke wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/Status.java, line 27
> > <http://review.cloudera.org/r/876/diff/5/?file=13581#file13581line27>
> >
> > Whitespace stuff
Should not be included in this patch. Fixing.
- Gary
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1380
-----------------------------------------------------------
On 2010-10-02 23:53:05, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-02 23:53:05)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> conf/hbase-policy.xml PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 365e4b9
> src/main/java/org/apache/hadoop/hbase/HServerAddress.java 3859968
> src/main/java/org/apache/hadoop/hbase/HServerInfo.java aaf4835
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0feb17c
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 3a429c0
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 19dbf2b
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bacf8b1
> src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 71a0447
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 0a4fbce
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java e5b1a30
> src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 63a6956
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 377e6b1
> src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java dc55407
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Lars Francke <la...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1380
-----------------------------------------------------------
Sorry for all the whitespace comments :)
There are a bunch more in the test classes.
src/main/java/org/apache/hadoop/hbase/HServerInfo.java
<http://review.cloudera.org/r/876/#comment4581>
The ternary operator does not need braces.
src/main/java/org/apache/hadoop/hbase/client/Action.java
<http://review.cloudera.org/r/876/#comment4582>
The ternary operator does not need braces.
src/main/java/org/apache/hadoop/hbase/client/Batch.java
<http://review.cloudera.org/r/876/#comment4583>
Remove extra character(s)
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
<http://review.cloudera.org/r/876/#comment4586>
Should be of Type List<R> not ArrayList<R>
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
<http://review.cloudera.org/r/876/#comment4585>
Why is this necessary? You already set the size by using the correct constructor.
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
<http://review.cloudera.org/r/876/#comment4588>
Remove the "public", interfaces don't need that.
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
<http://review.cloudera.org/r/876/#comment4589>
Remove the "public", interfaces don't need that.
Also byte[] key in Map so every implementor has to make sure to use a Map that does this correctly.
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
<http://review.cloudera.org/r/876/#comment4590>
Remove the "public", interfaces don't need that.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
<http://review.cloudera.org/r/876/#comment4591>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
<http://review.cloudera.org/r/876/#comment4592>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
<http://review.cloudera.org/r/876/#comment4593>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4594>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4595>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4596>
Inconsistent formatting
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4597>
Inconsistent formatting
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4598>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4599>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4600>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4601>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4602>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4603>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4604>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment4605>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4612>
Remove public static final
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4606>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4613>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4614>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4615>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4607>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4616>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4617>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4608>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4618>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4609>
Whitespace stuff
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4610>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4619>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4611>
Whitespace stuff
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4620>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment4621>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4622>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4623>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4624>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4625>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4626>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4627>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4628>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment4629>
Remove public
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
<http://review.cloudera.org/r/876/#comment4630>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
<http://review.cloudera.org/r/876/#comment4631>
Remove extra space behind the brace
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4642>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4632>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4643>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4644>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4645>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4633>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4646>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4634>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4647>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4648>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4649>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4635>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4650>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4651>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4636>
Whitespace stuff
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4637>
Whitespace stuff
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4638>
Whitespace stuff
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4639>
Whitespace stuff
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4652>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4653>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4654>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4655>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4656>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4640>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4657>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4658>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4641>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment4659>
Remove "public"
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
<http://review.cloudera.org/r/876/#comment4707>
Lots of whitespace stuff in this file. I've not listed them all.
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
<http://review.cloudera.org/r/876/#comment4708>
public Log?
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java
<http://review.cloudera.org/r/876/#comment4709>
private static final
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java
<http://review.cloudera.org/r/876/#comment4710>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
<http://review.cloudera.org/r/876/#comment4711>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
<http://review.cloudera.org/r/876/#comment4712>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
<http://review.cloudera.org/r/876/#comment4713>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
<http://review.cloudera.org/r/876/#comment4714>
Wrong formatting
src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
<http://review.cloudera.org/r/876/#comment4715>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/Status.java
<http://review.cloudera.org/r/876/#comment4716>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
<http://review.cloudera.org/r/876/#comment4717>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
<http://review.cloudera.org/r/876/#comment4718>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
<http://review.cloudera.org/r/876/#comment4719>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
<http://review.cloudera.org/r/876/#comment4720>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
<http://review.cloudera.org/r/876/#comment4721>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
<http://review.cloudera.org/r/876/#comment4722>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
<http://review.cloudera.org/r/876/#comment4723>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4724>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4725>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4726>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4727>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4728>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4729>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4730>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4731>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4732>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4733>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4734>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4735>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4736>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4739>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4737>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4740>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4741>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4742>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4743>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4745>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4744>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4746>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4747>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4748>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4749>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4750>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4751>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4752>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4753>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4754>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4755>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4756>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4757>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4758>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4759>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4761>
Inconsistent indentation with most of the rest of the file
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4760>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4762>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4763>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4764>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4765>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4766>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4768>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4767>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4770>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4771>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4773>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4772>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4774>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4775>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4776>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4777>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4778>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4779>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4780>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4781>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment4782>
Lock should be acquired before the try block
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4783>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4784>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4785>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4786>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4787>
Whitespace stuff and no comment for this parameter
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4788>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4789>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4790>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4791>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4792>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<http://review.cloudera.org/r/876/#comment4793>
src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
<http://review.cloudera.org/r/876/#comment4794>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
<http://review.cloudera.org/r/876/#comment4795>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
<http://review.cloudera.org/r/876/#comment4796>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
<http://review.cloudera.org/r/876/#comment4797>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
<http://review.cloudera.org/r/876/#comment4798>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
<http://review.cloudera.org/r/876/#comment4799>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
<http://review.cloudera.org/r/876/#comment4800>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
<http://review.cloudera.org/r/876/#comment4801>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
<http://review.cloudera.org/r/876/#comment4802>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
<http://review.cloudera.org/r/876/#comment4803>
Whitespace stuff
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
<http://review.cloudera.org/r/876/#comment4805>
Unnecessary, comments are automatically inherited
src/main/resources/hbase-default.xml
<http://review.cloudera.org/r/876/#comment4806>
Whitespace stuff
src/main/resources/hbase-default.xml
<http://review.cloudera.org/r/876/#comment4807>
Whitespace stuff
src/main/resources/hbase-default.xml
<http://review.cloudera.org/r/876/#comment4808>
Whitespace stuff
src/main/resources/hbase-default.xml
<http://review.cloudera.org/r/876/#comment4810>
Whitespace stuff
src/main/resources/hbase-default.xml
<http://review.cloudera.org/r/876/#comment4811>
Whitespace stuff
- Lars
On 2010-10-02 23:53:05, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-02 23:53:05)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> conf/hbase-policy.xml PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 365e4b9
> src/main/java/org/apache/hadoop/hbase/HServerAddress.java 3859968
> src/main/java/org/apache/hadoop/hbase/HServerInfo.java aaf4835
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0feb17c
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 3a429c0
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 19dbf2b
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bacf8b1
> src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 71a0447
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 0a4fbce
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java e5b1a30
> src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 63a6956
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 377e6b1
> src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java dc55407
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Andrew Purtell <ap...@apache.org>.
> On 2010-10-25 06:49:15, Himanshu Vashishtha wrote:
> > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java, line 343
> > <http://review.cloudera.org/r/876/diff/7/?file=14190#file14190line343>
> >
> > What is its purpose here? I couldn't see it being used as of now. Is it for some future functionality.
The access controller coprocessor (HBASE-3025) needs a CatalogTracker. Other future functionality is also considered.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1646
-----------------------------------------------------------
On 2010-10-05 18:22:16, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-05 18:22:16)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Himanshu Vashishtha <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1646
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment5525>
I see. A coprocessor can invoke getTable(Tablename) any number of times, with arbitrary table names. Though its instance is kept by CoprocessorHost, which is HRegion specific (making it one HTable specific), it is not limited in access to other tables per se.Thanks.
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment5526>
What is its purpose here? I couldn't see it being used as of now. Is it for some future functionality.
- Himanshu
On 2010-10-05 18:22:16, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-05 18:22:16)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Andrew Purtell <ap...@apache.org>.
> On 2010-11-15 16:51:18, stack wrote:
> > +1 on commit to TRUNK. I think all below can be cleaned up on commit (Andrew, you going to commit?)
Stack, Yes I plan to commit the patches for HBASE-2001/HBASE-2002/HBASE-2321 onto trunk this week. The dynamic RPC and coprocessor framework changes are largely independent and will go in separately to make the change history in the commit log more informative. We will address your comments before doing so.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1930
-----------------------------------------------------------
On 2010-11-08 14:41:41, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-11-08 14:41:41)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * Endpoint: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> Endpoint:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through Endpoint and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can create a new method, specify passed parameters and return types for the method. And the new Endpoint methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 3235a9b
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 1748a4f
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 69dc916
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java ade3e21
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 1ac7671
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 30dd877
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f8d8af7
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 4f4828b
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java cf30cf9
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java 209c291
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java dd2955d
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ab183e5
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1309f93
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1bcde8c
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ace7997
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java 5f05079
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by st...@duboce.net.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1930
-----------------------------------------------------------
Ship it!
+1 on commit to TRUNK. I think all below can be cleaned up on commit (Andrew, you going to commit?)
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
<http://review.cloudera.org/r/876/#comment6139>
Check in here. Looks like tabs? review board reporting it as whitespace.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
<http://review.cloudera.org/r/876/#comment6140>
Usually in hbase code base there are spaces around operations; e.g. around '+'.
src/main/java/org/apache/hadoop/hbase/client/coprocessor/ExecResult.java
<http://review.cloudera.org/r/876/#comment6142>
Be careful. In hbase lines are 80 characters long normally. Fix on commit?
src/main/java/org/apache/hadoop/hbase/client/coprocessor/ExecResult.java
<http://review.cloudera.org/r/876/#comment6143>
I think its ok if these lines > 80 characters
src/main/java/org/apache/hadoop/hbase/client/coprocessor/package-info.java
<http://review.cloudera.org/r/876/#comment6144>
Excellent
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
<http://review.cloudera.org/r/876/#comment6146>
Lots of white space in here.
- stack
On 2010-11-08 14:41:41, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-11-08 14:41:41)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * Endpoint: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> Endpoint:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through Endpoint and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can create a new method, specify passed parameters and return types for the method. And the new Endpoint methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 3235a9b
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 1748a4f
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 69dc916
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java ade3e21
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 1ac7671
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 30dd877
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f8d8af7
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 4f4828b
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java cf30cf9
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java 209c291
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java dd2955d
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ab183e5
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1309f93
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1bcde8c
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ace7997
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java 5f05079
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Andrew Purtell <ap...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1961
-----------------------------------------------------------
Ship it!
Will commit after running unit tests and verifying all pass.
- Andrew
On 2010-11-19 14:39:18, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-11-19 14:39:18)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * Endpoint: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> Endpoint:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through Endpoint and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can create a new method, specify passed parameters and return types for the method. And the new Endpoint methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java d669de1
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 2ef6432
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1309f93
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1bcde8c
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ace7997
> src/main/resources/hbase-default.xml 630c041
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Mingjie Lai <mj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
(Updated 2010-11-19 14:39:18.378849)
Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
Changes
-------
Final patch, ready to be checked in:
- Truncated white space at line end
- Rebuilt the patch after HBase-2002 checked in.
Summary
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java
src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
* Endpoint: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
Endpoint:
Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
Through Endpoint and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can create a new method, specify passed parameters and return types for the method. And the new Endpoint methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java d669de1
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 2ef6432
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1309f93
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1bcde8c
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ace7997
src/main/resources/hbase-default.xml 630c041
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Mingjie Lai <mj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
(Updated 2010-11-08 14:41:41.575886)
Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
Changes
-------
Changes:
- Addressed Stack's comments.
- Renamed CommandTarget to Endpoint (still up to decide whether it's better or not)
- Improved package-info.
- Refined some of RegionObserver method signatures, i.e., preGet(), etc.
- Added hooks for HTable.increment(Increment).
Summary (updated)
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java
src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
* Endpoint: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
Endpoint:
Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
Through Endpoint and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can create a new method, specify passed parameters and return types for the method. And the new Endpoint methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 3235a9b
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 1748a4f
src/main/java/org/apache/hadoop/hbase/client/HTable.java 69dc916
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java ade3e21
src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 1ac7671
src/main/java/org/apache/hadoop/hbase/client/coprocessor/Batch.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/coprocessor/Exec.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/coprocessor/ExecResult.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 30dd877
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f8d8af7
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 4f4828b
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java cf30cf9
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 209c291
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java dd2955d
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ab183e5
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1309f93
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1bcde8c
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ace7997
src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java 5f05079
src/main/resources/hbase-default.xml 5fafe65
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Andrew Purtell <ap...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1436
-----------------------------------------------------------
Ship it!
- Andrew
On 2010-10-05 18:22:16, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-05 18:22:16)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by st...@duboce.net.
> On 2010-10-05 23:10:58, stack wrote:
> > src/main/java/org/apache/hadoop/hbase/client/Action.java, line 30
> > <http://review.cloudera.org/r/876/diff/7/?file=14158#file14158line30>
> >
> > I took a look at the package-info.html. Very nice doc. One thought though was that the batch methods do not seem to be instrumented. Are they? The bulk of inserts are done by multiput now.
> >
> > Maybe link to the wiki page when you say this in package-info.html '....implement role-based access control for HBase'
> >
> > Fix this 'These code will be triggerd with existing...'
> >
> > BaseRegionObserver as the name of the class that implements BOTH Coprocessor and RegionObserver with sensible defaults seems off... it'd make sense as the name of an implemenation of RegionObserver but not of both. Is there a better name to give it -- even BaseRegionObserverCoprocessor? Unless BaseObserver already implements Coprocessor?
> >
> > Should this also say that methods can be new also? '...i.e., you can specify new passed parameters and return types for a method. '
> >
> > CommandTarget is a strange name for an host of arbitrary user-designed methods. Can we come up w/ something more telling? Notions that come to mind are Substrate, Platform -- i.e. stuff you build up on.
> >
> > Minor.. fix '...the actually implemention class running...'
> >
> > Fix this '...How is the client side example of calling...'
> >
> > The example is missing a bit of code that would help along its illustration.... a few comments would help too.... but this is a minor criticism. Not important. I get the gist (Folks interested in CP need to start with this page -- it makes grokking the code the easier).
> >
> > This page would seem to indicate CPs can be chained. Am I reading that wrong? (See 'Load from configuration') Over in Gary review, he was saying on CP per region only.
> >
> >
> > Usually attribute names are upper-cased. Here we have 'Coprocessor$1' (that $1is intentional right?)
> >
> > This functionality, if its working, is amazing.
> >
> >
> >
>
> Mingjie Lai wrote:
> @stack:
> I didn't realize you posted a comment until last week, since your comments here didn't get pushed to jira, neither emails sent to dev@hbase.
>
> Thanks for your comments. I will address them very soon. But before that I'd like to finalize the name of ``CommandTarget'':
>
> You said, ``CommandTarget is a strange name for an host of arbitrary user-designed methods. Can we come up w/ something more telling? Notions that come to mind are Substrate, Platform -- i.e. stuff you build up on.''
>
> Some of us suggested to use ``Endpoint'' instead of CommandTarget. Do you like it better? (I'm not really good at naming stuff)
>
> After finalizing the name, I will make the changes to both source code and package-info. And post a patch here.
>
> Thanks,
> Mingjie
>
I'm not good at naming either.... Endpoint seems more 'generic', less loaded than 'CommandTarget'. If you fellas working with this stuff think that a better name then thats good by me.
- stack
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1438
-----------------------------------------------------------
On 2010-11-08 14:41:41, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-11-08 14:41:41)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * Endpoint: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> Endpoint:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through Endpoint and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can create a new method, specify passed parameters and return types for the method. And the new Endpoint methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 3235a9b
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 1748a4f
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 69dc916
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java ade3e21
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 1ac7671
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 30dd877
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f8d8af7
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 4f4828b
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java cf30cf9
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java 209c291
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java dd2955d
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ab183e5
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1309f93
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1bcde8c
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ace7997
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java 5f05079
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Mingjie Lai <mj...@gmail.com>.
> On 2010-10-05 23:10:58, stack wrote:
> > src/main/java/org/apache/hadoop/hbase/client/Action.java, line 30
> > <http://review.cloudera.org/r/876/diff/7/?file=14158#file14158line30>
> >
> > I took a look at the package-info.html. Very nice doc. One thought though was that the batch methods do not seem to be instrumented. Are they? The bulk of inserts are done by multiput now.
> >
> > Maybe link to the wiki page when you say this in package-info.html '....implement role-based access control for HBase'
> >
> > Fix this 'These code will be triggerd with existing...'
> >
> > BaseRegionObserver as the name of the class that implements BOTH Coprocessor and RegionObserver with sensible defaults seems off... it'd make sense as the name of an implemenation of RegionObserver but not of both. Is there a better name to give it -- even BaseRegionObserverCoprocessor? Unless BaseObserver already implements Coprocessor?
> >
> > Should this also say that methods can be new also? '...i.e., you can specify new passed parameters and return types for a method. '
> >
> > CommandTarget is a strange name for an host of arbitrary user-designed methods. Can we come up w/ something more telling? Notions that come to mind are Substrate, Platform -- i.e. stuff you build up on.
> >
> > Minor.. fix '...the actually implemention class running...'
> >
> > Fix this '...How is the client side example of calling...'
> >
> > The example is missing a bit of code that would help along its illustration.... a few comments would help too.... but this is a minor criticism. Not important. I get the gist (Folks interested in CP need to start with this page -- it makes grokking the code the easier).
> >
> > This page would seem to indicate CPs can be chained. Am I reading that wrong? (See 'Load from configuration') Over in Gary review, he was saying on CP per region only.
> >
> >
> > Usually attribute names are upper-cased. Here we have 'Coprocessor$1' (that $1is intentional right?)
> >
> > This functionality, if its working, is amazing.
> >
> >
> >
@stack:
I didn't realize you posted a comment until last week, since your comments here didn't get pushed to jira, neither emails sent to dev@hbase.
Thanks for your comments. I will address them very soon. But before that I'd like to finalize the name of ``CommandTarget'':
You said, ``CommandTarget is a strange name for an host of arbitrary user-designed methods. Can we come up w/ something more telling? Notions that come to mind are Substrate, Platform -- i.e. stuff you build up on.''
Some of us suggested to use ``Endpoint'' instead of CommandTarget. Do you like it better? (I'm not really good at naming stuff)
After finalizing the name, I will make the changes to both source code and package-info. And post a patch here.
Thanks,
Mingjie
- Mingjie
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1438
-----------------------------------------------------------
On 2010-10-05 18:22:16, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-05 18:22:16)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by st...@duboce.net.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1438
-----------------------------------------------------------
I went through about 2/3rds of the classes particular to hbase-2001 and made some comments. As stated in the below, this functionality is pretty amazing. IMO, it could do with a few more rounds of new-patch/review before ready for commit.
src/main/java/org/apache/hadoop/hbase/client/Action.java
<http://review.cloudera.org/r/876/#comment5026>
I took a look at the package-info.html. Very nice doc. One thought though was that the batch methods do not seem to be instrumented. Are they? The bulk of inserts are done by multiput now.
Maybe link to the wiki page when you say this in package-info.html '....implement role-based access control for HBase'
Fix this 'These code will be triggerd with existing...'
BaseRegionObserver as the name of the class that implements BOTH Coprocessor and RegionObserver with sensible defaults seems off... it'd make sense as the name of an implemenation of RegionObserver but not of both. Is there a better name to give it -- even BaseRegionObserverCoprocessor? Unless BaseObserver already implements Coprocessor?
Should this also say that methods can be new also? '...i.e., you can specify new passed parameters and return types for a method. '
CommandTarget is a strange name for an host of arbitrary user-designed methods. Can we come up w/ something more telling? Notions that come to mind are Substrate, Platform -- i.e. stuff you build up on.
Minor.. fix '...the actually implemention class running...'
Fix this '...How is the client side example of calling...'
The example is missing a bit of code that would help along its illustration.... a few comments would help too.... but this is a minor criticism. Not important. I get the gist (Folks interested in CP need to start with this page -- it makes grokking the code the easier).
This page would seem to indicate CPs can be chained. Am I reading that wrong? (See 'Load from configuration') Over in Gary review, he was saying on CP per region only.
Usually attribute names are upper-cased. Here we have 'Coprocessor$1' (that $1is intentional right?)
This functionality, if its working, is amazing.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
<http://review.cloudera.org/r/876/#comment5036>
Fix this line.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
<http://review.cloudera.org/r/876/#comment5037>
What is this result if its a pre call? Whats in it? Is it empty? Would a pre call add extras to the Result or something?
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment5027>
This would seem to disagree with the package-info doc. Is that intentional? Doc is stale?
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment5028>
Is this per region? Multiple CPs per region?
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
<http://review.cloudera.org/r/876/#comment5029>
In previous review I think I asked where's the chance to reject or amend each key that passes through the compaction? (This was probably answered already -- need to check back in reviews)
What if I wanted to stop the compaction? Thats too intrusive? Same for split and flush?
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment5030>
Missing license
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
<http://review.cloudera.org/r/876/#comment5031>
Good.
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment5032>
How does this work if its a pre call, how can it return a result?
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment5033>
Should this method be passed the result of actual call to getClosestRowBefore?
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment5034>
ditto
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
<http://review.cloudera.org/r/876/#comment5035>
Whats missing from this interface?
Should this be done via reflection somehow given as folks will probably be adding and removing methods over time? It'll be a bit of a pain keeping the two in sync otherwise?
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment5038>
Should say something here no? I could imagine that fellas could be pulling their hair out trying to figure why their CP didn't load yet nothing printed in log.
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment5039>
So any CP can amend Result. I've yet to look at the amended HRegion? It'll adorn the passed in Result rather than create one of its own?
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment5040>
Its intentional that this went away?
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment5041>
Oh, I see whats going on... fair enough.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment5042>
CoprocessorHost is a good name for what it does.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment5043>
If the pre call put something into Result, this will overwrite or lose it?
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment5044>
ok
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment5047>
NOt important but I'd swap these.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment5048>
Ok, looks like pre can adorn get results... its being handled properly here.
- stack
On 2010-10-05 18:22:16, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-05 18:22:16)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Andrew Purtell <ap...@apache.org>.
> On 2010-10-24 18:44:32, Himanshu Vashishtha wrote:
> > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java, line 250
> > <http://review.cloudera.org/r/876/diff/7/?file=14190#file14190line250>
> >
> > Is this list is for scalability. I am not able to visualise a scenario where a Cp has reference to more than one table. Given a region has only one CpHost instance, and though it can a multi CpImpls in it, but each of them will have its own Environment instance (thus HTable instance). (I might be missing sth though :) )
No that is not correct. At this time a coprocessor can via the environment get HTable instances to arbitrary tables. We keep track of them so if a coprocessor is forcibly terminated we can clean up.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1639
-----------------------------------------------------------
On 2010-10-05 18:22:16, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-05 18:22:16)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Himanshu Vashishtha <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1639
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/876/#comment5505>
Is this list is for scalability. I am not able to visualise a scenario where a Cp has reference to more than one table. Given a region has only one CpHost instance, and though it can a multi CpImpls in it, but each of them will have its own Environment instance (thus HTable instance). (I might be missing sth though :) )
- Himanshu
On 2010-10-05 18:22:16, Mingjie Lai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
>
> (Updated 2010-10-05 18:22:16)
>
>
> Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
>
>
> Summary
> -------
>
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
>
> Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
>
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
>
>
> ==========================
>
> (Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
>
>
> Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
>
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
>
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
>
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
>
> Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
>
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
>
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
>
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
>
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
>
> (Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
>
>
> This addresses bug HBase-2001.
> http://issues.apache.org/jira/browse/HBase-2001
>
>
> Diffs
> -----
>
> src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
> src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
> src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
> src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
> src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
> src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
> src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
> src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
> src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
> src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
> src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
> src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
> src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
> src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
> src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
> src/main/resources/hbase-default.xml 5fafe65
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
> src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/876/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mingjie
>
>
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Mingjie Lai <mj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
(Updated 2010-10-05 18:22:16.199544)
Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
Changes
-------
Discarded other 2 useless files which were added by mistake.
Summary
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
src/main/resources/hbase-default.xml
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
* CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
CommandTarget:
Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
src/main/resources/hbase-default.xml 5fafe65
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Mingjie Lai <mj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
(Updated 2010-10-05 01:50:11.904139)
Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
Changes
-------
I added some unused file by mistake last time in rev4 and rev5. Took them out this time in rev6. And truncated line end white spaces.
Summary
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
src/main/resources/hbase-default.xml
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
* CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
CommandTarget:
Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 365e4b9
src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 866ba92
src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 3a429c0
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java df7412c
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5f829e4
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
src/main/resources/hbase-default.xml 5fafe65
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Mingjie Lai <mj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
(Updated 2010-10-02 23:53:05.329075)
Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
Changes
-------
Patch recreated by the latest [HBASE-2321] [HBASE-2002] patch:
https://review.cloudera.org/r/816/diff/8/
Summary
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
src/main/resources/hbase-default.xml
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
* CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
CommandTarget:
Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs (updated)
-----
conf/hbase-policy.xml PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 365e4b9
src/main/java/org/apache/hadoop/hbase/HServerAddress.java 3859968
src/main/java/org/apache/hadoop/hbase/HServerInfo.java aaf4835
src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0feb17c
src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 74593bf
src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 3a429c0
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 19dbf2b
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bacf8b1
src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 71a0447
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 0a4fbce
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 89f499a
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java d4166cf
src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
src/main/resources/hbase-default.xml 5fafe65
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java e5b1a30
src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 63a6956
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 377e6b1
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java dc55407
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie
Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Posted by Mingjie Lai <mj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
(Updated 2010-10-01 15:13:50.901219)
Review request for hbase, stack, Andrew Purtell, and Jonathan Gray.
Changes
-------
Updated package-info.java to reflect latest interface changes for RegionObserver: added hooks for CheckAndPut(), CheckAndDelete().
Also modified description.
Summary (updated)
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.
Eventually the patch here should be committed after 2001 and 2321. I will make another patch after they got checked in.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines are more than 7k. I turned back and forth, but still don't have a good idea to create the patch in order to reduce the review pain. However right now I'm putting the whole patch for all the 3 issues. Here the list of file which are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
src/main/resources/hbase-default.xml
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the package-info.java in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions contain references to the coprocessor implementation classes associated with them. Coprocessor classes will be loaded either from local jars on the region server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side, such as table get/put/scan/delete, etc.
* CommandTarget: provides on demand triggers for any arbitrary function executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and mediate client actions on the region.
CommandTarget:
Coprocessor and RegionObserver provide certain hooks for injecting user code running at each region. These code will be triggerd with existing HTable and HBaseAdmin operations at the certain hook points.
Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated between client and region server, i.e., you can specify new passed parameters and return types for a method. And the new CommandTarget methods can be triggered by calling client side dynamic RPC functions -- HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration, or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for opened regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs (updated)
-----
conf/hbase-policy.xml PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 365e4b9
src/main/java/org/apache/hadoop/hbase/HServerAddress.java 3859968
src/main/java/org/apache/hadoop/hbase/HServerInfo.java aaf4835
src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
src/main/java/org/apache/hadoop/hbase/client/Batch.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java fbdec0b
src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 3a429c0
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 19dbf2b
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bacf8b1
src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 71a0447
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java ee5dd8f
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb1e834
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 0a4fbce
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e80ece8
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 4ba63d8
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 9aaf7c3
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 01475d5
src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
src/main/resources/hbase-default.xml 5fafe65
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java e5b1a30
src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 63a6956
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 377e6b1
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java dc55407
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie