You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2016/09/05 12:48:20 UTC
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints
and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15464927#comment-15464927 ]
stack commented on HBASE-16264:
-------------------------------
I updated the attached doc. with where I am at on Approach #3. I repeat status below. I've uncovered some new kinks while on this tack; i.e. some CPEPs are core and Query and Mutation make use of CPEP AccessControl protos. Next up (Approach #4) is a disruptive renaming of proto files used internally to remove our clash on proto names.
h2. Tactic #3
Same as Tactic #1 only move the new hbase-endpoint module with all CPEPs in it AFTER hbase-server (and hbase-client) so all CPEPs bundled in with hbase are in the new hbase-endpoint module. CPEPs will then load their versions of non-shaded ClientProtos, etc., rather than the systems' shaded versions. Should work since CPEPs have own classloader so if issues, we can tinker (Root issue is that there are two versions of proto files; versions where protobuf is shaded used internally everywhere, and then the versions of protos that are not shaded so CPEPs can keep working -- we can’t change Class naming/layout if we want CPEPs to keep working).
This approach #3 fails for a reason similar to why approach #1 fails (only it is the inverse); the unit tests in hbase-endpoint are picking up the local non-shaded versions of ClientProtos when Master/RegionServer are starting up the minihbasecluster. Tests are failing to run with NoSuchMethodError.
As it turns out, three of the bundled CPEPs are depended upon by internals so we cannot move them out of hbase-client/hbase-server. The three modules are Authentication (used by rpc), MulitRowMutation (used to update meta), and AccessControl. I worked on refactoring these three CPEPs that internals depend on, redoing as base functionality so they are no longer CPEPs. Would mean that changing ACLs would require you to use an hbase-2.0.0 client. Multi was easy enough to do as an added internal PB Interface addition. Auth seems fine since it is not actually a CPEP though cast as one. TO FIX. AC is an issue since its PBs are referenced internally by Query and Mutation...TODO.
I was able to build the shaded hbase-protocol and then have hbase-client and hbase-server depend on it. Tests would run if I added an explicit dependency upon hbase-protocol using the ‘system’ label with a path to the built hbase-protocol module jar (so we picked up shaded artifacts at test time since they are only available in the hbase-protocol jar, post-package). TODO: Make this better.
356 <artifactId>hbase-protocol</artifactId>
357 <scope>system</scope>
358 <systemPath>/Users/stack/.m2/repository/org/apache/hbase/hbase-protocol/2.0.0-SNAPSHOT/hbase-protocol-2.0.0-SNAPSHOT.jar</systemPath>
359 </dependency>
Hbase-endpoints picks up a lot of tests moved here from the hbase-server. Problem is that when the minihbase cluster starts up, it goes to register supported Interfaces and picks up the local ClientProtos rather than the shaded ones from hbase-protocol jar referenced against using ‘system’.
420 Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService.newReflectiveBlockingService(Lorg/apache/hadoop/hbase/protobuf/generated/ClientProtos$ClientService$BlockingInterface;)Lorg/ apache/hadoop/hbase/shaded/com/google/protobuf/BlockingService;
421 at org.apache.hadoop.hbase.regionserver.RSRpcServices.getServices(RSRpcServices.java:1277)
422 at org.apache.hadoop.hbase.master.MasterRpcServices.getServices(MasterRpcServices.java:214)
423 at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:1065)
424 at org.apache.hadoop.hbase.master.MasterRpcServices.<init>(MasterRpcServices.java:132)
425 at org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:567)
426 at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:557)
427 at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:408)
428 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
429 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
430 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
431 at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
432 at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
433 ... 37 more
TODO: Tests of old school endpoints to ensure they still work.
PROBLEM: IDE Need to remove hbase-protocol and then add dependency on built hbase-protocol.
PROBLEM: @Ignore @Test // FIX THIS? BROKE AS PART OF SHADING PROJECT!!
public void testCoDelScheduling() throws Exception {
PROBLEM: TestDefaultCompactSelection.testCompactionRatio:94->TestCompactionPolicy.compactEquals:185->TestCompactionPolicy.compactEquals:204 expected:<[[50, 25, 12, 12]]> but was:<[[]]>
@Ignore @Test // TODO: FIXEEEE
public void testCompactionRatio() throws IOException {
> Figure how to deal with endpoints and shaded pb
> -----------------------------------------------
>
> Key: HBASE-16264
> URL: https://issues.apache.org/jira/browse/HBASE-16264
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, Protobufs
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, HBASE-16264.master.006.patch, HBASE-16264.master.007.patch
>
>
> Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. Would be sweet if could make it so all just worked. At worst, come up w/ a prescription for how to migrate existing CPs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)