Re: [DISCUSS] More Shading

[Stack <st...@duboce.net>]

On Thu, Jul 6, 2017 at 11:17 PM, Stack <st...@duboce.net> wrote:

> On Thu, Jul 6, 2017 at 11:52 AM, Stack <st...@duboce.net> wrote:
>
>> FYI:
>>
>> hbase-thirdparty has had its first release. Yesterday I
>> committed HBASE-17056 "Remove checked in PB generated files" which apart
>> from purging all checked-in generated files (30MB), it moves our hbase core
>> (master and branch-2) to start using the thirdparty jar.
>>
>> Things might be interesting over next few days so shout if you run into
>> issues.
>>
>>
> FYI, I reverted HBASE-17056 for the moment. Build seems unstable. OOMEs
> and other interesting issues along w/ some awkwardness w/ dependencies. Let
> me spend some more time on it. Will try again later.
>
>

Just a heads-up. With Chia-Ping Tsai and Guanghao Zhang's help, I was able
to figure a few odd issues in previous push. I just pushed again. Kick me
if you see OOMEs or weirdness.... I'll be keeping an eye out myself too.

Thanks for your patience.

St.Ack




> St.Ack
>
>
>
>> One issue is the need for mvn install where before mvn compile might have
>> been enough (see below for example of the issue you'd see now if you did
>> mvn clean compile only). We need the mvn install because we need to shade
>> the generated files so they use the relocated protobuf; shade happens after
>> we've made a module jar.
>>
>> IDEs will complain too if they pick up generated src from target dirs
>> since they'll be unsatisfied references to protobuf3 objects -- unless you
>> point at the shaded jar. Let me see if I can do something about the latter.
>>
>> Let me know if HBASE-17056 is too much for devs to bear. Can always
>> revert (though nice having the protobuf generation in-line w/ the build).
>> It is sort of a side-benefit on the general shading project so could back
>> it out and still have the shading of netty, guava, protobuf, etc.
>>
>> Thanks,
>>
>> St.Ack
>>
>> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
>> (default-compile) on project hbase-procedure: Compilation failure:
>> Compilation failure:
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[105,11]
>> cannot access com.google.protobuf.GeneratedMessageV3
>> [ERROR] class file for com.google.protobuf.GeneratedMessageV3 not found
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[129,7]
>> cannot access com.google.protobuf.GeneratedMessageV3.Builder
>> [ERROR] class file for com.google.protobuf.GeneratedMessageV3$Builder
>> not found
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[133,22]
>> cannot find symbol
>> [ERROR] symbol:   method writeDelimitedTo(org.apache.ha
>> doop.fs.FSDataOutputStream)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureStoreTracker
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[217,20]
>> cannot find symbol
>> [ERROR] symbol:   method writeDelimitedTo(org.apache.ha
>> doop.hbase.procedure2.util.ByteSlot)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureWALEntry
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[240,20]
>> cannot find symbol
>> [ERROR] symbol:   method writeDelimitedTo(org.apache.ha
>> doop.hbase.procedure2.util.ByteSlot)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureWALEntry
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[254,20]
>> cannot find symbol
>> [ERROR] symbol:   method writeDelimitedTo(org.apache.ha
>> doop.hbase.procedure2.util.ByteSlot)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureWALEntry
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/RemoteProcedureException.java:[98,30] cannot
>> find symbol
>> [ERROR] symbol:   method toByteArray()
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ErrorHandlingProtos.ForeignExceptionMessage
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/StateMachineProcedure.java:[267,17] cannot
>> find symbol
>> [ERROR] symbol:   method writeDelimitedTo(java.io.OutputStream)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.StateMachineProcedureData
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/ProcedureUtil.java:[130,56] incompatible
>> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
>> cannot be converted to com.google.protobuf.ByteString
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/ProcedureUtil.java:[137,54] incompatible
>> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
>> cannot be converted to com.google.protobuf.ByteString
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/ProcedureUtil.java:[237,56] incompatible
>> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
>> cannot be converted to com.google.protobuf.ByteString
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/SequentialProcedure.java:[75,17] cannot find
>> symbol
>> [ERROR] symbol:   method writeDelimitedTo(java.io.OutputStream)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.SequentialProcedureData
>> [ERROR] -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1] http://cwiki.apache.org/conflu
>> ence/display/MAVEN/MojoFailureException
>> [ERROR]
>> [ERROR] After correcting the problems, you can resume the build with the
>> command
>> [ERROR]   mvn <goals> -rf :hbase-procedure
>>
>> On Fri, Jun 30, 2017 at 3:09 PM, Stack <st...@duboce.net> wrote:
>>
>>> I just started a VOTE on hbase-thirdparty and the first RC made from it.
>>> Thanks,
>>> St.Ack
>>>
>>> On Tue, Jun 27, 2017 at 3:02 PM, Stack <st...@duboce.net> wrote:
>>>
>>>> Bit of an update.
>>>>
>>>> I'd suggest we go ahead w/ the hbase-thirdparty project [2]. It took a
>>>> while but in its current form -- a few poms that package a few jars [1]--
>>>> it at least enables the below:
>>>>
>>>> + Allows us to skip checking in protobuf generated files (25MB!); they
>>>> can be generated inline w/ the build because the hackery patching protobuf
>>>> has been moved out to hbase-thirdparty. There is a patch up on HBASE-17056.
>>>> + Update our guava from 12.0 to 22.0 w/o clashing w/ the guava of
>>>> others. There is a patch at HBASE-17908. It is taking a bit of wrangling
>>>> getting it to land because I pared back transitive includes from hadoop and
>>>> it takes a while to work through the failures.
>>>>
>>>> Other benefits are the protobuf-util lib is on the classpath now -- its
>>>> in hbase-thirdparty relocated; depends on pb and guava -- so we have
>>>> facility to goat "HBASE-18106 Redo ProcedureInfo and LockInfo" and shading
>>>> netty is almost done so we can do with netty as we wilt independent of
>>>> hadoop and downstreamers (the hard part -- relocation of the .so -- should
>>>> be done).
>>>>
>>>> Let me figure how to run a vote for a couple of poms.....
>>>>
>>>> St.Ack
>>>>
>>>> 1. https://repository.apache.org/content/groups/snapshots/or
>>>> g/apache/hbase/thirdparty/ (see hbase-shaded-thirdparty and
>>>> hbase-shaded-protobuf)
>>>> 2. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
>>>>
>>>>
>>>> On Tue, Jun 20, 2017 at 11:04 AM, Josh Elser <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> On 6/20/17 1:28 AM, Stack wrote:
>>>>>
>>>>>> On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser<el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> ...
>>>>>>>
>>>>>>> I think pushing this part forward with some code is the next logical
>>>>>>> step.
>>>>>>> Seems to be consensus about taking our known internal dependencies
>>>>>>> and
>>>>>>> performing this shade magic.
>>>>>>>
>>>>>>>
>>>>>>> I opened HBASE-18240 "Add hbase-auxillary, a project with hbase
>>>>>> utility
>>>>>> including an hbase-shaded-thirdparty module with guava, netty, etc."
>>>>>>
>>>>>> It has a tarball attached that bundles the outline of an
>>>>>> hbase-auxillary
>>>>>> project (groupId:org.apache.hbase.auxillary). This project is
>>>>>> intended to
>>>>>> be standalone, in its own repository, publishing its own artifacts
>>>>>> under
>>>>>> the aegis of this project's PMC.
>>>>>>
>>>>>> It includes the first instance of an auxillary utility, a module named
>>>>>> hbase-thirdparty-shaded (artifactId:hbase-thirdparty-shaded). Herein
>>>>>> we'll
>>>>>> pull down 3rd party libs and republish at an offset; e.g.
>>>>>> com.google.common.* from guava will be at
>>>>>> org.apache.hbase.thirdparty.shaded.com.google.common.*. Currently it
>>>>>> builds
>>>>>> a jar that includes a relocated guava 22.0.
>>>>>>
>>>>>> I then messed around making hbase-common use it (You have to build the
>>>>>> hbase-auxillary into your local repo). I put up a patch on the issue.
>>>>>> Mostly its mass find-and-replace w/ some clean up of transitive
>>>>>> includes of
>>>>>> guava from hadoop-common and some small fixup of methods renamed
>>>>>> between
>>>>>> guava 12.0 and 22.0.
>>>>>>
>>>>>> Unless objection, I was going to press on. Sean offered to help set
>>>>>> up new
>>>>>> repo. We can always undo and delete it if this project fails.
>>>>>>
>>>>>> When done, the hope is we are on a modern version of guava and our
>>>>>> netty
>>>>>> and protobuf 3 will be be relocated, 'hidden' from downstream (and
>>>>>> won't
>>>>>> clash w/ upstream). I hope to also purge the pre-build we have in our
>>>>>> modules that do protobuf moving this hackery out and under
>>>>>> hbase-thirdparty-shaded.
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>
>>>>> Kudos on the JFDI approach :). I think having something concrete to
>>>>> show is the best way to judge success of it.
>>>>>
>>>>> Will keep an eye on HBASE-18240.
>>>>>
>>>>>
>>>>
>>>
>>
>