You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2017/07/06 18:52:19 UTC
Re: [DISCUSS] More Shading
FYI:
hbase-thirdparty has had its first release. Yesterday I
committed HBASE-17056 "Remove checked in PB generated files" which apart
from purging all checked-in generated files (30MB), it moves our hbase core
(master and branch-2) to start using the thirdparty jar.
Things might be interesting over next few days so shout if you run into
issues.
One issue is the need for mvn install where before mvn compile might have
been enough (see below for example of the issue you'd see now if you did
mvn clean compile only). We need the mvn install because we need to shade
the generated files so they use the relocated protobuf; shade happens after
we've made a module jar.
IDEs will complain too if they pick up generated src from target dirs since
they'll be unsatisfied references to protobuf3 objects -- unless you point
at the shaded jar. Let me see if I can do something about the latter.
Let me know if HBASE-17056 is too much for devs to bear. Can always revert
(though nice having the protobuf generation in-line w/ the build). It is
sort of a side-benefit on the general shading project so could back it out
and still have the shading of netty, guava, protobuf, etc.
Thanks,
St.Ack
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
(default-compile) on project hbase-procedure: Compilation failure:
Compilation failure:
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[105,11]
cannot access com.google.protobuf.GeneratedMessageV3
[ERROR] class file for com.google.protobuf.GeneratedMessageV3 not found
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[129,7]
cannot access com.google.protobuf.GeneratedMessageV3.Builder
[ERROR] class file for com.google.protobuf.GeneratedMessageV3$Builder not
found
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[133,22]
cannot find symbol
[ERROR] symbol: method
writeDelimitedTo(org.apache.hadoop.fs.FSDataOutputStream)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureStoreTracker
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[217,20]
cannot find symbol
[ERROR] symbol: method
writeDelimitedTo(org.apache.hadoop.hbase.procedure2.util.ByteSlot)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureWALEntry
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[240,20]
cannot find symbol
[ERROR] symbol: method
writeDelimitedTo(org.apache.hadoop.hbase.procedure2.util.ByteSlot)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureWALEntry
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[254,20]
cannot find symbol
[ERROR] symbol: method
writeDelimitedTo(org.apache.hadoop.hbase.procedure2.util.ByteSlot)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureWALEntry
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureException.java:[98,30]
cannot find symbol
[ERROR] symbol: method toByteArray()
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ErrorHandlingProtos.ForeignExceptionMessage
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/StateMachineProcedure.java:[267,17]
cannot find symbol
[ERROR] symbol: method writeDelimitedTo(java.io.OutputStream)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.StateMachineProcedureData
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureUtil.java:[130,56]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString cannot be
converted to com.google.protobuf.ByteString
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureUtil.java:[137,54]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString cannot be
converted to com.google.protobuf.ByteString
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureUtil.java:[237,56]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString cannot be
converted to com.google.protobuf.ByteString
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/SequentialProcedure.java:[75,17]
cannot find symbol
[ERROR] symbol: method writeDelimitedTo(java.io.OutputStream)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.SequentialProcedureData
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn <goals> -rf :hbase-procedure
On Fri, Jun 30, 2017 at 3:09 PM, Stack <st...@duboce.net> wrote:
> I just started a VOTE on hbase-thirdparty and the first RC made from it.
> Thanks,
> St.Ack
>
> On Tue, Jun 27, 2017 at 3:02 PM, Stack <st...@duboce.net> wrote:
>
>> Bit of an update.
>>
>> I'd suggest we go ahead w/ the hbase-thirdparty project [2]. It took a
>> while but in its current form -- a few poms that package a few jars [1]--
>> it at least enables the below:
>>
>> + Allows us to skip checking in protobuf generated files (25MB!); they
>> can be generated inline w/ the build because the hackery patching protobuf
>> has been moved out to hbase-thirdparty. There is a patch up on HBASE-17056.
>> + Update our guava from 12.0 to 22.0 w/o clashing w/ the guava of others.
>> There is a patch at HBASE-17908. It is taking a bit of wrangling getting it
>> to land because I pared back transitive includes from hadoop and it takes a
>> while to work through the failures.
>>
>> Other benefits are the protobuf-util lib is on the classpath now -- its
>> in hbase-thirdparty relocated; depends on pb and guava -- so we have
>> facility to goat "HBASE-18106 Redo ProcedureInfo and LockInfo" and shading
>> netty is almost done so we can do with netty as we wilt independent of
>> hadoop and downstreamers (the hard part -- relocation of the .so -- should
>> be done).
>>
>> Let me figure how to run a vote for a couple of poms.....
>>
>> St.Ack
>>
>> 1. https://repository.apache.org/content/groups/snapshots/or
>> g/apache/hbase/thirdparty/ (see hbase-shaded-thirdparty and
>> hbase-shaded-protobuf)
>> 2. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
>>
>>
>> On Tue, Jun 20, 2017 at 11:04 AM, Josh Elser <jo...@gmail.com>
>> wrote:
>>
>>> On 6/20/17 1:28 AM, Stack wrote:
>>>
>>>> On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser<el...@apache.org> wrote:
>>>>
>>>> ...
>>>>>
>>>>> I think pushing this part forward with some code is the next logical
>>>>> step.
>>>>> Seems to be consensus about taking our known internal dependencies and
>>>>> performing this shade magic.
>>>>>
>>>>>
>>>>> I opened HBASE-18240 "Add hbase-auxillary, a project with hbase utility
>>>> including an hbase-shaded-thirdparty module with guava, netty, etc."
>>>>
>>>> It has a tarball attached that bundles the outline of an hbase-auxillary
>>>> project (groupId:org.apache.hbase.auxillary). This project is intended
>>>> to
>>>> be standalone, in its own repository, publishing its own artifacts under
>>>> the aegis of this project's PMC.
>>>>
>>>> It includes the first instance of an auxillary utility, a module named
>>>> hbase-thirdparty-shaded (artifactId:hbase-thirdparty-shaded). Herein
>>>> we'll
>>>> pull down 3rd party libs and republish at an offset; e.g.
>>>> com.google.common.* from guava will be at
>>>> org.apache.hbase.thirdparty.shaded.com.google.common.*. Currently it
>>>> builds
>>>> a jar that includes a relocated guava 22.0.
>>>>
>>>> I then messed around making hbase-common use it (You have to build the
>>>> hbase-auxillary into your local repo). I put up a patch on the issue.
>>>> Mostly its mass find-and-replace w/ some clean up of transitive
>>>> includes of
>>>> guava from hadoop-common and some small fixup of methods renamed between
>>>> guava 12.0 and 22.0.
>>>>
>>>> Unless objection, I was going to press on. Sean offered to help set up
>>>> new
>>>> repo. We can always undo and delete it if this project fails.
>>>>
>>>> When done, the hope is we are on a modern version of guava and our netty
>>>> and protobuf 3 will be be relocated, 'hidden' from downstream (and won't
>>>> clash w/ upstream). I hope to also purge the pre-build we have in our
>>>> modules that do protobuf moving this hackery out and under
>>>> hbase-thirdparty-shaded.
>>>>
>>>> St.Ack
>>>>
>>>
>>> Kudos on the JFDI approach :). I think having something concrete to show
>>> is the best way to judge success of it.
>>>
>>> Will keep an eye on HBASE-18240.
>>>
>>>
>>
>
Re: [DISCUSS] More Shading
Posted by Stack <st...@duboce.net>.
On Thu, Jul 6, 2017 at 11:17 PM, Stack <st...@duboce.net> wrote:
> On Thu, Jul 6, 2017 at 11:52 AM, Stack <st...@duboce.net> wrote:
>
>> FYI:
>>
>> hbase-thirdparty has had its first release. Yesterday I
>> committed HBASE-17056 "Remove checked in PB generated files" which apart
>> from purging all checked-in generated files (30MB), it moves our hbase core
>> (master and branch-2) to start using the thirdparty jar.
>>
>> Things might be interesting over next few days so shout if you run into
>> issues.
>>
>>
> FYI, I reverted HBASE-17056 for the moment. Build seems unstable. OOMEs
> and other interesting issues along w/ some awkwardness w/ dependencies. Let
> me spend some more time on it. Will try again later.
>
>
Just a heads-up. With Chia-Ping Tsai and Guanghao Zhang's help, I was able
to figure a few odd issues in previous push. I just pushed again. Kick me
if you see OOMEs or weirdness.... I'll be keeping an eye out myself too.
Thanks for your patience.
St.Ack
> St.Ack
>
>
>
>> One issue is the need for mvn install where before mvn compile might have
>> been enough (see below for example of the issue you'd see now if you did
>> mvn clean compile only). We need the mvn install because we need to shade
>> the generated files so they use the relocated protobuf; shade happens after
>> we've made a module jar.
>>
>> IDEs will complain too if they pick up generated src from target dirs
>> since they'll be unsatisfied references to protobuf3 objects -- unless you
>> point at the shaded jar. Let me see if I can do something about the latter.
>>
>> Let me know if HBASE-17056 is too much for devs to bear. Can always
>> revert (though nice having the protobuf generation in-line w/ the build).
>> It is sort of a side-benefit on the general shading project so could back
>> it out and still have the shading of netty, guava, protobuf, etc.
>>
>> Thanks,
>>
>> St.Ack
>>
>> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
>> (default-compile) on project hbase-procedure: Compilation failure:
>> Compilation failure:
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[105,11]
>> cannot access com.google.protobuf.GeneratedMessageV3
>> [ERROR] class file for com.google.protobuf.GeneratedMessageV3 not found
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[129,7]
>> cannot access com.google.protobuf.GeneratedMessageV3.Builder
>> [ERROR] class file for com.google.protobuf.GeneratedMessageV3$Builder
>> not found
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[133,22]
>> cannot find symbol
>> [ERROR] symbol: method writeDelimitedTo(org.apache.ha
>> doop.fs.FSDataOutputStream)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureStoreTracker
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[217,20]
>> cannot find symbol
>> [ERROR] symbol: method writeDelimitedTo(org.apache.ha
>> doop.hbase.procedure2.util.ByteSlot)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureWALEntry
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[240,20]
>> cannot find symbol
>> [ERROR] symbol: method writeDelimitedTo(org.apache.ha
>> doop.hbase.procedure2.util.ByteSlot)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureWALEntry
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[254,20]
>> cannot find symbol
>> [ERROR] symbol: method writeDelimitedTo(org.apache.ha
>> doop.hbase.procedure2.util.ByteSlot)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.ProcedureWALEntry
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/RemoteProcedureException.java:[98,30] cannot
>> find symbol
>> [ERROR] symbol: method toByteArray()
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ErrorHandlingProtos.ForeignExceptionMessage
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/StateMachineProcedure.java:[267,17] cannot
>> find symbol
>> [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.StateMachineProcedureData
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/ProcedureUtil.java:[130,56] incompatible
>> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
>> cannot be converted to com.google.protobuf.ByteString
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/ProcedureUtil.java:[137,54] incompatible
>> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
>> cannot be converted to com.google.protobuf.ByteString
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/ProcedureUtil.java:[237,56] incompatible
>> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
>> cannot be converted to com.google.protobuf.ByteString
>> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac
>> he/hadoop/hbase/procedure2/SequentialProcedure.java:[75,17] cannot find
>> symbol
>> [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream)
>> [ERROR] location: class org.apache.hadoop.hbase.shaded
>> .protobuf.generated.ProcedureProtos.SequentialProcedureData
>> [ERROR] -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1] http://cwiki.apache.org/conflu
>> ence/display/MAVEN/MojoFailureException
>> [ERROR]
>> [ERROR] After correcting the problems, you can resume the build with the
>> command
>> [ERROR] mvn <goals> -rf :hbase-procedure
>>
>> On Fri, Jun 30, 2017 at 3:09 PM, Stack <st...@duboce.net> wrote:
>>
>>> I just started a VOTE on hbase-thirdparty and the first RC made from it.
>>> Thanks,
>>> St.Ack
>>>
>>> On Tue, Jun 27, 2017 at 3:02 PM, Stack <st...@duboce.net> wrote:
>>>
>>>> Bit of an update.
>>>>
>>>> I'd suggest we go ahead w/ the hbase-thirdparty project [2]. It took a
>>>> while but in its current form -- a few poms that package a few jars [1]--
>>>> it at least enables the below:
>>>>
>>>> + Allows us to skip checking in protobuf generated files (25MB!); they
>>>> can be generated inline w/ the build because the hackery patching protobuf
>>>> has been moved out to hbase-thirdparty. There is a patch up on HBASE-17056.
>>>> + Update our guava from 12.0 to 22.0 w/o clashing w/ the guava of
>>>> others. There is a patch at HBASE-17908. It is taking a bit of wrangling
>>>> getting it to land because I pared back transitive includes from hadoop and
>>>> it takes a while to work through the failures.
>>>>
>>>> Other benefits are the protobuf-util lib is on the classpath now -- its
>>>> in hbase-thirdparty relocated; depends on pb and guava -- so we have
>>>> facility to goat "HBASE-18106 Redo ProcedureInfo and LockInfo" and shading
>>>> netty is almost done so we can do with netty as we wilt independent of
>>>> hadoop and downstreamers (the hard part -- relocation of the .so -- should
>>>> be done).
>>>>
>>>> Let me figure how to run a vote for a couple of poms.....
>>>>
>>>> St.Ack
>>>>
>>>> 1. https://repository.apache.org/content/groups/snapshots/or
>>>> g/apache/hbase/thirdparty/ (see hbase-shaded-thirdparty and
>>>> hbase-shaded-protobuf)
>>>> 2. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
>>>>
>>>>
>>>> On Tue, Jun 20, 2017 at 11:04 AM, Josh Elser <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> On 6/20/17 1:28 AM, Stack wrote:
>>>>>
>>>>>> On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser<el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> ...
>>>>>>>
>>>>>>> I think pushing this part forward with some code is the next logical
>>>>>>> step.
>>>>>>> Seems to be consensus about taking our known internal dependencies
>>>>>>> and
>>>>>>> performing this shade magic.
>>>>>>>
>>>>>>>
>>>>>>> I opened HBASE-18240 "Add hbase-auxillary, a project with hbase
>>>>>> utility
>>>>>> including an hbase-shaded-thirdparty module with guava, netty, etc."
>>>>>>
>>>>>> It has a tarball attached that bundles the outline of an
>>>>>> hbase-auxillary
>>>>>> project (groupId:org.apache.hbase.auxillary). This project is
>>>>>> intended to
>>>>>> be standalone, in its own repository, publishing its own artifacts
>>>>>> under
>>>>>> the aegis of this project's PMC.
>>>>>>
>>>>>> It includes the first instance of an auxillary utility, a module named
>>>>>> hbase-thirdparty-shaded (artifactId:hbase-thirdparty-shaded). Herein
>>>>>> we'll
>>>>>> pull down 3rd party libs and republish at an offset; e.g.
>>>>>> com.google.common.* from guava will be at
>>>>>> org.apache.hbase.thirdparty.shaded.com.google.common.*. Currently it
>>>>>> builds
>>>>>> a jar that includes a relocated guava 22.0.
>>>>>>
>>>>>> I then messed around making hbase-common use it (You have to build the
>>>>>> hbase-auxillary into your local repo). I put up a patch on the issue.
>>>>>> Mostly its mass find-and-replace w/ some clean up of transitive
>>>>>> includes of
>>>>>> guava from hadoop-common and some small fixup of methods renamed
>>>>>> between
>>>>>> guava 12.0 and 22.0.
>>>>>>
>>>>>> Unless objection, I was going to press on. Sean offered to help set
>>>>>> up new
>>>>>> repo. We can always undo and delete it if this project fails.
>>>>>>
>>>>>> When done, the hope is we are on a modern version of guava and our
>>>>>> netty
>>>>>> and protobuf 3 will be be relocated, 'hidden' from downstream (and
>>>>>> won't
>>>>>> clash w/ upstream). I hope to also purge the pre-build we have in our
>>>>>> modules that do protobuf moving this hackery out and under
>>>>>> hbase-thirdparty-shaded.
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>
>>>>> Kudos on the JFDI approach :). I think having something concrete to
>>>>> show is the best way to judge success of it.
>>>>>
>>>>> Will keep an eye on HBASE-18240.
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] More Shading
Posted by Stack <st...@duboce.net>.
On Thu, Jul 6, 2017 at 11:52 AM, Stack <st...@duboce.net> wrote:
> FYI:
>
> hbase-thirdparty has had its first release. Yesterday I
> committed HBASE-17056 "Remove checked in PB generated files" which apart
> from purging all checked-in generated files (30MB), it moves our hbase core
> (master and branch-2) to start using the thirdparty jar.
>
> Things might be interesting over next few days so shout if you run into
> issues.
>
>
FYI, I reverted HBASE-17056 for the moment. Build seems unstable. OOMEs and
other interesting issues along w/ some awkwardness w/ dependencies. Let me
spend some more time on it. Will try again later.
St.Ack
> One issue is the need for mvn install where before mvn compile might have
> been enough (see below for example of the issue you'd see now if you did
> mvn clean compile only). We need the mvn install because we need to shade
> the generated files so they use the relocated protobuf; shade happens after
> we've made a module jar.
>
> IDEs will complain too if they pick up generated src from target dirs
> since they'll be unsatisfied references to protobuf3 objects -- unless you
> point at the shaded jar. Let me see if I can do something about the latter.
>
> Let me know if HBASE-17056 is too much for devs to bear. Can always revert
> (though nice having the protobuf generation in-line w/ the build). It is
> sort of a side-benefit on the general shading project so could back it out
> and still have the shading of netty, guava, protobuf, etc.
>
> Thanks,
>
> St.Ack
>
> [ERROR] Failed to execute goal org.apache.maven.plugins:
> maven-compiler-plugin:3.6.1:compile (default-compile) on project
> hbase-procedure: Compilation failure: Compilation failure:
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[105,11]
> cannot access com.google.protobuf.GeneratedMessageV3
> [ERROR] class file for com.google.protobuf.GeneratedMessageV3 not found
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[129,7]
> cannot access com.google.protobuf.GeneratedMessageV3.Builder
> [ERROR] class file for com.google.protobuf.GeneratedMessageV3$Builder not
> found
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[133,22]
> cannot find symbol
> [ERROR] symbol: method writeDelimitedTo(org.apache.
> hadoop.fs.FSDataOutputStream)
> [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated.
> ProcedureProtos.ProcedureStoreTracker
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[217,20]
> cannot find symbol
> [ERROR] symbol: method writeDelimitedTo(org.apache.
> hadoop.hbase.procedure2.util.ByteSlot)
> [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated.
> ProcedureProtos.ProcedureWALEntry
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[240,20]
> cannot find symbol
> [ERROR] symbol: method writeDelimitedTo(org.apache.
> hadoop.hbase.procedure2.util.ByteSlot)
> [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated.
> ProcedureProtos.ProcedureWALEntry
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[254,20]
> cannot find symbol
> [ERROR] symbol: method writeDelimitedTo(org.apache.
> hadoop.hbase.procedure2.util.ByteSlot)
> [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated.
> ProcedureProtos.ProcedureWALEntry
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/RemoteProcedureException.java:[98,30]
> cannot find symbol
> [ERROR] symbol: method toByteArray()
> [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated.
> ErrorHandlingProtos.ForeignExceptionMessage
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/StateMachineProcedure.java:[267,17] cannot
> find symbol
> [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream)
> [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated.
> ProcedureProtos.StateMachineProcedureData
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/ProcedureUtil.java:[130,56] incompatible
> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
> cannot be converted to com.google.protobuf.ByteString
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/ProcedureUtil.java:[137,54] incompatible
> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
> cannot be converted to com.google.protobuf.ByteString
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/ProcedureUtil.java:[237,56] incompatible
> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString
> cannot be converted to com.google.protobuf.ByteString
> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/
> apache/hadoop/hbase/procedure2/SequentialProcedure.java:[75,17] cannot
> find symbol
> [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream)
> [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated.
> ProcedureProtos.SequentialProcedureData
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/
> MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn <goals> -rf :hbase-procedure
>
> On Fri, Jun 30, 2017 at 3:09 PM, Stack <st...@duboce.net> wrote:
>
>> I just started a VOTE on hbase-thirdparty and the first RC made from it.
>> Thanks,
>> St.Ack
>>
>> On Tue, Jun 27, 2017 at 3:02 PM, Stack <st...@duboce.net> wrote:
>>
>>> Bit of an update.
>>>
>>> I'd suggest we go ahead w/ the hbase-thirdparty project [2]. It took a
>>> while but in its current form -- a few poms that package a few jars [1]--
>>> it at least enables the below:
>>>
>>> + Allows us to skip checking in protobuf generated files (25MB!); they
>>> can be generated inline w/ the build because the hackery patching protobuf
>>> has been moved out to hbase-thirdparty. There is a patch up on HBASE-17056.
>>> + Update our guava from 12.0 to 22.0 w/o clashing w/ the guava of
>>> others. There is a patch at HBASE-17908. It is taking a bit of wrangling
>>> getting it to land because I pared back transitive includes from hadoop and
>>> it takes a while to work through the failures.
>>>
>>> Other benefits are the protobuf-util lib is on the classpath now -- its
>>> in hbase-thirdparty relocated; depends on pb and guava -- so we have
>>> facility to goat "HBASE-18106 Redo ProcedureInfo and LockInfo" and shading
>>> netty is almost done so we can do with netty as we wilt independent of
>>> hadoop and downstreamers (the hard part -- relocation of the .so -- should
>>> be done).
>>>
>>> Let me figure how to run a vote for a couple of poms.....
>>>
>>> St.Ack
>>>
>>> 1. https://repository.apache.org/content/groups/snapshots/or
>>> g/apache/hbase/thirdparty/ (see hbase-shaded-thirdparty and
>>> hbase-shaded-protobuf)
>>> 2. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
>>>
>>>
>>> On Tue, Jun 20, 2017 at 11:04 AM, Josh Elser <jo...@gmail.com>
>>> wrote:
>>>
>>>> On 6/20/17 1:28 AM, Stack wrote:
>>>>
>>>>> On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser<el...@apache.org> wrote:
>>>>>
>>>>> ...
>>>>>>
>>>>>> I think pushing this part forward with some code is the next logical
>>>>>> step.
>>>>>> Seems to be consensus about taking our known internal dependencies and
>>>>>> performing this shade magic.
>>>>>>
>>>>>>
>>>>>> I opened HBASE-18240 "Add hbase-auxillary, a project with hbase
>>>>> utility
>>>>> including an hbase-shaded-thirdparty module with guava, netty, etc."
>>>>>
>>>>> It has a tarball attached that bundles the outline of an
>>>>> hbase-auxillary
>>>>> project (groupId:org.apache.hbase.auxillary). This project is
>>>>> intended to
>>>>> be standalone, in its own repository, publishing its own artifacts
>>>>> under
>>>>> the aegis of this project's PMC.
>>>>>
>>>>> It includes the first instance of an auxillary utility, a module named
>>>>> hbase-thirdparty-shaded (artifactId:hbase-thirdparty-shaded). Herein
>>>>> we'll
>>>>> pull down 3rd party libs and republish at an offset; e.g.
>>>>> com.google.common.* from guava will be at
>>>>> org.apache.hbase.thirdparty.shaded.com.google.common.*. Currently it
>>>>> builds
>>>>> a jar that includes a relocated guava 22.0.
>>>>>
>>>>> I then messed around making hbase-common use it (You have to build the
>>>>> hbase-auxillary into your local repo). I put up a patch on the issue.
>>>>> Mostly its mass find-and-replace w/ some clean up of transitive
>>>>> includes of
>>>>> guava from hadoop-common and some small fixup of methods renamed
>>>>> between
>>>>> guava 12.0 and 22.0.
>>>>>
>>>>> Unless objection, I was going to press on. Sean offered to help set up
>>>>> new
>>>>> repo. We can always undo and delete it if this project fails.
>>>>>
>>>>> When done, the hope is we are on a modern version of guava and our
>>>>> netty
>>>>> and protobuf 3 will be be relocated, 'hidden' from downstream (and
>>>>> won't
>>>>> clash w/ upstream). I hope to also purge the pre-build we have in our
>>>>> modules that do protobuf moving this hackery out and under
>>>>> hbase-thirdparty-shaded.
>>>>>
>>>>> St.Ack
>>>>>
>>>>
>>>> Kudos on the JFDI approach :). I think having something concrete to
>>>> show is the best way to judge success of it.
>>>>
>>>> Will keep an eye on HBASE-18240.
>>>>
>>>>
>>>
>>
>