Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Jerry He <je...@gmail.com>]

Mike from another thread asked about shading jackson into
hbase-thirdparty. That reminded me of a question I was thinking
recently,
in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
as gateway to an hbase cluster going forward".

We have shaded artifacts in hbase-thirdparty
(hbase-shaded-miscellaneous, hbase-shaded-protobuf and
hbase-shaded-netty). The purpose is to shade and bundle the jars with
versions hbase uses internally.

Then in hbase-shaded-client we do shading again for some of the same
jars, but they are more non-hbase, transitive dependencies.
For example, in hbase-shaded-miscellaneous, we shade guava 22. But
hbase-client will bring in guava 11 from hadoop-common. In
hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
protobuf 2.5.
We will be doing shading of all these in hbase-shaded-client (and the
hbase-shaded-mapreduce), with the same relocation
(org.apache.hadoop.hbase.shaded).

The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf 2.5?
I checked hbase-shaded-client and found it had the shaded guava 22 classes.
Looks like the shading process will only add one version of the
classes with the same name.

This will potentially have problem, and potentially make it unusable
in unlucky situations.

I bring it up here for awareness and discussion. First, is this really
a problem?
Second, how to fix it?  Maybe a different relocation offset in hbase-thirdparty?

Thanks,

Jerry

Re: Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Stack <st...@duboce.net>]

On Thu, Nov 2, 2017 at 3:28 PM, Jerry He <je...@gmail.com> wrote:

> o.a.h.hbase.thirdparty does look like a better choice.
> Let me open two JIRAs to change the hbase-thirdparty and then all the
> references in the main code (more tedious part).
> Thanks, guys!
>
> Thanks Jerry.
S



>
>
> On Thu, Nov 2, 2017 at 1:54 PM Sean Busbey <bu...@apache.org> wrote:
>
> > I think something like o.a.h.hbase.thirdparty might be better. then we
> > can differentiate between "shaded" for downstream of us vs
> > "thirdparty" for our internal use.
> >
> > On Thu, Nov 2, 2017 at 3:52 PM, Stack <st...@duboce.net> wrote:
> > > Good one Jerry He.
> > >
> > > I can try changing the offset in hbase-thirdparty in a new release
> > > (o.a.hbase.shaded?). There used to be a reason why it had to be
> > > o.a.h.hbase.shaded but IIRC it went away after all shaded dependencies
> > made
> > > it out to hbase-thirdparty (including the pb3 that we used to have
> > checked
> > > into core).
> > >
> > > S
> > >
> > > On Thu, Nov 2, 2017 at 11:45 AM, Jerry He <je...@gmail.com> wrote:
> > >
> > >> Mike from another thread asked about shading jackson into
> > >> hbase-thirdparty. That reminded me of a question I was thinking
> > >> recently,
> > >> in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
> > >> as gateway to an hbase cluster going forward".
> > >>
> > >> We have shaded artifacts in hbase-thirdparty
> > >> (hbase-shaded-miscellaneous, hbase-shaded-protobuf and
> > >> hbase-shaded-netty). The purpose is to shade and bundle the jars with
> > >> versions hbase uses internally.
> > >>
> > >> Then in hbase-shaded-client we do shading again for some of the same
> > >> jars, but they are more non-hbase, transitive dependencies.
> > >> For example, in hbase-shaded-miscellaneous, we shade guava 22. But
> > >> hbase-client will bring in guava 11 from hadoop-common. In
> > >> hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
> > >> protobuf 2.5.
> > >> We will be doing shading of all these in hbase-shaded-client (and the
> > >> hbase-shaded-mapreduce), with the same relocation
> > >> (org.apache.hadoop.hbase.shaded).
> > >>
> > >> The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf
> > 2.5?
> > >> I checked hbase-shaded-client and found it had the shaded guava 22
> > classes.
> > >> Looks like the shading process will only add one version of the
> > >> classes with the same name.
> > >>
> > >> This will potentially have problem, and potentially make it unusable
> > >> in unlucky situations.
> > >>
> > >> I bring it up here for awareness and discussion. First, is this really
> > >> a problem?
> > >> Second, how to fix it?  Maybe a different relocation offset in
> > >> hbase-thirdparty?
> > >>
> > >> Thanks,
> > >>
> > >> Jerry
> > >>
> >
>

Re: Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Jerry He <je...@gmail.com>]

o.a.h.hbase.thirdparty does look like a better choice.
Let me open two JIRAs to change the hbase-thirdparty and then all the
references in the main code (more tedious part).
Thanks, guys!



On Thu, Nov 2, 2017 at 1:54 PM Sean Busbey <bu...@apache.org> wrote:

> I think something like o.a.h.hbase.thirdparty might be better. then we
> can differentiate between "shaded" for downstream of us vs
> "thirdparty" for our internal use.
>
> On Thu, Nov 2, 2017 at 3:52 PM, Stack <st...@duboce.net> wrote:
> > Good one Jerry He.
> >
> > I can try changing the offset in hbase-thirdparty in a new release
> > (o.a.hbase.shaded?). There used to be a reason why it had to be
> > o.a.h.hbase.shaded but IIRC it went away after all shaded dependencies
> made
> > it out to hbase-thirdparty (including the pb3 that we used to have
> checked
> > into core).
> >
> > S
> >
> > On Thu, Nov 2, 2017 at 11:45 AM, Jerry He <je...@gmail.com> wrote:
> >
> >> Mike from another thread asked about shading jackson into
> >> hbase-thirdparty. That reminded me of a question I was thinking
> >> recently,
> >> in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
> >> as gateway to an hbase cluster going forward".
> >>
> >> We have shaded artifacts in hbase-thirdparty
> >> (hbase-shaded-miscellaneous, hbase-shaded-protobuf and
> >> hbase-shaded-netty). The purpose is to shade and bundle the jars with
> >> versions hbase uses internally.
> >>
> >> Then in hbase-shaded-client we do shading again for some of the same
> >> jars, but they are more non-hbase, transitive dependencies.
> >> For example, in hbase-shaded-miscellaneous, we shade guava 22. But
> >> hbase-client will bring in guava 11 from hadoop-common. In
> >> hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
> >> protobuf 2.5.
> >> We will be doing shading of all these in hbase-shaded-client (and the
> >> hbase-shaded-mapreduce), with the same relocation
> >> (org.apache.hadoop.hbase.shaded).
> >>
> >> The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf
> 2.5?
> >> I checked hbase-shaded-client and found it had the shaded guava 22
> classes.
> >> Looks like the shading process will only add one version of the
> >> classes with the same name.
> >>
> >> This will potentially have problem, and potentially make it unusable
> >> in unlucky situations.
> >>
> >> I bring it up here for awareness and discussion. First, is this really
> >> a problem?
> >> Second, how to fix it?  Maybe a different relocation offset in
> >> hbase-thirdparty?
> >>
> >> Thanks,
> >>
> >> Jerry
> >>
>

Re: Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Sean Busbey <bu...@apache.org>]

I think something like o.a.h.hbase.thirdparty might be better. then we
can differentiate between "shaded" for downstream of us vs
"thirdparty" for our internal use.

On Thu, Nov 2, 2017 at 3:52 PM, Stack <st...@duboce.net> wrote:
> Good one Jerry He.
>
> I can try changing the offset in hbase-thirdparty in a new release
> (o.a.hbase.shaded?). There used to be a reason why it had to be
> o.a.h.hbase.shaded but IIRC it went away after all shaded dependencies made
> it out to hbase-thirdparty (including the pb3 that we used to have checked
> into core).
>
> S
>
> On Thu, Nov 2, 2017 at 11:45 AM, Jerry He <je...@gmail.com> wrote:
>
>> Mike from another thread asked about shading jackson into
>> hbase-thirdparty. That reminded me of a question I was thinking
>> recently,
>> in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
>> as gateway to an hbase cluster going forward".
>>
>> We have shaded artifacts in hbase-thirdparty
>> (hbase-shaded-miscellaneous, hbase-shaded-protobuf and
>> hbase-shaded-netty). The purpose is to shade and bundle the jars with
>> versions hbase uses internally.
>>
>> Then in hbase-shaded-client we do shading again for some of the same
>> jars, but they are more non-hbase, transitive dependencies.
>> For example, in hbase-shaded-miscellaneous, we shade guava 22. But
>> hbase-client will bring in guava 11 from hadoop-common. In
>> hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
>> protobuf 2.5.
>> We will be doing shading of all these in hbase-shaded-client (and the
>> hbase-shaded-mapreduce), with the same relocation
>> (org.apache.hadoop.hbase.shaded).
>>
>> The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf 2.5?
>> I checked hbase-shaded-client and found it had the shaded guava 22 classes.
>> Looks like the shading process will only add one version of the
>> classes with the same name.
>>
>> This will potentially have problem, and potentially make it unusable
>> in unlucky situations.
>>
>> I bring it up here for awareness and discussion. First, is this really
>> a problem?
>> Second, how to fix it?  Maybe a different relocation offset in
>> hbase-thirdparty?
>>
>> Thanks,
>>
>> Jerry
>>

Re: Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Stack <st...@duboce.net>]

Good one Jerry He.

I can try changing the offset in hbase-thirdparty in a new release
(o.a.hbase.shaded?). There used to be a reason why it had to be
o.a.h.hbase.shaded but IIRC it went away after all shaded dependencies made
it out to hbase-thirdparty (including the pb3 that we used to have checked
into core).

S

On Thu, Nov 2, 2017 at 11:45 AM, Jerry He <je...@gmail.com> wrote:

> Mike from another thread asked about shading jackson into
> hbase-thirdparty. That reminded me of a question I was thinking
> recently,
> in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
> as gateway to an hbase cluster going forward".
>
> We have shaded artifacts in hbase-thirdparty
> (hbase-shaded-miscellaneous, hbase-shaded-protobuf and
> hbase-shaded-netty). The purpose is to shade and bundle the jars with
> versions hbase uses internally.
>
> Then in hbase-shaded-client we do shading again for some of the same
> jars, but they are more non-hbase, transitive dependencies.
> For example, in hbase-shaded-miscellaneous, we shade guava 22. But
> hbase-client will bring in guava 11 from hadoop-common. In
> hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
> protobuf 2.5.
> We will be doing shading of all these in hbase-shaded-client (and the
> hbase-shaded-mapreduce), with the same relocation
> (org.apache.hadoop.hbase.shaded).
>
> The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf 2.5?
> I checked hbase-shaded-client and found it had the shaded guava 22 classes.
> Looks like the shading process will only add one version of the
> classes with the same name.
>
> This will potentially have problem, and potentially make it unusable
> in unlucky situations.
>
> I bring it up here for awareness and discussion. First, is this really
> a problem?
> Second, how to fix it?  Maybe a different relocation offset in
> hbase-thirdparty?
>
> Thanks,
>
> Jerry
>

Re: Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Sean Busbey <bu...@apache.org>]

yeah, probably will cause a problem at some point. A different offset
for hbase-thirdparty is a good idea, I think.

On Thu, Nov 2, 2017 at 1:45 PM, Jerry He <je...@gmail.com> wrote:
> Mike from another thread asked about shading jackson into
> hbase-thirdparty. That reminded me of a question I was thinking
> recently,
> in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
> as gateway to an hbase cluster going forward".
>
> We have shaded artifacts in hbase-thirdparty
> (hbase-shaded-miscellaneous, hbase-shaded-protobuf and
> hbase-shaded-netty). The purpose is to shade and bundle the jars with
> versions hbase uses internally.
>
> Then in hbase-shaded-client we do shading again for some of the same
> jars, but they are more non-hbase, transitive dependencies.
> For example, in hbase-shaded-miscellaneous, we shade guava 22. But
> hbase-client will bring in guava 11 from hadoop-common. In
> hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
> protobuf 2.5.
> We will be doing shading of all these in hbase-shaded-client (and the
> hbase-shaded-mapreduce), with the same relocation
> (org.apache.hadoop.hbase.shaded).
>
> The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf 2.5?
> I checked hbase-shaded-client and found it had the shaded guava 22 classes.
> Looks like the shading process will only add one version of the
> classes with the same name.
>
> This will potentially have problem, and potentially make it unusable
> in unlucky situations.
>
> I bring it up here for awareness and discussion. First, is this really
> a problem?
> Second, how to fix it?  Maybe a different relocation offset in hbase-thirdparty?
>
> Thanks,
>
> Jerry