You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2017/06/30 22:07:00 UTC

[VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

This is an interesting vote. The vote is on an RC made out of a new hbase
project, hbase-thirdparty.

First the usual stuff.

The 1.0.0RC0 is available at:

 https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty/1.0.0RC0/

Maven artifacts are in the following staging repository:

 https://repository.apache.org/content/repositories/orgapachehbase-1171

Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS file
http://www-us.apache.org/dist/hbase/KEYS.

I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04cdf9e224f3 in
the new hbase-thirdparty repository at [1].

Now to the unusual.

The artifact was made from a new repository, hbase-thirdparty[1].
hbase-thirdparty is a project overseen by the hbase PMC. It came of
discussion on the dev list in the thread '[DISCUSS] More Shading' [2]. Its
charge is the hosting of ornery thirdparty libs -- e.g. protobuf, guava,
netty -- that need patching and/or shading so we are no longer bound by
whatever the version of a lib Hadoop happens to ship (and we can change
versions w/o damaging out glorious downstreamers). In the past, when we've
been able, we've done this dirty patching and shading work as an
unacknowledged 'pre-build' hack step appended to our protobuf file
generation. Moving this mess out of our mainline build to live elsewhere,
pulled in as a legit 'dependency', simplifies our build particularly around
the shading of third-part libs; the narrower scope of a dedicated project
that produces transformed jars is easier to reason about, is less likely to
clash with existing dependencies whether transitive or otherwise, and it
makes for less spaghetti poms.

Other (minor) benefits include our being able to generate protobuf files
inline now; no need of the pre-build anymore nor checking in of generated
protobuf files (25MB).  There is a patch to remove them (predicated on this
hbase-thirdparty release): "HBASE-17056 Remove checked in PB generated
files".

Downside is having to explicitly reference our version in code as in
org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather than just
com.google.protobuf.Message and so on for guava references, etc. (We've
been doing this for a while now so hopefully it not too strange
developing)*.

There is little to this new project other than a few poms to do packaging
and a couple of patches we apply on build. It works by pulling down
dependencies and doing whatever shading, renaming, or patching we need.
Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava (12.0 =>
22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes three jars
for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and then
hbase-shaded-miscellaneous for the remaining libs. Philosophy is if the lib
is troublesome, protobuf needs patching and netty includes an .so that
needs a rename, then give it its own module (See README in the release for
more). I've arrived at this layout for hbase-thirdparty after a bunch of
experimentation and messing with hbase dependencies (See sub-tasks under
HBASE-17898 Update dependencies). That said, as we head to hbase-2.0.0, we
might have to make a an odd new release of hbase-thirdparty but thereafter
I'd imagine it should settle down to be an infrequent update.

VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty project
and a vote for this RC.

Questions on how it will go w/ this project welcome on this thread. If
heavy-detail on how the shading works, I suggest ask on the '[DISCUSS] More
Shading' thread.

Thanks,
St.Ack
1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
2. See the '[DISCUSS] More Shading' thread,
http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html

* Doc on how to make use of hbase-thirdparty is coming for the refguide.

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Stack <st...@duboce.net>.

On Sun, Jul 2, 2017 at 11:01 PM, 张铎(Duo Zhang) <pa...@gmail.com>
wrote:

> Then it will be a bit pain if developers want to see the source code of
> the

relocated classes in IDE?
>
>
Yes. Will be a pain. I can work on this for next versions. Interesting was
that some of the dependencies of guava had no src code seemingly (would
have to dig more). Filed HBASE-18313 [hbase-thirdparty] Produce src
jars/tgz.

St.Ack



> 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
>
> > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Checked sums and signatures: OK
> > > 'mvn clean install': OK, the size of the generated jar files are
> > reasonable
> > > Unzip the generated jar files: OK, all classes are relocated under
> > > org/apache/hadoop/hbase/shaded
> > >
> > > Only one minor question, seems we do not generate sources jars? There
> is
> > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the
> > size
> > > is only 22 bytes.
> > >
> > >
> > Good question. This project is a little bit odd. It is just poms (and a
> few
> > patches). The poms do a download, relocation, and then push to deploy the
> > jars to mvn repo. There is no real 'source' -- not anything that makes
> > sense to stuff in a src.jar --  other than the poms that are here in the
> > tgz (mvn install puts the jars w/ relocation into your local repo....
> add a
> > 'deploy' to push to mvn repository -- see README).
> >
> > Thanks Duo,
> > S
> >
> >
> >
> > > Thanks.
> > >
> > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > > > +1 from me.
> > > > St.Ack
> > > >
> > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > This is an interesting vote. The vote is on an RC made out of a new
> > > hbase
> > > > > project, hbase-thirdparty.
> > > > >
> > > > > First the usual stuff.
> > > > >
> > > > > The 1.0.0RC0 is available at:
> > > > >
> > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > thirdparty/1.0.0RC0/
> > > > >
> > > > > Maven artifacts are in the following staging repository:
> > > > >
> > > > >  https://repository.apache.org/content/repositories/
> > > orgapachehbase-1171
> > > > >
> > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> > > file
> > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > >
> > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > cdf9e224f3
> > > > > in the new hbase-thirdparty repository at [1].
> > > > >
> > > > > Now to the unusual.
> > > > >
> > > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> > [2].
> > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> protobuf,
> > > > > guava, netty -- that need patching and/or shading so we are no
> longer
> > > > bound
> > > > > by whatever the version of a lib Hadoop happens to ship (and we can
> > > > change
> > > > > versions w/o damaging out glorious downstreamers). In the past,
> when
> > > > we've
> > > > > been able, we've done this dirty patching and shading work as an
> > > > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > > > generation. Moving this mess out of our mainline build to live
> > > elsewhere,
> > > > > pulled in as a legit 'dependency', simplifies our build
> particularly
> > > > around
> > > > > the shading of third-part libs; the narrower scope of a dedicated
> > > project
> > > > > that produces transformed jars is easier to reason about, is less
> > > likely
> > > > to
> > > > > clash with existing dependencies whether transitive or otherwise,
> and
> > > it
> > > > > makes for less spaghetti poms.
> > > > >
> > > > > Other (minor) benefits include our being able to generate protobuf
> > > files
> > > > > inline now; no need of the pre-build anymore nor checking in of
> > > generated
> > > > > protobuf files (25MB).  There is a patch to remove them (predicated
> > on
> > > > this
> > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > generated
> > > > > files".
> > > > >
> > > > > Downside is having to explicitly reference our version in code as
> in
> > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather
> > than
> > > > > just com.google.protobuf.Message and so on for guava references,
> etc.
> > > > > (We've been doing this for a while now so hopefully it not too
> > strange
> > > > > developing)*.
> > > > >
> > > > > There is little to this new project other than a few poms to do
> > > packaging
> > > > > and a couple of patches we apply on build. It works by pulling down
> > > > > dependencies and doing whatever shading, renaming, or patching we
> > need.
> > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> > (12.0
> > > > =>
> > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes
> > three
> > > > jars
> > > > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty,
> and
> > > then
> > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if
> > the
> > > > lib
> > > > > is troublesome, protobuf needs patching and netty includes an .so
> > that
> > > > > needs a rename, then give it its own module (See README in the
> > release
> > > > for
> > > > > more). I've arrived at this layout for hbase-thirdparty after a
> bunch
> > > of
> > > > > experimentation and messing with hbase dependencies (See sub-tasks
> > > under
> > > > > HBASE-17898 Update dependencies). That said, as we head to
> > hbase-2.0.0,
> > > > we
> > > > > might have to make a an odd new release of hbase-thirdparty but
> > > > thereafter
> > > > > I'd imagine it should settle down to be an infrequent update.
> > > > >
> > > > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> > > project
> > > > > and a vote for this RC.
> > > > >
> > > > > Questions on how it will go w/ this project welcome on this thread.
> > If
> > > > > heavy-detail on how the shading works, I suggest ask on the
> > '[DISCUSS]
> > > > More
> > > > > Shading' thread.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > > 2. See the '[DISCUSS] More Shading' thread,
> > > > http://apache-hbase.679495.n3.
> > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > > >
> > > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > > refguide.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Stack <st...@duboce.net>.

On Mon, Jul 3, 2017 at 10:42 PM, Yu Li <ca...@gmail.com> wrote:

> bq. In branch-1, relocate all probobuf and guava references?
> Yes sir, IMHO this is nice to have (smile). Thanks.
>
>
This sounds 'safe' for a branch-1.4 or 1.5... if we are just relocating but
still referring to the same version of the lib (with the adventurous such
as yourselves slotting in newer libs).

S


> Best Regards,
> Yu
>
> On 4 July 2017 at 11:41, Stack <st...@duboce.net> wrote:
>
> > On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <ca...@gmail.com> wrote:
> > >
> > > bq. No plans to up guava or protobuf versions on branch-1. You think we
> > > should?
> > > We met with guava conflict when using hbase in Flink, and I could see
> > > similar issue mentioned in spark. Since branch-1 is still the stable
> > > release line and will be in the near future, possible to do similar
> > shading
> > > w/o upgrading the guava/protobuf version? Thanks.
> > >
> > > In branch-1, relocate all probobuf and guava references?
> >
> > Thanks Yu Li,
> > S
> >
> >
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 3 July 2017 at 14:01, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> > >
> > > > Then it will be a bit pain if developers want to see the source code
> of
> > > the
> > > > relocated classes in IDE?
> > > >
> > > > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
> > > >
> > > > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <
> palomino219@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Checked sums and signatures: OK
> > > > > > 'mvn clean install': OK, the size of the generated jar files are
> > > > > reasonable
> > > > > > Unzip the generated jar files: OK, all classes are relocated
> under
> > > > > > org/apache/hadoop/hbase/shaded
> > > > > >
> > > > > > Only one minor question, seems we do not generate sources jars?
> > There
> > > > is
> > > > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install'
> but
> > > the
> > > > > size
> > > > > > is only 22 bytes.
> > > > > >
> > > > > >
> > > > > Good question. This project is a little bit odd. It is just poms
> > (and a
> > > > few
> > > > > patches). The poms do a download, relocation, and then push to
> deploy
> > > the
> > > > > jars to mvn repo. There is no real 'source' -- not anything that
> > makes
> > > > > sense to stuff in a src.jar --  other than the poms that are here
> in
> > > the
> > > > > tgz (mvn install puts the jars w/ relocation into your local
> repo....
> > > > add a
> > > > > 'deploy' to push to mvn repository -- see README).
> > > > >
> > > > > Thanks Duo,
> > > > > S
> > > > >
> > > > >
> > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> > > > > >
> > > > > > > +1 from me.
> > > > > > > St.Ack
> > > > > > >
> > > > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net>
> wrote:
> > > > > > >
> > > > > > > > This is an interesting vote. The vote is on an RC made out
> of a
> > > new
> > > > > > hbase
> > > > > > > > project, hbase-thirdparty.
> > > > > > > >
> > > > > > > > First the usual stuff.
> > > > > > > >
> > > > > > > > The 1.0.0RC0 is available at:
> > > > > > > >
> > > > > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > > > > thirdparty/1.0.0RC0/
> > > > > > > >
> > > > > > > > Maven artifacts are in the following staging repository:
> > > > > > > >
> > > > > > > >  https://repository.apache.org/content/repositories/
> > > > > > orgapachehbase-1171
> > > > > > > >
> > > > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of
> our
> > > KEYS
> > > > > > file
> > > > > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > > > > >
> > > > > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > > > > cdf9e224f3
> > > > > > > > in the new hbase-thirdparty repository at [1].
> > > > > > > >
> > > > > > > > Now to the unusual.
> > > > > > > >
> > > > > > > > The artifact was made from a new repository,
> > hbase-thirdparty[1].
> > > > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It
> > came
> > > of
> > > > > > > > discussion on the dev list in the thread '[DISCUSS] More
> > Shading'
> > > > > [2].
> > > > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> > > > protobuf,
> > > > > > > > guava, netty -- that need patching and/or shading so we are
> no
> > > > longer
> > > > > > > bound
> > > > > > > > by whatever the version of a lib Hadoop happens to ship (and
> we
> > > can
> > > > > > > change
> > > > > > > > versions w/o damaging out glorious downstreamers). In the
> past,
> > > > when
> > > > > > > we've
> > > > > > > > been able, we've done this dirty patching and shading work as
> > an
> > > > > > > > unacknowledged 'pre-build' hack step appended to our protobuf
> > > file
> > > > > > > > generation. Moving this mess out of our mainline build to
> live
> > > > > > elsewhere,
> > > > > > > > pulled in as a legit 'dependency', simplifies our build
> > > > particularly
> > > > > > > around
> > > > > > > > the shading of third-part libs; the narrower scope of a
> > dedicated
> > > > > > project
> > > > > > > > that produces transformed jars is easier to reason about, is
> > less
> > > > > > likely
> > > > > > > to
> > > > > > > > clash with existing dependencies whether transitive or
> > otherwise,
> > > > and
> > > > > > it
> > > > > > > > makes for less spaghetti poms.
> > > > > > > >
> > > > > > > > Other (minor) benefits include our being able to generate
> > > protobuf
> > > > > > files
> > > > > > > > inline now; no need of the pre-build anymore nor checking in
> of
> > > > > > generated
> > > > > > > > protobuf files (25MB).  There is a patch to remove them
> > > (predicated
> > > > > on
> > > > > > > this
> > > > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > > > > generated
> > > > > > > > files".
> > > > > > > >
> > > > > > > > Downside is having to explicitly reference our version in
> code
> > as
> > > > in
> > > > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> > > rather
> > > > > than
> > > > > > > > just com.google.protobuf.Message and so on for guava
> > references,
> > > > etc.
> > > > > > > > (We've been doing this for a while now so hopefully it not
> too
> > > > > strange
> > > > > > > > developing)*.
> > > > > > > >
> > > > > > > > There is little to this new project other than a few poms to
> do
> > > > > > packaging
> > > > > > > > and a couple of patches we apply on build. It works by
> pulling
> > > down
> > > > > > > > dependencies and doing whatever shading, renaming, or
> patching
> > we
> > > > > need.
> > > > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1),
> > guava
> > > > > (12.0
> > > > > > > =>
> > > > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It
> > publishes
> > > > > three
> > > > > > > jars
> > > > > > > > for hbase to consume: hbase-shaded-protobuf,
> > hbase-shaded-netty,
> > > > and
> > > > > > then
> > > > > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy
> > is
> > > if
> > > > > the
> > > > > > > lib
> > > > > > > > is troublesome, protobuf needs patching and netty includes an
> > .so
> > > > > that
> > > > > > > > needs a rename, then give it its own module (See README in
> the
> > > > > release
> > > > > > > for
> > > > > > > > more). I've arrived at this layout for hbase-thirdparty
> after a
> > > > bunch
> > > > > > of
> > > > > > > > experimentation and messing with hbase dependencies (See
> > > sub-tasks
> > > > > > under
> > > > > > > > HBASE-17898 Update dependencies). That said, as we head to
> > > > > hbase-2.0.0,
> > > > > > > we
> > > > > > > > might have to make a an odd new release of hbase-thirdparty
> but
> > > > > > > thereafter
> > > > > > > > I'd imagine it should settle down to be an infrequent update.
> > > > > > > >
> > > > > > > > VOTE lasts 72 hours. Let +1 be an assent to the
> > hbase-thirdparty
> > > > > > project
> > > > > > > > and a vote for this RC.
> > > > > > > >
> > > > > > > > Questions on how it will go w/ this project welcome on this
> > > thread.
> > > > > If
> > > > > > > > heavy-detail on how the shading works, I suggest ask on the
> > > > > '[DISCUSS]
> > > > > > > More
> > > > > > > > Shading' thread.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > St.Ack
> > > > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > > > > > 2. See the '[DISCUSS] More Shading' thread,
> > > > > > > http://apache-hbase.679495.n3.
> > > > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > > > > > >
> > > > > > > > * Doc on how to make use of hbase-thirdparty is coming for
> the
> > > > > > refguide.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Stack <st...@duboce.net>.

On Tue, Jul 4, 2017 at 6:16 AM, Chunhui Shen <zj...@163.com> wrote:

> A small thing, we can't get the real version of the thirdparty  library
> after depend on the new generated hbase-shaded-*.jar.
> With the above doubt, I found the file 'META-INF/DEPENDENCIES' which is
> included in hbase-shaded-*.jar,  it contains the information about the
> shaded thirdparty library name and its version.
> However,  'META-INF/DEPENDENCIES' of hbase-shaded-protobuf-1.0.0.jar
> contains none...
>
>
Thanks Chunhui. Makes sense that this should be populated by the
hbase-thirdparty build. Let me fix in next release. I filed HBASE-18321
[hbase-thirdparty] Fix generation of META-INF/DEPENDENCIES to include
dependency list and versions

Thanks,
S


>
> Just realized such a different point,  it's not a problem for most
> developers I think, but better if we could get the real information about
> thirdparty library in hbase project.
>
> At 2017-07-04 13:42:44, "Yu Li" <ca...@gmail.com> wrote:
> >bq. In branch-1, relocate all probobuf and guava references?
> >Yes sir, IMHO this is nice to have (smile). Thanks.
> >
> >Best Regards,
> >Yu
> >
> >On 4 July 2017 at 11:41, Stack <st...@duboce.net> wrote:
> >
> >> On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <ca...@gmail.com> wrote:
> >> >
> >> > bq. No plans to up guava or protobuf versions on branch-1. You think
> we
> >> > should?
> >> > We met with guava conflict when using hbase in Flink, and I could see
> >> > similar issue mentioned in spark. Since branch-1 is still the stable
> >> > release line and will be in the near future, possible to do similar
> >> shading
> >> > w/o upgrading the guava/protobuf version? Thanks.
> >> >
> >> > In branch-1, relocate all probobuf and guava references?
> >>
> >> Thanks Yu Li,
> >> S
> >>
> >>
> >> >
> >> > Best Regards,
> >> > Yu
> >> >
> >> > On 3 July 2017 at 14:01, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >> >
> >> > > Then it will be a bit pain if developers want to see the source
> code of
> >> > the
> >> > > relocated classes in IDE?
> >> > >
> >> > > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
> >> > >
> >> > > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <
> palomino219@gmail.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > +1 (binding)
> >> > > > >
> >> > > > > Checked sums and signatures: OK
> >> > > > > 'mvn clean install': OK, the size of the generated jar files are
> >> > > > reasonable
> >> > > > > Unzip the generated jar files: OK, all classes are relocated
> under
> >> > > > > org/apache/hadoop/hbase/shaded
> >> > > > >
> >> > > > > Only one minor question, seems we do not generate sources jars?
> >> There
> >> > > is
> >> > > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install'
> but
> >> > the
> >> > > > size
> >> > > > > is only 22 bytes.
> >> > > > >
> >> > > > >
> >> > > > Good question. This project is a little bit odd. It is just poms
> >> (and a
> >> > > few
> >> > > > patches). The poms do a download, relocation, and then push to
> deploy
> >> > the
> >> > > > jars to mvn repo. There is no real 'source' -- not anything that
> >> makes
> >> > > > sense to stuff in a src.jar --  other than the poms that are here
> in
> >> > the
> >> > > > tgz (mvn install puts the jars w/ relocation into your local
> repo....
> >> > > add a
> >> > > > 'deploy' to push to mvn repository -- see README).
> >> > > >
> >> > > > Thanks Duo,
> >> > > > S
> >> > > >
> >> > > >
> >> > > >
> >> > > > > Thanks.
> >> > > > >
> >> > > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> >> > > > >
> >> > > > > > +1 from me.
> >> > > > > > St.Ack
> >> > > > > >
> >> > > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net>
> wrote:
> >> > > > > >
> >> > > > > > > This is an interesting vote. The vote is on an RC made out
> of a
> >> > new
> >> > > > > hbase
> >> > > > > > > project, hbase-thirdparty.
> >> > > > > > >
> >> > > > > > > First the usual stuff.
> >> > > > > > >
> >> > > > > > > The 1.0.0RC0 is available at:
> >> > > > > > >
> >> > > > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> >> > > > > thirdparty/1.0.0RC0/
> >> > > > > > >
> >> > > > > > > Maven artifacts are in the following staging repository:
> >> > > > > > >
> >> > > > > > >  https://repository.apache.org/content/repositories/
> >> > > > > orgapachehbase-1171
> >> > > > > > >
> >> > > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of
> our
> >> > KEYS
> >> > > > > file
> >> > > > > > > http://www-us.apache.org/dist/hbase/KEYS.
> >> > > > > > >
> >> > > > > > > I tagged this RC as 1.0.0RC at
> 8ffaf3dd561052bcc71772148ecd04
> >> > > > > cdf9e224f3
> >> > > > > > > in the new hbase-thirdparty repository at [1].
> >> > > > > > >
> >> > > > > > > Now to the unusual.
> >> > > > > > >
> >> > > > > > > The artifact was made from a new repository,
> >> hbase-thirdparty[1].
> >> > > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It
> >> came
> >> > of
> >> > > > > > > discussion on the dev list in the thread '[DISCUSS] More
> >> Shading'
> >> > > > [2].
> >> > > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> >> > > protobuf,
> >> > > > > > > guava, netty -- that need patching and/or shading so we are
> no
> >> > > longer
> >> > > > > > bound
> >> > > > > > > by whatever the version of a lib Hadoop happens to ship
> (and we
> >> > can
> >> > > > > > change
> >> > > > > > > versions w/o damaging out glorious downstreamers). In the
> past,
> >> > > when
> >> > > > > > we've
> >> > > > > > > been able, we've done this dirty patching and shading work
> as
> >> an
> >> > > > > > > unacknowledged 'pre-build' hack step appended to our
> protobuf
> >> > file
> >> > > > > > > generation. Moving this mess out of our mainline build to
> live
> >> > > > > elsewhere,
> >> > > > > > > pulled in as a legit 'dependency', simplifies our build
> >> > > particularly
> >> > > > > > around
> >> > > > > > > the shading of third-part libs; the narrower scope of a
> >> dedicated
> >> > > > > project
> >> > > > > > > that produces transformed jars is easier to reason about, is
> >> less
> >> > > > > likely
> >> > > > > > to
> >> > > > > > > clash with existing dependencies whether transitive or
> >> otherwise,
> >> > > and
> >> > > > > it
> >> > > > > > > makes for less spaghetti poms.
> >> > > > > > >
> >> > > > > > > Other (minor) benefits include our being able to generate
> >> > protobuf
> >> > > > > files
> >> > > > > > > inline now; no need of the pre-build anymore nor checking
> in of
> >> > > > > generated
> >> > > > > > > protobuf files (25MB).  There is a patch to remove them
> >> > (predicated
> >> > > > on
> >> > > > > > this
> >> > > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> >> > > > generated
> >> > > > > > > files".
> >> > > > > > >
> >> > > > > > > Downside is having to explicitly reference our version in
> code
> >> as
> >> > > in
> >> > > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> >> > rather
> >> > > > than
> >> > > > > > > just com.google.protobuf.Message and so on for guava
> >> references,
> >> > > etc.
> >> > > > > > > (We've been doing this for a while now so hopefully it not
> too
> >> > > > strange
> >> > > > > > > developing)*.
> >> > > > > > >
> >> > > > > > > There is little to this new project other than a few poms
> to do
> >> > > > > packaging
> >> > > > > > > and a couple of patches we apply on build. It works by
> pulling
> >> > down
> >> > > > > > > dependencies and doing whatever shading, renaming, or
> patching
> >> we
> >> > > > need.
> >> > > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1),
> >> guava
> >> > > > (12.0
> >> > > > > > =>
> >> > > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It
> >> publishes
> >> > > > three
> >> > > > > > jars
> >> > > > > > > for hbase to consume: hbase-shaded-protobuf,
> >> hbase-shaded-netty,
> >> > > and
> >> > > > > then
> >> > > > > > > hbase-shaded-miscellaneous for the remaining libs.
> Philosophy
> >> is
> >> > if
> >> > > > the
> >> > > > > > lib
> >> > > > > > > is troublesome, protobuf needs patching and netty includes
> an
> >> .so
> >> > > > that
> >> > > > > > > needs a rename, then give it its own module (See README in
> the
> >> > > > release
> >> > > > > > for
> >> > > > > > > more). I've arrived at this layout for hbase-thirdparty
> after a
> >> > > bunch
> >> > > > > of
> >> > > > > > > experimentation and messing with hbase dependencies (See
> >> > sub-tasks
> >> > > > > under
> >> > > > > > > HBASE-17898 Update dependencies). That said, as we head to
> >> > > > hbase-2.0.0,
> >> > > > > > we
> >> > > > > > > might have to make a an odd new release of hbase-thirdparty
> but
> >> > > > > > thereafter
> >> > > > > > > I'd imagine it should settle down to be an infrequent
> update.
> >> > > > > > >
> >> > > > > > > VOTE lasts 72 hours. Let +1 be an assent to the
> >> hbase-thirdparty
> >> > > > > project
> >> > > > > > > and a vote for this RC.
> >> > > > > > >
> >> > > > > > > Questions on how it will go w/ this project welcome on this
> >> > thread.
> >> > > > If
> >> > > > > > > heavy-detail on how the shading works, I suggest ask on the
> >> > > > '[DISCUSS]
> >> > > > > > More
> >> > > > > > > Shading' thread.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > St.Ack
> >> > > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> >> > > > > > > 2. See the '[DISCUSS] More Shading' thread,
> >> > > > > > http://apache-hbase.679495.n3.
> >> > > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> >> > > > > > >
> >> > > > > > > * Doc on how to make use of hbase-thirdparty is coming for
> the
> >> > > > > refguide.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>

Re:Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Chunhui Shen <zj...@163.com>.

A small thing, we can't get the real version of the thirdparty  library after depend on the new generated hbase-shaded-*.jar.
With the above doubt, I found the file 'META-INF/DEPENDENCIES' which is included in hbase-shaded-*.jar,  it contains the information about the shaded thirdparty library name and its version.
However,  'META-INF/DEPENDENCIES' of hbase-shaded-protobuf-1.0.0.jar contains none...


Just realized such a different point,  it's not a problem for most developers I think, but better if we could get the real information about thirdparty library in hbase project.

At 2017-07-04 13:42:44, "Yu Li" <ca...@gmail.com> wrote:
>bq. In branch-1, relocate all probobuf and guava references?
>Yes sir, IMHO this is nice to have (smile). Thanks.
>
>Best Regards,
>Yu
>
>On 4 July 2017 at 11:41, Stack <st...@duboce.net> wrote:
>
>> On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <ca...@gmail.com> wrote:
>> >
>> > bq. No plans to up guava or protobuf versions on branch-1. You think we
>> > should?
>> > We met with guava conflict when using hbase in Flink, and I could see
>> > similar issue mentioned in spark. Since branch-1 is still the stable
>> > release line and will be in the near future, possible to do similar
>> shading
>> > w/o upgrading the guava/protobuf version? Thanks.
>> >
>> > In branch-1, relocate all probobuf and guava references?
>>
>> Thanks Yu Li,
>> S
>>
>>
>> >
>> > Best Regards,
>> > Yu
>> >
>> > On 3 July 2017 at 14:01, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>> >
>> > > Then it will be a bit pain if developers want to see the source code of
>> > the
>> > > relocated classes in IDE?
>> > >
>> > > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
>> > >
>> > > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <palomino219@gmail.com
>> >
>> > > > wrote:
>> > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > Checked sums and signatures: OK
>> > > > > 'mvn clean install': OK, the size of the generated jar files are
>> > > > reasonable
>> > > > > Unzip the generated jar files: OK, all classes are relocated under
>> > > > > org/apache/hadoop/hbase/shaded
>> > > > >
>> > > > > Only one minor question, seems we do not generate sources jars?
>> There
>> > > is
>> > > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but
>> > the
>> > > > size
>> > > > > is only 22 bytes.
>> > > > >
>> > > > >
>> > > > Good question. This project is a little bit odd. It is just poms
>> (and a
>> > > few
>> > > > patches). The poms do a download, relocation, and then push to deploy
>> > the
>> > > > jars to mvn repo. There is no real 'source' -- not anything that
>> makes
>> > > > sense to stuff in a src.jar --  other than the poms that are here in
>> > the
>> > > > tgz (mvn install puts the jars w/ relocation into your local repo....
>> > > add a
>> > > > 'deploy' to push to mvn repository -- see README).
>> > > >
>> > > > Thanks Duo,
>> > > > S
>> > > >
>> > > >
>> > > >
>> > > > > Thanks.
>> > > > >
>> > > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
>> > > > >
>> > > > > > +1 from me.
>> > > > > > St.Ack
>> > > > > >
>> > > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
>> > > > > >
>> > > > > > > This is an interesting vote. The vote is on an RC made out of a
>> > new
>> > > > > hbase
>> > > > > > > project, hbase-thirdparty.
>> > > > > > >
>> > > > > > > First the usual stuff.
>> > > > > > >
>> > > > > > > The 1.0.0RC0 is available at:
>> > > > > > >
>> > > > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
>> > > > > thirdparty/1.0.0RC0/
>> > > > > > >
>> > > > > > > Maven artifacts are in the following staging repository:
>> > > > > > >
>> > > > > > >  https://repository.apache.org/content/repositories/
>> > > > > orgapachehbase-1171
>> > > > > > >
>> > > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our
>> > KEYS
>> > > > > file
>> > > > > > > http://www-us.apache.org/dist/hbase/KEYS.
>> > > > > > >
>> > > > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
>> > > > > cdf9e224f3
>> > > > > > > in the new hbase-thirdparty repository at [1].
>> > > > > > >
>> > > > > > > Now to the unusual.
>> > > > > > >
>> > > > > > > The artifact was made from a new repository,
>> hbase-thirdparty[1].
>> > > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It
>> came
>> > of
>> > > > > > > discussion on the dev list in the thread '[DISCUSS] More
>> Shading'
>> > > > [2].
>> > > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
>> > > protobuf,
>> > > > > > > guava, netty -- that need patching and/or shading so we are no
>> > > longer
>> > > > > > bound
>> > > > > > > by whatever the version of a lib Hadoop happens to ship (and we
>> > can
>> > > > > > change
>> > > > > > > versions w/o damaging out glorious downstreamers). In the past,
>> > > when
>> > > > > > we've
>> > > > > > > been able, we've done this dirty patching and shading work as
>> an
>> > > > > > > unacknowledged 'pre-build' hack step appended to our protobuf
>> > file
>> > > > > > > generation. Moving this mess out of our mainline build to live
>> > > > > elsewhere,
>> > > > > > > pulled in as a legit 'dependency', simplifies our build
>> > > particularly
>> > > > > > around
>> > > > > > > the shading of third-part libs; the narrower scope of a
>> dedicated
>> > > > > project
>> > > > > > > that produces transformed jars is easier to reason about, is
>> less
>> > > > > likely
>> > > > > > to
>> > > > > > > clash with existing dependencies whether transitive or
>> otherwise,
>> > > and
>> > > > > it
>> > > > > > > makes for less spaghetti poms.
>> > > > > > >
>> > > > > > > Other (minor) benefits include our being able to generate
>> > protobuf
>> > > > > files
>> > > > > > > inline now; no need of the pre-build anymore nor checking in of
>> > > > > generated
>> > > > > > > protobuf files (25MB).  There is a patch to remove them
>> > (predicated
>> > > > on
>> > > > > > this
>> > > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
>> > > > generated
>> > > > > > > files".
>> > > > > > >
>> > > > > > > Downside is having to explicitly reference our version in code
>> as
>> > > in
>> > > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
>> > rather
>> > > > than
>> > > > > > > just com.google.protobuf.Message and so on for guava
>> references,
>> > > etc.
>> > > > > > > (We've been doing this for a while now so hopefully it not too
>> > > > strange
>> > > > > > > developing)*.
>> > > > > > >
>> > > > > > > There is little to this new project other than a few poms to do
>> > > > > packaging
>> > > > > > > and a couple of patches we apply on build. It works by pulling
>> > down
>> > > > > > > dependencies and doing whatever shading, renaming, or patching
>> we
>> > > > need.
>> > > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1),
>> guava
>> > > > (12.0
>> > > > > > =>
>> > > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It
>> publishes
>> > > > three
>> > > > > > jars
>> > > > > > > for hbase to consume: hbase-shaded-protobuf,
>> hbase-shaded-netty,
>> > > and
>> > > > > then
>> > > > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy
>> is
>> > if
>> > > > the
>> > > > > > lib
>> > > > > > > is troublesome, protobuf needs patching and netty includes an
>> .so
>> > > > that
>> > > > > > > needs a rename, then give it its own module (See README in the
>> > > > release
>> > > > > > for
>> > > > > > > more). I've arrived at this layout for hbase-thirdparty after a
>> > > bunch
>> > > > > of
>> > > > > > > experimentation and messing with hbase dependencies (See
>> > sub-tasks
>> > > > > under
>> > > > > > > HBASE-17898 Update dependencies). That said, as we head to
>> > > > hbase-2.0.0,
>> > > > > > we
>> > > > > > > might have to make a an odd new release of hbase-thirdparty but
>> > > > > > thereafter
>> > > > > > > I'd imagine it should settle down to be an infrequent update.
>> > > > > > >
>> > > > > > > VOTE lasts 72 hours. Let +1 be an assent to the
>> hbase-thirdparty
>> > > > > project
>> > > > > > > and a vote for this RC.
>> > > > > > >
>> > > > > > > Questions on how it will go w/ this project welcome on this
>> > thread.
>> > > > If
>> > > > > > > heavy-detail on how the shading works, I suggest ask on the
>> > > > '[DISCUSS]
>> > > > > > More
>> > > > > > > Shading' thread.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > St.Ack
>> > > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
>> > > > > > > 2. See the '[DISCUSS] More Shading' thread,
>> > > > > > http://apache-hbase.679495.n3.
>> > > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
>> > > > > > >
>> > > > > > > * Doc on how to make use of hbase-thirdparty is coming for the
>> > > > > refguide.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Yu Li <ca...@gmail.com>.

bq. In branch-1, relocate all probobuf and guava references?
Yes sir, IMHO this is nice to have (smile). Thanks.

Best Regards,
Yu

On 4 July 2017 at 11:41, Stack <st...@duboce.net> wrote:

> On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <ca...@gmail.com> wrote:
> >
> > bq. No plans to up guava or protobuf versions on branch-1. You think we
> > should?
> > We met with guava conflict when using hbase in Flink, and I could see
> > similar issue mentioned in spark. Since branch-1 is still the stable
> > release line and will be in the near future, possible to do similar
> shading
> > w/o upgrading the guava/protobuf version? Thanks.
> >
> > In branch-1, relocate all probobuf and guava references?
>
> Thanks Yu Li,
> S
>
>
> >
> > Best Regards,
> > Yu
> >
> > On 3 July 2017 at 14:01, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > > Then it will be a bit pain if developers want to see the source code of
> > the
> > > relocated classes in IDE?
> > >
> > > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <palomino219@gmail.com
> >
> > > > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Checked sums and signatures: OK
> > > > > 'mvn clean install': OK, the size of the generated jar files are
> > > > reasonable
> > > > > Unzip the generated jar files: OK, all classes are relocated under
> > > > > org/apache/hadoop/hbase/shaded
> > > > >
> > > > > Only one minor question, seems we do not generate sources jars?
> There
> > > is
> > > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but
> > the
> > > > size
> > > > > is only 22 bytes.
> > > > >
> > > > >
> > > > Good question. This project is a little bit odd. It is just poms
> (and a
> > > few
> > > > patches). The poms do a download, relocation, and then push to deploy
> > the
> > > > jars to mvn repo. There is no real 'source' -- not anything that
> makes
> > > > sense to stuff in a src.jar --  other than the poms that are here in
> > the
> > > > tgz (mvn install puts the jars w/ relocation into your local repo....
> > > add a
> > > > 'deploy' to push to mvn repository -- see README).
> > > >
> > > > Thanks Duo,
> > > > S
> > > >
> > > >
> > > >
> > > > > Thanks.
> > > > >
> > > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> > > > >
> > > > > > +1 from me.
> > > > > > St.Ack
> > > > > >
> > > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > > > > >
> > > > > > > This is an interesting vote. The vote is on an RC made out of a
> > new
> > > > > hbase
> > > > > > > project, hbase-thirdparty.
> > > > > > >
> > > > > > > First the usual stuff.
> > > > > > >
> > > > > > > The 1.0.0RC0 is available at:
> > > > > > >
> > > > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > > > thirdparty/1.0.0RC0/
> > > > > > >
> > > > > > > Maven artifacts are in the following staging repository:
> > > > > > >
> > > > > > >  https://repository.apache.org/content/repositories/
> > > > > orgapachehbase-1171
> > > > > > >
> > > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our
> > KEYS
> > > > > file
> > > > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > > > >
> > > > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > > > cdf9e224f3
> > > > > > > in the new hbase-thirdparty repository at [1].
> > > > > > >
> > > > > > > Now to the unusual.
> > > > > > >
> > > > > > > The artifact was made from a new repository,
> hbase-thirdparty[1].
> > > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It
> came
> > of
> > > > > > > discussion on the dev list in the thread '[DISCUSS] More
> Shading'
> > > > [2].
> > > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> > > protobuf,
> > > > > > > guava, netty -- that need patching and/or shading so we are no
> > > longer
> > > > > > bound
> > > > > > > by whatever the version of a lib Hadoop happens to ship (and we
> > can
> > > > > > change
> > > > > > > versions w/o damaging out glorious downstreamers). In the past,
> > > when
> > > > > > we've
> > > > > > > been able, we've done this dirty patching and shading work as
> an
> > > > > > > unacknowledged 'pre-build' hack step appended to our protobuf
> > file
> > > > > > > generation. Moving this mess out of our mainline build to live
> > > > > elsewhere,
> > > > > > > pulled in as a legit 'dependency', simplifies our build
> > > particularly
> > > > > > around
> > > > > > > the shading of third-part libs; the narrower scope of a
> dedicated
> > > > > project
> > > > > > > that produces transformed jars is easier to reason about, is
> less
> > > > > likely
> > > > > > to
> > > > > > > clash with existing dependencies whether transitive or
> otherwise,
> > > and
> > > > > it
> > > > > > > makes for less spaghetti poms.
> > > > > > >
> > > > > > > Other (minor) benefits include our being able to generate
> > protobuf
> > > > > files
> > > > > > > inline now; no need of the pre-build anymore nor checking in of
> > > > > generated
> > > > > > > protobuf files (25MB).  There is a patch to remove them
> > (predicated
> > > > on
> > > > > > this
> > > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > > > generated
> > > > > > > files".
> > > > > > >
> > > > > > > Downside is having to explicitly reference our version in code
> as
> > > in
> > > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> > rather
> > > > than
> > > > > > > just com.google.protobuf.Message and so on for guava
> references,
> > > etc.
> > > > > > > (We've been doing this for a while now so hopefully it not too
> > > > strange
> > > > > > > developing)*.
> > > > > > >
> > > > > > > There is little to this new project other than a few poms to do
> > > > > packaging
> > > > > > > and a couple of patches we apply on build. It works by pulling
> > down
> > > > > > > dependencies and doing whatever shading, renaming, or patching
> we
> > > > need.
> > > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1),
> guava
> > > > (12.0
> > > > > > =>
> > > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It
> publishes
> > > > three
> > > > > > jars
> > > > > > > for hbase to consume: hbase-shaded-protobuf,
> hbase-shaded-netty,
> > > and
> > > > > then
> > > > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy
> is
> > if
> > > > the
> > > > > > lib
> > > > > > > is troublesome, protobuf needs patching and netty includes an
> .so
> > > > that
> > > > > > > needs a rename, then give it its own module (See README in the
> > > > release
> > > > > > for
> > > > > > > more). I've arrived at this layout for hbase-thirdparty after a
> > > bunch
> > > > > of
> > > > > > > experimentation and messing with hbase dependencies (See
> > sub-tasks
> > > > > under
> > > > > > > HBASE-17898 Update dependencies). That said, as we head to
> > > > hbase-2.0.0,
> > > > > > we
> > > > > > > might have to make a an odd new release of hbase-thirdparty but
> > > > > > thereafter
> > > > > > > I'd imagine it should settle down to be an infrequent update.
> > > > > > >
> > > > > > > VOTE lasts 72 hours. Let +1 be an assent to the
> hbase-thirdparty
> > > > > project
> > > > > > > and a vote for this RC.
> > > > > > >
> > > > > > > Questions on how it will go w/ this project welcome on this
> > thread.
> > > > If
> > > > > > > heavy-detail on how the shading works, I suggest ask on the
> > > > '[DISCUSS]
> > > > > > More
> > > > > > > Shading' thread.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > St.Ack
> > > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > > > > 2. See the '[DISCUSS] More Shading' thread,
> > > > > > http://apache-hbase.679495.n3.
> > > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > > > > >
> > > > > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > > > > refguide.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Stack <st...@duboce.net>.

On Sun, Jul 2, 2017 at 11:44 PM, Yu Li <ca...@gmail.com> wrote:
>
> bq. No plans to up guava or protobuf versions on branch-1. You think we
> should?
> We met with guava conflict when using hbase in Flink, and I could see
> similar issue mentioned in spark. Since branch-1 is still the stable
> release line and will be in the near future, possible to do similar shading
> w/o upgrading the guava/protobuf version? Thanks.
>
> In branch-1, relocate all probobuf and guava references?

Thanks Yu Li,
S


>
> Best Regards,
> Yu
>
> On 3 July 2017 at 14:01, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> > Then it will be a bit pain if developers want to see the source code of
> the
> > relocated classes in IDE?
> >
> > 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
> >
> > > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Checked sums and signatures: OK
> > > > 'mvn clean install': OK, the size of the generated jar files are
> > > reasonable
> > > > Unzip the generated jar files: OK, all classes are relocated under
> > > > org/apache/hadoop/hbase/shaded
> > > >
> > > > Only one minor question, seems we do not generate sources jars? There
> > is
> > > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but
> the
> > > size
> > > > is only 22 bytes.
> > > >
> > > >
> > > Good question. This project is a little bit odd. It is just poms (and a
> > few
> > > patches). The poms do a download, relocation, and then push to deploy
> the
> > > jars to mvn repo. There is no real 'source' -- not anything that makes
> > > sense to stuff in a src.jar --  other than the poms that are here in
> the
> > > tgz (mvn install puts the jars w/ relocation into your local repo....
> > add a
> > > 'deploy' to push to mvn repository -- see README).
> > >
> > > Thanks Duo,
> > > S
> > >
> > >
> > >
> > > > Thanks.
> > > >
> > > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> > > >
> > > > > +1 from me.
> > > > > St.Ack
> > > > >
> > > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > This is an interesting vote. The vote is on an RC made out of a
> new
> > > > hbase
> > > > > > project, hbase-thirdparty.
> > > > > >
> > > > > > First the usual stuff.
> > > > > >
> > > > > > The 1.0.0RC0 is available at:
> > > > > >
> > > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > > thirdparty/1.0.0RC0/
> > > > > >
> > > > > > Maven artifacts are in the following staging repository:
> > > > > >
> > > > > >  https://repository.apache.org/content/repositories/
> > > > orgapachehbase-1171
> > > > > >
> > > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our
> KEYS
> > > > file
> > > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > > >
> > > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > > cdf9e224f3
> > > > > > in the new hbase-thirdparty repository at [1].
> > > > > >
> > > > > > Now to the unusual.
> > > > > >
> > > > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > > > hbase-thirdparty is a project overseen by the hbase PMC. It came
> of
> > > > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> > > [2].
> > > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> > protobuf,
> > > > > > guava, netty -- that need patching and/or shading so we are no
> > longer
> > > > > bound
> > > > > > by whatever the version of a lib Hadoop happens to ship (and we
> can
> > > > > change
> > > > > > versions w/o damaging out glorious downstreamers). In the past,
> > when
> > > > > we've
> > > > > > been able, we've done this dirty patching and shading work as an
> > > > > > unacknowledged 'pre-build' hack step appended to our protobuf
> file
> > > > > > generation. Moving this mess out of our mainline build to live
> > > > elsewhere,
> > > > > > pulled in as a legit 'dependency', simplifies our build
> > particularly
> > > > > around
> > > > > > the shading of third-part libs; the narrower scope of a dedicated
> > > > project
> > > > > > that produces transformed jars is easier to reason about, is less
> > > > likely
> > > > > to
> > > > > > clash with existing dependencies whether transitive or otherwise,
> > and
> > > > it
> > > > > > makes for less spaghetti poms.
> > > > > >
> > > > > > Other (minor) benefits include our being able to generate
> protobuf
> > > > files
> > > > > > inline now; no need of the pre-build anymore nor checking in of
> > > > generated
> > > > > > protobuf files (25MB).  There is a patch to remove them
> (predicated
> > > on
> > > > > this
> > > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > > generated
> > > > > > files".
> > > > > >
> > > > > > Downside is having to explicitly reference our version in code as
> > in
> > > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> rather
> > > than
> > > > > > just com.google.protobuf.Message and so on for guava references,
> > etc.
> > > > > > (We've been doing this for a while now so hopefully it not too
> > > strange
> > > > > > developing)*.
> > > > > >
> > > > > > There is little to this new project other than a few poms to do
> > > > packaging
> > > > > > and a couple of patches we apply on build. It works by pulling
> down
> > > > > > dependencies and doing whatever shading, renaming, or patching we
> > > need.
> > > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> > > (12.0
> > > > > =>
> > > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes
> > > three
> > > > > jars
> > > > > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty,
> > and
> > > > then
> > > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is
> if
> > > the
> > > > > lib
> > > > > > is troublesome, protobuf needs patching and netty includes an .so
> > > that
> > > > > > needs a rename, then give it its own module (See README in the
> > > release
> > > > > for
> > > > > > more). I've arrived at this layout for hbase-thirdparty after a
> > bunch
> > > > of
> > > > > > experimentation and messing with hbase dependencies (See
> sub-tasks
> > > > under
> > > > > > HBASE-17898 Update dependencies). That said, as we head to
> > > hbase-2.0.0,
> > > > > we
> > > > > > might have to make a an odd new release of hbase-thirdparty but
> > > > > thereafter
> > > > > > I'd imagine it should settle down to be an infrequent update.
> > > > > >
> > > > > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> > > > project
> > > > > > and a vote for this RC.
> > > > > >
> > > > > > Questions on how it will go w/ this project welcome on this
> thread.
> > > If
> > > > > > heavy-detail on how the shading works, I suggest ask on the
> > > '[DISCUSS]
> > > > > More
> > > > > > Shading' thread.
> > > > > >
> > > > > > Thanks,
> > > > > > St.Ack
> > > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > > > 2. See the '[DISCUSS] More Shading' thread,
> > > > > http://apache-hbase.679495.n3.
> > > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > > > >
> > > > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > > > refguide.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Yu Li <ca...@gmail.com>.

bq. I could try w/ a 3.0.x for future release if it a pain?
Never mind boss, I just used 3.3.1 in my local env and found it requires
newer version, so left a note for new voters to save some of their time
(smile).

bq. No plans to up guava or protobuf versions on branch-1. You think we
should?
We met with guava conflict when using hbase in Flink, and I could see
similar issue mentioned in spark. Since branch-1 is still the stable
release line and will be in the near future, possible to do similar shading
w/o upgrading the guava/protobuf version? Thanks.


Best Regards,
Yu

On 3 July 2017 at 14:01, 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Then it will be a bit pain if developers want to see the source code of the
> relocated classes in IDE?
>
> 2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:
>
> > On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Checked sums and signatures: OK
> > > 'mvn clean install': OK, the size of the generated jar files are
> > reasonable
> > > Unzip the generated jar files: OK, all classes are relocated under
> > > org/apache/hadoop/hbase/shaded
> > >
> > > Only one minor question, seems we do not generate sources jars? There
> is
> > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the
> > size
> > > is only 22 bytes.
> > >
> > >
> > Good question. This project is a little bit odd. It is just poms (and a
> few
> > patches). The poms do a download, relocation, and then push to deploy the
> > jars to mvn repo. There is no real 'source' -- not anything that makes
> > sense to stuff in a src.jar --  other than the poms that are here in the
> > tgz (mvn install puts the jars w/ relocation into your local repo....
> add a
> > 'deploy' to push to mvn repository -- see README).
> >
> > Thanks Duo,
> > S
> >
> >
> >
> > > Thanks.
> > >
> > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > > > +1 from me.
> > > > St.Ack
> > > >
> > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > This is an interesting vote. The vote is on an RC made out of a new
> > > hbase
> > > > > project, hbase-thirdparty.
> > > > >
> > > > > First the usual stuff.
> > > > >
> > > > > The 1.0.0RC0 is available at:
> > > > >
> > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > thirdparty/1.0.0RC0/
> > > > >
> > > > > Maven artifacts are in the following staging repository:
> > > > >
> > > > >  https://repository.apache.org/content/repositories/
> > > orgapachehbase-1171
> > > > >
> > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> > > file
> > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > >
> > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > cdf9e224f3
> > > > > in the new hbase-thirdparty repository at [1].
> > > > >
> > > > > Now to the unusual.
> > > > >
> > > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> > [2].
> > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> protobuf,
> > > > > guava, netty -- that need patching and/or shading so we are no
> longer
> > > > bound
> > > > > by whatever the version of a lib Hadoop happens to ship (and we can
> > > > change
> > > > > versions w/o damaging out glorious downstreamers). In the past,
> when
> > > > we've
> > > > > been able, we've done this dirty patching and shading work as an
> > > > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > > > generation. Moving this mess out of our mainline build to live
> > > elsewhere,
> > > > > pulled in as a legit 'dependency', simplifies our build
> particularly
> > > > around
> > > > > the shading of third-part libs; the narrower scope of a dedicated
> > > project
> > > > > that produces transformed jars is easier to reason about, is less
> > > likely
> > > > to
> > > > > clash with existing dependencies whether transitive or otherwise,
> and
> > > it
> > > > > makes for less spaghetti poms.
> > > > >
> > > > > Other (minor) benefits include our being able to generate protobuf
> > > files
> > > > > inline now; no need of the pre-build anymore nor checking in of
> > > generated
> > > > > protobuf files (25MB).  There is a patch to remove them (predicated
> > on
> > > > this
> > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > generated
> > > > > files".
> > > > >
> > > > > Downside is having to explicitly reference our version in code as
> in
> > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather
> > than
> > > > > just com.google.protobuf.Message and so on for guava references,
> etc.
> > > > > (We've been doing this for a while now so hopefully it not too
> > strange
> > > > > developing)*.
> > > > >
> > > > > There is little to this new project other than a few poms to do
> > > packaging
> > > > > and a couple of patches we apply on build. It works by pulling down
> > > > > dependencies and doing whatever shading, renaming, or patching we
> > need.
> > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> > (12.0
> > > > =>
> > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes
> > three
> > > > jars
> > > > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty,
> and
> > > then
> > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if
> > the
> > > > lib
> > > > > is troublesome, protobuf needs patching and netty includes an .so
> > that
> > > > > needs a rename, then give it its own module (See README in the
> > release
> > > > for
> > > > > more). I've arrived at this layout for hbase-thirdparty after a
> bunch
> > > of
> > > > > experimentation and messing with hbase dependencies (See sub-tasks
> > > under
> > > > > HBASE-17898 Update dependencies). That said, as we head to
> > hbase-2.0.0,
> > > > we
> > > > > might have to make a an odd new release of hbase-thirdparty but
> > > > thereafter
> > > > > I'd imagine it should settle down to be an infrequent update.
> > > > >
> > > > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> > > project
> > > > > and a vote for this RC.
> > > > >
> > > > > Questions on how it will go w/ this project welcome on this thread.
> > If
> > > > > heavy-detail on how the shading works, I suggest ask on the
> > '[DISCUSS]
> > > > More
> > > > > Shading' thread.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > > 2. See the '[DISCUSS] More Shading' thread,
> > > > http://apache-hbase.679495.n3.
> > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > > >
> > > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > > refguide.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.

Then it will be a bit pain if developers want to see the source code of the
relocated classes in IDE?

2017-07-03 12:31 GMT+08:00 Stack <st...@duboce.net>:

> On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
> > +1 (binding)
> >
> > Checked sums and signatures: OK
> > 'mvn clean install': OK, the size of the generated jar files are
> reasonable
> > Unzip the generated jar files: OK, all classes are relocated under
> > org/apache/hadoop/hbase/shaded
> >
> > Only one minor question, seems we do not generate sources jars? There is
> > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the
> size
> > is only 22 bytes.
> >
> >
> Good question. This project is a little bit odd. It is just poms (and a few
> patches). The poms do a download, relocation, and then push to deploy the
> jars to mvn repo. There is no real 'source' -- not anything that makes
> sense to stuff in a src.jar --  other than the poms that are here in the
> tgz (mvn install puts the jars w/ relocation into your local repo.... add a
> 'deploy' to push to mvn repository -- see README).
>
> Thanks Duo,
> S
>
>
>
> > Thanks.
> >
> > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> >
> > > +1 from me.
> > > St.Ack
> > >
> > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > This is an interesting vote. The vote is on an RC made out of a new
> > hbase
> > > > project, hbase-thirdparty.
> > > >
> > > > First the usual stuff.
> > > >
> > > > The 1.0.0RC0 is available at:
> > > >
> > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > thirdparty/1.0.0RC0/
> > > >
> > > > Maven artifacts are in the following staging repository:
> > > >
> > > >  https://repository.apache.org/content/repositories/
> > orgapachehbase-1171
> > > >
> > > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> > file
> > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > >
> > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > cdf9e224f3
> > > > in the new hbase-thirdparty repository at [1].
> > > >
> > > > Now to the unusual.
> > > >
> > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> [2].
> > > > Its charge is the hosting of ornery thirdparty libs -- e.g. protobuf,
> > > > guava, netty -- that need patching and/or shading so we are no longer
> > > bound
> > > > by whatever the version of a lib Hadoop happens to ship (and we can
> > > change
> > > > versions w/o damaging out glorious downstreamers). In the past, when
> > > we've
> > > > been able, we've done this dirty patching and shading work as an
> > > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > > generation. Moving this mess out of our mainline build to live
> > elsewhere,
> > > > pulled in as a legit 'dependency', simplifies our build particularly
> > > around
> > > > the shading of third-part libs; the narrower scope of a dedicated
> > project
> > > > that produces transformed jars is easier to reason about, is less
> > likely
> > > to
> > > > clash with existing dependencies whether transitive or otherwise, and
> > it
> > > > makes for less spaghetti poms.
> > > >
> > > > Other (minor) benefits include our being able to generate protobuf
> > files
> > > > inline now; no need of the pre-build anymore nor checking in of
> > generated
> > > > protobuf files (25MB).  There is a patch to remove them (predicated
> on
> > > this
> > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> generated
> > > > files".
> > > >
> > > > Downside is having to explicitly reference our version in code as in
> > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather
> than
> > > > just com.google.protobuf.Message and so on for guava references, etc.
> > > > (We've been doing this for a while now so hopefully it not too
> strange
> > > > developing)*.
> > > >
> > > > There is little to this new project other than a few poms to do
> > packaging
> > > > and a couple of patches we apply on build. It works by pulling down
> > > > dependencies and doing whatever shading, renaming, or patching we
> need.
> > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> (12.0
> > > =>
> > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes
> three
> > > jars
> > > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and
> > then
> > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if
> the
> > > lib
> > > > is troublesome, protobuf needs patching and netty includes an .so
> that
> > > > needs a rename, then give it its own module (See README in the
> release
> > > for
> > > > more). I've arrived at this layout for hbase-thirdparty after a bunch
> > of
> > > > experimentation and messing with hbase dependencies (See sub-tasks
> > under
> > > > HBASE-17898 Update dependencies). That said, as we head to
> hbase-2.0.0,
> > > we
> > > > might have to make a an odd new release of hbase-thirdparty but
> > > thereafter
> > > > I'd imagine it should settle down to be an infrequent update.
> > > >
> > > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> > project
> > > > and a vote for this RC.
> > > >
> > > > Questions on how it will go w/ this project welcome on this thread.
> If
> > > > heavy-detail on how the shading works, I suggest ask on the
> '[DISCUSS]
> > > More
> > > > Shading' thread.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > 2. See the '[DISCUSS] More Shading' thread,
> > > http://apache-hbase.679495.n3.
> > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > >
> > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > refguide.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Stack <st...@duboce.net>.

On Sat, Jul 1, 2017 at 8:16 PM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> +1 (binding)
>
> Checked sums and signatures: OK
> 'mvn clean install': OK, the size of the generated jar files are reasonable
> Unzip the generated jar files: OK, all classes are relocated under
> org/apache/hadoop/hbase/shaded
>
> Only one minor question, seems we do not generate sources jars? There is
> a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the size
> is only 22 bytes.
>
>
Good question. This project is a little bit odd. It is just poms (and a few
patches). The poms do a download, relocation, and then push to deploy the
jars to mvn repo. There is no real 'source' -- not anything that makes
sense to stuff in a src.jar --  other than the poms that are here in the
tgz (mvn install puts the jars w/ relocation into your local repo.... add a
'deploy' to push to mvn repository -- see README).

Thanks Duo,
S



> Thanks.
>
> 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
>
> > +1 from me.
> > St.Ack
> >
> > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> >
> > > This is an interesting vote. The vote is on an RC made out of a new
> hbase
> > > project, hbase-thirdparty.
> > >
> > > First the usual stuff.
> > >
> > > The 1.0.0RC0 is available at:
> > >
> > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> thirdparty/1.0.0RC0/
> > >
> > > Maven artifacts are in the following staging repository:
> > >
> > >  https://repository.apache.org/content/repositories/
> orgapachehbase-1171
> > >
> > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> file
> > > http://www-us.apache.org/dist/hbase/KEYS.
> > >
> > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> cdf9e224f3
> > > in the new hbase-thirdparty repository at [1].
> > >
> > > Now to the unusual.
> > >
> > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > discussion on the dev list in the thread '[DISCUSS] More Shading' [2].
> > > Its charge is the hosting of ornery thirdparty libs -- e.g. protobuf,
> > > guava, netty -- that need patching and/or shading so we are no longer
> > bound
> > > by whatever the version of a lib Hadoop happens to ship (and we can
> > change
> > > versions w/o damaging out glorious downstreamers). In the past, when
> > we've
> > > been able, we've done this dirty patching and shading work as an
> > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > generation. Moving this mess out of our mainline build to live
> elsewhere,
> > > pulled in as a legit 'dependency', simplifies our build particularly
> > around
> > > the shading of third-part libs; the narrower scope of a dedicated
> project
> > > that produces transformed jars is easier to reason about, is less
> likely
> > to
> > > clash with existing dependencies whether transitive or otherwise, and
> it
> > > makes for less spaghetti poms.
> > >
> > > Other (minor) benefits include our being able to generate protobuf
> files
> > > inline now; no need of the pre-build anymore nor checking in of
> generated
> > > protobuf files (25MB).  There is a patch to remove them (predicated on
> > this
> > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB generated
> > > files".
> > >
> > > Downside is having to explicitly reference our version in code as in
> > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather than
> > > just com.google.protobuf.Message and so on for guava references, etc.
> > > (We've been doing this for a while now so hopefully it not too strange
> > > developing)*.
> > >
> > > There is little to this new project other than a few poms to do
> packaging
> > > and a couple of patches we apply on build. It works by pulling down
> > > dependencies and doing whatever shading, renaming, or patching we need.
> > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava (12.0
> > =>
> > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes three
> > jars
> > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and
> then
> > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if the
> > lib
> > > is troublesome, protobuf needs patching and netty includes an .so that
> > > needs a rename, then give it its own module (See README in the release
> > for
> > > more). I've arrived at this layout for hbase-thirdparty after a bunch
> of
> > > experimentation and messing with hbase dependencies (See sub-tasks
> under
> > > HBASE-17898 Update dependencies). That said, as we head to hbase-2.0.0,
> > we
> > > might have to make a an odd new release of hbase-thirdparty but
> > thereafter
> > > I'd imagine it should settle down to be an infrequent update.
> > >
> > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> project
> > > and a vote for this RC.
> > >
> > > Questions on how it will go w/ this project welcome on this thread. If
> > > heavy-detail on how the shading works, I suggest ask on the '[DISCUSS]
> > More
> > > Shading' thread.
> > >
> > > Thanks,
> > > St.Ack
> > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > 2. See the '[DISCUSS] More Shading' thread,
> > http://apache-hbase.679495.n3.
> > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > >
> > > * Doc on how to make use of hbase-thirdparty is coming for the
> refguide.
> > >
> > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by ramkrishna vasudevan <ra...@gmail.com>.

+1.
downloaded the src tar ball.
did 'mvn clean install' - Works fine.

It asks for maven greater than 3.3. But Hbase trunk compiles with maven
3.1.0. So may be unifying both will be better in this case.

Regards
Ram

On Mon, Jul 3, 2017 at 10:03 AM, Stack <st...@duboce.net> wrote:

> On Sun, Jul 2, 2017 at 9:17 PM, Yu Li <ca...@gmail.com> wrote:
>
> > +1
> >
> > Checked sums and signatures: ok
> > 'mvn install' succeeds: ok
> > Checked generated jar files: ok
> >
> > Note: it requires maven version > 3.3.3 to compile
> >
> >
> Yeah. I picked an arbitrary version. Older versions would probably work but
> 3.3.3 is old now. Figured it would be ok. I could try w/ a 3.0.x for future
> release if it a pain?
>
>
> > btw, is this only for branch-2 or branch-1 will also follow? Thanks.
> >
> >
> Plan is for hbase2 only currently. No plans to up guava or protobuf
> versions on branch-1. You think we should?
>
> Thanks for voting Yu Li.
>
> St.Ack
>
>
> > Best Regards,
> > Yu
> >
> > On 2 July 2017 at 11:16, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > > +1 (binding)
> > >
> > > Checked sums and signatures: OK
> > > 'mvn clean install': OK, the size of the generated jar files are
> > reasonable
> > > Unzip the generated jar files: OK, all classes are relocated under
> > > org/apache/hadoop/hbase/shaded
> > >
> > > Only one minor question, seems we do not generate sources jars? There
> is
> > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the
> > size
> > > is only 22 bytes.
> > >
> > > Thanks.
> > >
> > > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > > > +1 from me.
> > > > St.Ack
> > > >
> > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > This is an interesting vote. The vote is on an RC made out of a new
> > > hbase
> > > > > project, hbase-thirdparty.
> > > > >
> > > > > First the usual stuff.
> > > > >
> > > > > The 1.0.0RC0 is available at:
> > > > >
> > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > thirdparty/1.0.0RC0/
> > > > >
> > > > > Maven artifacts are in the following staging repository:
> > > > >
> > > > >  https://repository.apache.org/content/repositories/
> > > orgapachehbase-1171
> > > > >
> > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> > > file
> > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > >
> > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > cdf9e224f3
> > > > > in the new hbase-thirdparty repository at [1].
> > > > >
> > > > > Now to the unusual.
> > > > >
> > > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> > [2].
> > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> protobuf,
> > > > > guava, netty -- that need patching and/or shading so we are no
> longer
> > > > bound
> > > > > by whatever the version of a lib Hadoop happens to ship (and we can
> > > > change
> > > > > versions w/o damaging out glorious downstreamers). In the past,
> when
> > > > we've
> > > > > been able, we've done this dirty patching and shading work as an
> > > > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > > > generation. Moving this mess out of our mainline build to live
> > > elsewhere,
> > > > > pulled in as a legit 'dependency', simplifies our build
> particularly
> > > > around
> > > > > the shading of third-part libs; the narrower scope of a dedicated
> > > project
> > > > > that produces transformed jars is easier to reason about, is less
> > > likely
> > > > to
> > > > > clash with existing dependencies whether transitive or otherwise,
> and
> > > it
> > > > > makes for less spaghetti poms.
> > > > >
> > > > > Other (minor) benefits include our being able to generate protobuf
> > > files
> > > > > inline now; no need of the pre-build anymore nor checking in of
> > > generated
> > > > > protobuf files (25MB).  There is a patch to remove them (predicated
> > on
> > > > this
> > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > generated
> > > > > files".
> > > > >
> > > > > Downside is having to explicitly reference our version in code as
> in
> > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather
> > than
> > > > > just com.google.protobuf.Message and so on for guava references,
> etc.
> > > > > (We've been doing this for a while now so hopefully it not too
> > strange
> > > > > developing)*.
> > > > >
> > > > > There is little to this new project other than a few poms to do
> > > packaging
> > > > > and a couple of patches we apply on build. It works by pulling down
> > > > > dependencies and doing whatever shading, renaming, or patching we
> > need.
> > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> > (12.0
> > > > =>
> > > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes
> > three
> > > > jars
> > > > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty,
> and
> > > then
> > > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if
> > the
> > > > lib
> > > > > is troublesome, protobuf needs patching and netty includes an .so
> > that
> > > > > needs a rename, then give it its own module (See README in the
> > release
> > > > for
> > > > > more). I've arrived at this layout for hbase-thirdparty after a
> bunch
> > > of
> > > > > experimentation and messing with hbase dependencies (See sub-tasks
> > > under
> > > > > HBASE-17898 Update dependencies). That said, as we head to
> > hbase-2.0.0,
> > > > we
> > > > > might have to make a an odd new release of hbase-thirdparty but
> > > > thereafter
> > > > > I'd imagine it should settle down to be an infrequent update.
> > > > >
> > > > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> > > project
> > > > > and a vote for this RC.
> > > > >
> > > > > Questions on how it will go w/ this project welcome on this thread.
> > If
> > > > > heavy-detail on how the shading works, I suggest ask on the
> > '[DISCUSS]
> > > > More
> > > > > Shading' thread.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > > 2. See the '[DISCUSS] More Shading' thread,
> > > > http://apache-hbase.679495.n3.
> > > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > > >
> > > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > > refguide.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Stack <st...@duboce.net>.

On Sun, Jul 2, 2017 at 9:17 PM, Yu Li <ca...@gmail.com> wrote:

> +1
>
> Checked sums and signatures: ok
> 'mvn install' succeeds: ok
> Checked generated jar files: ok
>
> Note: it requires maven version > 3.3.3 to compile
>
>
Yeah. I picked an arbitrary version. Older versions would probably work but
3.3.3 is old now. Figured it would be ok. I could try w/ a 3.0.x for future
release if it a pain?


> btw, is this only for branch-2 or branch-1 will also follow? Thanks.
>
>
Plan is for hbase2 only currently. No plans to up guava or protobuf
versions on branch-1. You think we should?

Thanks for voting Yu Li.

St.Ack


> Best Regards,
> Yu
>
> On 2 July 2017 at 11:16, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> > +1 (binding)
> >
> > Checked sums and signatures: OK
> > 'mvn clean install': OK, the size of the generated jar files are
> reasonable
> > Unzip the generated jar files: OK, all classes are relocated under
> > org/apache/hadoop/hbase/shaded
> >
> > Only one minor question, seems we do not generate sources jars? There is
> > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the
> size
> > is only 22 bytes.
> >
> > Thanks.
> >
> > 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
> >
> > > +1 from me.
> > > St.Ack
> > >
> > > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > This is an interesting vote. The vote is on an RC made out of a new
> > hbase
> > > > project, hbase-thirdparty.
> > > >
> > > > First the usual stuff.
> > > >
> > > > The 1.0.0RC0 is available at:
> > > >
> > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > thirdparty/1.0.0RC0/
> > > >
> > > > Maven artifacts are in the following staging repository:
> > > >
> > > >  https://repository.apache.org/content/repositories/
> > orgapachehbase-1171
> > > >
> > > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> > file
> > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > >
> > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > cdf9e224f3
> > > > in the new hbase-thirdparty repository at [1].
> > > >
> > > > Now to the unusual.
> > > >
> > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> [2].
> > > > Its charge is the hosting of ornery thirdparty libs -- e.g. protobuf,
> > > > guava, netty -- that need patching and/or shading so we are no longer
> > > bound
> > > > by whatever the version of a lib Hadoop happens to ship (and we can
> > > change
> > > > versions w/o damaging out glorious downstreamers). In the past, when
> > > we've
> > > > been able, we've done this dirty patching and shading work as an
> > > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > > generation. Moving this mess out of our mainline build to live
> > elsewhere,
> > > > pulled in as a legit 'dependency', simplifies our build particularly
> > > around
> > > > the shading of third-part libs; the narrower scope of a dedicated
> > project
> > > > that produces transformed jars is easier to reason about, is less
> > likely
> > > to
> > > > clash with existing dependencies whether transitive or otherwise, and
> > it
> > > > makes for less spaghetti poms.
> > > >
> > > > Other (minor) benefits include our being able to generate protobuf
> > files
> > > > inline now; no need of the pre-build anymore nor checking in of
> > generated
> > > > protobuf files (25MB).  There is a patch to remove them (predicated
> on
> > > this
> > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> generated
> > > > files".
> > > >
> > > > Downside is having to explicitly reference our version in code as in
> > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather
> than
> > > > just com.google.protobuf.Message and so on for guava references, etc.
> > > > (We've been doing this for a while now so hopefully it not too
> strange
> > > > developing)*.
> > > >
> > > > There is little to this new project other than a few poms to do
> > packaging
> > > > and a couple of patches we apply on build. It works by pulling down
> > > > dependencies and doing whatever shading, renaming, or patching we
> need.
> > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> (12.0
> > > =>
> > > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes
> three
> > > jars
> > > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and
> > then
> > > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if
> the
> > > lib
> > > > is troublesome, protobuf needs patching and netty includes an .so
> that
> > > > needs a rename, then give it its own module (See README in the
> release
> > > for
> > > > more). I've arrived at this layout for hbase-thirdparty after a bunch
> > of
> > > > experimentation and messing with hbase dependencies (See sub-tasks
> > under
> > > > HBASE-17898 Update dependencies). That said, as we head to
> hbase-2.0.0,
> > > we
> > > > might have to make a an odd new release of hbase-thirdparty but
> > > thereafter
> > > > I'd imagine it should settle down to be an infrequent update.
> > > >
> > > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> > project
> > > > and a vote for this RC.
> > > >
> > > > Questions on how it will go w/ this project welcome on this thread.
> If
> > > > heavy-detail on how the shading works, I suggest ask on the
> '[DISCUSS]
> > > More
> > > > Shading' thread.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > > 2. See the '[DISCUSS] More Shading' thread,
> > > http://apache-hbase.679495.n3.
> > > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > > >
> > > > * Doc on how to make use of hbase-thirdparty is coming for the
> > refguide.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Yu Li <ca...@gmail.com>.

+1

Checked sums and signatures: ok
'mvn install' succeeds: ok
Checked generated jar files: ok

Note: it requires maven version > 3.3.3 to compile

btw, is this only for branch-2 or branch-1 will also follow? Thanks.

Best Regards,
Yu

On 2 July 2017 at 11:16, 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> +1 (binding)
>
> Checked sums and signatures: OK
> 'mvn clean install': OK, the size of the generated jar files are reasonable
> Unzip the generated jar files: OK, all classes are relocated under
> org/apache/hadoop/hbase/shaded
>
> Only one minor question, seems we do not generate sources jars? There is
> a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the size
> is only 22 bytes.
>
> Thanks.
>
> 2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:
>
> > +1 from me.
> > St.Ack
> >
> > On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
> >
> > > This is an interesting vote. The vote is on an RC made out of a new
> hbase
> > > project, hbase-thirdparty.
> > >
> > > First the usual stuff.
> > >
> > > The 1.0.0RC0 is available at:
> > >
> > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> thirdparty/1.0.0RC0/
> > >
> > > Maven artifacts are in the following staging repository:
> > >
> > >  https://repository.apache.org/content/repositories/
> orgapachehbase-1171
> > >
> > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> file
> > > http://www-us.apache.org/dist/hbase/KEYS.
> > >
> > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> cdf9e224f3
> > > in the new hbase-thirdparty repository at [1].
> > >
> > > Now to the unusual.
> > >
> > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > discussion on the dev list in the thread '[DISCUSS] More Shading' [2].
> > > Its charge is the hosting of ornery thirdparty libs -- e.g. protobuf,
> > > guava, netty -- that need patching and/or shading so we are no longer
> > bound
> > > by whatever the version of a lib Hadoop happens to ship (and we can
> > change
> > > versions w/o damaging out glorious downstreamers). In the past, when
> > we've
> > > been able, we've done this dirty patching and shading work as an
> > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > generation. Moving this mess out of our mainline build to live
> elsewhere,
> > > pulled in as a legit 'dependency', simplifies our build particularly
> > around
> > > the shading of third-part libs; the narrower scope of a dedicated
> project
> > > that produces transformed jars is easier to reason about, is less
> likely
> > to
> > > clash with existing dependencies whether transitive or otherwise, and
> it
> > > makes for less spaghetti poms.
> > >
> > > Other (minor) benefits include our being able to generate protobuf
> files
> > > inline now; no need of the pre-build anymore nor checking in of
> generated
> > > protobuf files (25MB).  There is a patch to remove them (predicated on
> > this
> > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB generated
> > > files".
> > >
> > > Downside is having to explicitly reference our version in code as in
> > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather than
> > > just com.google.protobuf.Message and so on for guava references, etc.
> > > (We've been doing this for a while now so hopefully it not too strange
> > > developing)*.
> > >
> > > There is little to this new project other than a few poms to do
> packaging
> > > and a couple of patches we apply on build. It works by pulling down
> > > dependencies and doing whatever shading, renaming, or patching we need.
> > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava (12.0
> > =>
> > > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes three
> > jars
> > > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and
> then
> > > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if the
> > lib
> > > is troublesome, protobuf needs patching and netty includes an .so that
> > > needs a rename, then give it its own module (See README in the release
> > for
> > > more). I've arrived at this layout for hbase-thirdparty after a bunch
> of
> > > experimentation and messing with hbase dependencies (See sub-tasks
> under
> > > HBASE-17898 Update dependencies). That said, as we head to hbase-2.0.0,
> > we
> > > might have to make a an odd new release of hbase-thirdparty but
> > thereafter
> > > I'd imagine it should settle down to be an infrequent update.
> > >
> > > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty
> project
> > > and a vote for this RC.
> > >
> > > Questions on how it will go w/ this project welcome on this thread. If
> > > heavy-detail on how the shading works, I suggest ask on the '[DISCUSS]
> > More
> > > Shading' thread.
> > >
> > > Thanks,
> > > St.Ack
> > > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > > 2. See the '[DISCUSS] More Shading' thread,
> > http://apache-hbase.679495.n3.
> > > nabble.com/DISCUSS-More-Shading-td4083025.html
> > >
> > > * Doc on how to make use of hbase-thirdparty is coming for the
> refguide.
> > >
> > >
> > >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.

+1 (binding)

Checked sums and signatures: OK
'mvn clean install': OK, the size of the generated jar files are reasonable
Unzip the generated jar files: OK, all classes are relocated under
org/apache/hadoop/hbase/shaded

Only one minor question, seems we do not generate sources jars? There is
a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the size
is only 22 bytes.

Thanks.

2017-07-02 2:38 GMT+08:00 Stack <st...@duboce.net>:

> +1 from me.
> St.Ack
>
> On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:
>
> > This is an interesting vote. The vote is on an RC made out of a new hbase
> > project, hbase-thirdparty.
> >
> > First the usual stuff.
> >
> > The 1.0.0RC0 is available at:
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty/1.0.0RC0/
> >
> > Maven artifacts are in the following staging repository:
> >
> >  https://repository.apache.org/content/repositories/orgapachehbase-1171
> >
> > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS file
> > http://www-us.apache.org/dist/hbase/KEYS.
> >
> > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04cdf9e224f3
> > in the new hbase-thirdparty repository at [1].
> >
> > Now to the unusual.
> >
> > The artifact was made from a new repository, hbase-thirdparty[1].
> > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > discussion on the dev list in the thread '[DISCUSS] More Shading' [2].
> > Its charge is the hosting of ornery thirdparty libs -- e.g. protobuf,
> > guava, netty -- that need patching and/or shading so we are no longer
> bound
> > by whatever the version of a lib Hadoop happens to ship (and we can
> change
> > versions w/o damaging out glorious downstreamers). In the past, when
> we've
> > been able, we've done this dirty patching and shading work as an
> > unacknowledged 'pre-build' hack step appended to our protobuf file
> > generation. Moving this mess out of our mainline build to live elsewhere,
> > pulled in as a legit 'dependency', simplifies our build particularly
> around
> > the shading of third-part libs; the narrower scope of a dedicated project
> > that produces transformed jars is easier to reason about, is less likely
> to
> > clash with existing dependencies whether transitive or otherwise, and it
> > makes for less spaghetti poms.
> >
> > Other (minor) benefits include our being able to generate protobuf files
> > inline now; no need of the pre-build anymore nor checking in of generated
> > protobuf files (25MB).  There is a patch to remove them (predicated on
> this
> > hbase-thirdparty release): "HBASE-17056 Remove checked in PB generated
> > files".
> >
> > Downside is having to explicitly reference our version in code as in
> > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather than
> > just com.google.protobuf.Message and so on for guava references, etc.
> > (We've been doing this for a while now so hopefully it not too strange
> > developing)*.
> >
> > There is little to this new project other than a few poms to do packaging
> > and a couple of patches we apply on build. It works by pulling down
> > dependencies and doing whatever shading, renaming, or patching we need.
> > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava (12.0
> =>
> > 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes three
> jars
> > for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and then
> > hbase-shaded-miscellaneous for the remaining libs. Philosophy is if the
> lib
> > is troublesome, protobuf needs patching and netty includes an .so that
> > needs a rename, then give it its own module (See README in the release
> for
> > more). I've arrived at this layout for hbase-thirdparty after a bunch of
> > experimentation and messing with hbase dependencies (See sub-tasks under
> > HBASE-17898 Update dependencies). That said, as we head to hbase-2.0.0,
> we
> > might have to make a an odd new release of hbase-thirdparty but
> thereafter
> > I'd imagine it should settle down to be an infrequent update.
> >
> > VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty project
> > and a vote for this RC.
> >
> > Questions on how it will go w/ this project welcome on this thread. If
> > heavy-detail on how the shading works, I suggest ask on the '[DISCUSS]
> More
> > Shading' thread.
> >
> > Thanks,
> > St.Ack
> > 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> > 2. See the '[DISCUSS] More Shading' thread,
> http://apache-hbase.679495.n3.
> > nabble.com/DISCUSS-More-Shading-td4083025.html
> >
> > * Doc on how to make use of hbase-thirdparty is coming for the refguide.
> >
> >
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Stack <st...@duboce.net>.

+1 from me.
St.Ack

On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:

> This is an interesting vote. The vote is on an RC made out of a new hbase
> project, hbase-thirdparty.
>
> First the usual stuff.
>
> The 1.0.0RC0 is available at:
>
>  https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty/1.0.0RC0/
>
> Maven artifacts are in the following staging repository:
>
>  https://repository.apache.org/content/repositories/orgapachehbase-1171
>
> Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS file
> http://www-us.apache.org/dist/hbase/KEYS.
>
> I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04cdf9e224f3
> in the new hbase-thirdparty repository at [1].
>
> Now to the unusual.
>
> The artifact was made from a new repository, hbase-thirdparty[1].
> hbase-thirdparty is a project overseen by the hbase PMC. It came of
> discussion on the dev list in the thread '[DISCUSS] More Shading' [2].
> Its charge is the hosting of ornery thirdparty libs -- e.g. protobuf,
> guava, netty -- that need patching and/or shading so we are no longer bound
> by whatever the version of a lib Hadoop happens to ship (and we can change
> versions w/o damaging out glorious downstreamers). In the past, when we've
> been able, we've done this dirty patching and shading work as an
> unacknowledged 'pre-build' hack step appended to our protobuf file
> generation. Moving this mess out of our mainline build to live elsewhere,
> pulled in as a legit 'dependency', simplifies our build particularly around
> the shading of third-part libs; the narrower scope of a dedicated project
> that produces transformed jars is easier to reason about, is less likely to
> clash with existing dependencies whether transitive or otherwise, and it
> makes for less spaghetti poms.
>
> Other (minor) benefits include our being able to generate protobuf files
> inline now; no need of the pre-build anymore nor checking in of generated
> protobuf files (25MB).  There is a patch to remove them (predicated on this
> hbase-thirdparty release): "HBASE-17056 Remove checked in PB generated
> files".
>
> Downside is having to explicitly reference our version in code as in
> org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather than
> just com.google.protobuf.Message and so on for guava references, etc.
> (We've been doing this for a while now so hopefully it not too strange
> developing)*.
>
> There is little to this new project other than a few poms to do packaging
> and a couple of patches we apply on build. It works by pulling down
> dependencies and doing whatever shading, renaming, or patching we need.
> Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava (12.0 =>
> 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes three jars
> for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and then
> hbase-shaded-miscellaneous for the remaining libs. Philosophy is if the lib
> is troublesome, protobuf needs patching and netty includes an .so that
> needs a rename, then give it its own module (See README in the release for
> more). I've arrived at this layout for hbase-thirdparty after a bunch of
> experimentation and messing with hbase dependencies (See sub-tasks under
> HBASE-17898 Update dependencies). That said, as we head to hbase-2.0.0, we
> might have to make a an odd new release of hbase-thirdparty but thereafter
> I'd imagine it should settle down to be an infrequent update.
>
> VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty project
> and a vote for this RC.
>
> Questions on how it will go w/ this project welcome on this thread. If
> heavy-detail on how the shading works, I suggest ask on the '[DISCUSS] More
> Shading' thread.
>
> Thanks,
> St.Ack
> 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> 2. See the '[DISCUSS] More Shading' thread, http://apache-hbase.679495.n3.
> nabble.com/DISCUSS-More-Shading-td4083025.html
>
> * Doc on how to make use of hbase-thirdparty is coming for the refguide.
>
>
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

Posted by Andrew Purtell <ap...@apache.org>.

+1

Checked sums and signatures: ok
NOTICE and LICENSE looks good: ok
'mvn install' succeeds: ok


On Fri, Jun 30, 2017 at 3:07 PM, Stack <st...@duboce.net> wrote:

> This is an interesting vote. The vote is on an RC made out of a new hbase
> project, hbase-thirdparty.
>
> First the usual stuff.
>
> The 1.0.0RC0 is available at:
>
>  https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty/1.0.0RC0/
>
> Maven artifacts are in the following staging repository:
>
>  https://repository.apache.org/content/repositories/orgapachehbase-1171
>
> Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS file
> http://www-us.apache.org/dist/hbase/KEYS.
>
> I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04cdf9e224f3 in
> the new hbase-thirdparty repository at [1].
>
> Now to the unusual.
>
> The artifact was made from a new repository, hbase-thirdparty[1].
> hbase-thirdparty is a project overseen by the hbase PMC. It came of
> discussion on the dev list in the thread '[DISCUSS] More Shading' [2]. Its
> charge is the hosting of ornery thirdparty libs -- e.g. protobuf, guava,
> netty -- that need patching and/or shading so we are no longer bound by
> whatever the version of a lib Hadoop happens to ship (and we can change
> versions w/o damaging out glorious downstreamers). In the past, when we've
> been able, we've done this dirty patching and shading work as an
> unacknowledged 'pre-build' hack step appended to our protobuf file
> generation. Moving this mess out of our mainline build to live elsewhere,
> pulled in as a legit 'dependency', simplifies our build particularly around
> the shading of third-part libs; the narrower scope of a dedicated project
> that produces transformed jars is easier to reason about, is less likely to
> clash with existing dependencies whether transitive or otherwise, and it
> makes for less spaghetti poms.
>
> Other (minor) benefits include our being able to generate protobuf files
> inline now; no need of the pre-build anymore nor checking in of generated
> protobuf files (25MB).  There is a patch to remove them (predicated on this
> hbase-thirdparty release): "HBASE-17056 Remove checked in PB generated
> files".
>
> Downside is having to explicitly reference our version in code as in
> org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather than
> just
> com.google.protobuf.Message and so on for guava references, etc. (We've
> been doing this for a while now so hopefully it not too strange
> developing)*.
>
> There is little to this new project other than a few poms to do packaging
> and a couple of patches we apply on build. It works by pulling down
> dependencies and doing whatever shading, renaming, or patching we need.
> Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava (12.0 =>
> 22.0), gson, protobuf-util, and netty 4.1.12-FINAL. It publishes three jars
> for hbase to consume: hbase-shaded-protobuf, hbase-shaded-netty, and then
> hbase-shaded-miscellaneous for the remaining libs. Philosophy is if the lib
> is troublesome, protobuf needs patching and netty includes an .so that
> needs a rename, then give it its own module (See README in the release for
> more). I've arrived at this layout for hbase-thirdparty after a bunch of
> experimentation and messing with hbase dependencies (See sub-tasks under
> HBASE-17898 Update dependencies). That said, as we head to hbase-2.0.0, we
> might have to make a an odd new release of hbase-thirdparty but thereafter
> I'd imagine it should settle down to be an infrequent update.
>
> VOTE lasts 72 hours. Let +1 be an assent to the hbase-thirdparty project
> and a vote for this RC.
>
> Questions on how it will go w/ this project welcome on this thread. If
> heavy-detail on how the shading works, I suggest ask on the '[DISCUSS] More
> Shading' thread.
>
> Thanks,
> St.Ack
> 1. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty
> 2. See the '[DISCUSS] More Shading' thread,
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-
> Shading-td4083025.html
>
> * Doc on how to make use of hbase-thirdparty is coming for the refguide.
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk