You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Apekshit Sharma <ap...@cloudera.com> on 2017/10/25 21:56:54 UTC
Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Hi folks,
Found some discrepancies in moduleSet <include> list in src.xml and
hadoop-two-compat.xml. Got a crash course on hbase-assembly today by stack.
Throwing some larger questions here;
1. Do we want h-archetypes in binary tar?
2. Shouldn't we be building h-shaded, h-examples and h-archetypes before
h-assembly so that the corresponding jars get included in source/binary
tar? Here's the current build order :
....
Apache HBase - Archetypes .......................... SUCCESS [ 0.006 s]
Apache HBase - Assembly ............................ SUCCESS [ 0.281 s]
Apache HBase - Shaded .............................. SUCCESS [ 0.006 s]
Apache HBase - Shaded - Client ..................... SUCCESS [ 0.010 s]
Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.011 s]
Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.007 s]
Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.096 s]
Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 0.095
s]
Apache HBase - Archetype builder ................... SUCCESS [ 0.008 s]
-- Appy
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Sean Busbey <bu...@apache.org>.
We don't do both at once because the maven assembly plugin isn't very
good at making source tarballs. There's lots of ways for things to go
wrong and give us bad artifacts. most recent example, our 2.0.0-alpha3
release is ~100MiB because it includes shaded jars because prior
commands matter.
Personally, I'd rather update our release process to just use git
archive to create source tarballs off of RC tags.
For example, you can do this to make a source tarball from the
2.0.0-alpha-3 RC tag:
git archive --format=tar.gz
--output=/path/to/rc/artifacts/hbase-2.0.0-alpha-3.tar.gz
--prefix=hbase-2.0.0-alpha-3/ 2.0.0-alpha-3RC0.2
nice and simple. doesn't need a assembly descriptor; says right in the
command which tag the source will correspond to.
In fact, I think I'll do this for the next 1.2 release.
On Wed, Oct 25, 2017 at 6:27 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> Any reason why we don't build both together?
>
> On Wed, Oct 25, 2017 at 3:17 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
>
>> Thanks for the quick reply Busbey. Let me work on fixing the includes list
>> (HBASE-19089 <https://issues.apache.org/jira/browse/HBASE-19089>). Will
>> come back here if more questions arise.
>>
>> --Appy
>>
>> On Wed, Oct 25, 2017 at 3:03 PM, Sean Busbey <bu...@apache.org> wrote:
>>
>>> Thus far, the shaded jars and the archetypes have only been aimed at
>>> consumption during downstream build processes. Namely folks using maven to
>>> build an app.
>>>
>>> For those users, only being published in the Apache Nexus repo matters, so
>>> we deployed there (via the deploy step of our release process with release
>>> and apache-release maven profiles). We have not, thus far, included those
>>> jars in our binary tarball. Thus they aren't listed as dependencies of the
>>> assembly module and get built after it.
>>>
>>> Adding them would nearly double our binary tarball size, so I'm inclined
>>> to
>>> not include them without a compelling use case.
>>>
>>> The source tarball isn't made by a module, despite the descriptor living
>>> in
>>> the hbase-assembly module. It could just as easily be in dev-support. The
>>> step of our release process that creates the source tarball does a manual
>>> invocation of the maven assembly plug-in and points at the source
>>> descriptor directly.
>>>
>>> On Oct 25, 2017 4:57 PM, "Apekshit Sharma" <ap...@cloudera.com> wrote:
>>>
>>> Hi folks,
>>>
>>> Found some discrepancies in moduleSet <include> list in src.xml and
>>> hadoop-two-compat.xml. Got a crash course on hbase-assembly today by
>>> stack.
>>> Throwing some larger questions here;
>>>
>>> 1. Do we want h-archetypes in binary tar?
>>> 2. Shouldn't we be building h-shaded, h-examples and h-archetypes before
>>> h-assembly so that the corresponding jars get included in source/binary
>>> tar? Here's the current build order :
>>>
>>> ....
>>> Apache HBase - Archetypes .......................... SUCCESS [ 0.006 s]
>>> Apache HBase - Assembly ............................ SUCCESS [ 0.281 s]
>>> Apache HBase - Shaded .............................. SUCCESS [ 0.006 s]
>>> Apache HBase - Shaded - Client ..................... SUCCESS [ 0.010 s]
>>> Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.011 s]
>>> Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.007 s]
>>> Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.096 s]
>>> Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 0.095
>>> s]
>>> Apache HBase - Archetype builder ................... SUCCESS [ 0.008 s]
>>>
>>>
>>> -- Appy
>>>
>>
>>
>>
>> --
>>
>> -- Appy
>>
>
>
>
> --
>
> -- Appy
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Apekshit Sharma <ap...@cloudera.com>.
Any reason why we don't build both together?
On Wed, Oct 25, 2017 at 3:17 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> Thanks for the quick reply Busbey. Let me work on fixing the includes list
> (HBASE-19089 <https://issues.apache.org/jira/browse/HBASE-19089>). Will
> come back here if more questions arise.
>
> --Appy
>
> On Wed, Oct 25, 2017 at 3:03 PM, Sean Busbey <bu...@apache.org> wrote:
>
>> Thus far, the shaded jars and the archetypes have only been aimed at
>> consumption during downstream build processes. Namely folks using maven to
>> build an app.
>>
>> For those users, only being published in the Apache Nexus repo matters, so
>> we deployed there (via the deploy step of our release process with release
>> and apache-release maven profiles). We have not, thus far, included those
>> jars in our binary tarball. Thus they aren't listed as dependencies of the
>> assembly module and get built after it.
>>
>> Adding them would nearly double our binary tarball size, so I'm inclined
>> to
>> not include them without a compelling use case.
>>
>> The source tarball isn't made by a module, despite the descriptor living
>> in
>> the hbase-assembly module. It could just as easily be in dev-support. The
>> step of our release process that creates the source tarball does a manual
>> invocation of the maven assembly plug-in and points at the source
>> descriptor directly.
>>
>> On Oct 25, 2017 4:57 PM, "Apekshit Sharma" <ap...@cloudera.com> wrote:
>>
>> Hi folks,
>>
>> Found some discrepancies in moduleSet <include> list in src.xml and
>> hadoop-two-compat.xml. Got a crash course on hbase-assembly today by
>> stack.
>> Throwing some larger questions here;
>>
>> 1. Do we want h-archetypes in binary tar?
>> 2. Shouldn't we be building h-shaded, h-examples and h-archetypes before
>> h-assembly so that the corresponding jars get included in source/binary
>> tar? Here's the current build order :
>>
>> ....
>> Apache HBase - Archetypes .......................... SUCCESS [ 0.006 s]
>> Apache HBase - Assembly ............................ SUCCESS [ 0.281 s]
>> Apache HBase - Shaded .............................. SUCCESS [ 0.006 s]
>> Apache HBase - Shaded - Client ..................... SUCCESS [ 0.010 s]
>> Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.011 s]
>> Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.007 s]
>> Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.096 s]
>> Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 0.095
>> s]
>> Apache HBase - Archetype builder ................... SUCCESS [ 0.008 s]
>>
>>
>> -- Appy
>>
>
>
>
> --
>
> -- Appy
>
--
-- Appy
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Apekshit Sharma <ap...@cloudera.com>.
Thanks for the quick reply Busbey. Let me work on fixing the includes list (
HBASE-19089 <https://issues.apache.org/jira/browse/HBASE-19089>). Will come
back here if more questions arise.
--Appy
On Wed, Oct 25, 2017 at 3:03 PM, Sean Busbey <bu...@apache.org> wrote:
> Thus far, the shaded jars and the archetypes have only been aimed at
> consumption during downstream build processes. Namely folks using maven to
> build an app.
>
> For those users, only being published in the Apache Nexus repo matters, so
> we deployed there (via the deploy step of our release process with release
> and apache-release maven profiles). We have not, thus far, included those
> jars in our binary tarball. Thus they aren't listed as dependencies of the
> assembly module and get built after it.
>
> Adding them would nearly double our binary tarball size, so I'm inclined to
> not include them without a compelling use case.
>
> The source tarball isn't made by a module, despite the descriptor living in
> the hbase-assembly module. It could just as easily be in dev-support. The
> step of our release process that creates the source tarball does a manual
> invocation of the maven assembly plug-in and points at the source
> descriptor directly.
>
> On Oct 25, 2017 4:57 PM, "Apekshit Sharma" <ap...@cloudera.com> wrote:
>
> Hi folks,
>
> Found some discrepancies in moduleSet <include> list in src.xml and
> hadoop-two-compat.xml. Got a crash course on hbase-assembly today by stack.
> Throwing some larger questions here;
>
> 1. Do we want h-archetypes in binary tar?
> 2. Shouldn't we be building h-shaded, h-examples and h-archetypes before
> h-assembly so that the corresponding jars get included in source/binary
> tar? Here's the current build order :
>
> ....
> Apache HBase - Archetypes .......................... SUCCESS [ 0.006 s]
> Apache HBase - Assembly ............................ SUCCESS [ 0.281 s]
> Apache HBase - Shaded .............................. SUCCESS [ 0.006 s]
> Apache HBase - Shaded - Client ..................... SUCCESS [ 0.010 s]
> Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.011 s]
> Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.007 s]
> Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.096 s]
> Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 0.095
> s]
> Apache HBase - Archetype builder ................... SUCCESS [ 0.008 s]
>
>
> -- Appy
>
--
-- Appy
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Nick Dimiduk <nd...@gmail.com>.
IMHO, you go to the bin.tgz for a runnable executable. Everything else
(archetypes, client jars, sources) is pulled down from Maven Central
through dev tooling. I don't see a strong need for ops to receive the
latter in the bin.tgz. The only use I can think of is ignoring someone
wants to automate a complex mirror situation based on the content of a
single download. That's a bit of a stretch.
That said, maybe the Foundation mandates the completeness of a binary
convenience release?
On Wed, Oct 25, 2017 at 7:14 PM Stack <st...@duboce.net> wrote:
> On Wed, Oct 25, 2017 at 3:03 PM, Sean Busbey <bu...@apache.org> wrote:
>
> > Thus far, the shaded jars and the archetypes have only been aimed at
> > consumption during downstream build processes. Namely folks using maven
> to
> > build an app.
> >
> > For those users, only being published in the Apache Nexus repo matters,
> so
> > we deployed there (via the deploy step of our release process with
> release
> > and apache-release maven profiles). We have not, thus far, included those
> > jars in our binary tarball. Thus they aren't listed as dependencies of
> the
> > assembly module and get built after it.
> >
> > Adding them would nearly double our binary tarball size, so I'm inclined
> to
> > not include them without a compelling use case.
> >
> >
> Interesting. I'd have said our bin should have all our built product but
> yeah, as you say, archetypes depends on maven context and would make no
> sense in bin tgz.
>
> If we want to evangelize shaded client as primary access point, would that
> be justification bundling it in bin tgz?
>
>
>
> > The source tarball isn't made by a module, despite the descriptor living
> in
> > the hbase-assembly module. It could just as easily be in dev-support. The
> > step of our release process that creates the source tarball does a manual
> > invocation of the maven assembly plug-in and points at the source
> > descriptor directly.
> >
> >
> Our build could do w/ a revamp. It still has a form from when we used emit
> multiple products, src and a bin tarball for hadoop1 and hadoop2.
>
> St.Ack
>
>
>
>
> > On Oct 25, 2017 4:57 PM, "Apekshit Sharma" <ap...@cloudera.com> wrote:
> >
> > Hi folks,
> >
> > Found some discrepancies in moduleSet <include> list in src.xml and
> > hadoop-two-compat.xml. Got a crash course on hbase-assembly today by
> stack.
> > Throwing some larger questions here;
> >
> > 1. Do we want h-archetypes in binary tar?
> > 2. Shouldn't we be building h-shaded, h-examples and h-archetypes before
> > h-assembly so that the corresponding jars get included in source/binary
> > tar? Here's the current build order :
> >
> > ....
> > Apache HBase - Archetypes .......................... SUCCESS [ 0.006 s]
> > Apache HBase - Assembly ............................ SUCCESS [ 0.281 s]
> > Apache HBase - Shaded .............................. SUCCESS [ 0.006 s]
> > Apache HBase - Shaded - Client ..................... SUCCESS [ 0.010 s]
> > Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.011 s]
> > Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.007 s]
> > Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.096 s]
> > Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [
> 0.095
> > s]
> > Apache HBase - Archetype builder ................... SUCCESS [ 0.008 s]
> >
> >
> > -- Appy
> >
>
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Nick Dimiduk <nd...@gmail.com>.
On Wed, Oct 25, 2017 at 9:53 PM, Sean Busbey <bu...@apache.org> wrote:
>
> okay so here's a use case. our return from `hbase classpath` is a
> nightmare. if you're a downstream user that just wants "hey hbase give
> me jars I need to run against you right now." it's garbage. we could
> include the shaded jars and return the shaded client for something
> like `hbase classpath-client`. Same with replacing `hbase mapredcp`
> current output with the shaded mapreduce jar.
>
I think the listing of dependencies provided by `hbase classpath` family of
utilities an anti-pattern in light of shaded artifacts. Can we replace
these commands with instructions to build your application jar with the
appropriate dependencies? Are there use-cases where the user doesn't
control their classpath, they have no choice but to use `hadoop`/`yarn` to
launch their application instead of `hbase`?
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Sean Busbey <bu...@apache.org>.
On Wed, Oct 25, 2017 at 9:14 PM, Stack <st...@duboce.net> wrote:
> On Wed, Oct 25, 2017 at 3:03 PM, Sean Busbey <bu...@apache.org> wrote:
>
>> Thus far, the shaded jars and the archetypes have only been aimed at
>> consumption during downstream build processes. Namely folks using maven to
>> build an app.
>>
>> For those users, only being published in the Apache Nexus repo matters, so
>> we deployed there (via the deploy step of our release process with release
>> and apache-release maven profiles). We have not, thus far, included those
>> jars in our binary tarball. Thus they aren't listed as dependencies of the
>> assembly module and get built after it.
>>
>> Adding them would nearly double our binary tarball size, so I'm inclined to
>> not include them without a compelling use case.
>>
>>
> Interesting. I'd have said our bin should have all our built product but
> yeah, as you say, archetypes depends on maven context and would make no
> sense in bin tgz.
>
> If we want to evangelize shaded client as primary access point, would that
> be justification bundling it in bin tgz?
>
>
okay so here's a use case. our return from `hbase classpath` is a
nightmare. if you're a downstream user that just wants "hey hbase give
me jars I need to run against you right now." it's garbage. we could
include the shaded jars and return the shaded client for something
like `hbase classpath-client`. Same with replacing `hbase mapredcp`
current output with the shaded mapreduce jar.
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Stack <st...@duboce.net>.
On Wed, Oct 25, 2017 at 3:03 PM, Sean Busbey <bu...@apache.org> wrote:
> Thus far, the shaded jars and the archetypes have only been aimed at
> consumption during downstream build processes. Namely folks using maven to
> build an app.
>
> For those users, only being published in the Apache Nexus repo matters, so
> we deployed there (via the deploy step of our release process with release
> and apache-release maven profiles). We have not, thus far, included those
> jars in our binary tarball. Thus they aren't listed as dependencies of the
> assembly module and get built after it.
>
> Adding them would nearly double our binary tarball size, so I'm inclined to
> not include them without a compelling use case.
>
>
Interesting. I'd have said our bin should have all our built product but
yeah, as you say, archetypes depends on maven context and would make no
sense in bin tgz.
If we want to evangelize shaded client as primary access point, would that
be justification bundling it in bin tgz?
> The source tarball isn't made by a module, despite the descriptor living in
> the hbase-assembly module. It could just as easily be in dev-support. The
> step of our release process that creates the source tarball does a manual
> invocation of the maven assembly plug-in and points at the source
> descriptor directly.
>
>
Our build could do w/ a revamp. It still has a form from when we used emit
multiple products, src and a bin tarball for hadoop1 and hadoop2.
St.Ack
> On Oct 25, 2017 4:57 PM, "Apekshit Sharma" <ap...@cloudera.com> wrote:
>
> Hi folks,
>
> Found some discrepancies in moduleSet <include> list in src.xml and
> hadoop-two-compat.xml. Got a crash course on hbase-assembly today by stack.
> Throwing some larger questions here;
>
> 1. Do we want h-archetypes in binary tar?
> 2. Shouldn't we be building h-shaded, h-examples and h-archetypes before
> h-assembly so that the corresponding jars get included in source/binary
> tar? Here's the current build order :
>
> ....
> Apache HBase - Archetypes .......................... SUCCESS [ 0.006 s]
> Apache HBase - Assembly ............................ SUCCESS [ 0.281 s]
> Apache HBase - Shaded .............................. SUCCESS [ 0.006 s]
> Apache HBase - Shaded - Client ..................... SUCCESS [ 0.010 s]
> Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.011 s]
> Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.007 s]
> Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.096 s]
> Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 0.095
> s]
> Apache HBase - Archetype builder ................... SUCCESS [ 0.008 s]
>
>
> -- Appy
>
Re: Is our maven build order correct? Do we have required jars in
binary/source tar.gz?
Posted by Sean Busbey <bu...@apache.org>.
Thus far, the shaded jars and the archetypes have only been aimed at
consumption during downstream build processes. Namely folks using maven to
build an app.
For those users, only being published in the Apache Nexus repo matters, so
we deployed there (via the deploy step of our release process with release
and apache-release maven profiles). We have not, thus far, included those
jars in our binary tarball. Thus they aren't listed as dependencies of the
assembly module and get built after it.
Adding them would nearly double our binary tarball size, so I'm inclined to
not include them without a compelling use case.
The source tarball isn't made by a module, despite the descriptor living in
the hbase-assembly module. It could just as easily be in dev-support. The
step of our release process that creates the source tarball does a manual
invocation of the maven assembly plug-in and points at the source
descriptor directly.
On Oct 25, 2017 4:57 PM, "Apekshit Sharma" <ap...@cloudera.com> wrote:
Hi folks,
Found some discrepancies in moduleSet <include> list in src.xml and
hadoop-two-compat.xml. Got a crash course on hbase-assembly today by stack.
Throwing some larger questions here;
1. Do we want h-archetypes in binary tar?
2. Shouldn't we be building h-shaded, h-examples and h-archetypes before
h-assembly so that the corresponding jars get included in source/binary
tar? Here's the current build order :
....
Apache HBase - Archetypes .......................... SUCCESS [ 0.006 s]
Apache HBase - Assembly ............................ SUCCESS [ 0.281 s]
Apache HBase - Shaded .............................. SUCCESS [ 0.006 s]
Apache HBase - Shaded - Client ..................... SUCCESS [ 0.010 s]
Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.011 s]
Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.007 s]
Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.096 s]
Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 0.095
s]
Apache HBase - Archetype builder ................... SUCCESS [ 0.008 s]
-- Appy