[DISCUSS] including shaded artifacts in the convenience binary

[Sean Busbey <bu...@apache.org>]

Hi folks!

Over in HBASE-20331 I'm trying to polish up our story around how
downstreamers make use of our shaded artifacts. As a part of that I'd
like have them present as a part of a "normal" hbase installation.

Previously when we've discussed this topic, the assumption was
downstream folks would package up the shaded client with their
application themselves. Presumably this would be done via maven or the
like.

Having worked with them for awhile, I think we're better off including
them after all.

1) If most applications are going to use the shaded clients, then by
not shipping them we're encouraging a situation where you end up with
a copy per application.

2) If we ship them we can simplify the default path for some uses,
namely making hbase mapredcp return the shaded mapreduce client.
Similarly, we could make a "client classpath" command that gave the
shaded artifact as an alternative to the current bloat in the hbase
classpath

3) If we ship them we can update the docs that walk through using the
example mapreduce tools to make use of the shaded mapreduce client. If
we don't make that update we'll essentially have docs that say "here's
how you run _our_ MR jobs that talk to HBase, but you shouldn't do
that when running _yours_", which is confusing.

I have a POC patch for just adding them up on HBASE-20615. It keeps
them out of the normal server classpath entirely.

An alternative is that I could help Josh finish up HBASE-19735 "create
a minimal client tarball" and we could start pushing folks to install
it on nodes that they expect to use for connecting to hbase. (I'd want
to bring it back into 2.1 in that case.)

What do folks think?

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Stack <st...@duboce.net>]

On Tue, May 29, 2018 at 7:09 AM, Sean Busbey <bu...@apache.org> wrote:

> Any other feedback here folks?
>
>
Bundling shaded artifacts AND a client-only access tgz would be doing our
users a useful service.

Aside: Is it going to be hell to rig the build to autogenerate these new
artifacts?

S



> Mike D, you leaning towards the client tarball or just making sure
> I've considered it?
>
> (FWIW, I think the client tarball would work fine practically speaking
> and don't feel strongly about either approach.)
>
> On Thu, May 24, 2018 at 1:01 AM, Sean Busbey <bu...@apache.org> wrote:
> > Hi folks!
> >
> > Over in HBASE-20331 I'm trying to polish up our story around how
> > downstreamers make use of our shaded artifacts. As a part of that I'd
> > like have them present as a part of a "normal" hbase installation.
> >
> > Previously when we've discussed this topic, the assumption was
> > downstream folks would package up the shaded client with their
> > application themselves. Presumably this would be done via maven or the
> > like.
> >
> > Having worked with them for awhile, I think we're better off including
> > them after all.
> >
> > 1) If most applications are going to use the shaded clients, then by
> > not shipping them we're encouraging a situation where you end up with
> > a copy per application.
> >
> > 2) If we ship them we can simplify the default path for some uses,
> > namely making hbase mapredcp return the shaded mapreduce client.
> > Similarly, we could make a "client classpath" command that gave the
> > shaded artifact as an alternative to the current bloat in the hbase
> > classpath
> >
> > 3) If we ship them we can update the docs that walk through using the
> > example mapreduce tools to make use of the shaded mapreduce client. If
> > we don't make that update we'll essentially have docs that say "here's
> > how you run _our_ MR jobs that talk to HBase, but you shouldn't do
> > that when running _yours_", which is confusing.
> >
> > I have a POC patch for just adding them up on HBASE-20615. It keeps
> > them out of the normal server classpath entirely.
> >
> > An alternative is that I could help Josh finish up HBASE-19735 "create
> > a minimal client tarball" and we could start pushing folks to install
> > it on nodes that they expect to use for connecting to hbase. (I'd want
> > to bring it back into 2.1 in that case.)
> >
> > What do folks think?
>

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Sean Busbey <bu...@apache.org>]

Any other feedback here folks?

Mike D, you leaning towards the client tarball or just making sure
I've considered it?

(FWIW, I think the client tarball would work fine practically speaking
and don't feel strongly about either approach.)

On Thu, May 24, 2018 at 1:01 AM, Sean Busbey <bu...@apache.org> wrote:
> Hi folks!
>
> Over in HBASE-20331 I'm trying to polish up our story around how
> downstreamers make use of our shaded artifacts. As a part of that I'd
> like have them present as a part of a "normal" hbase installation.
>
> Previously when we've discussed this topic, the assumption was
> downstream folks would package up the shaded client with their
> application themselves. Presumably this would be done via maven or the
> like.
>
> Having worked with them for awhile, I think we're better off including
> them after all.
>
> 1) If most applications are going to use the shaded clients, then by
> not shipping them we're encouraging a situation where you end up with
> a copy per application.
>
> 2) If we ship them we can simplify the default path for some uses,
> namely making hbase mapredcp return the shaded mapreduce client.
> Similarly, we could make a "client classpath" command that gave the
> shaded artifact as an alternative to the current bloat in the hbase
> classpath
>
> 3) If we ship them we can update the docs that walk through using the
> example mapreduce tools to make use of the shaded mapreduce client. If
> we don't make that update we'll essentially have docs that say "here's
> how you run _our_ MR jobs that talk to HBase, but you shouldn't do
> that when running _yours_", which is confusing.
>
> I have a POC patch for just adding them up on HBASE-20615. It keeps
> them out of the normal server classpath entirely.
>
> An alternative is that I could help Josh finish up HBASE-19735 "create
> a minimal client tarball" and we could start pushing folks to install
> it on nodes that they expect to use for connecting to hbase. (I'd want
> to bring it back into 2.1 in that case.)
>
> What do folks think?

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Josh Elser <el...@apache.org>]

On 5/31/18 3:57 PM, Sean Busbey wrote:
> On Thu, May 31, 2018 at 2:56 PM, Josh Elser <el...@apache.org> wrote:
>>
>>
>> On 5/24/18 11:16 AM, Sean Busbey wrote:
>>>
>>> On Thu, May 24, 2018 at 9:41 AM, Mike Drob<md...@apache.org>  wrote:
>>>>
>>>> Are the two issues (HBASE-20615 and HBASE-19735) conflicting, redundant,
>>>> or
>>>> orthogonal?
>>>>
>>> I think they're orthogonal, at least as they stand. I think the fact
>>> that HBASE-19735 doesn't e.g. remove the shell from the main tarball
>>> is a good way to illustrate that inclusion in one doesn't obviate the
>>> possibility of inclusion in the other.
>>>
>>> That said, if HBASE-19735 lands then we could adapt the testing I'm
>>> getting in place as a part of HBASE-20331 to check that the generated
>>> artifact works as expected. If both landed then I'd probably want to
>>> test both using the same test.
>>>
>>
>> If it makes sense to consolidate, I'd be happy to try to support that too.
>>
>> We don't have any outstanding shaded jar issues, do we? I haven't made the
>> time to circle back around and do more testing.
> 
> there are some under the umbrella HBASE-20331

4/5 in patch available? Let me add these to my queue to review :)

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Sean Busbey <bu...@apache.org>]

On Thu, May 31, 2018 at 2:56 PM, Josh Elser <el...@apache.org> wrote:
>
>
> On 5/24/18 11:16 AM, Sean Busbey wrote:
>>
>> On Thu, May 24, 2018 at 9:41 AM, Mike Drob<md...@apache.org>  wrote:
>>>
>>> Are the two issues (HBASE-20615 and HBASE-19735) conflicting, redundant,
>>> or
>>> orthogonal?
>>>
>> I think they're orthogonal, at least as they stand. I think the fact
>> that HBASE-19735 doesn't e.g. remove the shell from the main tarball
>> is a good way to illustrate that inclusion in one doesn't obviate the
>> possibility of inclusion in the other.
>>
>> That said, if HBASE-19735 lands then we could adapt the testing I'm
>> getting in place as a part of HBASE-20331 to check that the generated
>> artifact works as expected. If both landed then I'd probably want to
>> test both using the same test.
>>
>
> If it makes sense to consolidate, I'd be happy to try to support that too.
>
> We don't have any outstanding shaded jar issues, do we? I haven't made the
> time to circle back around and do more testing.

there are some under the umbrella HBASE-20331

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Josh Elser <el...@apache.org>]

On 5/24/18 11:16 AM, Sean Busbey wrote:
> On Thu, May 24, 2018 at 9:41 AM, Mike Drob<md...@apache.org>  wrote:
>> Are the two issues (HBASE-20615 and HBASE-19735) conflicting, redundant, or
>> orthogonal?
>>
> I think they're orthogonal, at least as they stand. I think the fact
> that HBASE-19735 doesn't e.g. remove the shell from the main tarball
> is a good way to illustrate that inclusion in one doesn't obviate the
> possibility of inclusion in the other.
> 
> That said, if HBASE-19735 lands then we could adapt the testing I'm
> getting in place as a part of HBASE-20331 to check that the generated
> artifact works as expected. If both landed then I'd probably want to
> test both using the same test.
> 

If it makes sense to consolidate, I'd be happy to try to support that too.

We don't have any outstanding shaded jar issues, do we? I haven't made 
the time to circle back around and do more testing.

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Josh Elser <el...@apache.org>]

On 5/24/18 11:16 AM, Sean Busbey wrote:
>> If we have a minimal client tarball that includes the shaded client and
>> shaded mapreduce modules (and nothing else?) then are we good to go?
>>
> I think if we have a client tarball that includes the shaded client
> modules (and other things they need like logging jars) then we can
> make a go of things. I think Josh's current plan to also include the
> shell in the tarball is a good idea.
> 
> I don't know if practically speaking this will just mean we end up
> making everyone install two tarballs instead of one. We could always
> push forward with it and then revisit the issue as we get feedback.

+1

I think if we create something that pushes folks towards how we want 
them to use it, we'll have more success :)

Feel free to ping on JIRA issues if you want some eyes (I'm terrible at 
proactively following issues these days)

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Sean Busbey <bu...@apache.org>]

On Thu, May 24, 2018 at 9:41 AM, Mike Drob <md...@apache.org> wrote:
> Are the two issues (HBASE-20615 and HBASE-19735) conflicting, redundant, or
> orthogonal?
>

I think they're orthogonal, at least as they stand. I think the fact
that HBASE-19735 doesn't e.g. remove the shell from the main tarball
is a good way to illustrate that inclusion in one doesn't obviate the
possibility of inclusion in the other.

That said, if HBASE-19735 lands then we could adapt the testing I'm
getting in place as a part of HBASE-20331 to check that the generated
artifact works as expected. If both landed then I'd probably want to
test both using the same test.

> If we have a minimal client tarball that includes the shaded client and
> shaded mapreduce modules (and nothing else?) then are we good to go?
>

I think if we have a client tarball that includes the shaded client
modules (and other things they need like logging jars) then we can
make a go of things. I think Josh's current plan to also include the
shell in the tarball is a good idea.

I don't know if practically speaking this will just mean we end up
making everyone install two tarballs instead of one. We could always
push forward with it and then revisit the issue as we get feedback.

Re: [DISCUSS] including shaded artifacts in the convenience binary

[Mike Drob <md...@apache.org>]

Are the two issues (HBASE-20615 and HBASE-19735) conflicting, redundant, or
orthogonal?

If we have a minimal client tarball that includes the shaded client and
shaded mapreduce modules (and nothing else?) then are we good to go?

On Thu, May 24, 2018 at 1:01 AM, Sean Busbey <bu...@apache.org> wrote:

> Hi folks!
>
> Over in HBASE-20331 I'm trying to polish up our story around how
> downstreamers make use of our shaded artifacts. As a part of that I'd
> like have them present as a part of a "normal" hbase installation.
>
> Previously when we've discussed this topic, the assumption was
> downstream folks would package up the shaded client with their
> application themselves. Presumably this would be done via maven or the
> like.
>
> Having worked with them for awhile, I think we're better off including
> them after all.
>
> 1) If most applications are going to use the shaded clients, then by
> not shipping them we're encouraging a situation where you end up with
> a copy per application.
>
> 2) If we ship them we can simplify the default path for some uses,
> namely making hbase mapredcp return the shaded mapreduce client.
> Similarly, we could make a "client classpath" command that gave the
> shaded artifact as an alternative to the current bloat in the hbase
> classpath
>
> 3) If we ship them we can update the docs that walk through using the
> example mapreduce tools to make use of the shaded mapreduce client. If
> we don't make that update we'll essentially have docs that say "here's
> how you run _our_ MR jobs that talk to HBase, but you shouldn't do
> that when running _yours_", which is confusing.
>
> I have a POC patch for just adding them up on HBASE-20615. It keeps
> them out of the normal server classpath entirely.
>
> An alternative is that I could help Josh finish up HBASE-19735 "create
> a minimal client tarball" and we could start pushing folks to install
> it on nodes that they expect to use for connecting to hbase. (I'd want
> to bring it back into 2.1 in that case.)
>
> What do folks think?
>