You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Istvan Toth <st...@apache.org> on 2024/03/05 08:13:32 UTC
[DISCUSS] Removing tests and/or Hadoop from the binary assemblies
DISCLAIMER: I don't have a patch ready, or even an elegant way mapped out
to achieve this, this is about discussing whether we even want to make
these changes.
These are also substantial changes, but they could be targeted for HBase
3.0.
One issue I have noticed is that we ship test jars and test dependencies in
the assembly.
I can't see anyone using those, but it bloats the assembly and classpath,
and adds unnecessary JARs with possible CVE issues. (for example Kerby
which is a Hadoop minicluster dependency)
My proposal is to exclude the test jars and the test scope dependencies
from the assembly.
The advantages would be:
* Smaller distro size
* Faster startup (this is marginal)
* Less CVE-prone JARs in the binary assemblies
The other issue is that the assembly includes much of the Hadoop
distribution.
The basic assumption in all scripts and instructions is that the node has a
fully configured Hadoop installation, and we include it in the classpath of
HBase.
If that is true, then there is no reason to include Hadoop in the assembly,
HBase and its direct dependencies should be enough.
One could argue that it would simplify the client side, which is true to
some extent (though 95% of the client distro use cases are served better by
simply using hbase-shaded-client).
We could either remove the Hadoop libraries from either or both of the
assemblies unconditionally, or provide two variants for either or both
assemblies, one with Hadoop included, and one without it.
Spark already does this, it has binary distributions both with and without
Hadoop.
The advantages would be:
* Smaller distro size
* Faster startup (this is marginal)
* Less chance of conflicts with the Hadoop jars
* Less CVE-prone JARs in the binary assemblies
Thirdly, we could consider excluding the
full-fat org.apache.hbase:hbase-shaded-client JAR from the Hadoop-less
binary assemblies. It is not used by the assembly, and AFAIK it is not
included in any of the 'hbase classpath' command variants.
This would make sure that no Hadoop libraries are included (even in shaded
form) and would make the HBase distribution fully insulated from Hadoop's
CVE issues.
(The full-fat hbase-shaded-client works best as direct build-time
dependency anyway)
best regards
Istvan
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
For me a profile for building the binary which could run ITBLL is
enough, as only developers need to run ITBLL for finding issues,
developers should have the ability to build HBase by their own.
But let's start a new discussion thread both on dev and user mailing
list to see if there are any other opinions.
Thanks.
Istvan Toth <st...@cloudera.com.invalid> 于2025年3月17日周一 14:32写道：
>
> If we don't want to publish the testing assembly, we could also just add a
> profile to hbase-assembly,
> so that the testing assembly can be built from the same module by enabling
> the profile.
>
>
> On Mon, Mar 17, 2025 at 7:22 AM Istvan Toth <st...@cloudera.com> wrote:
>
> > I have opened HBASE-29187 for this.
> >
> > Should we publish this testing assembly for releases ?
> > Should this testing assembly be built by default, or should we gate it
> > with some property ?
> >
> > Istvan
> >
> >
> > On Sat, Mar 15, 2025 at 7:13 PM Istvan Toth <st...@cloudera.com> wrote:
> >
> >> I will work on it next week.
> >>
> >> On Sat, Mar 15, 2025 at 3:01 PM 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >>
> >>> Do we have an issue for it?
> >>>
> >>> At least for releasing 3.0.0, we need to run ITBLL...
> >>>
> >>> Thanks.
> >>>
> >>> Istvan Toth <st...@apache.org> 于2025年3月15日周六 12:32写道：
> >>> >
> >>> > Yes, we've discussed this issue, but deferred solving it.
> >>> >
> >>> > IMO the easiest way is to add a third assembly which is functionally
> >>> > equivalent to the 2.x assembly.
> >>> > That could work for running hbase-it, ITBLL, chaos monkey, etc.
> >>> > We'd also have to decide whether to build it by default, and whether we
> >>> > want to publish it as part of official releases.
> >>> >
> >>> > In theory could also make a delta assembly that only includes the
> >>> > additional test related stuff, but I'm afraid that that would require
> >>> a lot
> >>> > of maintenance.
> >>> > We could also add a script/maven target that downloads test-related
> >>> JARs
> >>> > from maven, but keeping that one up to date would also be problematic.
> >>> >
> >>> > Istvan
> >>> >
> >>> > On Sat, Mar 15, 2025 at 4:51 AM 张铎(Duo Zhang) <pa...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > After this change, we can not run ITBLL on 3.0.0 because hbase-it is
> >>> > > also excluded...
> >>> > >
> >>> > > I tried manually copying all the tests jar and hbase-it jar to the
> >>> lib
> >>> > > directory but it did not work, I guess we still missed several hadoop
> >>> > > jars...
> >>> > >
> >>> > > So what is the suggested way to run ITBLL after this change?
> >>> > >
> >>> > > Thanks.
> >>> > >
> >>> > > Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
> >>> > > >
> >>> > > > This is almost done.
> >>> > > >
> >>> > > > The final outstanding patch is
> >>> https://github.com/apache/hbase/pull/5766
> >>> > > > for the new Hadoop-less assembly.
> >>> > > >
> >>> > > > Could you please review it ?
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com>
> >>> > > wrote:
> >>> > > >
> >>> > > > > I have created sub tasks with necessary details in the umbrella
> >>> jira.
> >>> > > Will
> >>> > > > > take them up in coming days. Also will add more sub tasks later
> >>> if
> >>> > > needed.
> >>> > > > >
> >>> > > > > Regards
> >>> > > > > Nihal
> >>> > > > >
> >>> > > > > On Sat, 9 Mar 2024, 11:53 Istvan Toth,
> >>> <st...@cloudera.com.invalid>
> >>> > > wrote:
> >>> > > > >
> >>> > > > > > Thank you Nihal.
> >>> > > > > > I'm not very familiar with the tools in the test code, so you
> >>> can
> >>> > > > > probably
> >>> > > > > > plan that work better.
> >>> > > > > > I just have some generic steps in mind:
> >>> > > > > > * Identify all the tools / scripts in the test jars
> >>> > > > > > * Identify and analyze their dependencies (compared to the
> >>> current
> >>> > > > > runtime
> >>> > > > > > deps)
> >>> > > > > > * Decide which ones to move to the runtime JARs.
> >>> > > > > > * Move them to the runtime code (or perhaps a separate module)
> >>> > > > > >
> >>> > > > > > I have created
> >>> https://issues.apache.org/jira/browse/HBASE-28431 as
> >>> > > an
> >>> > > > > > umbrella ticket to organize the sub-tasks.
> >>> > > > > >
> >>> > > > > > Istvan
> >>> > > > > >
> >>> > > > > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <
> >>> nihaljain.cs@gmail.com>
> >>> > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Sure I will be able to take up. Please create tasks with
> >>> necessary
> >>> > > > > > details
> >>> > > > > > > or let me know if you want me to create.
> >>> > > > > > >
> >>> > > > > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth,
> >>> <stoty@cloudera.com.invalid
> >>> > > >
> >>> > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Thanks for volunteering, Nihal.
> >>> > > > > > > >
> >>> > > > > > > > I could work on the Hadoop-less, and assemblies, and you
> >>> could
> >>> > > work
> >>> > > > > on
> >>> > > > > > > > cleaning up the test jars.
> >>> > > > > > > > Would that work for you ?
> >>> > > > > > > > I know that I'm picking the smaller part, but it turns out
> >>> that I
> >>> > > > > won't
> >>> > > > > > > > have as much time to work on this as I hoped.
> >>> > > > > > > >
> >>> > > > > > > > (Unless there are other volunteers, of course)
> >>> > > > > > > >
> >>> > > > > > > > Istvan
> >>> > > > > > > >
> >>> > > > > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <
> >>> stoty@cloudera.com>
> >>> > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > We seem to be in agreement in principle, however the
> >>> devil is
> >>> > > in
> >>> > > > > the
> >>> > > > > > > > > details.
> >>> > > > > > > > >
> >>> > > > > > > > > The first step should be moving the diagnostic tools out
> >>> of the
> >>> > > > > test
> >>> > > > > > > > jars.
> >>> > > > > > > > > Are there any tools we don't want to move out ?
> >>> > > > > > > > > Do the diagnostic tools pull in extra dependencies
> >>> compared to
> >>> > > the
> >>> > > > > > > > current
> >>> > > > > > > > > runtime JARs, and if they do, what are those ?
> >>> > > > > > > > > I haven't thought of the chaosmonkey tests yet, do those
> >>> have
> >>> > > > > > specific
> >>> > > > > > > > > additional dependencies / scripts ?
> >>> > > > > > > > >
> >>> > > > > > > > > Should we move the tools simply to the normal jars, or
> >>> should
> >>> > > we
> >>> > > > > move
> >>> > > > > > > > them
> >>> > > > > > > > > to a new module (could be called hbase-diagnostics) ?
> >>> > > > > > > > >
> >>> > > > > > > > > Istvan
> >>> > > > > > > > >
> >>> > > > > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> >>> > > > > > > > bbeaudreault@apache.org>
> >>> > > > > > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > >> I'm +0 on hbase-examples, but +1000000 on any
> >>> improvements we
> >>> > > can
> >>> > > > > > make
> >>> > > > > > > > to
> >>> > > > > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely
> >>> frustrating how
> >>> > > much
> >>> > > > > > > > reliance
> >>> > > > > > > > >> we have on test jars both generally but also
> >>> specifically
> >>> > > around
> >>> > > > > > these
> >>> > > > > > > > >> core
> >>> > > > > > > > >> test executables. Unfortunately I haven't had time to
> >>> > > dedicate to
> >>> > > > > > > these
> >>> > > > > > > > >> frustrations myself, but happy to help with review, etc.
> >>> > > > > > > > >>
> >>> > > > > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <
> >>> > > nihaljain.cs@gmail.com
> >>> > > > > >
> >>> > > > > > > > wrote:
> >>> > > > > > > > >>
> >>> > > > > > > > >> > Thank you for bringing this up.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > +1 for this change.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > In fact, some time back, we had faced similar problem.
> >>> > > Security
> >>> > > > > > > scans
> >>> > > > > > > > >> found
> >>> > > > > > > > >> > that we were bundling some vulnerable hadoop test
> >>> jar. To
> >>> > > deal
> >>> > > > > > with
> >>> > > > > > > > >> that we
> >>> > > > > > > > >> > had to make a change in our internal HBase fork to
> >>> exclude
> >>> > > all
> >>> > > > > > HBase
> >>> > > > > > > > and
> >>> > > > > > > > >> > Hadoop test jars from assembly. This helped us get
> >>> rid of
> >>> > > > > > vulnerable
> >>> > > > > > > > >> jar.
> >>> > > > > > > > >> > (Although I hadn't dealt with test scope dependencies
> >>> > > there.)
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > But, I have been thinking of pushing this change in
> >>> Apache
> >>> > > > > HBase,
> >>> > > > > > > just
> >>> > > > > > > > >> > wasn't sure if this was even acceptable. It's great
> >>> to see
> >>> > > same
> >>> > > > > > has
> >>> > > > > > > > been
> >>> > > > > > > > >> > brought up here today.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote
> >>> a
> >>> > > script
> >>> > > > > to
> >>> > > > > > > > >> download
> >>> > > > > > > > >> > them on demand to avoid massive code change in
> >>> internal
> >>> > > fork.
> >>> > > > > But
> >>> > > > > > I
> >>> > > > > > > > >> have a
> >>> > > > > > > > >> > +1 on the idea of identifying and moving all such
> >>> tools to
> >>> > > a new
> >>> > > > > > > > module.
> >>> > > > > > > > >> > This would be great and make things easier for us as
> >>> well.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > Also, a way we could help new users easily get
> >>> started, in
> >>> > > case
> >>> > > > > we
> >>> > > > > > > > >> > completely stop bundling hadoop jars, is by providing
> >>> a
> >>> > > script
> >>> > > > > > which
> >>> > > > > > > > >> starts
> >>> > > > > > > > >> > a hbase cluster in a single node setup. In fact I had
> >>> > > written a
> >>> > > > > > > simple
> >>> > > > > > > > >> > script sometime back that automates this process
> >>> given a
> >>> > > release
> >>> > > > > > > link
> >>> > > > > > > > >> for
> >>> > > > > > > > >> > both. It first downloads Hadoop and HBase binaries
> >>> and then
> >>> > > > > starts
> >>> > > > > > > > both
> >>> > > > > > > > >> > with the hbase root directory set to be on hdfs. We
> >>> could
> >>> > > > > provide
> >>> > > > > > > > >> something
> >>> > > > > > > > >> > similar to help new users to get started easily.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > Although I am also +1 on the idea to provide both
> >>> variants
> >>> > > as
> >>> > > > > > > > mentioned
> >>> > > > > > > > >> by
> >>> > > > > > > > >> > Nick, which might not even need any such script.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > Also, I am willing to volunteer for help towards this
> >>> > > effort.
> >>> > > > > > Please
> >>> > > > > > > > >> let me
> >>> > > > > > > > >> > know if anything is needed.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > Thanks,
> >>> > > > > > > > >> > Nihal
> >>> > > > > > > > >> >
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <
> >>> > > ndimiduk@apache.org>
> >>> > > > > > > wrote:
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > > This would be great cleanup, big +1 from me for all
> >>> three
> >>> > > of
> >>> > > > > > these
> >>> > > > > > > > >> > > adjustments, including the promotion of pe, ltt, and
> >>> > > friends
> >>> > > > > out
> >>> > > > > > > of
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > test scope.
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > I believe that we included hbase test jars because
> >>> we
> >>> > > used to
> >>> > > > > > > freely
> >>> > > > > > > > >> mix
> >>> > > > > > > > >> > > classes needed for minicluster between runtime and
> >>> test
> >>> > > jars,
> >>> > > > > > > which
> >>> > > > > > > > in
> >>> > > > > > > > >> > turn
> >>> > > > > > > > >> > > relied on Hadoop minicluster capabilities. The big
> >>> cleanup
> >>> > > > > > around
> >>> > > > > > > > >> > > HBaseTestingUtil/it addressed much (or all) of these
> >>> > > issues on
> >>> > > > > > > > >> branch-3.
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > I believe that we include a Hadoop distribution in
> >>> our
> >>> > > > > assembly
> >>> > > > > > > > >> because
> >>> > > > > > > > >> > > that makes it easy for a new user to download our
> >>> release
> >>> > > > > > bin.tgz
> >>> > > > > > > > and
> >>> > > > > > > > >> get
> >>> > > > > > > > >> > > started immediately with learning. I guess it’s
> >>> high time
> >>> > > that
> >>> > > > > > we
> >>> > > > > > > > work
> >>> > > > > > > > >> > out
> >>> > > > > > > > >> > > the with- and without-Hadoop variants.
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > Thanks,
> >>> > > > > > > > >> > > Nick
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <
> >>> > > stoty@apache.org>
> >>> > > > > > > wrote:
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an
> >>> > > elegant
> >>> > > > > way
> >>> > > > > > > > >> mapped
> >>> > > > > > > > >> > out
> >>> > > > > > > > >> > > > to achieve this, this is about discussing whether
> >>> we
> >>> > > even
> >>> > > > > want
> >>> > > > > > > to
> >>> > > > > > > > >> make
> >>> > > > > > > > >> > > > these changes.
> >>> > > > > > > > >> > > > These are also substantial changes, but they
> >>> could be
> >>> > > > > targeted
> >>> > > > > > > for
> >>> > > > > > > > >> > HBase
> >>> > > > > > > > >> > > > 3.0.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > One issue I have noticed is that we ship test
> >>> jars and
> >>> > > test
> >>> > > > > > > > >> > dependencies
> >>> > > > > > > > >> > > in
> >>> > > > > > > > >> > > > the assembly.
> >>> > > > > > > > >> > > > I can't see anyone using those, but it bloats the
> >>> > > assembly
> >>> > > > > and
> >>> > > > > > > > >> > classpath,
> >>> > > > > > > > >> > > > and adds unnecessary JARs with possible CVE
> >>> issues. (for
> >>> > > > > > example
> >>> > > > > > > > >> Kerby
> >>> > > > > > > > >> > > > which is a Hadoop minicluster dependency)
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > My proposal is to exclude the test jars and the
> >>> test
> >>> > > scope
> >>> > > > > > > > >> dependencies
> >>> > > > > > > > >> > > > from the assembly.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > The advantages would be:
> >>> > > > > > > > >> > > > * Smaller distro size
> >>> > > > > > > > >> > > > * Faster startup (this is marginal)
> >>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > The other issue is that the assembly includes
> >>> much of
> >>> > > the
> >>> > > > > > Hadoop
> >>> > > > > > > > >> > > > distribution.
> >>> > > > > > > > >> > > > The basic assumption in all scripts and
> >>> instructions is
> >>> > > that
> >>> > > > > > the
> >>> > > > > > > > >> node
> >>> > > > > > > > >> > > has a
> >>> > > > > > > > >> > > > fully configured Hadoop installation, and we
> >>> include it
> >>> > > in
> >>> > > > > the
> >>> > > > > > > > >> > classpath
> >>> > > > > > > > >> > > of
> >>> > > > > > > > >> > > > HBase.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > If that is true, then there is no reason to
> >>> include
> >>> > > Hadoop
> >>> > > > > in
> >>> > > > > > > the
> >>> > > > > > > > >> > > assembly,
> >>> > > > > > > > >> > > > HBase and its direct dependencies should be
> >>> enough.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > One could argue that it would simplify the client
> >>> side,
> >>> > > > > which
> >>> > > > > > is
> >>> > > > > > > > >> true
> >>> > > > > > > > >> > to
> >>> > > > > > > > >> > > > some extent (though 95% of the client distro use
> >>> cases
> >>> > > are
> >>> > > > > > > served
> >>> > > > > > > > >> > better
> >>> > > > > > > > >> > > by
> >>> > > > > > > > >> > > > simply using hbase-shaded-client).
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > We could either remove the Hadoop libraries from
> >>> either
> >>> > > or
> >>> > > > > > both
> >>> > > > > > > of
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > > assemblies unconditionally, or provide two
> >>> variants for
> >>> > > > > either
> >>> > > > > > > or
> >>> > > > > > > > >> both
> >>> > > > > > > > >> > > > assemblies, one with Hadoop included, and one
> >>> without
> >>> > > it.
> >>> > > > > > > > >> > > > Spark already does this, it has binary
> >>> distributions
> >>> > > both
> >>> > > > > with
> >>> > > > > > > and
> >>> > > > > > > > >> > > without
> >>> > > > > > > > >> > > > Hadoop.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > The advantages would be:
> >>> > > > > > > > >> > > > * Smaller distro size
> >>> > > > > > > > >> > > > * Faster startup (this is marginal)
> >>> > > > > > > > >> > > > * Less chance of conflicts with the Hadoop jars
> >>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > Thirdly, we could consider excluding the
> >>> > > > > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR
> >>> from
> >>> > > the
> >>> > > > > > > > >> Hadoop-less
> >>> > > > > > > > >> > > > binary assemblies. It is not used by the
> >>> assembly, and
> >>> > > AFAIK
> >>> > > > > > it
> >>> > > > > > > is
> >>> > > > > > > > >> not
> >>> > > > > > > > >> > > > included in any of the 'hbase classpath' command
> >>> > > variants.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > This would make sure that no Hadoop libraries are
> >>> > > included
> >>> > > > > > (even
> >>> > > > > > > > in
> >>> > > > > > > > >> > > shaded
> >>> > > > > > > > >> > > > form) and would make the HBase distribution fully
> >>> > > insulated
> >>> > > > > > from
> >>> > > > > > > > >> > Hadoop's
> >>> > > > > > > > >> > > > CVE issues.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > (The full-fat hbase-shaded-client works best as
> >>> direct
> >>> > > > > > > build-time
> >>> > > > > > > > >> > > > dependency anyway)
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > best regards
> >>> > > > > > > > >> > > > Istvan
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> >
> >>> > > > > > > > >>
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > --
> >>> > > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> >>> > > > > > > > > *Email*: stoty@cloudera.com
> >>> > > > > > > > > cloudera.com <https://www.cloudera.com>
> >>> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> >>> > > > > > > > > [image: Cloudera on Twitter] <
> >>> https://twitter.com/cloudera>
> >>> > > > > [image:
> >>> > > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera
> >>> >
> >>> > > [image:
> >>> > > > > > > > > Cloudera on LinkedIn] <
> >>> > > https://www.linkedin.com/company/cloudera>
> >>> > > > > > > > > ------------------------------
> >>> > > > > > > > > ------------------------------
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > --
> >>> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> >>> > > > > > > > *Email*: stoty@cloudera.com
> >>> > > > > > > > cloudera.com <https://www.cloudera.com>
> >>> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> >>> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera
> >>> >
> >>> > > [image:
> >>> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> >>> > > [image:
> >>> > > > > > > Cloudera
> >>> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >>> > > > > > > > ------------------------------
> >>> > > > > > > > ------------------------------
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > --
> >>> > > > > > *István Tóth* | Sr. Staff Software Engineer
> >>> > > > > > *Email*: stoty@cloudera.com
> >>> > > > > > cloudera.com <https://www.cloudera.com>
> >>> > > > > > [image: Cloudera] <https://www.cloudera.com/>
> >>> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> >>> [image:
> >>> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> >>> [image:
> >>> > > > > Cloudera
> >>> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >>> > > > > > ------------------------------
> >>> > > > > > ------------------------------
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > > >
> >>> > > > --
> >>> > > > *István Tóth* | Sr. Staff Software Engineer
> >>> > > > *Email*: stoty@cloudera.com
> >>> > > > cloudera.com <https://www.cloudera.com>
> >>> > > > [image: Cloudera] <https://www.cloudera.com/>
> >>> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> >>> [image:
> >>> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> >>> > > Cloudera
> >>> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >>> > > > ------------------------------
> >>> > > > ------------------------------
> >>> > >
> >>>
> >>
> >>
> >> --
> >> *István Tóth* | Sr. Staff Software Engineer
> >> *Email*: stoty@cloudera.com
> >> cloudera.com <https://www.cloudera.com>
> >> [image: Cloudera] <https://www.cloudera.com/>
> >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >> ------------------------------
> >> ------------------------------
> >>
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
I've uploaded the PR. I was able to run some integration tests with it
successfully on a pseudo distributed cluster without HDFS.
However, the org.apache.hadoop.hbase.test.IntegrationTestMonkeys test
failed.
Has anyone tried that recently ? It's not obvious if it is related to the
assembly changes, or if that's an unrelated regression, or if I'm just not
starting it as intended.
2025-03-17T08:38:03,450 ERROR [ChaosMonkey {}] util.Monkeys: Exception
> occurred when running chaos monkeys:
> java.lang.NullPointerException: Cannot invoke
> "org.apache.hadoop.hbase.HBaseClusterInterface.getInitialClusterMetrics()"
> because "this.cluster" is null
> at org.apache.hadoop.hbase.chaos.actions.Action.init(Action.java:113)
> ~[hbase-it-4.0.0-alpha-1-SNAPSHOT-tests.jar:4.0.0-alpha-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.chaos.policies.PeriodicRandomActionPolicy.init(PeriodicRandomActionPolicy.java:67)
> ~[hbase-it-4.0.0-alpha-1-SNAPSHOT-tests.jar:4.0.0-alpha-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.chaos.monkies.PolicyBasedChaosMonkey.start(PolicyBasedChaosMonkey.java:138)
> ~[hbase-it-4.0.0-alpha-1-SNAPSHOT-tests.jar:4.0.0-alpha-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.chaos.util.ChaosMonkeyRunner.getAndStartMonkey(ChaosMonkeyRunner.java:142)
> ~[hbase-it-4.0.0-alpha-1-SNAPSHOT-tests.jar:4.0.0-alpha-1-SNAPSHOT]
> at org.apache.hadoop.hbase.chaos.util.Monkeys.lambda$0(Monkeys.java:54)
> ~[hbase-it-4.0.0-alpha-1-SNAPSHOT-tests.jar:4.0.0-alpha-1-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:840) ~[?:?]
> 2025-03-17T08:38:18,416 INFO [main {}] util.Monkeys: Chaos monkeys are
> stopped.
> 2025-03-17T08:38:28,417 INFO [main {}] util.Monkeys: Chaos monkeys are
> running.
Istvan
On Mon, Mar 17, 2025 at 7:32 AM Istvan Toth <st...@cloudera.com> wrote:
> If we don't want to publish the testing assembly, we could also just add a
> profile to hbase-assembly,
> so that the testing assembly can be built from the same module by enabling
> the profile.
>
>
> On Mon, Mar 17, 2025 at 7:22 AM Istvan Toth <st...@cloudera.com> wrote:
>
>> I have opened HBASE-29187 for this.
>>
>> Should we publish this testing assembly for releases ?
>> Should this testing assembly be built by default, or should we gate it
>> with some property ?
>>
>> Istvan
>>
>>
>> On Sat, Mar 15, 2025 at 7:13 PM Istvan Toth <st...@cloudera.com> wrote:
>>
>>> I will work on it next week.
>>>
>>> On Sat, Mar 15, 2025 at 3:01 PM 张铎(Duo Zhang) <pa...@gmail.com>
>>> wrote:
>>>
>>>> Do we have an issue for it?
>>>>
>>>> At least for releasing 3.0.0, we need to run ITBLL...
>>>>
>>>> Thanks.
>>>>
>>>> Istvan Toth <st...@apache.org> 于2025年3月15日周六 12:32写道：
>>>> >
>>>> > Yes, we've discussed this issue, but deferred solving it.
>>>> >
>>>> > IMO the easiest way is to add a third assembly which is functionally
>>>> > equivalent to the 2.x assembly.
>>>> > That could work for running hbase-it, ITBLL, chaos monkey, etc.
>>>> > We'd also have to decide whether to build it by default, and whether
>>>> we
>>>> > want to publish it as part of official releases.
>>>> >
>>>> > In theory could also make a delta assembly that only includes the
>>>> > additional test related stuff, but I'm afraid that that would require
>>>> a lot
>>>> > of maintenance.
>>>> > We could also add a script/maven target that downloads test-related
>>>> JARs
>>>> > from maven, but keeping that one up to date would also be problematic.
>>>> >
>>>> > Istvan
>>>> >
>>>> > On Sat, Mar 15, 2025 at 4:51 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > After this change, we can not run ITBLL on 3.0.0 because hbase-it is
>>>> > > also excluded...
>>>> > >
>>>> > > I tried manually copying all the tests jar and hbase-it jar to the
>>>> lib
>>>> > > directory but it did not work, I guess we still missed several
>>>> hadoop
>>>> > > jars...
>>>> > >
>>>> > > So what is the suggested way to run ITBLL after this change?
>>>> > >
>>>> > > Thanks.
>>>> > >
>>>> > > Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
>>>> > > >
>>>> > > > This is almost done.
>>>> > > >
>>>> > > > The final outstanding patch is
>>>> https://github.com/apache/hbase/pull/5766
>>>> > > > for the new Hadoop-less assembly.
>>>> > > >
>>>> > > > Could you please review it ?
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <nihaljain.cs@gmail.com
>>>> >
>>>> > > wrote:
>>>> > > >
>>>> > > > > I have created sub tasks with necessary details in the umbrella
>>>> jira.
>>>> > > Will
>>>> > > > > take them up in coming days. Also will add more sub tasks later
>>>> if
>>>> > > needed.
>>>> > > > >
>>>> > > > > Regards
>>>> > > > > Nihal
>>>> > > > >
>>>> > > > > On Sat, 9 Mar 2024, 11:53 Istvan Toth,
>>>> <st...@cloudera.com.invalid>
>>>> > > wrote:
>>>> > > > >
>>>> > > > > > Thank you Nihal.
>>>> > > > > > I'm not very familiar with the tools in the test code, so you
>>>> can
>>>> > > > > probably
>>>> > > > > > plan that work better.
>>>> > > > > > I just have some generic steps in mind:
>>>> > > > > > * Identify all the tools / scripts in the test jars
>>>> > > > > > * Identify and analyze their dependencies (compared to the
>>>> current
>>>> > > > > runtime
>>>> > > > > > deps)
>>>> > > > > > * Decide which ones to move to the runtime JARs.
>>>> > > > > > * Move them to the runtime code (or perhaps a separate module)
>>>> > > > > >
>>>> > > > > > I have created
>>>> https://issues.apache.org/jira/browse/HBASE-28431 as
>>>> > > an
>>>> > > > > > umbrella ticket to organize the sub-tasks.
>>>> > > > > >
>>>> > > > > > Istvan
>>>> > > > > >
>>>> > > > > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <
>>>> nihaljain.cs@gmail.com>
>>>> > > > > wrote:
>>>> > > > > >
>>>> > > > > > > Sure I will be able to take up. Please create tasks with
>>>> necessary
>>>> > > > > > details
>>>> > > > > > > or let me know if you want me to create.
>>>> > > > > > >
>>>> > > > > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth,
>>>> <stoty@cloudera.com.invalid
>>>> > > >
>>>> > > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > > Thanks for volunteering, Nihal.
>>>> > > > > > > >
>>>> > > > > > > > I could work on the Hadoop-less, and assemblies, and you
>>>> could
>>>> > > work
>>>> > > > > on
>>>> > > > > > > > cleaning up the test jars.
>>>> > > > > > > > Would that work for you ?
>>>> > > > > > > > I know that I'm picking the smaller part, but it turns
>>>> out that I
>>>> > > > > won't
>>>> > > > > > > > have as much time to work on this as I hoped.
>>>> > > > > > > >
>>>> > > > > > > > (Unless there are other volunteers, of course)
>>>> > > > > > > >
>>>> > > > > > > > Istvan
>>>> > > > > > > >
>>>> > > > > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <
>>>> stoty@cloudera.com>
>>>> > > > > wrote:
>>>> > > > > > > >
>>>> > > > > > > > > We seem to be in agreement in principle, however the
>>>> devil is
>>>> > > in
>>>> > > > > the
>>>> > > > > > > > > details.
>>>> > > > > > > > >
>>>> > > > > > > > > The first step should be moving the diagnostic tools
>>>> out of the
>>>> > > > > test
>>>> > > > > > > > jars.
>>>> > > > > > > > > Are there any tools we don't want to move out ?
>>>> > > > > > > > > Do the diagnostic tools pull in extra dependencies
>>>> compared to
>>>> > > the
>>>> > > > > > > > current
>>>> > > > > > > > > runtime JARs, and if they do, what are those ?
>>>> > > > > > > > > I haven't thought of the chaosmonkey tests yet, do
>>>> those have
>>>> > > > > > specific
>>>> > > > > > > > > additional dependencies / scripts ?
>>>> > > > > > > > >
>>>> > > > > > > > > Should we move the tools simply to the normal jars, or
>>>> should
>>>> > > we
>>>> > > > > move
>>>> > > > > > > > them
>>>> > > > > > > > > to a new module (could be called hbase-diagnostics) ?
>>>> > > > > > > > >
>>>> > > > > > > > > Istvan
>>>> > > > > > > > >
>>>> > > > > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
>>>> > > > > > > > bbeaudreault@apache.org>
>>>> > > > > > > > > wrote:
>>>> > > > > > > > >
>>>> > > > > > > > >> I'm +0 on hbase-examples, but +1000000 on any
>>>> improvements we
>>>> > > can
>>>> > > > > > make
>>>> > > > > > > > to
>>>> > > > > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely
>>>> frustrating how
>>>> > > much
>>>> > > > > > > > reliance
>>>> > > > > > > > >> we have on test jars both generally but also
>>>> specifically
>>>> > > around
>>>> > > > > > these
>>>> > > > > > > > >> core
>>>> > > > > > > > >> test executables. Unfortunately I haven't had time to
>>>> > > dedicate to
>>>> > > > > > > these
>>>> > > > > > > > >> frustrations myself, but happy to help with review,
>>>> etc.
>>>> > > > > > > > >>
>>>> > > > > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <
>>>> > > nihaljain.cs@gmail.com
>>>> > > > > >
>>>> > > > > > > > wrote:
>>>> > > > > > > > >>
>>>> > > > > > > > >> > Thank you for bringing this up.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > +1 for this change.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > In fact, some time back, we had faced similar
>>>> problem.
>>>> > > Security
>>>> > > > > > > scans
>>>> > > > > > > > >> found
>>>> > > > > > > > >> > that we were bundling some vulnerable hadoop test
>>>> jar. To
>>>> > > deal
>>>> > > > > > with
>>>> > > > > > > > >> that we
>>>> > > > > > > > >> > had to make a change in our internal HBase fork to
>>>> exclude
>>>> > > all
>>>> > > > > > HBase
>>>> > > > > > > > and
>>>> > > > > > > > >> > Hadoop test jars from assembly. This helped us get
>>>> rid of
>>>> > > > > > vulnerable
>>>> > > > > > > > >> jar.
>>>> > > > > > > > >> > (Although I hadn't dealt with test scope dependencies
>>>> > > there.)
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > But, I have been thinking of pushing this change in
>>>> Apache
>>>> > > > > HBase,
>>>> > > > > > > just
>>>> > > > > > > > >> > wasn't sure if this was even acceptable. It's great
>>>> to see
>>>> > > same
>>>> > > > > > has
>>>> > > > > > > > been
>>>> > > > > > > > >> > brought up here today.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > We hadn't dealt with the ltt, pe etc. tools and
>>>> wrote a
>>>> > > script
>>>> > > > > to
>>>> > > > > > > > >> download
>>>> > > > > > > > >> > them on demand to avoid massive code change in
>>>> internal
>>>> > > fork.
>>>> > > > > But
>>>> > > > > > I
>>>> > > > > > > > >> have a
>>>> > > > > > > > >> > +1 on the idea of identifying and moving all such
>>>> tools to
>>>> > > a new
>>>> > > > > > > > module.
>>>> > > > > > > > >> > This would be great and make things easier for us as
>>>> well.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > Also, a way we could help new users easily get
>>>> started, in
>>>> > > case
>>>> > > > > we
>>>> > > > > > > > >> > completely stop bundling hadoop jars, is by
>>>> providing a
>>>> > > script
>>>> > > > > > which
>>>> > > > > > > > >> starts
>>>> > > > > > > > >> > a hbase cluster in a single node setup. In fact I had
>>>> > > written a
>>>> > > > > > > simple
>>>> > > > > > > > >> > script sometime back that automates this process
>>>> given a
>>>> > > release
>>>> > > > > > > link
>>>> > > > > > > > >> for
>>>> > > > > > > > >> > both. It first downloads Hadoop and HBase binaries
>>>> and then
>>>> > > > > starts
>>>> > > > > > > > both
>>>> > > > > > > > >> > with the hbase root directory set to be on hdfs. We
>>>> could
>>>> > > > > provide
>>>> > > > > > > > >> something
>>>> > > > > > > > >> > similar to help new users to get started easily.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > Although I am also +1 on the idea to provide both
>>>> variants
>>>> > > as
>>>> > > > > > > > mentioned
>>>> > > > > > > > >> by
>>>> > > > > > > > >> > Nick, which might not even need any such script.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > Also, I am willing to volunteer for help towards this
>>>> > > effort.
>>>> > > > > > Please
>>>> > > > > > > > >> let me
>>>> > > > > > > > >> > know if anything is needed.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > Thanks,
>>>> > > > > > > > >> > Nihal
>>>> > > > > > > > >> >
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <
>>>> > > ndimiduk@apache.org>
>>>> > > > > > > wrote:
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > > This would be great cleanup, big +1 from me for
>>>> all three
>>>> > > of
>>>> > > > > > these
>>>> > > > > > > > >> > > adjustments, including the promotion of pe, ltt,
>>>> and
>>>> > > friends
>>>> > > > > out
>>>> > > > > > > of
>>>> > > > > > > > >> the
>>>> > > > > > > > >> > > test scope.
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> > > I believe that we included hbase test jars because
>>>> we
>>>> > > used to
>>>> > > > > > > freely
>>>> > > > > > > > >> mix
>>>> > > > > > > > >> > > classes needed for minicluster between runtime and
>>>> test
>>>> > > jars,
>>>> > > > > > > which
>>>> > > > > > > > in
>>>> > > > > > > > >> > turn
>>>> > > > > > > > >> > > relied on Hadoop minicluster capabilities. The big
>>>> cleanup
>>>> > > > > > around
>>>> > > > > > > > >> > > HBaseTestingUtil/it addressed much (or all) of
>>>> these
>>>> > > issues on
>>>> > > > > > > > >> branch-3.
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> > > I believe that we include a Hadoop distribution in
>>>> our
>>>> > > > > assembly
>>>> > > > > > > > >> because
>>>> > > > > > > > >> > > that makes it easy for a new user to download our
>>>> release
>>>> > > > > > bin.tgz
>>>> > > > > > > > and
>>>> > > > > > > > >> get
>>>> > > > > > > > >> > > started immediately with learning. I guess it’s
>>>> high time
>>>> > > that
>>>> > > > > > we
>>>> > > > > > > > work
>>>> > > > > > > > >> > out
>>>> > > > > > > > >> > > the with- and without-Hadoop variants.
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> > > Thanks,
>>>> > > > > > > > >> > > Nick
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <
>>>> > > stoty@apache.org>
>>>> > > > > > > wrote:
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even
>>>> an
>>>> > > elegant
>>>> > > > > way
>>>> > > > > > > > >> mapped
>>>> > > > > > > > >> > out
>>>> > > > > > > > >> > > > to achieve this, this is about discussing
>>>> whether we
>>>> > > even
>>>> > > > > want
>>>> > > > > > > to
>>>> > > > > > > > >> make
>>>> > > > > > > > >> > > > these changes.
>>>> > > > > > > > >> > > > These are also substantial changes, but they
>>>> could be
>>>> > > > > targeted
>>>> > > > > > > for
>>>> > > > > > > > >> > HBase
>>>> > > > > > > > >> > > > 3.0.
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > One issue I have noticed is that we ship test
>>>> jars and
>>>> > > test
>>>> > > > > > > > >> > dependencies
>>>> > > > > > > > >> > > in
>>>> > > > > > > > >> > > > the assembly.
>>>> > > > > > > > >> > > > I can't see anyone using those, but it bloats the
>>>> > > assembly
>>>> > > > > and
>>>> > > > > > > > >> > classpath,
>>>> > > > > > > > >> > > > and adds unnecessary JARs with possible CVE
>>>> issues. (for
>>>> > > > > > example
>>>> > > > > > > > >> Kerby
>>>> > > > > > > > >> > > > which is a Hadoop minicluster dependency)
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > My proposal is to exclude the test jars and the
>>>> test
>>>> > > scope
>>>> > > > > > > > >> dependencies
>>>> > > > > > > > >> > > > from the assembly.
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > The advantages would be:
>>>> > > > > > > > >> > > > * Smaller distro size
>>>> > > > > > > > >> > > > * Faster startup (this is marginal)
>>>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > The other issue is that the assembly includes
>>>> much of
>>>> > > the
>>>> > > > > > Hadoop
>>>> > > > > > > > >> > > > distribution.
>>>> > > > > > > > >> > > > The basic assumption in all scripts and
>>>> instructions is
>>>> > > that
>>>> > > > > > the
>>>> > > > > > > > >> node
>>>> > > > > > > > >> > > has a
>>>> > > > > > > > >> > > > fully configured Hadoop installation, and we
>>>> include it
>>>> > > in
>>>> > > > > the
>>>> > > > > > > > >> > classpath
>>>> > > > > > > > >> > > of
>>>> > > > > > > > >> > > > HBase.
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > If that is true, then there is no reason to
>>>> include
>>>> > > Hadoop
>>>> > > > > in
>>>> > > > > > > the
>>>> > > > > > > > >> > > assembly,
>>>> > > > > > > > >> > > > HBase and its direct dependencies should be
>>>> enough.
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > One could argue that it would simplify the
>>>> client side,
>>>> > > > > which
>>>> > > > > > is
>>>> > > > > > > > >> true
>>>> > > > > > > > >> > to
>>>> > > > > > > > >> > > > some extent (though 95% of the client distro use
>>>> cases
>>>> > > are
>>>> > > > > > > served
>>>> > > > > > > > >> > better
>>>> > > > > > > > >> > > by
>>>> > > > > > > > >> > > > simply using hbase-shaded-client).
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > We could either remove the Hadoop libraries from
>>>> either
>>>> > > or
>>>> > > > > > both
>>>> > > > > > > of
>>>> > > > > > > > >> the
>>>> > > > > > > > >> > > > assemblies unconditionally, or provide two
>>>> variants for
>>>> > > > > either
>>>> > > > > > > or
>>>> > > > > > > > >> both
>>>> > > > > > > > >> > > > assemblies, one with Hadoop included, and one
>>>> without
>>>> > > it.
>>>> > > > > > > > >> > > > Spark already does this, it has binary
>>>> distributions
>>>> > > both
>>>> > > > > with
>>>> > > > > > > and
>>>> > > > > > > > >> > > without
>>>> > > > > > > > >> > > > Hadoop.
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > The advantages would be:
>>>> > > > > > > > >> > > > * Smaller distro size
>>>> > > > > > > > >> > > > * Faster startup (this is marginal)
>>>> > > > > > > > >> > > > * Less chance of conflicts with the Hadoop jars
>>>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > Thirdly, we could consider excluding the
>>>> > > > > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client
>>>> JAR from
>>>> > > the
>>>> > > > > > > > >> Hadoop-less
>>>> > > > > > > > >> > > > binary assemblies. It is not used by the
>>>> assembly, and
>>>> > > AFAIK
>>>> > > > > > it
>>>> > > > > > > is
>>>> > > > > > > > >> not
>>>> > > > > > > > >> > > > included in any of the 'hbase classpath' command
>>>> > > variants.
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > This would make sure that no Hadoop libraries are
>>>> > > included
>>>> > > > > > (even
>>>> > > > > > > > in
>>>> > > > > > > > >> > > shaded
>>>> > > > > > > > >> > > > form) and would make the HBase distribution fully
>>>> > > insulated
>>>> > > > > > from
>>>> > > > > > > > >> > Hadoop's
>>>> > > > > > > > >> > > > CVE issues.
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > (The full-fat hbase-shaded-client works best as
>>>> direct
>>>> > > > > > > build-time
>>>> > > > > > > > >> > > > dependency anyway)
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > > > best regards
>>>> > > > > > > > >> > > > Istvan
>>>> > > > > > > > >> > > >
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> >
>>>> > > > > > > > >>
>>>> > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > > --
>>>> > > > > > > > > *István Tóth* | Sr. Staff Software Engineer
>>>> > > > > > > > > *Email*: stoty@cloudera.com
>>>> > > > > > > > > cloudera.com <https://www.cloudera.com>
>>>> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
>>>> > > > > > > > > [image: Cloudera on Twitter] <
>>>> https://twitter.com/cloudera>
>>>> > > > > [image:
>>>> > > > > > > > > Cloudera on Facebook] <
>>>> https://www.facebook.com/cloudera>
>>>> > > [image:
>>>> > > > > > > > > Cloudera on LinkedIn] <
>>>> > > https://www.linkedin.com/company/cloudera>
>>>> > > > > > > > > ------------------------------
>>>> > > > > > > > > ------------------------------
>>>> > > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > --
>>>> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
>>>> > > > > > > > *Email*: stoty@cloudera.com
>>>> > > > > > > > cloudera.com <https://www.cloudera.com>
>>>> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
>>>> > > > > > > > [image: Cloudera on Twitter] <
>>>> https://twitter.com/cloudera>
>>>> > > [image:
>>>> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>>>> > > [image:
>>>> > > > > > > Cloudera
>>>> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>>> > > > > > > > ------------------------------
>>>> > > > > > > > ------------------------------
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > --
>>>> > > > > > *István Tóth* | Sr. Staff Software Engineer
>>>> > > > > > *Email*: stoty@cloudera.com
>>>> > > > > > cloudera.com <https://www.cloudera.com>
>>>> > > > > > [image: Cloudera] <https://www.cloudera.com/>
>>>> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>>>> [image:
>>>> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>>>> [image:
>>>> > > > > Cloudera
>>>> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>>> > > > > > ------------------------------
>>>> > > > > > ------------------------------
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > > >
>>>> > > > --
>>>> > > > *István Tóth* | Sr. Staff Software Engineer
>>>> > > > *Email*: stoty@cloudera.com
>>>> > > > cloudera.com <https://www.cloudera.com>
>>>> > > > [image: Cloudera] <https://www.cloudera.com/>
>>>> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>>>> [image:
>>>> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>>>> > > Cloudera
>>>> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>>> > > > ------------------------------
>>>> > > > ------------------------------
>>>> > >
>>>>
>>>
>>>
>>> --
>>> *István Tóth* | Sr. Staff Software Engineer
>>> *Email*: stoty@cloudera.com
>>> cloudera.com <https://www.cloudera.com>
>>> [image: Cloudera] <https://www.cloudera.com/>
>>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>>> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>> ------------------------------
>>> ------------------------------
>>>
>>
>>
>> --
>> *István Tóth* | Sr. Staff Software Engineer
>> *Email*: stoty@cloudera.com
>> cloudera.com <https://www.cloudera.com>
>> [image: Cloudera] <https://www.cloudera.com/>
>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> ------------------------------
>> ------------------------------
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
If we don't want to publish the testing assembly, we could also just add a
profile to hbase-assembly,
so that the testing assembly can be built from the same module by enabling
the profile.
On Mon, Mar 17, 2025 at 7:22 AM Istvan Toth <st...@cloudera.com> wrote:
> I have opened HBASE-29187 for this.
>
> Should we publish this testing assembly for releases ?
> Should this testing assembly be built by default, or should we gate it
> with some property ?
>
> Istvan
>
>
> On Sat, Mar 15, 2025 at 7:13 PM Istvan Toth <st...@cloudera.com> wrote:
>
>> I will work on it next week.
>>
>> On Sat, Mar 15, 2025 at 3:01 PM 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>>
>>> Do we have an issue for it?
>>>
>>> At least for releasing 3.0.0, we need to run ITBLL...
>>>
>>> Thanks.
>>>
>>> Istvan Toth <st...@apache.org> 于2025年3月15日周六 12:32写道：
>>> >
>>> > Yes, we've discussed this issue, but deferred solving it.
>>> >
>>> > IMO the easiest way is to add a third assembly which is functionally
>>> > equivalent to the 2.x assembly.
>>> > That could work for running hbase-it, ITBLL, chaos monkey, etc.
>>> > We'd also have to decide whether to build it by default, and whether we
>>> > want to publish it as part of official releases.
>>> >
>>> > In theory could also make a delta assembly that only includes the
>>> > additional test related stuff, but I'm afraid that that would require
>>> a lot
>>> > of maintenance.
>>> > We could also add a script/maven target that downloads test-related
>>> JARs
>>> > from maven, but keeping that one up to date would also be problematic.
>>> >
>>> > Istvan
>>> >
>>> > On Sat, Mar 15, 2025 at 4:51 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>> wrote:
>>> >
>>> > > After this change, we can not run ITBLL on 3.0.0 because hbase-it is
>>> > > also excluded...
>>> > >
>>> > > I tried manually copying all the tests jar and hbase-it jar to the
>>> lib
>>> > > directory but it did not work, I guess we still missed several hadoop
>>> > > jars...
>>> > >
>>> > > So what is the suggested way to run ITBLL after this change?
>>> > >
>>> > > Thanks.
>>> > >
>>> > > Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
>>> > > >
>>> > > > This is almost done.
>>> > > >
>>> > > > The final outstanding patch is
>>> https://github.com/apache/hbase/pull/5766
>>> > > > for the new Hadoop-less assembly.
>>> > > >
>>> > > > Could you please review it ?
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com>
>>> > > wrote:
>>> > > >
>>> > > > > I have created sub tasks with necessary details in the umbrella
>>> jira.
>>> > > Will
>>> > > > > take them up in coming days. Also will add more sub tasks later
>>> if
>>> > > needed.
>>> > > > >
>>> > > > > Regards
>>> > > > > Nihal
>>> > > > >
>>> > > > > On Sat, 9 Mar 2024, 11:53 Istvan Toth,
>>> <st...@cloudera.com.invalid>
>>> > > wrote:
>>> > > > >
>>> > > > > > Thank you Nihal.
>>> > > > > > I'm not very familiar with the tools in the test code, so you
>>> can
>>> > > > > probably
>>> > > > > > plan that work better.
>>> > > > > > I just have some generic steps in mind:
>>> > > > > > * Identify all the tools / scripts in the test jars
>>> > > > > > * Identify and analyze their dependencies (compared to the
>>> current
>>> > > > > runtime
>>> > > > > > deps)
>>> > > > > > * Decide which ones to move to the runtime JARs.
>>> > > > > > * Move them to the runtime code (or perhaps a separate module)
>>> > > > > >
>>> > > > > > I have created
>>> https://issues.apache.org/jira/browse/HBASE-28431 as
>>> > > an
>>> > > > > > umbrella ticket to organize the sub-tasks.
>>> > > > > >
>>> > > > > > Istvan
>>> > > > > >
>>> > > > > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <
>>> nihaljain.cs@gmail.com>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > Sure I will be able to take up. Please create tasks with
>>> necessary
>>> > > > > > details
>>> > > > > > > or let me know if you want me to create.
>>> > > > > > >
>>> > > > > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth,
>>> <stoty@cloudera.com.invalid
>>> > > >
>>> > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Thanks for volunteering, Nihal.
>>> > > > > > > >
>>> > > > > > > > I could work on the Hadoop-less, and assemblies, and you
>>> could
>>> > > work
>>> > > > > on
>>> > > > > > > > cleaning up the test jars.
>>> > > > > > > > Would that work for you ?
>>> > > > > > > > I know that I'm picking the smaller part, but it turns out
>>> that I
>>> > > > > won't
>>> > > > > > > > have as much time to work on this as I hoped.
>>> > > > > > > >
>>> > > > > > > > (Unless there are other volunteers, of course)
>>> > > > > > > >
>>> > > > > > > > Istvan
>>> > > > > > > >
>>> > > > > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <
>>> stoty@cloudera.com>
>>> > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > We seem to be in agreement in principle, however the
>>> devil is
>>> > > in
>>> > > > > the
>>> > > > > > > > > details.
>>> > > > > > > > >
>>> > > > > > > > > The first step should be moving the diagnostic tools out
>>> of the
>>> > > > > test
>>> > > > > > > > jars.
>>> > > > > > > > > Are there any tools we don't want to move out ?
>>> > > > > > > > > Do the diagnostic tools pull in extra dependencies
>>> compared to
>>> > > the
>>> > > > > > > > current
>>> > > > > > > > > runtime JARs, and if they do, what are those ?
>>> > > > > > > > > I haven't thought of the chaosmonkey tests yet, do those
>>> have
>>> > > > > > specific
>>> > > > > > > > > additional dependencies / scripts ?
>>> > > > > > > > >
>>> > > > > > > > > Should we move the tools simply to the normal jars, or
>>> should
>>> > > we
>>> > > > > move
>>> > > > > > > > them
>>> > > > > > > > > to a new module (could be called hbase-diagnostics) ?
>>> > > > > > > > >
>>> > > > > > > > > Istvan
>>> > > > > > > > >
>>> > > > > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
>>> > > > > > > > bbeaudreault@apache.org>
>>> > > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > >> I'm +0 on hbase-examples, but +1000000 on any
>>> improvements we
>>> > > can
>>> > > > > > make
>>> > > > > > > > to
>>> > > > > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely
>>> frustrating how
>>> > > much
>>> > > > > > > > reliance
>>> > > > > > > > >> we have on test jars both generally but also
>>> specifically
>>> > > around
>>> > > > > > these
>>> > > > > > > > >> core
>>> > > > > > > > >> test executables. Unfortunately I haven't had time to
>>> > > dedicate to
>>> > > > > > > these
>>> > > > > > > > >> frustrations myself, but happy to help with review, etc.
>>> > > > > > > > >>
>>> > > > > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <
>>> > > nihaljain.cs@gmail.com
>>> > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > > >>
>>> > > > > > > > >> > Thank you for bringing this up.
>>> > > > > > > > >> >
>>> > > > > > > > >> > +1 for this change.
>>> > > > > > > > >> >
>>> > > > > > > > >> > In fact, some time back, we had faced similar problem.
>>> > > Security
>>> > > > > > > scans
>>> > > > > > > > >> found
>>> > > > > > > > >> > that we were bundling some vulnerable hadoop test
>>> jar. To
>>> > > deal
>>> > > > > > with
>>> > > > > > > > >> that we
>>> > > > > > > > >> > had to make a change in our internal HBase fork to
>>> exclude
>>> > > all
>>> > > > > > HBase
>>> > > > > > > > and
>>> > > > > > > > >> > Hadoop test jars from assembly. This helped us get
>>> rid of
>>> > > > > > vulnerable
>>> > > > > > > > >> jar.
>>> > > > > > > > >> > (Although I hadn't dealt with test scope dependencies
>>> > > there.)
>>> > > > > > > > >> >
>>> > > > > > > > >> > But, I have been thinking of pushing this change in
>>> Apache
>>> > > > > HBase,
>>> > > > > > > just
>>> > > > > > > > >> > wasn't sure if this was even acceptable. It's great
>>> to see
>>> > > same
>>> > > > > > has
>>> > > > > > > > been
>>> > > > > > > > >> > brought up here today.
>>> > > > > > > > >> >
>>> > > > > > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote
>>> a
>>> > > script
>>> > > > > to
>>> > > > > > > > >> download
>>> > > > > > > > >> > them on demand to avoid massive code change in
>>> internal
>>> > > fork.
>>> > > > > But
>>> > > > > > I
>>> > > > > > > > >> have a
>>> > > > > > > > >> > +1 on the idea of identifying and moving all such
>>> tools to
>>> > > a new
>>> > > > > > > > module.
>>> > > > > > > > >> > This would be great and make things easier for us as
>>> well.
>>> > > > > > > > >> >
>>> > > > > > > > >> > Also, a way we could help new users easily get
>>> started, in
>>> > > case
>>> > > > > we
>>> > > > > > > > >> > completely stop bundling hadoop jars, is by providing
>>> a
>>> > > script
>>> > > > > > which
>>> > > > > > > > >> starts
>>> > > > > > > > >> > a hbase cluster in a single node setup. In fact I had
>>> > > written a
>>> > > > > > > simple
>>> > > > > > > > >> > script sometime back that automates this process
>>> given a
>>> > > release
>>> > > > > > > link
>>> > > > > > > > >> for
>>> > > > > > > > >> > both. It first downloads Hadoop and HBase binaries
>>> and then
>>> > > > > starts
>>> > > > > > > > both
>>> > > > > > > > >> > with the hbase root directory set to be on hdfs. We
>>> could
>>> > > > > provide
>>> > > > > > > > >> something
>>> > > > > > > > >> > similar to help new users to get started easily.
>>> > > > > > > > >> >
>>> > > > > > > > >> > Although I am also +1 on the idea to provide both
>>> variants
>>> > > as
>>> > > > > > > > mentioned
>>> > > > > > > > >> by
>>> > > > > > > > >> > Nick, which might not even need any such script.
>>> > > > > > > > >> >
>>> > > > > > > > >> > Also, I am willing to volunteer for help towards this
>>> > > effort.
>>> > > > > > Please
>>> > > > > > > > >> let me
>>> > > > > > > > >> > know if anything is needed.
>>> > > > > > > > >> >
>>> > > > > > > > >> > Thanks,
>>> > > > > > > > >> > Nihal
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <
>>> > > ndimiduk@apache.org>
>>> > > > > > > wrote:
>>> > > > > > > > >> >
>>> > > > > > > > >> > > This would be great cleanup, big +1 from me for all
>>> three
>>> > > of
>>> > > > > > these
>>> > > > > > > > >> > > adjustments, including the promotion of pe, ltt, and
>>> > > friends
>>> > > > > out
>>> > > > > > > of
>>> > > > > > > > >> the
>>> > > > > > > > >> > > test scope.
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > I believe that we included hbase test jars because
>>> we
>>> > > used to
>>> > > > > > > freely
>>> > > > > > > > >> mix
>>> > > > > > > > >> > > classes needed for minicluster between runtime and
>>> test
>>> > > jars,
>>> > > > > > > which
>>> > > > > > > > in
>>> > > > > > > > >> > turn
>>> > > > > > > > >> > > relied on Hadoop minicluster capabilities. The big
>>> cleanup
>>> > > > > > around
>>> > > > > > > > >> > > HBaseTestingUtil/it addressed much (or all) of these
>>> > > issues on
>>> > > > > > > > >> branch-3.
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > I believe that we include a Hadoop distribution in
>>> our
>>> > > > > assembly
>>> > > > > > > > >> because
>>> > > > > > > > >> > > that makes it easy for a new user to download our
>>> release
>>> > > > > > bin.tgz
>>> > > > > > > > and
>>> > > > > > > > >> get
>>> > > > > > > > >> > > started immediately with learning. I guess it’s
>>> high time
>>> > > that
>>> > > > > > we
>>> > > > > > > > work
>>> > > > > > > > >> > out
>>> > > > > > > > >> > > the with- and without-Hadoop variants.
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > Thanks,
>>> > > > > > > > >> > > Nick
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <
>>> > > stoty@apache.org>
>>> > > > > > > wrote:
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an
>>> > > elegant
>>> > > > > way
>>> > > > > > > > >> mapped
>>> > > > > > > > >> > out
>>> > > > > > > > >> > > > to achieve this, this is about discussing whether
>>> we
>>> > > even
>>> > > > > want
>>> > > > > > > to
>>> > > > > > > > >> make
>>> > > > > > > > >> > > > these changes.
>>> > > > > > > > >> > > > These are also substantial changes, but they
>>> could be
>>> > > > > targeted
>>> > > > > > > for
>>> > > > > > > > >> > HBase
>>> > > > > > > > >> > > > 3.0.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > One issue I have noticed is that we ship test
>>> jars and
>>> > > test
>>> > > > > > > > >> > dependencies
>>> > > > > > > > >> > > in
>>> > > > > > > > >> > > > the assembly.
>>> > > > > > > > >> > > > I can't see anyone using those, but it bloats the
>>> > > assembly
>>> > > > > and
>>> > > > > > > > >> > classpath,
>>> > > > > > > > >> > > > and adds unnecessary JARs with possible CVE
>>> issues. (for
>>> > > > > > example
>>> > > > > > > > >> Kerby
>>> > > > > > > > >> > > > which is a Hadoop minicluster dependency)
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > My proposal is to exclude the test jars and the
>>> test
>>> > > scope
>>> > > > > > > > >> dependencies
>>> > > > > > > > >> > > > from the assembly.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > The advantages would be:
>>> > > > > > > > >> > > > * Smaller distro size
>>> > > > > > > > >> > > > * Faster startup (this is marginal)
>>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > The other issue is that the assembly includes
>>> much of
>>> > > the
>>> > > > > > Hadoop
>>> > > > > > > > >> > > > distribution.
>>> > > > > > > > >> > > > The basic assumption in all scripts and
>>> instructions is
>>> > > that
>>> > > > > > the
>>> > > > > > > > >> node
>>> > > > > > > > >> > > has a
>>> > > > > > > > >> > > > fully configured Hadoop installation, and we
>>> include it
>>> > > in
>>> > > > > the
>>> > > > > > > > >> > classpath
>>> > > > > > > > >> > > of
>>> > > > > > > > >> > > > HBase.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > If that is true, then there is no reason to
>>> include
>>> > > Hadoop
>>> > > > > in
>>> > > > > > > the
>>> > > > > > > > >> > > assembly,
>>> > > > > > > > >> > > > HBase and its direct dependencies should be
>>> enough.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > One could argue that it would simplify the client
>>> side,
>>> > > > > which
>>> > > > > > is
>>> > > > > > > > >> true
>>> > > > > > > > >> > to
>>> > > > > > > > >> > > > some extent (though 95% of the client distro use
>>> cases
>>> > > are
>>> > > > > > > served
>>> > > > > > > > >> > better
>>> > > > > > > > >> > > by
>>> > > > > > > > >> > > > simply using hbase-shaded-client).
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > We could either remove the Hadoop libraries from
>>> either
>>> > > or
>>> > > > > > both
>>> > > > > > > of
>>> > > > > > > > >> the
>>> > > > > > > > >> > > > assemblies unconditionally, or provide two
>>> variants for
>>> > > > > either
>>> > > > > > > or
>>> > > > > > > > >> both
>>> > > > > > > > >> > > > assemblies, one with Hadoop included, and one
>>> without
>>> > > it.
>>> > > > > > > > >> > > > Spark already does this, it has binary
>>> distributions
>>> > > both
>>> > > > > with
>>> > > > > > > and
>>> > > > > > > > >> > > without
>>> > > > > > > > >> > > > Hadoop.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > The advantages would be:
>>> > > > > > > > >> > > > * Smaller distro size
>>> > > > > > > > >> > > > * Faster startup (this is marginal)
>>> > > > > > > > >> > > > * Less chance of conflicts with the Hadoop jars
>>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > Thirdly, we could consider excluding the
>>> > > > > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR
>>> from
>>> > > the
>>> > > > > > > > >> Hadoop-less
>>> > > > > > > > >> > > > binary assemblies. It is not used by the
>>> assembly, and
>>> > > AFAIK
>>> > > > > > it
>>> > > > > > > is
>>> > > > > > > > >> not
>>> > > > > > > > >> > > > included in any of the 'hbase classpath' command
>>> > > variants.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > This would make sure that no Hadoop libraries are
>>> > > included
>>> > > > > > (even
>>> > > > > > > > in
>>> > > > > > > > >> > > shaded
>>> > > > > > > > >> > > > form) and would make the HBase distribution fully
>>> > > insulated
>>> > > > > > from
>>> > > > > > > > >> > Hadoop's
>>> > > > > > > > >> > > > CVE issues.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > (The full-fat hbase-shaded-client works best as
>>> direct
>>> > > > > > > build-time
>>> > > > > > > > >> > > > dependency anyway)
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > best regards
>>> > > > > > > > >> > > > Istvan
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > --
>>> > > > > > > > > *István Tóth* | Sr. Staff Software Engineer
>>> > > > > > > > > *Email*: stoty@cloudera.com
>>> > > > > > > > > cloudera.com <https://www.cloudera.com>
>>> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
>>> > > > > > > > > [image: Cloudera on Twitter] <
>>> https://twitter.com/cloudera>
>>> > > > > [image:
>>> > > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera
>>> >
>>> > > [image:
>>> > > > > > > > > Cloudera on LinkedIn] <
>>> > > https://www.linkedin.com/company/cloudera>
>>> > > > > > > > > ------------------------------
>>> > > > > > > > > ------------------------------
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > --
>>> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
>>> > > > > > > > *Email*: stoty@cloudera.com
>>> > > > > > > > cloudera.com <https://www.cloudera.com>
>>> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
>>> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera
>>> >
>>> > > [image:
>>> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>>> > > [image:
>>> > > > > > > Cloudera
>>> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>> > > > > > > > ------------------------------
>>> > > > > > > > ------------------------------
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > *István Tóth* | Sr. Staff Software Engineer
>>> > > > > > *Email*: stoty@cloudera.com
>>> > > > > > cloudera.com <https://www.cloudera.com>
>>> > > > > > [image: Cloudera] <https://www.cloudera.com/>
>>> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>>> [image:
>>> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>>> [image:
>>> > > > > Cloudera
>>> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>> > > > > > ------------------------------
>>> > > > > > ------------------------------
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > *István Tóth* | Sr. Staff Software Engineer
>>> > > > *Email*: stoty@cloudera.com
>>> > > > cloudera.com <https://www.cloudera.com>
>>> > > > [image: Cloudera] <https://www.cloudera.com/>
>>> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>>> [image:
>>> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>>> > > Cloudera
>>> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>> > > > ------------------------------
>>> > > > ------------------------------
>>> > >
>>>
>>
>>
>> --
>> *István Tóth* | Sr. Staff Software Engineer
>> *Email*: stoty@cloudera.com
>> cloudera.com <https://www.cloudera.com>
>> [image: Cloudera] <https://www.cloudera.com/>
>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> ------------------------------
>> ------------------------------
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
I have opened HBASE-29187 for this.
Should we publish this testing assembly for releases ?
Should this testing assembly be built by default, or should we gate it with
some property ?
Istvan
On Sat, Mar 15, 2025 at 7:13 PM Istvan Toth <st...@cloudera.com> wrote:
> I will work on it next week.
>
> On Sat, Mar 15, 2025 at 3:01 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Do we have an issue for it?
>>
>> At least for releasing 3.0.0, we need to run ITBLL...
>>
>> Thanks.
>>
>> Istvan Toth <st...@apache.org> 于2025年3月15日周六 12:32写道：
>> >
>> > Yes, we've discussed this issue, but deferred solving it.
>> >
>> > IMO the easiest way is to add a third assembly which is functionally
>> > equivalent to the 2.x assembly.
>> > That could work for running hbase-it, ITBLL, chaos monkey, etc.
>> > We'd also have to decide whether to build it by default, and whether we
>> > want to publish it as part of official releases.
>> >
>> > In theory could also make a delta assembly that only includes the
>> > additional test related stuff, but I'm afraid that that would require a
>> lot
>> > of maintenance.
>> > We could also add a script/maven target that downloads test-related JARs
>> > from maven, but keeping that one up to date would also be problematic.
>> >
>> > Istvan
>> >
>> > On Sat, Mar 15, 2025 at 4:51 AM 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>> >
>> > > After this change, we can not run ITBLL on 3.0.0 because hbase-it is
>> > > also excluded...
>> > >
>> > > I tried manually copying all the tests jar and hbase-it jar to the lib
>> > > directory but it did not work, I guess we still missed several hadoop
>> > > jars...
>> > >
>> > > So what is the suggested way to run ITBLL after this change?
>> > >
>> > > Thanks.
>> > >
>> > > Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
>> > > >
>> > > > This is almost done.
>> > > >
>> > > > The final outstanding patch is
>> https://github.com/apache/hbase/pull/5766
>> > > > for the new Hadoop-less assembly.
>> > > >
>> > > > Could you please review it ?
>> > > >
>> > > >
>> > > >
>> > > > On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > I have created sub tasks with necessary details in the umbrella
>> jira.
>> > > Will
>> > > > > take them up in coming days. Also will add more sub tasks later if
>> > > needed.
>> > > > >
>> > > > > Regards
>> > > > > Nihal
>> > > > >
>> > > > > On Sat, 9 Mar 2024, 11:53 Istvan Toth, <stoty@cloudera.com.invalid
>> >
>> > > wrote:
>> > > > >
>> > > > > > Thank you Nihal.
>> > > > > > I'm not very familiar with the tools in the test code, so you
>> can
>> > > > > probably
>> > > > > > plan that work better.
>> > > > > > I just have some generic steps in mind:
>> > > > > > * Identify all the tools / scripts in the test jars
>> > > > > > * Identify and analyze their dependencies (compared to the
>> current
>> > > > > runtime
>> > > > > > deps)
>> > > > > > * Decide which ones to move to the runtime JARs.
>> > > > > > * Move them to the runtime code (or perhaps a separate module)
>> > > > > >
>> > > > > > I have created
>> https://issues.apache.org/jira/browse/HBASE-28431 as
>> > > an
>> > > > > > umbrella ticket to organize the sub-tasks.
>> > > > > >
>> > > > > > Istvan
>> > > > > >
>> > > > > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <
>> nihaljain.cs@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Sure I will be able to take up. Please create tasks with
>> necessary
>> > > > > > details
>> > > > > > > or let me know if you want me to create.
>> > > > > > >
>> > > > > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth,
>> <stoty@cloudera.com.invalid
>> > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Thanks for volunteering, Nihal.
>> > > > > > > >
>> > > > > > > > I could work on the Hadoop-less, and assemblies, and you
>> could
>> > > work
>> > > > > on
>> > > > > > > > cleaning up the test jars.
>> > > > > > > > Would that work for you ?
>> > > > > > > > I know that I'm picking the smaller part, but it turns out
>> that I
>> > > > > won't
>> > > > > > > > have as much time to work on this as I hoped.
>> > > > > > > >
>> > > > > > > > (Unless there are other volunteers, of course)
>> > > > > > > >
>> > > > > > > > Istvan
>> > > > > > > >
>> > > > > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <
>> stoty@cloudera.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > We seem to be in agreement in principle, however the
>> devil is
>> > > in
>> > > > > the
>> > > > > > > > > details.
>> > > > > > > > >
>> > > > > > > > > The first step should be moving the diagnostic tools out
>> of the
>> > > > > test
>> > > > > > > > jars.
>> > > > > > > > > Are there any tools we don't want to move out ?
>> > > > > > > > > Do the diagnostic tools pull in extra dependencies
>> compared to
>> > > the
>> > > > > > > > current
>> > > > > > > > > runtime JARs, and if they do, what are those ?
>> > > > > > > > > I haven't thought of the chaosmonkey tests yet, do those
>> have
>> > > > > > specific
>> > > > > > > > > additional dependencies / scripts ?
>> > > > > > > > >
>> > > > > > > > > Should we move the tools simply to the normal jars, or
>> should
>> > > we
>> > > > > move
>> > > > > > > > them
>> > > > > > > > > to a new module (could be called hbase-diagnostics) ?
>> > > > > > > > >
>> > > > > > > > > Istvan
>> > > > > > > > >
>> > > > > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
>> > > > > > > > bbeaudreault@apache.org>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> I'm +0 on hbase-examples, but +1000000 on any
>> improvements we
>> > > can
>> > > > > > make
>> > > > > > > > to
>> > > > > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating
>> how
>> > > much
>> > > > > > > > reliance
>> > > > > > > > >> we have on test jars both generally but also specifically
>> > > around
>> > > > > > these
>> > > > > > > > >> core
>> > > > > > > > >> test executables. Unfortunately I haven't had time to
>> > > dedicate to
>> > > > > > > these
>> > > > > > > > >> frustrations myself, but happy to help with review, etc.
>> > > > > > > > >>
>> > > > > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <
>> > > nihaljain.cs@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >>
>> > > > > > > > >> > Thank you for bringing this up.
>> > > > > > > > >> >
>> > > > > > > > >> > +1 for this change.
>> > > > > > > > >> >
>> > > > > > > > >> > In fact, some time back, we had faced similar problem.
>> > > Security
>> > > > > > > scans
>> > > > > > > > >> found
>> > > > > > > > >> > that we were bundling some vulnerable hadoop test jar.
>> To
>> > > deal
>> > > > > > with
>> > > > > > > > >> that we
>> > > > > > > > >> > had to make a change in our internal HBase fork to
>> exclude
>> > > all
>> > > > > > HBase
>> > > > > > > > and
>> > > > > > > > >> > Hadoop test jars from assembly. This helped us get rid
>> of
>> > > > > > vulnerable
>> > > > > > > > >> jar.
>> > > > > > > > >> > (Although I hadn't dealt with test scope dependencies
>> > > there.)
>> > > > > > > > >> >
>> > > > > > > > >> > But, I have been thinking of pushing this change in
>> Apache
>> > > > > HBase,
>> > > > > > > just
>> > > > > > > > >> > wasn't sure if this was even acceptable. It's great to
>> see
>> > > same
>> > > > > > has
>> > > > > > > > been
>> > > > > > > > >> > brought up here today.
>> > > > > > > > >> >
>> > > > > > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a
>> > > script
>> > > > > to
>> > > > > > > > >> download
>> > > > > > > > >> > them on demand to avoid massive code change in internal
>> > > fork.
>> > > > > But
>> > > > > > I
>> > > > > > > > >> have a
>> > > > > > > > >> > +1 on the idea of identifying and moving all such
>> tools to
>> > > a new
>> > > > > > > > module.
>> > > > > > > > >> > This would be great and make things easier for us as
>> well.
>> > > > > > > > >> >
>> > > > > > > > >> > Also, a way we could help new users easily get
>> started, in
>> > > case
>> > > > > we
>> > > > > > > > >> > completely stop bundling hadoop jars, is by providing a
>> > > script
>> > > > > > which
>> > > > > > > > >> starts
>> > > > > > > > >> > a hbase cluster in a single node setup. In fact I had
>> > > written a
>> > > > > > > simple
>> > > > > > > > >> > script sometime back that automates this process given
>> a
>> > > release
>> > > > > > > link
>> > > > > > > > >> for
>> > > > > > > > >> > both. It first downloads Hadoop and HBase binaries and
>> then
>> > > > > starts
>> > > > > > > > both
>> > > > > > > > >> > with the hbase root directory set to be on hdfs. We
>> could
>> > > > > provide
>> > > > > > > > >> something
>> > > > > > > > >> > similar to help new users to get started easily.
>> > > > > > > > >> >
>> > > > > > > > >> > Although I am also +1 on the idea to provide both
>> variants
>> > > as
>> > > > > > > > mentioned
>> > > > > > > > >> by
>> > > > > > > > >> > Nick, which might not even need any such script.
>> > > > > > > > >> >
>> > > > > > > > >> > Also, I am willing to volunteer for help towards this
>> > > effort.
>> > > > > > Please
>> > > > > > > > >> let me
>> > > > > > > > >> > know if anything is needed.
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks,
>> > > > > > > > >> > Nihal
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <
>> > > ndimiduk@apache.org>
>> > > > > > > wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > This would be great cleanup, big +1 from me for all
>> three
>> > > of
>> > > > > > these
>> > > > > > > > >> > > adjustments, including the promotion of pe, ltt, and
>> > > friends
>> > > > > out
>> > > > > > > of
>> > > > > > > > >> the
>> > > > > > > > >> > > test scope.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I believe that we included hbase test jars because we
>> > > used to
>> > > > > > > freely
>> > > > > > > > >> mix
>> > > > > > > > >> > > classes needed for minicluster between runtime and
>> test
>> > > jars,
>> > > > > > > which
>> > > > > > > > in
>> > > > > > > > >> > turn
>> > > > > > > > >> > > relied on Hadoop minicluster capabilities. The big
>> cleanup
>> > > > > > around
>> > > > > > > > >> > > HBaseTestingUtil/it addressed much (or all) of these
>> > > issues on
>> > > > > > > > >> branch-3.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I believe that we include a Hadoop distribution in
>> our
>> > > > > assembly
>> > > > > > > > >> because
>> > > > > > > > >> > > that makes it easy for a new user to download our
>> release
>> > > > > > bin.tgz
>> > > > > > > > and
>> > > > > > > > >> get
>> > > > > > > > >> > > started immediately with learning. I guess it’s high
>> time
>> > > that
>> > > > > > we
>> > > > > > > > work
>> > > > > > > > >> > out
>> > > > > > > > >> > > the with- and without-Hadoop variants.
>> > > > > > > > >> > >
>> > > > > > > > >> > > Thanks,
>> > > > > > > > >> > > Nick
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <
>> > > stoty@apache.org>
>> > > > > > > wrote:
>> > > > > > > > >> > >
>> > > > > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an
>> > > elegant
>> > > > > way
>> > > > > > > > >> mapped
>> > > > > > > > >> > out
>> > > > > > > > >> > > > to achieve this, this is about discussing whether
>> we
>> > > even
>> > > > > want
>> > > > > > > to
>> > > > > > > > >> make
>> > > > > > > > >> > > > these changes.
>> > > > > > > > >> > > > These are also substantial changes, but they could
>> be
>> > > > > targeted
>> > > > > > > for
>> > > > > > > > >> > HBase
>> > > > > > > > >> > > > 3.0.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > One issue I have noticed is that we ship test jars
>> and
>> > > test
>> > > > > > > > >> > dependencies
>> > > > > > > > >> > > in
>> > > > > > > > >> > > > the assembly.
>> > > > > > > > >> > > > I can't see anyone using those, but it bloats the
>> > > assembly
>> > > > > and
>> > > > > > > > >> > classpath,
>> > > > > > > > >> > > > and adds unnecessary JARs with possible CVE
>> issues. (for
>> > > > > > example
>> > > > > > > > >> Kerby
>> > > > > > > > >> > > > which is a Hadoop minicluster dependency)
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > My proposal is to exclude the test jars and the
>> test
>> > > scope
>> > > > > > > > >> dependencies
>> > > > > > > > >> > > > from the assembly.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > The advantages would be:
>> > > > > > > > >> > > > * Smaller distro size
>> > > > > > > > >> > > > * Faster startup (this is marginal)
>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > The other issue is that the assembly includes much
>> of
>> > > the
>> > > > > > Hadoop
>> > > > > > > > >> > > > distribution.
>> > > > > > > > >> > > > The basic assumption in all scripts and
>> instructions is
>> > > that
>> > > > > > the
>> > > > > > > > >> node
>> > > > > > > > >> > > has a
>> > > > > > > > >> > > > fully configured Hadoop installation, and we
>> include it
>> > > in
>> > > > > the
>> > > > > > > > >> > classpath
>> > > > > > > > >> > > of
>> > > > > > > > >> > > > HBase.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > If that is true, then there is no reason to include
>> > > Hadoop
>> > > > > in
>> > > > > > > the
>> > > > > > > > >> > > assembly,
>> > > > > > > > >> > > > HBase and its direct dependencies should be enough.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > One could argue that it would simplify the client
>> side,
>> > > > > which
>> > > > > > is
>> > > > > > > > >> true
>> > > > > > > > >> > to
>> > > > > > > > >> > > > some extent (though 95% of the client distro use
>> cases
>> > > are
>> > > > > > > served
>> > > > > > > > >> > better
>> > > > > > > > >> > > by
>> > > > > > > > >> > > > simply using hbase-shaded-client).
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > We could either remove the Hadoop libraries from
>> either
>> > > or
>> > > > > > both
>> > > > > > > of
>> > > > > > > > >> the
>> > > > > > > > >> > > > assemblies unconditionally, or provide two
>> variants for
>> > > > > either
>> > > > > > > or
>> > > > > > > > >> both
>> > > > > > > > >> > > > assemblies, one with Hadoop included, and one
>> without
>> > > it.
>> > > > > > > > >> > > > Spark already does this, it has binary
>> distributions
>> > > both
>> > > > > with
>> > > > > > > and
>> > > > > > > > >> > > without
>> > > > > > > > >> > > > Hadoop.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > The advantages would be:
>> > > > > > > > >> > > > * Smaller distro size
>> > > > > > > > >> > > > * Faster startup (this is marginal)
>> > > > > > > > >> > > > * Less chance of conflicts with the Hadoop jars
>> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
>> > > > > > > > >> > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Thirdly, we could consider excluding the
>> > > > > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR
>> from
>> > > the
>> > > > > > > > >> Hadoop-less
>> > > > > > > > >> > > > binary assemblies. It is not used by the assembly,
>> and
>> > > AFAIK
>> > > > > > it
>> > > > > > > is
>> > > > > > > > >> not
>> > > > > > > > >> > > > included in any of the 'hbase classpath' command
>> > > variants.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > This would make sure that no Hadoop libraries are
>> > > included
>> > > > > > (even
>> > > > > > > > in
>> > > > > > > > >> > > shaded
>> > > > > > > > >> > > > form) and would make the HBase distribution fully
>> > > insulated
>> > > > > > from
>> > > > > > > > >> > Hadoop's
>> > > > > > > > >> > > > CVE issues.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > (The full-fat hbase-shaded-client works best as
>> direct
>> > > > > > > build-time
>> > > > > > > > >> > > > dependency anyway)
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > best regards
>> > > > > > > > >> > > > Istvan
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > *István Tóth* | Sr. Staff Software Engineer
>> > > > > > > > > *Email*: stoty@cloudera.com
>> > > > > > > > > cloudera.com <https://www.cloudera.com>
>> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
>> > > > > > > > > [image: Cloudera on Twitter] <
>> https://twitter.com/cloudera>
>> > > > > [image:
>> > > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>> > > [image:
>> > > > > > > > > Cloudera on LinkedIn] <
>> > > https://www.linkedin.com/company/cloudera>
>> > > > > > > > > ------------------------------
>> > > > > > > > > ------------------------------
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
>> > > > > > > > *Email*: stoty@cloudera.com
>> > > > > > > > cloudera.com <https://www.cloudera.com>
>> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
>> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>> > > [image:
>> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>> > > [image:
>> > > > > > > Cloudera
>> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> > > > > > > > ------------------------------
>> > > > > > > > ------------------------------
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > *István Tóth* | Sr. Staff Software Engineer
>> > > > > > *Email*: stoty@cloudera.com
>> > > > > > cloudera.com <https://www.cloudera.com>
>> > > > > > [image: Cloudera] <https://www.cloudera.com/>
>> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>> [image:
>> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>> [image:
>> > > > > Cloudera
>> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> > > > > > ------------------------------
>> > > > > > ------------------------------
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > *István Tóth* | Sr. Staff Software Engineer
>> > > > *Email*: stoty@cloudera.com
>> > > > cloudera.com <https://www.cloudera.com>
>> > > > [image: Cloudera] <https://www.cloudera.com/>
>> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> > > Cloudera
>> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> > > > ------------------------------
>> > > > ------------------------------
>> > >
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
I will work on it next week.
On Sat, Mar 15, 2025 at 3:01 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> Do we have an issue for it?
>
> At least for releasing 3.0.0, we need to run ITBLL...
>
> Thanks.
>
> Istvan Toth <st...@apache.org> 于2025年3月15日周六 12:32写道：
> >
> > Yes, we've discussed this issue, but deferred solving it.
> >
> > IMO the easiest way is to add a third assembly which is functionally
> > equivalent to the 2.x assembly.
> > That could work for running hbase-it, ITBLL, chaos monkey, etc.
> > We'd also have to decide whether to build it by default, and whether we
> > want to publish it as part of official releases.
> >
> > In theory could also make a delta assembly that only includes the
> > additional test related stuff, but I'm afraid that that would require a
> lot
> > of maintenance.
> > We could also add a script/maven target that downloads test-related JARs
> > from maven, but keeping that one up to date would also be problematic.
> >
> > Istvan
> >
> > On Sat, Mar 15, 2025 at 4:51 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> >
> > > After this change, we can not run ITBLL on 3.0.0 because hbase-it is
> > > also excluded...
> > >
> > > I tried manually copying all the tests jar and hbase-it jar to the lib
> > > directory but it did not work, I guess we still missed several hadoop
> > > jars...
> > >
> > > So what is the suggested way to run ITBLL after this change?
> > >
> > > Thanks.
> > >
> > > Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
> > > >
> > > > This is almost done.
> > > >
> > > > The final outstanding patch is
> https://github.com/apache/hbase/pull/5766
> > > > for the new Hadoop-less assembly.
> > > >
> > > > Could you please review it ?
> > > >
> > > >
> > > >
> > > > On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com>
> > > wrote:
> > > >
> > > > > I have created sub tasks with necessary details in the umbrella
> jira.
> > > Will
> > > > > take them up in coming days. Also will add more sub tasks later if
> > > needed.
> > > > >
> > > > > Regards
> > > > > Nihal
> > > > >
> > > > > On Sat, 9 Mar 2024, 11:53 Istvan Toth, <stoty@cloudera.com.invalid
> >
> > > wrote:
> > > > >
> > > > > > Thank you Nihal.
> > > > > > I'm not very familiar with the tools in the test code, so you can
> > > > > probably
> > > > > > plan that work better.
> > > > > > I just have some generic steps in mind:
> > > > > > * Identify all the tools / scripts in the test jars
> > > > > > * Identify and analyze their dependencies (compared to the
> current
> > > > > runtime
> > > > > > deps)
> > > > > > * Decide which ones to move to the runtime JARs.
> > > > > > * Move them to the runtime code (or perhaps a separate module)
> > > > > >
> > > > > > I have created https://issues.apache.org/jira/browse/HBASE-28431
> as
> > > an
> > > > > > umbrella ticket to organize the sub-tasks.
> > > > > >
> > > > > > Istvan
> > > > > >
> > > > > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <
> nihaljain.cs@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Sure I will be able to take up. Please create tasks with
> necessary
> > > > > > details
> > > > > > > or let me know if you want me to create.
> > > > > > >
> > > > > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth,
> <stoty@cloudera.com.invalid
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for volunteering, Nihal.
> > > > > > > >
> > > > > > > > I could work on the Hadoop-less, and assemblies, and you
> could
> > > work
> > > > > on
> > > > > > > > cleaning up the test jars.
> > > > > > > > Would that work for you ?
> > > > > > > > I know that I'm picking the smaller part, but it turns out
> that I
> > > > > won't
> > > > > > > > have as much time to work on this as I hoped.
> > > > > > > >
> > > > > > > > (Unless there are other volunteers, of course)
> > > > > > > >
> > > > > > > > Istvan
> > > > > > > >
> > > > > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <
> stoty@cloudera.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > We seem to be in agreement in principle, however the devil
> is
> > > in
> > > > > the
> > > > > > > > > details.
> > > > > > > > >
> > > > > > > > > The first step should be moving the diagnostic tools out
> of the
> > > > > test
> > > > > > > > jars.
> > > > > > > > > Are there any tools we don't want to move out ?
> > > > > > > > > Do the diagnostic tools pull in extra dependencies
> compared to
> > > the
> > > > > > > > current
> > > > > > > > > runtime JARs, and if they do, what are those ?
> > > > > > > > > I haven't thought of the chaosmonkey tests yet, do those
> have
> > > > > > specific
> > > > > > > > > additional dependencies / scripts ?
> > > > > > > > >
> > > > > > > > > Should we move the tools simply to the normal jars, or
> should
> > > we
> > > > > move
> > > > > > > > them
> > > > > > > > > to a new module (could be called hbase-diagnostics) ?
> > > > > > > > >
> > > > > > > > > Istvan
> > > > > > > > >
> > > > > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> > > > > > > > bbeaudreault@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> I'm +0 on hbase-examples, but +1000000 on any
> improvements we
> > > can
> > > > > > make
> > > > > > > > to
> > > > > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating
> how
> > > much
> > > > > > > > reliance
> > > > > > > > >> we have on test jars both generally but also specifically
> > > around
> > > > > > these
> > > > > > > > >> core
> > > > > > > > >> test executables. Unfortunately I haven't had time to
> > > dedicate to
> > > > > > > these
> > > > > > > > >> frustrations myself, but happy to help with review, etc.
> > > > > > > > >>
> > > > > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <
> > > nihaljain.cs@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Thank you for bringing this up.
> > > > > > > > >> >
> > > > > > > > >> > +1 for this change.
> > > > > > > > >> >
> > > > > > > > >> > In fact, some time back, we had faced similar problem.
> > > Security
> > > > > > > scans
> > > > > > > > >> found
> > > > > > > > >> > that we were bundling some vulnerable hadoop test jar.
> To
> > > deal
> > > > > > with
> > > > > > > > >> that we
> > > > > > > > >> > had to make a change in our internal HBase fork to
> exclude
> > > all
> > > > > > HBase
> > > > > > > > and
> > > > > > > > >> > Hadoop test jars from assembly. This helped us get rid
> of
> > > > > > vulnerable
> > > > > > > > >> jar.
> > > > > > > > >> > (Although I hadn't dealt with test scope dependencies
> > > there.)
> > > > > > > > >> >
> > > > > > > > >> > But, I have been thinking of pushing this change in
> Apache
> > > > > HBase,
> > > > > > > just
> > > > > > > > >> > wasn't sure if this was even acceptable. It's great to
> see
> > > same
> > > > > > has
> > > > > > > > been
> > > > > > > > >> > brought up here today.
> > > > > > > > >> >
> > > > > > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a
> > > script
> > > > > to
> > > > > > > > >> download
> > > > > > > > >> > them on demand to avoid massive code change in internal
> > > fork.
> > > > > But
> > > > > > I
> > > > > > > > >> have a
> > > > > > > > >> > +1 on the idea of identifying and moving all such tools
> to
> > > a new
> > > > > > > > module.
> > > > > > > > >> > This would be great and make things easier for us as
> well.
> > > > > > > > >> >
> > > > > > > > >> > Also, a way we could help new users easily get started,
> in
> > > case
> > > > > we
> > > > > > > > >> > completely stop bundling hadoop jars, is by providing a
> > > script
> > > > > > which
> > > > > > > > >> starts
> > > > > > > > >> > a hbase cluster in a single node setup. In fact I had
> > > written a
> > > > > > > simple
> > > > > > > > >> > script sometime back that automates this process given a
> > > release
> > > > > > > link
> > > > > > > > >> for
> > > > > > > > >> > both. It first downloads Hadoop and HBase binaries and
> then
> > > > > starts
> > > > > > > > both
> > > > > > > > >> > with the hbase root directory set to be on hdfs. We
> could
> > > > > provide
> > > > > > > > >> something
> > > > > > > > >> > similar to help new users to get started easily.
> > > > > > > > >> >
> > > > > > > > >> > Although I am also +1 on the idea to provide both
> variants
> > > as
> > > > > > > > mentioned
> > > > > > > > >> by
> > > > > > > > >> > Nick, which might not even need any such script.
> > > > > > > > >> >
> > > > > > > > >> > Also, I am willing to volunteer for help towards this
> > > effort.
> > > > > > Please
> > > > > > > > >> let me
> > > > > > > > >> > know if anything is needed.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Nihal
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <
> > > ndimiduk@apache.org>
> > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > This would be great cleanup, big +1 from me for all
> three
> > > of
> > > > > > these
> > > > > > > > >> > > adjustments, including the promotion of pe, ltt, and
> > > friends
> > > > > out
> > > > > > > of
> > > > > > > > >> the
> > > > > > > > >> > > test scope.
> > > > > > > > >> > >
> > > > > > > > >> > > I believe that we included hbase test jars because we
> > > used to
> > > > > > > freely
> > > > > > > > >> mix
> > > > > > > > >> > > classes needed for minicluster between runtime and
> test
> > > jars,
> > > > > > > which
> > > > > > > > in
> > > > > > > > >> > turn
> > > > > > > > >> > > relied on Hadoop minicluster capabilities. The big
> cleanup
> > > > > > around
> > > > > > > > >> > > HBaseTestingUtil/it addressed much (or all) of these
> > > issues on
> > > > > > > > >> branch-3.
> > > > > > > > >> > >
> > > > > > > > >> > > I believe that we include a Hadoop distribution in our
> > > > > assembly
> > > > > > > > >> because
> > > > > > > > >> > > that makes it easy for a new user to download our
> release
> > > > > > bin.tgz
> > > > > > > > and
> > > > > > > > >> get
> > > > > > > > >> > > started immediately with learning. I guess it’s high
> time
> > > that
> > > > > > we
> > > > > > > > work
> > > > > > > > >> > out
> > > > > > > > >> > > the with- and without-Hadoop variants.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Nick
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <
> > > stoty@apache.org>
> > > > > > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an
> > > elegant
> > > > > way
> > > > > > > > >> mapped
> > > > > > > > >> > out
> > > > > > > > >> > > > to achieve this, this is about discussing whether we
> > > even
> > > > > want
> > > > > > > to
> > > > > > > > >> make
> > > > > > > > >> > > > these changes.
> > > > > > > > >> > > > These are also substantial changes, but they could
> be
> > > > > targeted
> > > > > > > for
> > > > > > > > >> > HBase
> > > > > > > > >> > > > 3.0.
> > > > > > > > >> > > >
> > > > > > > > >> > > > One issue I have noticed is that we ship test jars
> and
> > > test
> > > > > > > > >> > dependencies
> > > > > > > > >> > > in
> > > > > > > > >> > > > the assembly.
> > > > > > > > >> > > > I can't see anyone using those, but it bloats the
> > > assembly
> > > > > and
> > > > > > > > >> > classpath,
> > > > > > > > >> > > > and adds unnecessary JARs with possible CVE issues.
> (for
> > > > > > example
> > > > > > > > >> Kerby
> > > > > > > > >> > > > which is a Hadoop minicluster dependency)
> > > > > > > > >> > > >
> > > > > > > > >> > > > My proposal is to exclude the test jars and the test
> > > scope
> > > > > > > > >> dependencies
> > > > > > > > >> > > > from the assembly.
> > > > > > > > >> > > >
> > > > > > > > >> > > > The advantages would be:
> > > > > > > > >> > > > * Smaller distro size
> > > > > > > > >> > > > * Faster startup (this is marginal)
> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > > > > >> > > >
> > > > > > > > >> > > > The other issue is that the assembly includes much
> of
> > > the
> > > > > > Hadoop
> > > > > > > > >> > > > distribution.
> > > > > > > > >> > > > The basic assumption in all scripts and
> instructions is
> > > that
> > > > > > the
> > > > > > > > >> node
> > > > > > > > >> > > has a
> > > > > > > > >> > > > fully configured Hadoop installation, and we
> include it
> > > in
> > > > > the
> > > > > > > > >> > classpath
> > > > > > > > >> > > of
> > > > > > > > >> > > > HBase.
> > > > > > > > >> > > >
> > > > > > > > >> > > > If that is true, then there is no reason to include
> > > Hadoop
> > > > > in
> > > > > > > the
> > > > > > > > >> > > assembly,
> > > > > > > > >> > > > HBase and its direct dependencies should be enough.
> > > > > > > > >> > > >
> > > > > > > > >> > > > One could argue that it would simplify the client
> side,
> > > > > which
> > > > > > is
> > > > > > > > >> true
> > > > > > > > >> > to
> > > > > > > > >> > > > some extent (though 95% of the client distro use
> cases
> > > are
> > > > > > > served
> > > > > > > > >> > better
> > > > > > > > >> > > by
> > > > > > > > >> > > > simply using hbase-shaded-client).
> > > > > > > > >> > > >
> > > > > > > > >> > > > We could either remove the Hadoop libraries from
> either
> > > or
> > > > > > both
> > > > > > > of
> > > > > > > > >> the
> > > > > > > > >> > > > assemblies unconditionally, or provide two variants
> for
> > > > > either
> > > > > > > or
> > > > > > > > >> both
> > > > > > > > >> > > > assemblies, one with Hadoop included, and one
> without
> > > it.
> > > > > > > > >> > > > Spark already does this, it has binary distributions
> > > both
> > > > > with
> > > > > > > and
> > > > > > > > >> > > without
> > > > > > > > >> > > > Hadoop.
> > > > > > > > >> > > >
> > > > > > > > >> > > > The advantages would be:
> > > > > > > > >> > > > * Smaller distro size
> > > > > > > > >> > > > * Faster startup (this is marginal)
> > > > > > > > >> > > > * Less chance of conflicts with the Hadoop jars
> > > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thirdly, we could consider excluding the
> > > > > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR
> from
> > > the
> > > > > > > > >> Hadoop-less
> > > > > > > > >> > > > binary assemblies. It is not used by the assembly,
> and
> > > AFAIK
> > > > > > it
> > > > > > > is
> > > > > > > > >> not
> > > > > > > > >> > > > included in any of the 'hbase classpath' command
> > > variants.
> > > > > > > > >> > > >
> > > > > > > > >> > > > This would make sure that no Hadoop libraries are
> > > included
> > > > > > (even
> > > > > > > > in
> > > > > > > > >> > > shaded
> > > > > > > > >> > > > form) and would make the HBase distribution fully
> > > insulated
> > > > > > from
> > > > > > > > >> > Hadoop's
> > > > > > > > >> > > > CVE issues.
> > > > > > > > >> > > >
> > > > > > > > >> > > > (The full-fat hbase-shaded-client works best as
> direct
> > > > > > > build-time
> > > > > > > > >> > > > dependency anyway)
> > > > > > > > >> > > >
> > > > > > > > >> > > > best regards
> > > > > > > > >> > > > Istvan
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > > > > *Email*: stoty@cloudera.com
> > > > > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera
> >
> > > > > [image:
> > > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > > [image:
> > > > > > > > > Cloudera on LinkedIn] <
> > > https://www.linkedin.com/company/cloudera>
> > > > > > > > > ------------------------------
> > > > > > > > > ------------------------------
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > > > *Email*: stoty@cloudera.com
> > > > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > > [image:
> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > > [image:
> > > > > > > Cloudera
> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > > > ------------------------------
> > > > > > > > ------------------------------
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > *Email*: stoty@cloudera.com
> > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> [image:
> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> [image:
> > > > > Cloudera
> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > ------------------------------
> > > > > > ------------------------------
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Sr. Staff Software Engineer
> > > > *Email*: stoty@cloudera.com
> > > > cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > > ------------------------------
> > >
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Do we have an issue for it?
At least for releasing 3.0.0, we need to run ITBLL...
Thanks.
Istvan Toth <st...@apache.org> 于2025年3月15日周六 12:32写道：
>
> Yes, we've discussed this issue, but deferred solving it.
>
> IMO the easiest way is to add a third assembly which is functionally
> equivalent to the 2.x assembly.
> That could work for running hbase-it, ITBLL, chaos monkey, etc.
> We'd also have to decide whether to build it by default, and whether we
> want to publish it as part of official releases.
>
> In theory could also make a delta assembly that only includes the
> additional test related stuff, but I'm afraid that that would require a lot
> of maintenance.
> We could also add a script/maven target that downloads test-related JARs
> from maven, but keeping that one up to date would also be problematic.
>
> Istvan
>
> On Sat, Mar 15, 2025 at 4:51 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> > After this change, we can not run ITBLL on 3.0.0 because hbase-it is
> > also excluded...
> >
> > I tried manually copying all the tests jar and hbase-it jar to the lib
> > directory but it did not work, I guess we still missed several hadoop
> > jars...
> >
> > So what is the suggested way to run ITBLL after this change?
> >
> > Thanks.
> >
> > Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
> > >
> > > This is almost done.
> > >
> > > The final outstanding patch is https://github.com/apache/hbase/pull/5766
> > > for the new Hadoop-less assembly.
> > >
> > > Could you please review it ?
> > >
> > >
> > >
> > > On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com>
> > wrote:
> > >
> > > > I have created sub tasks with necessary details in the umbrella jira.
> > Will
> > > > take them up in coming days. Also will add more sub tasks later if
> > needed.
> > > >
> > > > Regards
> > > > Nihal
> > > >
> > > > On Sat, 9 Mar 2024, 11:53 Istvan Toth, <st...@cloudera.com.invalid>
> > wrote:
> > > >
> > > > > Thank you Nihal.
> > > > > I'm not very familiar with the tools in the test code, so you can
> > > > probably
> > > > > plan that work better.
> > > > > I just have some generic steps in mind:
> > > > > * Identify all the tools / scripts in the test jars
> > > > > * Identify and analyze their dependencies (compared to the current
> > > > runtime
> > > > > deps)
> > > > > * Decide which ones to move to the runtime JARs.
> > > > > * Move them to the runtime code (or perhaps a separate module)
> > > > >
> > > > > I have created https://issues.apache.org/jira/browse/HBASE-28431 as
> > an
> > > > > umbrella ticket to organize the sub-tasks.
> > > > >
> > > > > Istvan
> > > > >
> > > > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <ni...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Sure I will be able to take up. Please create tasks with necessary
> > > > > details
> > > > > > or let me know if you want me to create.
> > > > > >
> > > > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth, <stoty@cloudera.com.invalid
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Thanks for volunteering, Nihal.
> > > > > > >
> > > > > > > I could work on the Hadoop-less, and assemblies, and you could
> > work
> > > > on
> > > > > > > cleaning up the test jars.
> > > > > > > Would that work for you ?
> > > > > > > I know that I'm picking the smaller part, but it turns out that I
> > > > won't
> > > > > > > have as much time to work on this as I hoped.
> > > > > > >
> > > > > > > (Unless there are other volunteers, of course)
> > > > > > >
> > > > > > > Istvan
> > > > > > >
> > > > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com>
> > > > wrote:
> > > > > > >
> > > > > > > > We seem to be in agreement in principle, however the devil is
> > in
> > > > the
> > > > > > > > details.
> > > > > > > >
> > > > > > > > The first step should be moving the diagnostic tools out of the
> > > > test
> > > > > > > jars.
> > > > > > > > Are there any tools we don't want to move out ?
> > > > > > > > Do the diagnostic tools pull in extra dependencies compared to
> > the
> > > > > > > current
> > > > > > > > runtime JARs, and if they do, what are those ?
> > > > > > > > I haven't thought of the chaosmonkey tests yet, do those have
> > > > > specific
> > > > > > > > additional dependencies / scripts ?
> > > > > > > >
> > > > > > > > Should we move the tools simply to the normal jars, or should
> > we
> > > > move
> > > > > > > them
> > > > > > > > to a new module (could be called hbase-diagnostics) ?
> > > > > > > >
> > > > > > > > Istvan
> > > > > > > >
> > > > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> > > > > > > bbeaudreault@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> I'm +0 on hbase-examples, but +1000000 on any improvements we
> > can
> > > > > make
> > > > > > > to
> > > > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how
> > much
> > > > > > > reliance
> > > > > > > >> we have on test jars both generally but also specifically
> > around
> > > > > these
> > > > > > > >> core
> > > > > > > >> test executables. Unfortunately I haven't had time to
> > dedicate to
> > > > > > these
> > > > > > > >> frustrations myself, but happy to help with review, etc.
> > > > > > > >>
> > > > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <
> > nihaljain.cs@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Thank you for bringing this up.
> > > > > > > >> >
> > > > > > > >> > +1 for this change.
> > > > > > > >> >
> > > > > > > >> > In fact, some time back, we had faced similar problem.
> > Security
> > > > > > scans
> > > > > > > >> found
> > > > > > > >> > that we were bundling some vulnerable hadoop test jar. To
> > deal
> > > > > with
> > > > > > > >> that we
> > > > > > > >> > had to make a change in our internal HBase fork to exclude
> > all
> > > > > HBase
> > > > > > > and
> > > > > > > >> > Hadoop test jars from assembly. This helped us get rid of
> > > > > vulnerable
> > > > > > > >> jar.
> > > > > > > >> > (Although I hadn't dealt with test scope dependencies
> > there.)
> > > > > > > >> >
> > > > > > > >> > But, I have been thinking of pushing this change in Apache
> > > > HBase,
> > > > > > just
> > > > > > > >> > wasn't sure if this was even acceptable. It's great to see
> > same
> > > > > has
> > > > > > > been
> > > > > > > >> > brought up here today.
> > > > > > > >> >
> > > > > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a
> > script
> > > > to
> > > > > > > >> download
> > > > > > > >> > them on demand to avoid massive code change in internal
> > fork.
> > > > But
> > > > > I
> > > > > > > >> have a
> > > > > > > >> > +1 on the idea of identifying and moving all such tools to
> > a new
> > > > > > > module.
> > > > > > > >> > This would be great and make things easier for us as well.
> > > > > > > >> >
> > > > > > > >> > Also, a way we could help new users easily get started, in
> > case
> > > > we
> > > > > > > >> > completely stop bundling hadoop jars, is by providing a
> > script
> > > > > which
> > > > > > > >> starts
> > > > > > > >> > a hbase cluster in a single node setup. In fact I had
> > written a
> > > > > > simple
> > > > > > > >> > script sometime back that automates this process given a
> > release
> > > > > > link
> > > > > > > >> for
> > > > > > > >> > both. It first downloads Hadoop and HBase binaries and then
> > > > starts
> > > > > > > both
> > > > > > > >> > with the hbase root directory set to be on hdfs. We could
> > > > provide
> > > > > > > >> something
> > > > > > > >> > similar to help new users to get started easily.
> > > > > > > >> >
> > > > > > > >> > Although I am also +1 on the idea to provide both variants
> > as
> > > > > > > mentioned
> > > > > > > >> by
> > > > > > > >> > Nick, which might not even need any such script.
> > > > > > > >> >
> > > > > > > >> > Also, I am willing to volunteer for help towards this
> > effort.
> > > > > Please
> > > > > > > >> let me
> > > > > > > >> > know if anything is needed.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Nihal
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <
> > ndimiduk@apache.org>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > This would be great cleanup, big +1 from me for all three
> > of
> > > > > these
> > > > > > > >> > > adjustments, including the promotion of pe, ltt, and
> > friends
> > > > out
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > test scope.
> > > > > > > >> > >
> > > > > > > >> > > I believe that we included hbase test jars because we
> > used to
> > > > > > freely
> > > > > > > >> mix
> > > > > > > >> > > classes needed for minicluster between runtime and test
> > jars,
> > > > > > which
> > > > > > > in
> > > > > > > >> > turn
> > > > > > > >> > > relied on Hadoop minicluster capabilities. The big cleanup
> > > > > around
> > > > > > > >> > > HBaseTestingUtil/it addressed much (or all) of these
> > issues on
> > > > > > > >> branch-3.
> > > > > > > >> > >
> > > > > > > >> > > I believe that we include a Hadoop distribution in our
> > > > assembly
> > > > > > > >> because
> > > > > > > >> > > that makes it easy for a new user to download our release
> > > > > bin.tgz
> > > > > > > and
> > > > > > > >> get
> > > > > > > >> > > started immediately with learning. I guess it’s high time
> > that
> > > > > we
> > > > > > > work
> > > > > > > >> > out
> > > > > > > >> > > the with- and without-Hadoop variants.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Nick
> > > > > > > >> > >
> > > > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <
> > stoty@apache.org>
> > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an
> > elegant
> > > > way
> > > > > > > >> mapped
> > > > > > > >> > out
> > > > > > > >> > > > to achieve this, this is about discussing whether we
> > even
> > > > want
> > > > > > to
> > > > > > > >> make
> > > > > > > >> > > > these changes.
> > > > > > > >> > > > These are also substantial changes, but they could be
> > > > targeted
> > > > > > for
> > > > > > > >> > HBase
> > > > > > > >> > > > 3.0.
> > > > > > > >> > > >
> > > > > > > >> > > > One issue I have noticed is that we ship test jars and
> > test
> > > > > > > >> > dependencies
> > > > > > > >> > > in
> > > > > > > >> > > > the assembly.
> > > > > > > >> > > > I can't see anyone using those, but it bloats the
> > assembly
> > > > and
> > > > > > > >> > classpath,
> > > > > > > >> > > > and adds unnecessary JARs with possible CVE issues. (for
> > > > > example
> > > > > > > >> Kerby
> > > > > > > >> > > > which is a Hadoop minicluster dependency)
> > > > > > > >> > > >
> > > > > > > >> > > > My proposal is to exclude the test jars and the test
> > scope
> > > > > > > >> dependencies
> > > > > > > >> > > > from the assembly.
> > > > > > > >> > > >
> > > > > > > >> > > > The advantages would be:
> > > > > > > >> > > > * Smaller distro size
> > > > > > > >> > > > * Faster startup (this is marginal)
> > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > > > >> > > >
> > > > > > > >> > > > The other issue is that the assembly includes much of
> > the
> > > > > Hadoop
> > > > > > > >> > > > distribution.
> > > > > > > >> > > > The basic assumption in all scripts and instructions is
> > that
> > > > > the
> > > > > > > >> node
> > > > > > > >> > > has a
> > > > > > > >> > > > fully configured Hadoop installation, and we include it
> > in
> > > > the
> > > > > > > >> > classpath
> > > > > > > >> > > of
> > > > > > > >> > > > HBase.
> > > > > > > >> > > >
> > > > > > > >> > > > If that is true, then there is no reason to include
> > Hadoop
> > > > in
> > > > > > the
> > > > > > > >> > > assembly,
> > > > > > > >> > > > HBase and its direct dependencies should be enough.
> > > > > > > >> > > >
> > > > > > > >> > > > One could argue that it would simplify the client side,
> > > > which
> > > > > is
> > > > > > > >> true
> > > > > > > >> > to
> > > > > > > >> > > > some extent (though 95% of the client distro use cases
> > are
> > > > > > served
> > > > > > > >> > better
> > > > > > > >> > > by
> > > > > > > >> > > > simply using hbase-shaded-client).
> > > > > > > >> > > >
> > > > > > > >> > > > We could either remove the Hadoop libraries from either
> > or
> > > > > both
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > > assemblies unconditionally, or provide two variants for
> > > > either
> > > > > > or
> > > > > > > >> both
> > > > > > > >> > > > assemblies, one with Hadoop included, and one without
> > it.
> > > > > > > >> > > > Spark already does this, it has binary distributions
> > both
> > > > with
> > > > > > and
> > > > > > > >> > > without
> > > > > > > >> > > > Hadoop.
> > > > > > > >> > > >
> > > > > > > >> > > > The advantages would be:
> > > > > > > >> > > > * Smaller distro size
> > > > > > > >> > > > * Faster startup (this is marginal)
> > > > > > > >> > > > * Less chance of conflicts with the Hadoop jars
> > > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > Thirdly, we could consider excluding the
> > > > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from
> > the
> > > > > > > >> Hadoop-less
> > > > > > > >> > > > binary assemblies. It is not used by the assembly, and
> > AFAIK
> > > > > it
> > > > > > is
> > > > > > > >> not
> > > > > > > >> > > > included in any of the 'hbase classpath' command
> > variants.
> > > > > > > >> > > >
> > > > > > > >> > > > This would make sure that no Hadoop libraries are
> > included
> > > > > (even
> > > > > > > in
> > > > > > > >> > > shaded
> > > > > > > >> > > > form) and would make the HBase distribution fully
> > insulated
> > > > > from
> > > > > > > >> > Hadoop's
> > > > > > > >> > > > CVE issues.
> > > > > > > >> > > >
> > > > > > > >> > > > (The full-fat hbase-shaded-client works best as direct
> > > > > > build-time
> > > > > > > >> > > > dependency anyway)
> > > > > > > >> > > >
> > > > > > > >> > > > best regards
> > > > > > > >> > > > Istvan
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > > > *Email*: stoty@cloudera.com
> > > > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > > > [image:
> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > [image:
> > > > > > > > Cloudera on LinkedIn] <
> > https://www.linkedin.com/company/cloudera>
> > > > > > > > ------------------------------
> > > > > > > > ------------------------------
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > > *Email*: stoty@cloudera.com
> > > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > [image:
> > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > [image:
> > > > > > Cloudera
> > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > > ------------------------------
> > > > > > > ------------------------------
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > *Email*: stoty@cloudera.com
> > > > > cloudera.com <https://www.cloudera.com>
> > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > Cloudera
> > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > ------------------------------
> > > > > ------------------------------
> > > > >
> > > >
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: stoty@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > > ------------------------------
> >
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@apache.org>.
Yes, we've discussed this issue, but deferred solving it.
IMO the easiest way is to add a third assembly which is functionally
equivalent to the 2.x assembly.
That could work for running hbase-it, ITBLL, chaos monkey, etc.
We'd also have to decide whether to build it by default, and whether we
want to publish it as part of official releases.
In theory could also make a delta assembly that only includes the
additional test related stuff, but I'm afraid that that would require a lot
of maintenance.
We could also add a script/maven target that downloads test-related JARs
from maven, but keeping that one up to date would also be problematic.
Istvan
On Sat, Mar 15, 2025 at 4:51 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> After this change, we can not run ITBLL on 3.0.0 because hbase-it is
> also excluded...
>
> I tried manually copying all the tests jar and hbase-it jar to the lib
> directory but it did not work, I guess we still missed several hadoop
> jars...
>
> So what is the suggested way to run ITBLL after this change?
>
> Thanks.
>
> Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
> >
> > This is almost done.
> >
> > The final outstanding patch is https://github.com/apache/hbase/pull/5766
> > for the new Hadoop-less assembly.
> >
> > Could you please review it ?
> >
> >
> >
> > On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com>
> wrote:
> >
> > > I have created sub tasks with necessary details in the umbrella jira.
> Will
> > > take them up in coming days. Also will add more sub tasks later if
> needed.
> > >
> > > Regards
> > > Nihal
> > >
> > > On Sat, 9 Mar 2024, 11:53 Istvan Toth, <st...@cloudera.com.invalid>
> wrote:
> > >
> > > > Thank you Nihal.
> > > > I'm not very familiar with the tools in the test code, so you can
> > > probably
> > > > plan that work better.
> > > > I just have some generic steps in mind:
> > > > * Identify all the tools / scripts in the test jars
> > > > * Identify and analyze their dependencies (compared to the current
> > > runtime
> > > > deps)
> > > > * Decide which ones to move to the runtime JARs.
> > > > * Move them to the runtime code (or perhaps a separate module)
> > > >
> > > > I have created https://issues.apache.org/jira/browse/HBASE-28431 as
> an
> > > > umbrella ticket to organize the sub-tasks.
> > > >
> > > > Istvan
> > > >
> > > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <ni...@gmail.com>
> > > wrote:
> > > >
> > > > > Sure I will be able to take up. Please create tasks with necessary
> > > > details
> > > > > or let me know if you want me to create.
> > > > >
> > > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth, <stoty@cloudera.com.invalid
> >
> > > > wrote:
> > > > >
> > > > > > Thanks for volunteering, Nihal.
> > > > > >
> > > > > > I could work on the Hadoop-less, and assemblies, and you could
> work
> > > on
> > > > > > cleaning up the test jars.
> > > > > > Would that work for you ?
> > > > > > I know that I'm picking the smaller part, but it turns out that I
> > > won't
> > > > > > have as much time to work on this as I hoped.
> > > > > >
> > > > > > (Unless there are other volunteers, of course)
> > > > > >
> > > > > > Istvan
> > > > > >
> > > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com>
> > > wrote:
> > > > > >
> > > > > > > We seem to be in agreement in principle, however the devil is
> in
> > > the
> > > > > > > details.
> > > > > > >
> > > > > > > The first step should be moving the diagnostic tools out of the
> > > test
> > > > > > jars.
> > > > > > > Are there any tools we don't want to move out ?
> > > > > > > Do the diagnostic tools pull in extra dependencies compared to
> the
> > > > > > current
> > > > > > > runtime JARs, and if they do, what are those ?
> > > > > > > I haven't thought of the chaosmonkey tests yet, do those have
> > > > specific
> > > > > > > additional dependencies / scripts ?
> > > > > > >
> > > > > > > Should we move the tools simply to the normal jars, or should
> we
> > > move
> > > > > > them
> > > > > > > to a new module (could be called hbase-diagnostics) ?
> > > > > > >
> > > > > > > Istvan
> > > > > > >
> > > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> > > > > > bbeaudreault@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> I'm +0 on hbase-examples, but +1000000 on any improvements we
> can
> > > > make
> > > > > > to
> > > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how
> much
> > > > > > reliance
> > > > > > >> we have on test jars both generally but also specifically
> around
> > > > these
> > > > > > >> core
> > > > > > >> test executables. Unfortunately I haven't had time to
> dedicate to
> > > > > these
> > > > > > >> frustrations myself, but happy to help with review, etc.
> > > > > > >>
> > > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <
> nihaljain.cs@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Thank you for bringing this up.
> > > > > > >> >
> > > > > > >> > +1 for this change.
> > > > > > >> >
> > > > > > >> > In fact, some time back, we had faced similar problem.
> Security
> > > > > scans
> > > > > > >> found
> > > > > > >> > that we were bundling some vulnerable hadoop test jar. To
> deal
> > > > with
> > > > > > >> that we
> > > > > > >> > had to make a change in our internal HBase fork to exclude
> all
> > > > HBase
> > > > > > and
> > > > > > >> > Hadoop test jars from assembly. This helped us get rid of
> > > > vulnerable
> > > > > > >> jar.
> > > > > > >> > (Although I hadn't dealt with test scope dependencies
> there.)
> > > > > > >> >
> > > > > > >> > But, I have been thinking of pushing this change in Apache
> > > HBase,
> > > > > just
> > > > > > >> > wasn't sure if this was even acceptable. It's great to see
> same
> > > > has
> > > > > > been
> > > > > > >> > brought up here today.
> > > > > > >> >
> > > > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a
> script
> > > to
> > > > > > >> download
> > > > > > >> > them on demand to avoid massive code change in internal
> fork.
> > > But
> > > > I
> > > > > > >> have a
> > > > > > >> > +1 on the idea of identifying and moving all such tools to
> a new
> > > > > > module.
> > > > > > >> > This would be great and make things easier for us as well.
> > > > > > >> >
> > > > > > >> > Also, a way we could help new users easily get started, in
> case
> > > we
> > > > > > >> > completely stop bundling hadoop jars, is by providing a
> script
> > > > which
> > > > > > >> starts
> > > > > > >> > a hbase cluster in a single node setup. In fact I had
> written a
> > > > > simple
> > > > > > >> > script sometime back that automates this process given a
> release
> > > > > link
> > > > > > >> for
> > > > > > >> > both. It first downloads Hadoop and HBase binaries and then
> > > starts
> > > > > > both
> > > > > > >> > with the hbase root directory set to be on hdfs. We could
> > > provide
> > > > > > >> something
> > > > > > >> > similar to help new users to get started easily.
> > > > > > >> >
> > > > > > >> > Although I am also +1 on the idea to provide both variants
> as
> > > > > > mentioned
> > > > > > >> by
> > > > > > >> > Nick, which might not even need any such script.
> > > > > > >> >
> > > > > > >> > Also, I am willing to volunteer for help towards this
> effort.
> > > > Please
> > > > > > >> let me
> > > > > > >> > know if anything is needed.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Nihal
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <
> ndimiduk@apache.org>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > This would be great cleanup, big +1 from me for all three
> of
> > > > these
> > > > > > >> > > adjustments, including the promotion of pe, ltt, and
> friends
> > > out
> > > > > of
> > > > > > >> the
> > > > > > >> > > test scope.
> > > > > > >> > >
> > > > > > >> > > I believe that we included hbase test jars because we
> used to
> > > > > freely
> > > > > > >> mix
> > > > > > >> > > classes needed for minicluster between runtime and test
> jars,
> > > > > which
> > > > > > in
> > > > > > >> > turn
> > > > > > >> > > relied on Hadoop minicluster capabilities. The big cleanup
> > > > around
> > > > > > >> > > HBaseTestingUtil/it addressed much (or all) of these
> issues on
> > > > > > >> branch-3.
> > > > > > >> > >
> > > > > > >> > > I believe that we include a Hadoop distribution in our
> > > assembly
> > > > > > >> because
> > > > > > >> > > that makes it easy for a new user to download our release
> > > > bin.tgz
> > > > > > and
> > > > > > >> get
> > > > > > >> > > started immediately with learning. I guess it’s high time
> that
> > > > we
> > > > > > work
> > > > > > >> > out
> > > > > > >> > > the with- and without-Hadoop variants.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Nick
> > > > > > >> > >
> > > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <
> stoty@apache.org>
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an
> elegant
> > > way
> > > > > > >> mapped
> > > > > > >> > out
> > > > > > >> > > > to achieve this, this is about discussing whether we
> even
> > > want
> > > > > to
> > > > > > >> make
> > > > > > >> > > > these changes.
> > > > > > >> > > > These are also substantial changes, but they could be
> > > targeted
> > > > > for
> > > > > > >> > HBase
> > > > > > >> > > > 3.0.
> > > > > > >> > > >
> > > > > > >> > > > One issue I have noticed is that we ship test jars and
> test
> > > > > > >> > dependencies
> > > > > > >> > > in
> > > > > > >> > > > the assembly.
> > > > > > >> > > > I can't see anyone using those, but it bloats the
> assembly
> > > and
> > > > > > >> > classpath,
> > > > > > >> > > > and adds unnecessary JARs with possible CVE issues. (for
> > > > example
> > > > > > >> Kerby
> > > > > > >> > > > which is a Hadoop minicluster dependency)
> > > > > > >> > > >
> > > > > > >> > > > My proposal is to exclude the test jars and the test
> scope
> > > > > > >> dependencies
> > > > > > >> > > > from the assembly.
> > > > > > >> > > >
> > > > > > >> > > > The advantages would be:
> > > > > > >> > > > * Smaller distro size
> > > > > > >> > > > * Faster startup (this is marginal)
> > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > > >> > > >
> > > > > > >> > > > The other issue is that the assembly includes much of
> the
> > > > Hadoop
> > > > > > >> > > > distribution.
> > > > > > >> > > > The basic assumption in all scripts and instructions is
> that
> > > > the
> > > > > > >> node
> > > > > > >> > > has a
> > > > > > >> > > > fully configured Hadoop installation, and we include it
> in
> > > the
> > > > > > >> > classpath
> > > > > > >> > > of
> > > > > > >> > > > HBase.
> > > > > > >> > > >
> > > > > > >> > > > If that is true, then there is no reason to include
> Hadoop
> > > in
> > > > > the
> > > > > > >> > > assembly,
> > > > > > >> > > > HBase and its direct dependencies should be enough.
> > > > > > >> > > >
> > > > > > >> > > > One could argue that it would simplify the client side,
> > > which
> > > > is
> > > > > > >> true
> > > > > > >> > to
> > > > > > >> > > > some extent (though 95% of the client distro use cases
> are
> > > > > served
> > > > > > >> > better
> > > > > > >> > > by
> > > > > > >> > > > simply using hbase-shaded-client).
> > > > > > >> > > >
> > > > > > >> > > > We could either remove the Hadoop libraries from either
> or
> > > > both
> > > > > of
> > > > > > >> the
> > > > > > >> > > > assemblies unconditionally, or provide two variants for
> > > either
> > > > > or
> > > > > > >> both
> > > > > > >> > > > assemblies, one with Hadoop included, and one without
> it.
> > > > > > >> > > > Spark already does this, it has binary distributions
> both
> > > with
> > > > > and
> > > > > > >> > > without
> > > > > > >> > > > Hadoop.
> > > > > > >> > > >
> > > > > > >> > > > The advantages would be:
> > > > > > >> > > > * Smaller distro size
> > > > > > >> > > > * Faster startup (this is marginal)
> > > > > > >> > > > * Less chance of conflicts with the Hadoop jars
> > > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > Thirdly, we could consider excluding the
> > > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from
> the
> > > > > > >> Hadoop-less
> > > > > > >> > > > binary assemblies. It is not used by the assembly, and
> AFAIK
> > > > it
> > > > > is
> > > > > > >> not
> > > > > > >> > > > included in any of the 'hbase classpath' command
> variants.
> > > > > > >> > > >
> > > > > > >> > > > This would make sure that no Hadoop libraries are
> included
> > > > (even
> > > > > > in
> > > > > > >> > > shaded
> > > > > > >> > > > form) and would make the HBase distribution fully
> insulated
> > > > from
> > > > > > >> > Hadoop's
> > > > > > >> > > > CVE issues.
> > > > > > >> > > >
> > > > > > >> > > > (The full-fat hbase-shaded-client works best as direct
> > > > > build-time
> > > > > > >> > > > dependency anyway)
> > > > > > >> > > >
> > > > > > >> > > > best regards
> > > > > > >> > > > Istvan
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > > *Email*: stoty@cloudera.com
> > > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > > [image:
> > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> [image:
> > > > > > > Cloudera on LinkedIn] <
> https://www.linkedin.com/company/cloudera>
> > > > > > > ------------------------------
> > > > > > > ------------------------------
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > *Email*: stoty@cloudera.com
> > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> [image:
> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> [image:
> > > > > Cloudera
> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > ------------------------------
> > > > > > ------------------------------
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Sr. Staff Software Engineer
> > > > *Email*: stoty@cloudera.com
> > > > cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > > ------------------------------
> > > >
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
>
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
After this change, we can not run ITBLL on 3.0.0 because hbase-it is
also excluded...
I tried manually copying all the tests jar and hbase-it jar to the lib
directory but it did not work, I guess we still missed several hadoop
jars...
So what is the suggested way to run ITBLL after this change?
Thanks.
Istvan Toth <st...@cloudera.com.invalid> 于2025年1月20日周一 14:20写道：
>
> This is almost done.
>
> The final outstanding patch is https://github.com/apache/hbase/pull/5766
> for the new Hadoop-less assembly.
>
> Could you please review it ?
>
>
>
> On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com> wrote:
>
> > I have created sub tasks with necessary details in the umbrella jira. Will
> > take them up in coming days. Also will add more sub tasks later if needed.
> >
> > Regards
> > Nihal
> >
> > On Sat, 9 Mar 2024, 11:53 Istvan Toth, <st...@cloudera.com.invalid> wrote:
> >
> > > Thank you Nihal.
> > > I'm not very familiar with the tools in the test code, so you can
> > probably
> > > plan that work better.
> > > I just have some generic steps in mind:
> > > * Identify all the tools / scripts in the test jars
> > > * Identify and analyze their dependencies (compared to the current
> > runtime
> > > deps)
> > > * Decide which ones to move to the runtime JARs.
> > > * Move them to the runtime code (or perhaps a separate module)
> > >
> > > I have created https://issues.apache.org/jira/browse/HBASE-28431 as an
> > > umbrella ticket to organize the sub-tasks.
> > >
> > > Istvan
> > >
> > > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <ni...@gmail.com>
> > wrote:
> > >
> > > > Sure I will be able to take up. Please create tasks with necessary
> > > details
> > > > or let me know if you want me to create.
> > > >
> > > > On Fri, 8 Mar 2024, 12:45 Istvan Toth, <st...@cloudera.com.invalid>
> > > wrote:
> > > >
> > > > > Thanks for volunteering, Nihal.
> > > > >
> > > > > I could work on the Hadoop-less, and assemblies, and you could work
> > on
> > > > > cleaning up the test jars.
> > > > > Would that work for you ?
> > > > > I know that I'm picking the smaller part, but it turns out that I
> > won't
> > > > > have as much time to work on this as I hoped.
> > > > >
> > > > > (Unless there are other volunteers, of course)
> > > > >
> > > > > Istvan
> > > > >
> > > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com>
> > wrote:
> > > > >
> > > > > > We seem to be in agreement in principle, however the devil is in
> > the
> > > > > > details.
> > > > > >
> > > > > > The first step should be moving the diagnostic tools out of the
> > test
> > > > > jars.
> > > > > > Are there any tools we don't want to move out ?
> > > > > > Do the diagnostic tools pull in extra dependencies compared to the
> > > > > current
> > > > > > runtime JARs, and if they do, what are those ?
> > > > > > I haven't thought of the chaosmonkey tests yet, do those have
> > > specific
> > > > > > additional dependencies / scripts ?
> > > > > >
> > > > > > Should we move the tools simply to the normal jars, or should we
> > move
> > > > > them
> > > > > > to a new module (could be called hbase-diagnostics) ?
> > > > > >
> > > > > > Istvan
> > > > > >
> > > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> > > > > bbeaudreault@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > >> I'm +0 on hbase-examples, but +1000000 on any improvements we can
> > > make
> > > > > to
> > > > > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much
> > > > > reliance
> > > > > >> we have on test jars both generally but also specifically around
> > > these
> > > > > >> core
> > > > > >> test executables. Unfortunately I haven't had time to dedicate to
> > > > these
> > > > > >> frustrations myself, but happy to help with review, etc.
> > > > > >>
> > > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <nihaljain.cs@gmail.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > Thank you for bringing this up.
> > > > > >> >
> > > > > >> > +1 for this change.
> > > > > >> >
> > > > > >> > In fact, some time back, we had faced similar problem. Security
> > > > scans
> > > > > >> found
> > > > > >> > that we were bundling some vulnerable hadoop test jar. To deal
> > > with
> > > > > >> that we
> > > > > >> > had to make a change in our internal HBase fork to exclude all
> > > HBase
> > > > > and
> > > > > >> > Hadoop test jars from assembly. This helped us get rid of
> > > vulnerable
> > > > > >> jar.
> > > > > >> > (Although I hadn't dealt with test scope dependencies there.)
> > > > > >> >
> > > > > >> > But, I have been thinking of pushing this change in Apache
> > HBase,
> > > > just
> > > > > >> > wasn't sure if this was even acceptable. It's great to see same
> > > has
> > > > > been
> > > > > >> > brought up here today.
> > > > > >> >
> > > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a script
> > to
> > > > > >> download
> > > > > >> > them on demand to avoid massive code change in internal fork.
> > But
> > > I
> > > > > >> have a
> > > > > >> > +1 on the idea of identifying and moving all such tools to a new
> > > > > module.
> > > > > >> > This would be great and make things easier for us as well.
> > > > > >> >
> > > > > >> > Also, a way we could help new users easily get started, in case
> > we
> > > > > >> > completely stop bundling hadoop jars, is by providing a script
> > > which
> > > > > >> starts
> > > > > >> > a hbase cluster in a single node setup. In fact I had written a
> > > > simple
> > > > > >> > script sometime back that automates this process given a release
> > > > link
> > > > > >> for
> > > > > >> > both. It first downloads Hadoop and HBase binaries and then
> > starts
> > > > > both
> > > > > >> > with the hbase root directory set to be on hdfs. We could
> > provide
> > > > > >> something
> > > > > >> > similar to help new users to get started easily.
> > > > > >> >
> > > > > >> > Although I am also +1 on the idea to provide both variants as
> > > > > mentioned
> > > > > >> by
> > > > > >> > Nick, which might not even need any such script.
> > > > > >> >
> > > > > >> > Also, I am willing to volunteer for help towards this effort.
> > > Please
> > > > > >> let me
> > > > > >> > know if anything is needed.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Nihal
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org>
> > > > wrote:
> > > > > >> >
> > > > > >> > > This would be great cleanup, big +1 from me for all three of
> > > these
> > > > > >> > > adjustments, including the promotion of pe, ltt, and friends
> > out
> > > > of
> > > > > >> the
> > > > > >> > > test scope.
> > > > > >> > >
> > > > > >> > > I believe that we included hbase test jars because we used to
> > > > freely
> > > > > >> mix
> > > > > >> > > classes needed for minicluster between runtime and test jars,
> > > > which
> > > > > in
> > > > > >> > turn
> > > > > >> > > relied on Hadoop minicluster capabilities. The big cleanup
> > > around
> > > > > >> > > HBaseTestingUtil/it addressed much (or all) of these issues on
> > > > > >> branch-3.
> > > > > >> > >
> > > > > >> > > I believe that we include a Hadoop distribution in our
> > assembly
> > > > > >> because
> > > > > >> > > that makes it easy for a new user to download our release
> > > bin.tgz
> > > > > and
> > > > > >> get
> > > > > >> > > started immediately with learning. I guess it’s high time that
> > > we
> > > > > work
> > > > > >> > out
> > > > > >> > > the with- and without-Hadoop variants.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Nick
> > > > > >> > >
> > > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org>
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an elegant
> > way
> > > > > >> mapped
> > > > > >> > out
> > > > > >> > > > to achieve this, this is about discussing whether we even
> > want
> > > > to
> > > > > >> make
> > > > > >> > > > these changes.
> > > > > >> > > > These are also substantial changes, but they could be
> > targeted
> > > > for
> > > > > >> > HBase
> > > > > >> > > > 3.0.
> > > > > >> > > >
> > > > > >> > > > One issue I have noticed is that we ship test jars and test
> > > > > >> > dependencies
> > > > > >> > > in
> > > > > >> > > > the assembly.
> > > > > >> > > > I can't see anyone using those, but it bloats the assembly
> > and
> > > > > >> > classpath,
> > > > > >> > > > and adds unnecessary JARs with possible CVE issues. (for
> > > example
> > > > > >> Kerby
> > > > > >> > > > which is a Hadoop minicluster dependency)
> > > > > >> > > >
> > > > > >> > > > My proposal is to exclude the test jars and the test scope
> > > > > >> dependencies
> > > > > >> > > > from the assembly.
> > > > > >> > > >
> > > > > >> > > > The advantages would be:
> > > > > >> > > > * Smaller distro size
> > > > > >> > > > * Faster startup (this is marginal)
> > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > >> > > >
> > > > > >> > > > The other issue is that the assembly includes much of the
> > > Hadoop
> > > > > >> > > > distribution.
> > > > > >> > > > The basic assumption in all scripts and instructions is that
> > > the
> > > > > >> node
> > > > > >> > > has a
> > > > > >> > > > fully configured Hadoop installation, and we include it in
> > the
> > > > > >> > classpath
> > > > > >> > > of
> > > > > >> > > > HBase.
> > > > > >> > > >
> > > > > >> > > > If that is true, then there is no reason to include Hadoop
> > in
> > > > the
> > > > > >> > > assembly,
> > > > > >> > > > HBase and its direct dependencies should be enough.
> > > > > >> > > >
> > > > > >> > > > One could argue that it would simplify the client side,
> > which
> > > is
> > > > > >> true
> > > > > >> > to
> > > > > >> > > > some extent (though 95% of the client distro use cases are
> > > > served
> > > > > >> > better
> > > > > >> > > by
> > > > > >> > > > simply using hbase-shaded-client).
> > > > > >> > > >
> > > > > >> > > > We could either remove the Hadoop libraries from either or
> > > both
> > > > of
> > > > > >> the
> > > > > >> > > > assemblies unconditionally, or provide two variants for
> > either
> > > > or
> > > > > >> both
> > > > > >> > > > assemblies, one with Hadoop included, and one without it.
> > > > > >> > > > Spark already does this, it has binary distributions both
> > with
> > > > and
> > > > > >> > > without
> > > > > >> > > > Hadoop.
> > > > > >> > > >
> > > > > >> > > > The advantages would be:
> > > > > >> > > > * Smaller distro size
> > > > > >> > > > * Faster startup (this is marginal)
> > > > > >> > > > * Less chance of conflicts with the Hadoop jars
> > > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Thirdly, we could consider excluding the
> > > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from the
> > > > > >> Hadoop-less
> > > > > >> > > > binary assemblies. It is not used by the assembly, and AFAIK
> > > it
> > > > is
> > > > > >> not
> > > > > >> > > > included in any of the 'hbase classpath' command variants.
> > > > > >> > > >
> > > > > >> > > > This would make sure that no Hadoop libraries are included
> > > (even
> > > > > in
> > > > > >> > > shaded
> > > > > >> > > > form) and would make the HBase distribution fully insulated
> > > from
> > > > > >> > Hadoop's
> > > > > >> > > > CVE issues.
> > > > > >> > > >
> > > > > >> > > > (The full-fat hbase-shaded-client works best as direct
> > > > build-time
> > > > > >> > > > dependency anyway)
> > > > > >> > > >
> > > > > >> > > > best regards
> > > > > >> > > > Istvan
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > *Email*: stoty@cloudera.com
> > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > [image:
> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > ------------------------------
> > > > > > ------------------------------
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > *Email*: stoty@cloudera.com
> > > > > cloudera.com <https://www.cloudera.com>
> > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > Cloudera
> > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > ------------------------------
> > > > > ------------------------------
> > > > >
> > > >
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: stoty@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > > ------------------------------
> > >
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
This is almost done.
The final outstanding patch is https://github.com/apache/hbase/pull/5766
for the new Hadoop-less assembly.
Could you please review it ?
On Sat, Mar 9, 2024 at 8:48 AM Nihal Jain <ni...@gmail.com> wrote:
> I have created sub tasks with necessary details in the umbrella jira. Will
> take them up in coming days. Also will add more sub tasks later if needed.
>
> Regards
> Nihal
>
> On Sat, 9 Mar 2024, 11:53 Istvan Toth, <st...@cloudera.com.invalid> wrote:
>
> > Thank you Nihal.
> > I'm not very familiar with the tools in the test code, so you can
> probably
> > plan that work better.
> > I just have some generic steps in mind:
> > * Identify all the tools / scripts in the test jars
> > * Identify and analyze their dependencies (compared to the current
> runtime
> > deps)
> > * Decide which ones to move to the runtime JARs.
> > * Move them to the runtime code (or perhaps a separate module)
> >
> > I have created https://issues.apache.org/jira/browse/HBASE-28431 as an
> > umbrella ticket to organize the sub-tasks.
> >
> > Istvan
> >
> > On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <ni...@gmail.com>
> wrote:
> >
> > > Sure I will be able to take up. Please create tasks with necessary
> > details
> > > or let me know if you want me to create.
> > >
> > > On Fri, 8 Mar 2024, 12:45 Istvan Toth, <st...@cloudera.com.invalid>
> > wrote:
> > >
> > > > Thanks for volunteering, Nihal.
> > > >
> > > > I could work on the Hadoop-less, and assemblies, and you could work
> on
> > > > cleaning up the test jars.
> > > > Would that work for you ?
> > > > I know that I'm picking the smaller part, but it turns out that I
> won't
> > > > have as much time to work on this as I hoped.
> > > >
> > > > (Unless there are other volunteers, of course)
> > > >
> > > > Istvan
> > > >
> > > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com>
> wrote:
> > > >
> > > > > We seem to be in agreement in principle, however the devil is in
> the
> > > > > details.
> > > > >
> > > > > The first step should be moving the diagnostic tools out of the
> test
> > > > jars.
> > > > > Are there any tools we don't want to move out ?
> > > > > Do the diagnostic tools pull in extra dependencies compared to the
> > > > current
> > > > > runtime JARs, and if they do, what are those ?
> > > > > I haven't thought of the chaosmonkey tests yet, do those have
> > specific
> > > > > additional dependencies / scripts ?
> > > > >
> > > > > Should we move the tools simply to the normal jars, or should we
> move
> > > > them
> > > > > to a new module (could be called hbase-diagnostics) ?
> > > > >
> > > > > Istvan
> > > > >
> > > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> > > > bbeaudreault@apache.org>
> > > > > wrote:
> > > > >
> > > > >> I'm +0 on hbase-examples, but +1000000 on any improvements we can
> > make
> > > > to
> > > > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much
> > > > reliance
> > > > >> we have on test jars both generally but also specifically around
> > these
> > > > >> core
> > > > >> test executables. Unfortunately I haven't had time to dedicate to
> > > these
> > > > >> frustrations myself, but happy to help with review, etc.
> > > > >>
> > > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <nihaljain.cs@gmail.com
> >
> > > > wrote:
> > > > >>
> > > > >> > Thank you for bringing this up.
> > > > >> >
> > > > >> > +1 for this change.
> > > > >> >
> > > > >> > In fact, some time back, we had faced similar problem. Security
> > > scans
> > > > >> found
> > > > >> > that we were bundling some vulnerable hadoop test jar. To deal
> > with
> > > > >> that we
> > > > >> > had to make a change in our internal HBase fork to exclude all
> > HBase
> > > > and
> > > > >> > Hadoop test jars from assembly. This helped us get rid of
> > vulnerable
> > > > >> jar.
> > > > >> > (Although I hadn't dealt with test scope dependencies there.)
> > > > >> >
> > > > >> > But, I have been thinking of pushing this change in Apache
> HBase,
> > > just
> > > > >> > wasn't sure if this was even acceptable. It's great to see same
> > has
> > > > been
> > > > >> > brought up here today.
> > > > >> >
> > > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a script
> to
> > > > >> download
> > > > >> > them on demand to avoid massive code change in internal fork.
> But
> > I
> > > > >> have a
> > > > >> > +1 on the idea of identifying and moving all such tools to a new
> > > > module.
> > > > >> > This would be great and make things easier for us as well.
> > > > >> >
> > > > >> > Also, a way we could help new users easily get started, in case
> we
> > > > >> > completely stop bundling hadoop jars, is by providing a script
> > which
> > > > >> starts
> > > > >> > a hbase cluster in a single node setup. In fact I had written a
> > > simple
> > > > >> > script sometime back that automates this process given a release
> > > link
> > > > >> for
> > > > >> > both. It first downloads Hadoop and HBase binaries and then
> starts
> > > > both
> > > > >> > with the hbase root directory set to be on hdfs. We could
> provide
> > > > >> something
> > > > >> > similar to help new users to get started easily.
> > > > >> >
> > > > >> > Although I am also +1 on the idea to provide both variants as
> > > > mentioned
> > > > >> by
> > > > >> > Nick, which might not even need any such script.
> > > > >> >
> > > > >> > Also, I am willing to volunteer for help towards this effort.
> > Please
> > > > >> let me
> > > > >> > know if anything is needed.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Nihal
> > > > >> >
> > > > >> >
> > > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> > > This would be great cleanup, big +1 from me for all three of
> > these
> > > > >> > > adjustments, including the promotion of pe, ltt, and friends
> out
> > > of
> > > > >> the
> > > > >> > > test scope.
> > > > >> > >
> > > > >> > > I believe that we included hbase test jars because we used to
> > > freely
> > > > >> mix
> > > > >> > > classes needed for minicluster between runtime and test jars,
> > > which
> > > > in
> > > > >> > turn
> > > > >> > > relied on Hadoop minicluster capabilities. The big cleanup
> > around
> > > > >> > > HBaseTestingUtil/it addressed much (or all) of these issues on
> > > > >> branch-3.
> > > > >> > >
> > > > >> > > I believe that we include a Hadoop distribution in our
> assembly
> > > > >> because
> > > > >> > > that makes it easy for a new user to download our release
> > bin.tgz
> > > > and
> > > > >> get
> > > > >> > > started immediately with learning. I guess it’s high time that
> > we
> > > > work
> > > > >> > out
> > > > >> > > the with- and without-Hadoop variants.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Nick
> > > > >> > >
> > > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org>
> > > wrote:
> > > > >> > >
> > > > >> > > > DISCLAIMER: I don't have a patch ready, or even an elegant
> way
> > > > >> mapped
> > > > >> > out
> > > > >> > > > to achieve this, this is about discussing whether we even
> want
> > > to
> > > > >> make
> > > > >> > > > these changes.
> > > > >> > > > These are also substantial changes, but they could be
> targeted
> > > for
> > > > >> > HBase
> > > > >> > > > 3.0.
> > > > >> > > >
> > > > >> > > > One issue I have noticed is that we ship test jars and test
> > > > >> > dependencies
> > > > >> > > in
> > > > >> > > > the assembly.
> > > > >> > > > I can't see anyone using those, but it bloats the assembly
> and
> > > > >> > classpath,
> > > > >> > > > and adds unnecessary JARs with possible CVE issues. (for
> > example
> > > > >> Kerby
> > > > >> > > > which is a Hadoop minicluster dependency)
> > > > >> > > >
> > > > >> > > > My proposal is to exclude the test jars and the test scope
> > > > >> dependencies
> > > > >> > > > from the assembly.
> > > > >> > > >
> > > > >> > > > The advantages would be:
> > > > >> > > > * Smaller distro size
> > > > >> > > > * Faster startup (this is marginal)
> > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > >> > > >
> > > > >> > > > The other issue is that the assembly includes much of the
> > Hadoop
> > > > >> > > > distribution.
> > > > >> > > > The basic assumption in all scripts and instructions is that
> > the
> > > > >> node
> > > > >> > > has a
> > > > >> > > > fully configured Hadoop installation, and we include it in
> the
> > > > >> > classpath
> > > > >> > > of
> > > > >> > > > HBase.
> > > > >> > > >
> > > > >> > > > If that is true, then there is no reason to include Hadoop
> in
> > > the
> > > > >> > > assembly,
> > > > >> > > > HBase and its direct dependencies should be enough.
> > > > >> > > >
> > > > >> > > > One could argue that it would simplify the client side,
> which
> > is
> > > > >> true
> > > > >> > to
> > > > >> > > > some extent (though 95% of the client distro use cases are
> > > served
> > > > >> > better
> > > > >> > > by
> > > > >> > > > simply using hbase-shaded-client).
> > > > >> > > >
> > > > >> > > > We could either remove the Hadoop libraries from either or
> > both
> > > of
> > > > >> the
> > > > >> > > > assemblies unconditionally, or provide two variants for
> either
> > > or
> > > > >> both
> > > > >> > > > assemblies, one with Hadoop included, and one without it.
> > > > >> > > > Spark already does this, it has binary distributions both
> with
> > > and
> > > > >> > > without
> > > > >> > > > Hadoop.
> > > > >> > > >
> > > > >> > > > The advantages would be:
> > > > >> > > > * Smaller distro size
> > > > >> > > > * Faster startup (this is marginal)
> > > > >> > > > * Less chance of conflicts with the Hadoop jars
> > > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Thirdly, we could consider excluding the
> > > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from the
> > > > >> Hadoop-less
> > > > >> > > > binary assemblies. It is not used by the assembly, and AFAIK
> > it
> > > is
> > > > >> not
> > > > >> > > > included in any of the 'hbase classpath' command variants.
> > > > >> > > >
> > > > >> > > > This would make sure that no Hadoop libraries are included
> > (even
> > > > in
> > > > >> > > shaded
> > > > >> > > > form) and would make the HBase distribution fully insulated
> > from
> > > > >> > Hadoop's
> > > > >> > > > CVE issues.
> > > > >> > > >
> > > > >> > > > (The full-fat hbase-shaded-client works best as direct
> > > build-time
> > > > >> > > > dependency anyway)
> > > > >> > > >
> > > > >> > > > best regards
> > > > >> > > > Istvan
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > *Email*: stoty@cloudera.com
> > > > > cloudera.com <https://www.cloudera.com>
> > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> [image:
> > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > ------------------------------
> > > > > ------------------------------
> > > > >
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Sr. Staff Software Engineer
> > > > *Email*: stoty@cloudera.com
> > > > cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > > ------------------------------
> > > >
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
> >
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Nihal Jain <ni...@gmail.com>.
I have created sub tasks with necessary details in the umbrella jira. Will
take them up in coming days. Also will add more sub tasks later if needed.
Regards
Nihal
On Sat, 9 Mar 2024, 11:53 Istvan Toth, <st...@cloudera.com.invalid> wrote:
> Thank you Nihal.
> I'm not very familiar with the tools in the test code, so you can probably
> plan that work better.
> I just have some generic steps in mind:
> * Identify all the tools / scripts in the test jars
> * Identify and analyze their dependencies (compared to the current runtime
> deps)
> * Decide which ones to move to the runtime JARs.
> * Move them to the runtime code (or perhaps a separate module)
>
> I have created https://issues.apache.org/jira/browse/HBASE-28431 as an
> umbrella ticket to organize the sub-tasks.
>
> Istvan
>
> On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <ni...@gmail.com> wrote:
>
> > Sure I will be able to take up. Please create tasks with necessary
> details
> > or let me know if you want me to create.
> >
> > On Fri, 8 Mar 2024, 12:45 Istvan Toth, <st...@cloudera.com.invalid>
> wrote:
> >
> > > Thanks for volunteering, Nihal.
> > >
> > > I could work on the Hadoop-less, and assemblies, and you could work on
> > > cleaning up the test jars.
> > > Would that work for you ?
> > > I know that I'm picking the smaller part, but it turns out that I won't
> > > have as much time to work on this as I hoped.
> > >
> > > (Unless there are other volunteers, of course)
> > >
> > > Istvan
> > >
> > > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com> wrote:
> > >
> > > > We seem to be in agreement in principle, however the devil is in the
> > > > details.
> > > >
> > > > The first step should be moving the diagnostic tools out of the test
> > > jars.
> > > > Are there any tools we don't want to move out ?
> > > > Do the diagnostic tools pull in extra dependencies compared to the
> > > current
> > > > runtime JARs, and if they do, what are those ?
> > > > I haven't thought of the chaosmonkey tests yet, do those have
> specific
> > > > additional dependencies / scripts ?
> > > >
> > > > Should we move the tools simply to the normal jars, or should we move
> > > them
> > > > to a new module (could be called hbase-diagnostics) ?
> > > >
> > > > Istvan
> > > >
> > > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> > > bbeaudreault@apache.org>
> > > > wrote:
> > > >
> > > >> I'm +0 on hbase-examples, but +1000000 on any improvements we can
> make
> > > to
> > > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much
> > > reliance
> > > >> we have on test jars both generally but also specifically around
> these
> > > >> core
> > > >> test executables. Unfortunately I haven't had time to dedicate to
> > these
> > > >> frustrations myself, but happy to help with review, etc.
> > > >>
> > > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <ni...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Thank you for bringing this up.
> > > >> >
> > > >> > +1 for this change.
> > > >> >
> > > >> > In fact, some time back, we had faced similar problem. Security
> > scans
> > > >> found
> > > >> > that we were bundling some vulnerable hadoop test jar. To deal
> with
> > > >> that we
> > > >> > had to make a change in our internal HBase fork to exclude all
> HBase
> > > and
> > > >> > Hadoop test jars from assembly. This helped us get rid of
> vulnerable
> > > >> jar.
> > > >> > (Although I hadn't dealt with test scope dependencies there.)
> > > >> >
> > > >> > But, I have been thinking of pushing this change in Apache HBase,
> > just
> > > >> > wasn't sure if this was even acceptable. It's great to see same
> has
> > > been
> > > >> > brought up here today.
> > > >> >
> > > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a script to
> > > >> download
> > > >> > them on demand to avoid massive code change in internal fork. But
> I
> > > >> have a
> > > >> > +1 on the idea of identifying and moving all such tools to a new
> > > module.
> > > >> > This would be great and make things easier for us as well.
> > > >> >
> > > >> > Also, a way we could help new users easily get started, in case we
> > > >> > completely stop bundling hadoop jars, is by providing a script
> which
> > > >> starts
> > > >> > a hbase cluster in a single node setup. In fact I had written a
> > simple
> > > >> > script sometime back that automates this process given a release
> > link
> > > >> for
> > > >> > both. It first downloads Hadoop and HBase binaries and then starts
> > > both
> > > >> > with the hbase root directory set to be on hdfs. We could provide
> > > >> something
> > > >> > similar to help new users to get started easily.
> > > >> >
> > > >> > Although I am also +1 on the idea to provide both variants as
> > > mentioned
> > > >> by
> > > >> > Nick, which might not even need any such script.
> > > >> >
> > > >> > Also, I am willing to volunteer for help towards this effort.
> Please
> > > >> let me
> > > >> > know if anything is needed.
> > > >> >
> > > >> > Thanks,
> > > >> > Nihal
> > > >> >
> > > >> >
> > > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org>
> > wrote:
> > > >> >
> > > >> > > This would be great cleanup, big +1 from me for all three of
> these
> > > >> > > adjustments, including the promotion of pe, ltt, and friends out
> > of
> > > >> the
> > > >> > > test scope.
> > > >> > >
> > > >> > > I believe that we included hbase test jars because we used to
> > freely
> > > >> mix
> > > >> > > classes needed for minicluster between runtime and test jars,
> > which
> > > in
> > > >> > turn
> > > >> > > relied on Hadoop minicluster capabilities. The big cleanup
> around
> > > >> > > HBaseTestingUtil/it addressed much (or all) of these issues on
> > > >> branch-3.
> > > >> > >
> > > >> > > I believe that we include a Hadoop distribution in our assembly
> > > >> because
> > > >> > > that makes it easy for a new user to download our release
> bin.tgz
> > > and
> > > >> get
> > > >> > > started immediately with learning. I guess it’s high time that
> we
> > > work
> > > >> > out
> > > >> > > the with- and without-Hadoop variants.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Nick
> > > >> > >
> > > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org>
> > wrote:
> > > >> > >
> > > >> > > > DISCLAIMER: I don't have a patch ready, or even an elegant way
> > > >> mapped
> > > >> > out
> > > >> > > > to achieve this, this is about discussing whether we even want
> > to
> > > >> make
> > > >> > > > these changes.
> > > >> > > > These are also substantial changes, but they could be targeted
> > for
> > > >> > HBase
> > > >> > > > 3.0.
> > > >> > > >
> > > >> > > > One issue I have noticed is that we ship test jars and test
> > > >> > dependencies
> > > >> > > in
> > > >> > > > the assembly.
> > > >> > > > I can't see anyone using those, but it bloats the assembly and
> > > >> > classpath,
> > > >> > > > and adds unnecessary JARs with possible CVE issues. (for
> example
> > > >> Kerby
> > > >> > > > which is a Hadoop minicluster dependency)
> > > >> > > >
> > > >> > > > My proposal is to exclude the test jars and the test scope
> > > >> dependencies
> > > >> > > > from the assembly.
> > > >> > > >
> > > >> > > > The advantages would be:
> > > >> > > > * Smaller distro size
> > > >> > > > * Faster startup (this is marginal)
> > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > >> > > >
> > > >> > > > The other issue is that the assembly includes much of the
> Hadoop
> > > >> > > > distribution.
> > > >> > > > The basic assumption in all scripts and instructions is that
> the
> > > >> node
> > > >> > > has a
> > > >> > > > fully configured Hadoop installation, and we include it in the
> > > >> > classpath
> > > >> > > of
> > > >> > > > HBase.
> > > >> > > >
> > > >> > > > If that is true, then there is no reason to include Hadoop in
> > the
> > > >> > > assembly,
> > > >> > > > HBase and its direct dependencies should be enough.
> > > >> > > >
> > > >> > > > One could argue that it would simplify the client side, which
> is
> > > >> true
> > > >> > to
> > > >> > > > some extent (though 95% of the client distro use cases are
> > served
> > > >> > better
> > > >> > > by
> > > >> > > > simply using hbase-shaded-client).
> > > >> > > >
> > > >> > > > We could either remove the Hadoop libraries from either or
> both
> > of
> > > >> the
> > > >> > > > assemblies unconditionally, or provide two variants for either
> > or
> > > >> both
> > > >> > > > assemblies, one with Hadoop included, and one without it.
> > > >> > > > Spark already does this, it has binary distributions both with
> > and
> > > >> > > without
> > > >> > > > Hadoop.
> > > >> > > >
> > > >> > > > The advantages would be:
> > > >> > > > * Smaller distro size
> > > >> > > > * Faster startup (this is marginal)
> > > >> > > > * Less chance of conflicts with the Hadoop jars
> > > >> > > > * Less CVE-prone JARs in the binary assemblies
> > > >> > > >
> > > >> > > >
> > > >> > > > Thirdly, we could consider excluding the
> > > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from the
> > > >> Hadoop-less
> > > >> > > > binary assemblies. It is not used by the assembly, and AFAIK
> it
> > is
> > > >> not
> > > >> > > > included in any of the 'hbase classpath' command variants.
> > > >> > > >
> > > >> > > > This would make sure that no Hadoop libraries are included
> (even
> > > in
> > > >> > > shaded
> > > >> > > > form) and would make the HBase distribution fully insulated
> from
> > > >> > Hadoop's
> > > >> > > > CVE issues.
> > > >> > > >
> > > >> > > > (The full-fat hbase-shaded-client works best as direct
> > build-time
> > > >> > > > dependency anyway)
> > > >> > > >
> > > >> > > > best regards
> > > >> > > > Istvan
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Sr. Staff Software Engineer
> > > > *Email*: stoty@cloudera.com
> > > > cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > > ------------------------------
> > > >
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: stoty@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > > ------------------------------
> > >
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Thank you Nihal.
I'm not very familiar with the tools in the test code, so you can probably
plan that work better.
I just have some generic steps in mind:
* Identify all the tools / scripts in the test jars
* Identify and analyze their dependencies (compared to the current runtime
deps)
* Decide which ones to move to the runtime JARs.
* Move them to the runtime code (or perhaps a separate module)
I have created https://issues.apache.org/jira/browse/HBASE-28431 as an
umbrella ticket to organize the sub-tasks.
Istvan
On Fri, Mar 8, 2024 at 7:06 PM Nihal Jain <ni...@gmail.com> wrote:
> Sure I will be able to take up. Please create tasks with necessary details
> or let me know if you want me to create.
>
> On Fri, 8 Mar 2024, 12:45 Istvan Toth, <st...@cloudera.com.invalid> wrote:
>
> > Thanks for volunteering, Nihal.
> >
> > I could work on the Hadoop-less, and assemblies, and you could work on
> > cleaning up the test jars.
> > Would that work for you ?
> > I know that I'm picking the smaller part, but it turns out that I won't
> > have as much time to work on this as I hoped.
> >
> > (Unless there are other volunteers, of course)
> >
> > Istvan
> >
> > On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com> wrote:
> >
> > > We seem to be in agreement in principle, however the devil is in the
> > > details.
> > >
> > > The first step should be moving the diagnostic tools out of the test
> > jars.
> > > Are there any tools we don't want to move out ?
> > > Do the diagnostic tools pull in extra dependencies compared to the
> > current
> > > runtime JARs, and if they do, what are those ?
> > > I haven't thought of the chaosmonkey tests yet, do those have specific
> > > additional dependencies / scripts ?
> > >
> > > Should we move the tools simply to the normal jars, or should we move
> > them
> > > to a new module (could be called hbase-diagnostics) ?
> > >
> > > Istvan
> > >
> > > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> > bbeaudreault@apache.org>
> > > wrote:
> > >
> > >> I'm +0 on hbase-examples, but +1000000 on any improvements we can make
> > to
> > >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much
> > reliance
> > >> we have on test jars both generally but also specifically around these
> > >> core
> > >> test executables. Unfortunately I haven't had time to dedicate to
> these
> > >> frustrations myself, but happy to help with review, etc.
> > >>
> > >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <ni...@gmail.com>
> > wrote:
> > >>
> > >> > Thank you for bringing this up.
> > >> >
> > >> > +1 for this change.
> > >> >
> > >> > In fact, some time back, we had faced similar problem. Security
> scans
> > >> found
> > >> > that we were bundling some vulnerable hadoop test jar. To deal with
> > >> that we
> > >> > had to make a change in our internal HBase fork to exclude all HBase
> > and
> > >> > Hadoop test jars from assembly. This helped us get rid of vulnerable
> > >> jar.
> > >> > (Although I hadn't dealt with test scope dependencies there.)
> > >> >
> > >> > But, I have been thinking of pushing this change in Apache HBase,
> just
> > >> > wasn't sure if this was even acceptable. It's great to see same has
> > been
> > >> > brought up here today.
> > >> >
> > >> > We hadn't dealt with the ltt, pe etc. tools and wrote a script to
> > >> download
> > >> > them on demand to avoid massive code change in internal fork. But I
> > >> have a
> > >> > +1 on the idea of identifying and moving all such tools to a new
> > module.
> > >> > This would be great and make things easier for us as well.
> > >> >
> > >> > Also, a way we could help new users easily get started, in case we
> > >> > completely stop bundling hadoop jars, is by providing a script which
> > >> starts
> > >> > a hbase cluster in a single node setup. In fact I had written a
> simple
> > >> > script sometime back that automates this process given a release
> link
> > >> for
> > >> > both. It first downloads Hadoop and HBase binaries and then starts
> > both
> > >> > with the hbase root directory set to be on hdfs. We could provide
> > >> something
> > >> > similar to help new users to get started easily.
> > >> >
> > >> > Although I am also +1 on the idea to provide both variants as
> > mentioned
> > >> by
> > >> > Nick, which might not even need any such script.
> > >> >
> > >> > Also, I am willing to volunteer for help towards this effort. Please
> > >> let me
> > >> > know if anything is needed.
> > >> >
> > >> > Thanks,
> > >> > Nihal
> > >> >
> > >> >
> > >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org>
> wrote:
> > >> >
> > >> > > This would be great cleanup, big +1 from me for all three of these
> > >> > > adjustments, including the promotion of pe, ltt, and friends out
> of
> > >> the
> > >> > > test scope.
> > >> > >
> > >> > > I believe that we included hbase test jars because we used to
> freely
> > >> mix
> > >> > > classes needed for minicluster between runtime and test jars,
> which
> > in
> > >> > turn
> > >> > > relied on Hadoop minicluster capabilities. The big cleanup around
> > >> > > HBaseTestingUtil/it addressed much (or all) of these issues on
> > >> branch-3.
> > >> > >
> > >> > > I believe that we include a Hadoop distribution in our assembly
> > >> because
> > >> > > that makes it easy for a new user to download our release bin.tgz
> > and
> > >> get
> > >> > > started immediately with learning. I guess it’s high time that we
> > work
> > >> > out
> > >> > > the with- and without-Hadoop variants.
> > >> > >
> > >> > > Thanks,
> > >> > > Nick
> > >> > >
> > >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org>
> wrote:
> > >> > >
> > >> > > > DISCLAIMER: I don't have a patch ready, or even an elegant way
> > >> mapped
> > >> > out
> > >> > > > to achieve this, this is about discussing whether we even want
> to
> > >> make
> > >> > > > these changes.
> > >> > > > These are also substantial changes, but they could be targeted
> for
> > >> > HBase
> > >> > > > 3.0.
> > >> > > >
> > >> > > > One issue I have noticed is that we ship test jars and test
> > >> > dependencies
> > >> > > in
> > >> > > > the assembly.
> > >> > > > I can't see anyone using those, but it bloats the assembly and
> > >> > classpath,
> > >> > > > and adds unnecessary JARs with possible CVE issues. (for example
> > >> Kerby
> > >> > > > which is a Hadoop minicluster dependency)
> > >> > > >
> > >> > > > My proposal is to exclude the test jars and the test scope
> > >> dependencies
> > >> > > > from the assembly.
> > >> > > >
> > >> > > > The advantages would be:
> > >> > > > * Smaller distro size
> > >> > > > * Faster startup (this is marginal)
> > >> > > > * Less CVE-prone JARs in the binary assemblies
> > >> > > >
> > >> > > > The other issue is that the assembly includes much of the Hadoop
> > >> > > > distribution.
> > >> > > > The basic assumption in all scripts and instructions is that the
> > >> node
> > >> > > has a
> > >> > > > fully configured Hadoop installation, and we include it in the
> > >> > classpath
> > >> > > of
> > >> > > > HBase.
> > >> > > >
> > >> > > > If that is true, then there is no reason to include Hadoop in
> the
> > >> > > assembly,
> > >> > > > HBase and its direct dependencies should be enough.
> > >> > > >
> > >> > > > One could argue that it would simplify the client side, which is
> > >> true
> > >> > to
> > >> > > > some extent (though 95% of the client distro use cases are
> served
> > >> > better
> > >> > > by
> > >> > > > simply using hbase-shaded-client).
> > >> > > >
> > >> > > > We could either remove the Hadoop libraries from either or both
> of
> > >> the
> > >> > > > assemblies unconditionally, or provide two variants for either
> or
> > >> both
> > >> > > > assemblies, one with Hadoop included, and one without it.
> > >> > > > Spark already does this, it has binary distributions both with
> and
> > >> > > without
> > >> > > > Hadoop.
> > >> > > >
> > >> > > > The advantages would be:
> > >> > > > * Smaller distro size
> > >> > > > * Faster startup (this is marginal)
> > >> > > > * Less chance of conflicts with the Hadoop jars
> > >> > > > * Less CVE-prone JARs in the binary assemblies
> > >> > > >
> > >> > > >
> > >> > > > Thirdly, we could consider excluding the
> > >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from the
> > >> Hadoop-less
> > >> > > > binary assemblies. It is not used by the assembly, and AFAIK it
> is
> > >> not
> > >> > > > included in any of the 'hbase classpath' command variants.
> > >> > > >
> > >> > > > This would make sure that no Hadoop libraries are included (even
> > in
> > >> > > shaded
> > >> > > > form) and would make the HBase distribution fully insulated from
> > >> > Hadoop's
> > >> > > > CVE issues.
> > >> > > >
> > >> > > > (The full-fat hbase-shaded-client works best as direct
> build-time
> > >> > > > dependency anyway)
> > >> > > >
> > >> > > > best regards
> > >> > > > Istvan
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: stoty@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > > ------------------------------
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
> >
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Nihal Jain <ni...@gmail.com>.
Sure I will be able to take up. Please create tasks with necessary details
or let me know if you want me to create.
On Fri, 8 Mar 2024, 12:45 Istvan Toth, <st...@cloudera.com.invalid> wrote:
> Thanks for volunteering, Nihal.
>
> I could work on the Hadoop-less, and assemblies, and you could work on
> cleaning up the test jars.
> Would that work for you ?
> I know that I'm picking the smaller part, but it turns out that I won't
> have as much time to work on this as I hoped.
>
> (Unless there are other volunteers, of course)
>
> Istvan
>
> On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com> wrote:
>
> > We seem to be in agreement in principle, however the devil is in the
> > details.
> >
> > The first step should be moving the diagnostic tools out of the test
> jars.
> > Are there any tools we don't want to move out ?
> > Do the diagnostic tools pull in extra dependencies compared to the
> current
> > runtime JARs, and if they do, what are those ?
> > I haven't thought of the chaosmonkey tests yet, do those have specific
> > additional dependencies / scripts ?
> >
> > Should we move the tools simply to the normal jars, or should we move
> them
> > to a new module (could be called hbase-diagnostics) ?
> >
> > Istvan
> >
> > On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <
> bbeaudreault@apache.org>
> > wrote:
> >
> >> I'm +0 on hbase-examples, but +1000000 on any improvements we can make
> to
> >> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much
> reliance
> >> we have on test jars both generally but also specifically around these
> >> core
> >> test executables. Unfortunately I haven't had time to dedicate to these
> >> frustrations myself, but happy to help with review, etc.
> >>
> >> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <ni...@gmail.com>
> wrote:
> >>
> >> > Thank you for bringing this up.
> >> >
> >> > +1 for this change.
> >> >
> >> > In fact, some time back, we had faced similar problem. Security scans
> >> found
> >> > that we were bundling some vulnerable hadoop test jar. To deal with
> >> that we
> >> > had to make a change in our internal HBase fork to exclude all HBase
> and
> >> > Hadoop test jars from assembly. This helped us get rid of vulnerable
> >> jar.
> >> > (Although I hadn't dealt with test scope dependencies there.)
> >> >
> >> > But, I have been thinking of pushing this change in Apache HBase, just
> >> > wasn't sure if this was even acceptable. It's great to see same has
> been
> >> > brought up here today.
> >> >
> >> > We hadn't dealt with the ltt, pe etc. tools and wrote a script to
> >> download
> >> > them on demand to avoid massive code change in internal fork. But I
> >> have a
> >> > +1 on the idea of identifying and moving all such tools to a new
> module.
> >> > This would be great and make things easier for us as well.
> >> >
> >> > Also, a way we could help new users easily get started, in case we
> >> > completely stop bundling hadoop jars, is by providing a script which
> >> starts
> >> > a hbase cluster in a single node setup. In fact I had written a simple
> >> > script sometime back that automates this process given a release link
> >> for
> >> > both. It first downloads Hadoop and HBase binaries and then starts
> both
> >> > with the hbase root directory set to be on hdfs. We could provide
> >> something
> >> > similar to help new users to get started easily.
> >> >
> >> > Although I am also +1 on the idea to provide both variants as
> mentioned
> >> by
> >> > Nick, which might not even need any such script.
> >> >
> >> > Also, I am willing to volunteer for help towards this effort. Please
> >> let me
> >> > know if anything is needed.
> >> >
> >> > Thanks,
> >> > Nihal
> >> >
> >> >
> >> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org> wrote:
> >> >
> >> > > This would be great cleanup, big +1 from me for all three of these
> >> > > adjustments, including the promotion of pe, ltt, and friends out of
> >> the
> >> > > test scope.
> >> > >
> >> > > I believe that we included hbase test jars because we used to freely
> >> mix
> >> > > classes needed for minicluster between runtime and test jars, which
> in
> >> > turn
> >> > > relied on Hadoop minicluster capabilities. The big cleanup around
> >> > > HBaseTestingUtil/it addressed much (or all) of these issues on
> >> branch-3.
> >> > >
> >> > > I believe that we include a Hadoop distribution in our assembly
> >> because
> >> > > that makes it easy for a new user to download our release bin.tgz
> and
> >> get
> >> > > started immediately with learning. I guess it’s high time that we
> work
> >> > out
> >> > > the with- and without-Hadoop variants.
> >> > >
> >> > > Thanks,
> >> > > Nick
> >> > >
> >> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org> wrote:
> >> > >
> >> > > > DISCLAIMER: I don't have a patch ready, or even an elegant way
> >> mapped
> >> > out
> >> > > > to achieve this, this is about discussing whether we even want to
> >> make
> >> > > > these changes.
> >> > > > These are also substantial changes, but they could be targeted for
> >> > HBase
> >> > > > 3.0.
> >> > > >
> >> > > > One issue I have noticed is that we ship test jars and test
> >> > dependencies
> >> > > in
> >> > > > the assembly.
> >> > > > I can't see anyone using those, but it bloats the assembly and
> >> > classpath,
> >> > > > and adds unnecessary JARs with possible CVE issues. (for example
> >> Kerby
> >> > > > which is a Hadoop minicluster dependency)
> >> > > >
> >> > > > My proposal is to exclude the test jars and the test scope
> >> dependencies
> >> > > > from the assembly.
> >> > > >
> >> > > > The advantages would be:
> >> > > > * Smaller distro size
> >> > > > * Faster startup (this is marginal)
> >> > > > * Less CVE-prone JARs in the binary assemblies
> >> > > >
> >> > > > The other issue is that the assembly includes much of the Hadoop
> >> > > > distribution.
> >> > > > The basic assumption in all scripts and instructions is that the
> >> node
> >> > > has a
> >> > > > fully configured Hadoop installation, and we include it in the
> >> > classpath
> >> > > of
> >> > > > HBase.
> >> > > >
> >> > > > If that is true, then there is no reason to include Hadoop in the
> >> > > assembly,
> >> > > > HBase and its direct dependencies should be enough.
> >> > > >
> >> > > > One could argue that it would simplify the client side, which is
> >> true
> >> > to
> >> > > > some extent (though 95% of the client distro use cases are served
> >> > better
> >> > > by
> >> > > > simply using hbase-shaded-client).
> >> > > >
> >> > > > We could either remove the Hadoop libraries from either or both of
> >> the
> >> > > > assemblies unconditionally, or provide two variants for either or
> >> both
> >> > > > assemblies, one with Hadoop included, and one without it.
> >> > > > Spark already does this, it has binary distributions both with and
> >> > > without
> >> > > > Hadoop.
> >> > > >
> >> > > > The advantages would be:
> >> > > > * Smaller distro size
> >> > > > * Faster startup (this is marginal)
> >> > > > * Less chance of conflicts with the Hadoop jars
> >> > > > * Less CVE-prone JARs in the binary assemblies
> >> > > >
> >> > > >
> >> > > > Thirdly, we could consider excluding the
> >> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from the
> >> Hadoop-less
> >> > > > binary assemblies. It is not used by the assembly, and AFAIK it is
> >> not
> >> > > > included in any of the 'hbase classpath' command variants.
> >> > > >
> >> > > > This would make sure that no Hadoop libraries are included (even
> in
> >> > > shaded
> >> > > > form) and would make the HBase distribution fully insulated from
> >> > Hadoop's
> >> > > > CVE issues.
> >> > > >
> >> > > > (The full-fat hbase-shaded-client works best as direct build-time
> >> > > > dependency anyway)
> >> > > >
> >> > > > best regards
> >> > > > Istvan
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Thanks for volunteering, Nihal.
I could work on the Hadoop-less, and assemblies, and you could work on
cleaning up the test jars.
Would that work for you ?
I know that I'm picking the smaller part, but it turns out that I won't
have as much time to work on this as I hoped.
(Unless there are other volunteers, of course)
Istvan
On Wed, Mar 6, 2024 at 7:03 PM Istvan Toth <st...@cloudera.com> wrote:
> We seem to be in agreement in principle, however the devil is in the
> details.
>
> The first step should be moving the diagnostic tools out of the test jars.
> Are there any tools we don't want to move out ?
> Do the diagnostic tools pull in extra dependencies compared to the current
> runtime JARs, and if they do, what are those ?
> I haven't thought of the chaosmonkey tests yet, do those have specific
> additional dependencies / scripts ?
>
> Should we move the tools simply to the normal jars, or should we move them
> to a new module (could be called hbase-diagnostics) ?
>
> Istvan
>
> On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <bb...@apache.org>
> wrote:
>
>> I'm +0 on hbase-examples, but +1000000 on any improvements we can make to
>> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much reliance
>> we have on test jars both generally but also specifically around these
>> core
>> test executables. Unfortunately I haven't had time to dedicate to these
>> frustrations myself, but happy to help with review, etc.
>>
>> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <ni...@gmail.com> wrote:
>>
>> > Thank you for bringing this up.
>> >
>> > +1 for this change.
>> >
>> > In fact, some time back, we had faced similar problem. Security scans
>> found
>> > that we were bundling some vulnerable hadoop test jar. To deal with
>> that we
>> > had to make a change in our internal HBase fork to exclude all HBase and
>> > Hadoop test jars from assembly. This helped us get rid of vulnerable
>> jar.
>> > (Although I hadn't dealt with test scope dependencies there.)
>> >
>> > But, I have been thinking of pushing this change in Apache HBase, just
>> > wasn't sure if this was even acceptable. It's great to see same has been
>> > brought up here today.
>> >
>> > We hadn't dealt with the ltt, pe etc. tools and wrote a script to
>> download
>> > them on demand to avoid massive code change in internal fork. But I
>> have a
>> > +1 on the idea of identifying and moving all such tools to a new module.
>> > This would be great and make things easier for us as well.
>> >
>> > Also, a way we could help new users easily get started, in case we
>> > completely stop bundling hadoop jars, is by providing a script which
>> starts
>> > a hbase cluster in a single node setup. In fact I had written a simple
>> > script sometime back that automates this process given a release link
>> for
>> > both. It first downloads Hadoop and HBase binaries and then starts both
>> > with the hbase root directory set to be on hdfs. We could provide
>> something
>> > similar to help new users to get started easily.
>> >
>> > Although I am also +1 on the idea to provide both variants as mentioned
>> by
>> > Nick, which might not even need any such script.
>> >
>> > Also, I am willing to volunteer for help towards this effort. Please
>> let me
>> > know if anything is needed.
>> >
>> > Thanks,
>> > Nihal
>> >
>> >
>> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org> wrote:
>> >
>> > > This would be great cleanup, big +1 from me for all three of these
>> > > adjustments, including the promotion of pe, ltt, and friends out of
>> the
>> > > test scope.
>> > >
>> > > I believe that we included hbase test jars because we used to freely
>> mix
>> > > classes needed for minicluster between runtime and test jars, which in
>> > turn
>> > > relied on Hadoop minicluster capabilities. The big cleanup around
>> > > HBaseTestingUtil/it addressed much (or all) of these issues on
>> branch-3.
>> > >
>> > > I believe that we include a Hadoop distribution in our assembly
>> because
>> > > that makes it easy for a new user to download our release bin.tgz and
>> get
>> > > started immediately with learning. I guess it’s high time that we work
>> > out
>> > > the with- and without-Hadoop variants.
>> > >
>> > > Thanks,
>> > > Nick
>> > >
>> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org> wrote:
>> > >
>> > > > DISCLAIMER: I don't have a patch ready, or even an elegant way
>> mapped
>> > out
>> > > > to achieve this, this is about discussing whether we even want to
>> make
>> > > > these changes.
>> > > > These are also substantial changes, but they could be targeted for
>> > HBase
>> > > > 3.0.
>> > > >
>> > > > One issue I have noticed is that we ship test jars and test
>> > dependencies
>> > > in
>> > > > the assembly.
>> > > > I can't see anyone using those, but it bloats the assembly and
>> > classpath,
>> > > > and adds unnecessary JARs with possible CVE issues. (for example
>> Kerby
>> > > > which is a Hadoop minicluster dependency)
>> > > >
>> > > > My proposal is to exclude the test jars and the test scope
>> dependencies
>> > > > from the assembly.
>> > > >
>> > > > The advantages would be:
>> > > > * Smaller distro size
>> > > > * Faster startup (this is marginal)
>> > > > * Less CVE-prone JARs in the binary assemblies
>> > > >
>> > > > The other issue is that the assembly includes much of the Hadoop
>> > > > distribution.
>> > > > The basic assumption in all scripts and instructions is that the
>> node
>> > > has a
>> > > > fully configured Hadoop installation, and we include it in the
>> > classpath
>> > > of
>> > > > HBase.
>> > > >
>> > > > If that is true, then there is no reason to include Hadoop in the
>> > > assembly,
>> > > > HBase and its direct dependencies should be enough.
>> > > >
>> > > > One could argue that it would simplify the client side, which is
>> true
>> > to
>> > > > some extent (though 95% of the client distro use cases are served
>> > better
>> > > by
>> > > > simply using hbase-shaded-client).
>> > > >
>> > > > We could either remove the Hadoop libraries from either or both of
>> the
>> > > > assemblies unconditionally, or provide two variants for either or
>> both
>> > > > assemblies, one with Hadoop included, and one without it.
>> > > > Spark already does this, it has binary distributions both with and
>> > > without
>> > > > Hadoop.
>> > > >
>> > > > The advantages would be:
>> > > > * Smaller distro size
>> > > > * Faster startup (this is marginal)
>> > > > * Less chance of conflicts with the Hadoop jars
>> > > > * Less CVE-prone JARs in the binary assemblies
>> > > >
>> > > >
>> > > > Thirdly, we could consider excluding the
>> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from the
>> Hadoop-less
>> > > > binary assemblies. It is not used by the assembly, and AFAIK it is
>> not
>> > > > included in any of the 'hbase classpath' command variants.
>> > > >
>> > > > This would make sure that no Hadoop libraries are included (even in
>> > > shaded
>> > > > form) and would make the HBase distribution fully insulated from
>> > Hadoop's
>> > > > CVE issues.
>> > > >
>> > > > (The full-fat hbase-shaded-client works best as direct build-time
>> > > > dependency anyway)
>> > > >
>> > > > best regards
>> > > > Istvan
>> > > >
>> > >
>> >
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
We seem to be in agreement in principle, however the devil is in the
details.
The first step should be moving the diagnostic tools out of the test jars.
Are there any tools we don't want to move out ?
Do the diagnostic tools pull in extra dependencies compared to the current
runtime JARs, and if they do, what are those ?
I haven't thought of the chaosmonkey tests yet, do those have specific
additional dependencies / scripts ?
Should we move the tools simply to the normal jars, or should we move them
to a new module (could be called hbase-diagnostics) ?
Istvan
On Tue, Mar 5, 2024 at 7:10 PM Bryan Beaudreault <bb...@apache.org>
wrote:
> I'm +0 on hbase-examples, but +1000000 on any improvements we can make to
> ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much reliance
> we have on test jars both generally but also specifically around these core
> test executables. Unfortunately I haven't had time to dedicate to these
> frustrations myself, but happy to help with review, etc.
>
> On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <ni...@gmail.com> wrote:
>
> > Thank you for bringing this up.
> >
> > +1 for this change.
> >
> > In fact, some time back, we had faced similar problem. Security scans
> found
> > that we were bundling some vulnerable hadoop test jar. To deal with that
> we
> > had to make a change in our internal HBase fork to exclude all HBase and
> > Hadoop test jars from assembly. This helped us get rid of vulnerable jar.
> > (Although I hadn't dealt with test scope dependencies there.)
> >
> > But, I have been thinking of pushing this change in Apache HBase, just
> > wasn't sure if this was even acceptable. It's great to see same has been
> > brought up here today.
> >
> > We hadn't dealt with the ltt, pe etc. tools and wrote a script to
> download
> > them on demand to avoid massive code change in internal fork. But I have
> a
> > +1 on the idea of identifying and moving all such tools to a new module.
> > This would be great and make things easier for us as well.
> >
> > Also, a way we could help new users easily get started, in case we
> > completely stop bundling hadoop jars, is by providing a script which
> starts
> > a hbase cluster in a single node setup. In fact I had written a simple
> > script sometime back that automates this process given a release link for
> > both. It first downloads Hadoop and HBase binaries and then starts both
> > with the hbase root directory set to be on hdfs. We could provide
> something
> > similar to help new users to get started easily.
> >
> > Although I am also +1 on the idea to provide both variants as mentioned
> by
> > Nick, which might not even need any such script.
> >
> > Also, I am willing to volunteer for help towards this effort. Please let
> me
> > know if anything is needed.
> >
> > Thanks,
> > Nihal
> >
> >
> > On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org> wrote:
> >
> > > This would be great cleanup, big +1 from me for all three of these
> > > adjustments, including the promotion of pe, ltt, and friends out of the
> > > test scope.
> > >
> > > I believe that we included hbase test jars because we used to freely
> mix
> > > classes needed for minicluster between runtime and test jars, which in
> > turn
> > > relied on Hadoop minicluster capabilities. The big cleanup around
> > > HBaseTestingUtil/it addressed much (or all) of these issues on
> branch-3.
> > >
> > > I believe that we include a Hadoop distribution in our assembly because
> > > that makes it easy for a new user to download our release bin.tgz and
> get
> > > started immediately with learning. I guess it’s high time that we work
> > out
> > > the with- and without-Hadoop variants.
> > >
> > > Thanks,
> > > Nick
> > >
> > > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org> wrote:
> > >
> > > > DISCLAIMER: I don't have a patch ready, or even an elegant way mapped
> > out
> > > > to achieve this, this is about discussing whether we even want to
> make
> > > > these changes.
> > > > These are also substantial changes, but they could be targeted for
> > HBase
> > > > 3.0.
> > > >
> > > > One issue I have noticed is that we ship test jars and test
> > dependencies
> > > in
> > > > the assembly.
> > > > I can't see anyone using those, but it bloats the assembly and
> > classpath,
> > > > and adds unnecessary JARs with possible CVE issues. (for example
> Kerby
> > > > which is a Hadoop minicluster dependency)
> > > >
> > > > My proposal is to exclude the test jars and the test scope
> dependencies
> > > > from the assembly.
> > > >
> > > > The advantages would be:
> > > > * Smaller distro size
> > > > * Faster startup (this is marginal)
> > > > * Less CVE-prone JARs in the binary assemblies
> > > >
> > > > The other issue is that the assembly includes much of the Hadoop
> > > > distribution.
> > > > The basic assumption in all scripts and instructions is that the node
> > > has a
> > > > fully configured Hadoop installation, and we include it in the
> > classpath
> > > of
> > > > HBase.
> > > >
> > > > If that is true, then there is no reason to include Hadoop in the
> > > assembly,
> > > > HBase and its direct dependencies should be enough.
> > > >
> > > > One could argue that it would simplify the client side, which is true
> > to
> > > > some extent (though 95% of the client distro use cases are served
> > better
> > > by
> > > > simply using hbase-shaded-client).
> > > >
> > > > We could either remove the Hadoop libraries from either or both of
> the
> > > > assemblies unconditionally, or provide two variants for either or
> both
> > > > assemblies, one with Hadoop included, and one without it.
> > > > Spark already does this, it has binary distributions both with and
> > > without
> > > > Hadoop.
> > > >
> > > > The advantages would be:
> > > > * Smaller distro size
> > > > * Faster startup (this is marginal)
> > > > * Less chance of conflicts with the Hadoop jars
> > > > * Less CVE-prone JARs in the binary assemblies
> > > >
> > > >
> > > > Thirdly, we could consider excluding the
> > > > full-fat org.apache.hbase:hbase-shaded-client JAR from the
> Hadoop-less
> > > > binary assemblies. It is not used by the assembly, and AFAIK it is
> not
> > > > included in any of the 'hbase classpath' command variants.
> > > >
> > > > This would make sure that no Hadoop libraries are included (even in
> > > shaded
> > > > form) and would make the HBase distribution fully insulated from
> > Hadoop's
> > > > CVE issues.
> > > >
> > > > (The full-fat hbase-shaded-client works best as direct build-time
> > > > dependency anyway)
> > > >
> > > > best regards
> > > > Istvan
> > > >
> > >
> >
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Bryan Beaudreault <bb...@apache.org>.
I'm +0 on hbase-examples, but +1000000 on any improvements we can make to
ltt/pe/chaos/minicluster/etc. It's extremely frustrating how much reliance
we have on test jars both generally but also specifically around these core
test executables. Unfortunately I haven't had time to dedicate to these
frustrations myself, but happy to help with review, etc.
On Tue, Mar 5, 2024 at 1:03 PM Nihal Jain <ni...@gmail.com> wrote:
> Thank you for bringing this up.
>
> +1 for this change.
>
> In fact, some time back, we had faced similar problem. Security scans found
> that we were bundling some vulnerable hadoop test jar. To deal with that we
> had to make a change in our internal HBase fork to exclude all HBase and
> Hadoop test jars from assembly. This helped us get rid of vulnerable jar.
> (Although I hadn't dealt with test scope dependencies there.)
>
> But, I have been thinking of pushing this change in Apache HBase, just
> wasn't sure if this was even acceptable. It's great to see same has been
> brought up here today.
>
> We hadn't dealt with the ltt, pe etc. tools and wrote a script to download
> them on demand to avoid massive code change in internal fork. But I have a
> +1 on the idea of identifying and moving all such tools to a new module.
> This would be great and make things easier for us as well.
>
> Also, a way we could help new users easily get started, in case we
> completely stop bundling hadoop jars, is by providing a script which starts
> a hbase cluster in a single node setup. In fact I had written a simple
> script sometime back that automates this process given a release link for
> both. It first downloads Hadoop and HBase binaries and then starts both
> with the hbase root directory set to be on hdfs. We could provide something
> similar to help new users to get started easily.
>
> Although I am also +1 on the idea to provide both variants as mentioned by
> Nick, which might not even need any such script.
>
> Also, I am willing to volunteer for help towards this effort. Please let me
> know if anything is needed.
>
> Thanks,
> Nihal
>
>
> On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org> wrote:
>
> > This would be great cleanup, big +1 from me for all three of these
> > adjustments, including the promotion of pe, ltt, and friends out of the
> > test scope.
> >
> > I believe that we included hbase test jars because we used to freely mix
> > classes needed for minicluster between runtime and test jars, which in
> turn
> > relied on Hadoop minicluster capabilities. The big cleanup around
> > HBaseTestingUtil/it addressed much (or all) of these issues on branch-3.
> >
> > I believe that we include a Hadoop distribution in our assembly because
> > that makes it easy for a new user to download our release bin.tgz and get
> > started immediately with learning. I guess it’s high time that we work
> out
> > the with- and without-Hadoop variants.
> >
> > Thanks,
> > Nick
> >
> > On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org> wrote:
> >
> > > DISCLAIMER: I don't have a patch ready, or even an elegant way mapped
> out
> > > to achieve this, this is about discussing whether we even want to make
> > > these changes.
> > > These are also substantial changes, but they could be targeted for
> HBase
> > > 3.0.
> > >
> > > One issue I have noticed is that we ship test jars and test
> dependencies
> > in
> > > the assembly.
> > > I can't see anyone using those, but it bloats the assembly and
> classpath,
> > > and adds unnecessary JARs with possible CVE issues. (for example Kerby
> > > which is a Hadoop minicluster dependency)
> > >
> > > My proposal is to exclude the test jars and the test scope dependencies
> > > from the assembly.
> > >
> > > The advantages would be:
> > > * Smaller distro size
> > > * Faster startup (this is marginal)
> > > * Less CVE-prone JARs in the binary assemblies
> > >
> > > The other issue is that the assembly includes much of the Hadoop
> > > distribution.
> > > The basic assumption in all scripts and instructions is that the node
> > has a
> > > fully configured Hadoop installation, and we include it in the
> classpath
> > of
> > > HBase.
> > >
> > > If that is true, then there is no reason to include Hadoop in the
> > assembly,
> > > HBase and its direct dependencies should be enough.
> > >
> > > One could argue that it would simplify the client side, which is true
> to
> > > some extent (though 95% of the client distro use cases are served
> better
> > by
> > > simply using hbase-shaded-client).
> > >
> > > We could either remove the Hadoop libraries from either or both of the
> > > assemblies unconditionally, or provide two variants for either or both
> > > assemblies, one with Hadoop included, and one without it.
> > > Spark already does this, it has binary distributions both with and
> > without
> > > Hadoop.
> > >
> > > The advantages would be:
> > > * Smaller distro size
> > > * Faster startup (this is marginal)
> > > * Less chance of conflicts with the Hadoop jars
> > > * Less CVE-prone JARs in the binary assemblies
> > >
> > >
> > > Thirdly, we could consider excluding the
> > > full-fat org.apache.hbase:hbase-shaded-client JAR from the Hadoop-less
> > > binary assemblies. It is not used by the assembly, and AFAIK it is not
> > > included in any of the 'hbase classpath' command variants.
> > >
> > > This would make sure that no Hadoop libraries are included (even in
> > shaded
> > > form) and would make the HBase distribution fully insulated from
> Hadoop's
> > > CVE issues.
> > >
> > > (The full-fat hbase-shaded-client works best as direct build-time
> > > dependency anyway)
> > >
> > > best regards
> > > Istvan
> > >
> >
>
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Nihal Jain <ni...@gmail.com>.
Thank you for bringing this up.
+1 for this change.
In fact, some time back, we had faced similar problem. Security scans found
that we were bundling some vulnerable hadoop test jar. To deal with that we
had to make a change in our internal HBase fork to exclude all HBase and
Hadoop test jars from assembly. This helped us get rid of vulnerable jar.
(Although I hadn't dealt with test scope dependencies there.)
But, I have been thinking of pushing this change in Apache HBase, just
wasn't sure if this was even acceptable. It's great to see same has been
brought up here today.
We hadn't dealt with the ltt, pe etc. tools and wrote a script to download
them on demand to avoid massive code change in internal fork. But I have a
+1 on the idea of identifying and moving all such tools to a new module.
This would be great and make things easier for us as well.
Also, a way we could help new users easily get started, in case we
completely stop bundling hadoop jars, is by providing a script which starts
a hbase cluster in a single node setup. In fact I had written a simple
script sometime back that automates this process given a release link for
both. It first downloads Hadoop and HBase binaries and then starts both
with the hbase root directory set to be on hdfs. We could provide something
similar to help new users to get started easily.
Although I am also +1 on the idea to provide both variants as mentioned by
Nick, which might not even need any such script.
Also, I am willing to volunteer for help towards this effort. Please let me
know if anything is needed.
Thanks,
Nihal
On Tue, 5 Mar 2024, 15:35 Nick Dimiduk, <nd...@apache.org> wrote:
> This would be great cleanup, big +1 from me for all three of these
> adjustments, including the promotion of pe, ltt, and friends out of the
> test scope.
>
> I believe that we included hbase test jars because we used to freely mix
> classes needed for minicluster between runtime and test jars, which in turn
> relied on Hadoop minicluster capabilities. The big cleanup around
> HBaseTestingUtil/it addressed much (or all) of these issues on branch-3.
>
> I believe that we include a Hadoop distribution in our assembly because
> that makes it easy for a new user to download our release bin.tgz and get
> started immediately with learning. I guess it’s high time that we work out
> the with- and without-Hadoop variants.
>
> Thanks,
> Nick
>
> On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org> wrote:
>
> > DISCLAIMER: I don't have a patch ready, or even an elegant way mapped out
> > to achieve this, this is about discussing whether we even want to make
> > these changes.
> > These are also substantial changes, but they could be targeted for HBase
> > 3.0.
> >
> > One issue I have noticed is that we ship test jars and test dependencies
> in
> > the assembly.
> > I can't see anyone using those, but it bloats the assembly and classpath,
> > and adds unnecessary JARs with possible CVE issues. (for example Kerby
> > which is a Hadoop minicluster dependency)
> >
> > My proposal is to exclude the test jars and the test scope dependencies
> > from the assembly.
> >
> > The advantages would be:
> > * Smaller distro size
> > * Faster startup (this is marginal)
> > * Less CVE-prone JARs in the binary assemblies
> >
> > The other issue is that the assembly includes much of the Hadoop
> > distribution.
> > The basic assumption in all scripts and instructions is that the node
> has a
> > fully configured Hadoop installation, and we include it in the classpath
> of
> > HBase.
> >
> > If that is true, then there is no reason to include Hadoop in the
> assembly,
> > HBase and its direct dependencies should be enough.
> >
> > One could argue that it would simplify the client side, which is true to
> > some extent (though 95% of the client distro use cases are served better
> by
> > simply using hbase-shaded-client).
> >
> > We could either remove the Hadoop libraries from either or both of the
> > assemblies unconditionally, or provide two variants for either or both
> > assemblies, one with Hadoop included, and one without it.
> > Spark already does this, it has binary distributions both with and
> without
> > Hadoop.
> >
> > The advantages would be:
> > * Smaller distro size
> > * Faster startup (this is marginal)
> > * Less chance of conflicts with the Hadoop jars
> > * Less CVE-prone JARs in the binary assemblies
> >
> >
> > Thirdly, we could consider excluding the
> > full-fat org.apache.hbase:hbase-shaded-client JAR from the Hadoop-less
> > binary assemblies. It is not used by the assembly, and AFAIK it is not
> > included in any of the 'hbase classpath' command variants.
> >
> > This would make sure that no Hadoop libraries are included (even in
> shaded
> > form) and would make the HBase distribution fully insulated from Hadoop's
> > CVE issues.
> >
> > (The full-fat hbase-shaded-client works best as direct build-time
> > dependency anyway)
> >
> > best regards
> > Istvan
> >
>
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Nick Dimiduk <nd...@apache.org>.
This would be great cleanup, big +1 from me for all three of these
adjustments, including the promotion of pe, ltt, and friends out of the
test scope.
I believe that we included hbase test jars because we used to freely mix
classes needed for minicluster between runtime and test jars, which in turn
relied on Hadoop minicluster capabilities. The big cleanup around
HBaseTestingUtil/it addressed much (or all) of these issues on branch-3.
I believe that we include a Hadoop distribution in our assembly because
that makes it easy for a new user to download our release bin.tgz and get
started immediately with learning. I guess it’s high time that we work out
the with- and without-Hadoop variants.
Thanks,
Nick
On Tue, 5 Mar 2024 at 09:14, Istvan Toth <st...@apache.org> wrote:
> DISCLAIMER: I don't have a patch ready, or even an elegant way mapped out
> to achieve this, this is about discussing whether we even want to make
> these changes.
> These are also substantial changes, but they could be targeted for HBase
> 3.0.
>
> One issue I have noticed is that we ship test jars and test dependencies in
> the assembly.
> I can't see anyone using those, but it bloats the assembly and classpath,
> and adds unnecessary JARs with possible CVE issues. (for example Kerby
> which is a Hadoop minicluster dependency)
>
> My proposal is to exclude the test jars and the test scope dependencies
> from the assembly.
>
> The advantages would be:
> * Smaller distro size
> * Faster startup (this is marginal)
> * Less CVE-prone JARs in the binary assemblies
>
> The other issue is that the assembly includes much of the Hadoop
> distribution.
> The basic assumption in all scripts and instructions is that the node has a
> fully configured Hadoop installation, and we include it in the classpath of
> HBase.
>
> If that is true, then there is no reason to include Hadoop in the assembly,
> HBase and its direct dependencies should be enough.
>
> One could argue that it would simplify the client side, which is true to
> some extent (though 95% of the client distro use cases are served better by
> simply using hbase-shaded-client).
>
> We could either remove the Hadoop libraries from either or both of the
> assemblies unconditionally, or provide two variants for either or both
> assemblies, one with Hadoop included, and one without it.
> Spark already does this, it has binary distributions both with and without
> Hadoop.
>
> The advantages would be:
> * Smaller distro size
> * Faster startup (this is marginal)
> * Less chance of conflicts with the Hadoop jars
> * Less CVE-prone JARs in the binary assemblies
>
>
> Thirdly, we could consider excluding the
> full-fat org.apache.hbase:hbase-shaded-client JAR from the Hadoop-less
> binary assemblies. It is not used by the assembly, and AFAIK it is not
> included in any of the 'hbase classpath' command variants.
>
> This would make sure that no Hadoop libraries are included (even in shaded
> form) and would make the HBase distribution fully insulated from Hadoop's
> CVE issues.
>
> (The full-fat hbase-shaded-client works best as direct build-time
> dependency anyway)
>
> best regards
> Istvan
>
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
I agree, we don't want to omit those from the binary distro.
We should identify what those tools are. (Should be easy based on the
presence of main() or the Tool interface)
Such tools could either be moved into a new module, like hbase-tools, or
simply moved to the runtime JARs.
Istvan
On Tue, Mar 5, 2024 at 10:34 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> There are some tools in the tests jar, such as PerformanceEvaluation.
>
> But anyway, maybe they should be moved to main...
>
> Istvan Toth <st...@apache.org> 于2024年3月5日周二 16:14写道：
> >
> > DISCLAIMER: I don't have a patch ready, or even an elegant way mapped out
> > to achieve this, this is about discussing whether we even want to make
> > these changes.
> > These are also substantial changes, but they could be targeted for HBase
> > 3.0.
> >
> > One issue I have noticed is that we ship test jars and test dependencies
> in
> > the assembly.
> > I can't see anyone using those, but it bloats the assembly and classpath,
> > and adds unnecessary JARs with possible CVE issues. (for example Kerby
> > which is a Hadoop minicluster dependency)
> >
> > My proposal is to exclude the test jars and the test scope dependencies
> > from the assembly.
> >
> > The advantages would be:
> > * Smaller distro size
> > * Faster startup (this is marginal)
> > * Less CVE-prone JARs in the binary assemblies
> >
> > The other issue is that the assembly includes much of the Hadoop
> > distribution.
> > The basic assumption in all scripts and instructions is that the node
> has a
> > fully configured Hadoop installation, and we include it in the classpath
> of
> > HBase.
> >
> > If that is true, then there is no reason to include Hadoop in the
> assembly,
> > HBase and its direct dependencies should be enough.
> >
> > One could argue that it would simplify the client side, which is true to
> > some extent (though 95% of the client distro use cases are served better
> by
> > simply using hbase-shaded-client).
> >
> > We could either remove the Hadoop libraries from either or both of the
> > assemblies unconditionally, or provide two variants for either or both
> > assemblies, one with Hadoop included, and one without it.
> > Spark already does this, it has binary distributions both with and
> without
> > Hadoop.
> >
> > The advantages would be:
> > * Smaller distro size
> > * Faster startup (this is marginal)
> > * Less chance of conflicts with the Hadoop jars
> > * Less CVE-prone JARs in the binary assemblies
> >
> >
> > Thirdly, we could consider excluding the
> > full-fat org.apache.hbase:hbase-shaded-client JAR from the Hadoop-less
> > binary assemblies. It is not used by the assembly, and AFAIK it is not
> > included in any of the 'hbase classpath' command variants.
> >
> > This would make sure that no Hadoop libraries are included (even in
> shaded
> > form) and would make the HBase distribution fully insulated from Hadoop's
> > CVE issues.
> >
> > (The full-fat hbase-shaded-client works best as direct build-time
> > dependency anyway)
> >
> > best regards
> > Istvan
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Removing tests and/or Hadoop from the binary assemblies
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
There are some tools in the tests jar, such as PerformanceEvaluation.
But anyway, maybe they should be moved to main...
Istvan Toth <st...@apache.org> 于2024年3月5日周二 16:14写道：
>
> DISCLAIMER: I don't have a patch ready, or even an elegant way mapped out
> to achieve this, this is about discussing whether we even want to make
> these changes.
> These are also substantial changes, but they could be targeted for HBase
> 3.0.
>
> One issue I have noticed is that we ship test jars and test dependencies in
> the assembly.
> I can't see anyone using those, but it bloats the assembly and classpath,
> and adds unnecessary JARs with possible CVE issues. (for example Kerby
> which is a Hadoop minicluster dependency)
>
> My proposal is to exclude the test jars and the test scope dependencies
> from the assembly.
>
> The advantages would be:
> * Smaller distro size
> * Faster startup (this is marginal)
> * Less CVE-prone JARs in the binary assemblies
>
> The other issue is that the assembly includes much of the Hadoop
> distribution.
> The basic assumption in all scripts and instructions is that the node has a
> fully configured Hadoop installation, and we include it in the classpath of
> HBase.
>
> If that is true, then there is no reason to include Hadoop in the assembly,
> HBase and its direct dependencies should be enough.
>
> One could argue that it would simplify the client side, which is true to
> some extent (though 95% of the client distro use cases are served better by
> simply using hbase-shaded-client).
>
> We could either remove the Hadoop libraries from either or both of the
> assemblies unconditionally, or provide two variants for either or both
> assemblies, one with Hadoop included, and one without it.
> Spark already does this, it has binary distributions both with and without
> Hadoop.
>
> The advantages would be:
> * Smaller distro size
> * Faster startup (this is marginal)
> * Less chance of conflicts with the Hadoop jars
> * Less CVE-prone JARs in the binary assemblies
>
>
> Thirdly, we could consider excluding the
> full-fat org.apache.hbase:hbase-shaded-client JAR from the Hadoop-less
> binary assemblies. It is not used by the assembly, and AFAIK it is not
> included in any of the 'hbase classpath' command variants.
>
> This would make sure that no Hadoop libraries are included (even in shaded
> form) and would make the HBase distribution fully insulated from Hadoop's
> CVE issues.
>
> (The full-fat hbase-shaded-client works best as direct build-time
> dependency anyway)
>
> best regards
> Istvan