You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by Istvan Toth <st...@apache.org> on 2024/09/04 07:58:43 UTC
[DISCUSS] Improving the official binary CVE situation
Hi!
In light of the recent discussions with the Trino team, we should think
about improving our CVE stance.
We are doing a good job of updating our direct dependencies, but we don't
have a good solution for our indirect dependencies, which are included in
our official shaded binary artifacts, and show up in CVE scans.
These are basically all coming from Hadoop, as HBase is also good at
keeping its direct dependencies in order.
The main problem is that in order to maximize compatibility, we build our
binaries with the oldest supported Hadoop major.minor version, which is
often long EOL.
I don't have experience with mixing Hadoop server and client versions, but
I recall that Hadoop major versions are supposed to be backwards wire
compatible.
The only binaries that include the Hadoop (and HBase) code are the
phoenix-client-embedded and phoenix-client-lite uberjars, which are
unfortunately the most widely used artifacts.
My first proposal is to use the latest supported Hadoop version instead of
the oldest one for each HBase profile. This should remove many of the
existing CVEs from the binaries.
Do you think this would cause backwards compatibility issues ?
If not, do you think that we should do this for 5.2.1 ?
My second discussion point is dependencyManage-ing transitive dependency
versions for the CVEs that present in whatever Hadoop version we build with.
This is a slippery slope, and I personally would like to minimize and
compartmentalize it as much as possible.
What is your opinion on this ?
Is this worth doing ?
Should we do it in the main code, or only for the uberjars (which would be
big a testing problem)
Should we put those dependencyManage entries in a profile, so that they can
be turned off when building with newer Hadoop versions ?
looking forward to hearing your thoughts on this
Istvan
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
We're changing Hbase to always build its releases with the latest Hadoop
version, which should improve things dramatically.
Of course, there is still going to be several months of lag until Hadoop
issues are fixed and released, then a new HBase release is built with the
fixed Hadoop version, but at least we won't depend on stuff with years old
CVEs.
This does not apply to 5.1.2, but should help with future releases.
Istvan
On Tue, Sep 10, 2024 at 4:40 PM Istvan Toth <st...@cloudera.com> wrote:
> Hi!
>
> I have checked the OWASP output with Hadoop 3.3.6 and managed avro,
> woodstox and jettison versions.
> (It's too big to attach, but BUILDING.md includes the command line to
> generate it)
>
> While about a third of them are false positives, there are a number of
> real ones.
>
> Opened https://issues.apache.org/jira/browse/PHOENIX-7405 for Jetty.
> We could bump jackson, though the CVE for it is disputed.
> htrace is also a problem, but I would not hold up 5.2.1 for removing it.
>
> For the others, we really need to think about where we draw the line.
>
> Let me know if there are other transitive dependencies we should meddle
> with.
>
> On Tue, Sep 10, 2024 at 1:54 PM Istvan Toth <st...@cloudera.com> wrote:
>
>> Thanks for the suggestion, but that is not realistic, HBase and Phoenix
>> are WAY too tightly coupled for that.
>> The Phoenix client JVM doesn't do that much, the majority of work and
>> interesting stuff happens in the HBase cluster.
>>
>> There is a subset of tests which CAN be run on separate clusters, and
>> there has been recent work done to improve on that.
>> Your suggestion of using Docker would make sense for those, for example
>> it could be used for testing interoperability of different Phoenix client
>> and server
>> versions, and upgrades (though we already have a - somewhat clumsy - test
>> framework for that)
>>
>> CLDR does have an internal test suite running on real clusters, 90%
>> effort and time is spent on configuring the clusters and re-starting them.
>> It is a good fit for some tests, mostly involving larger amounts of data,
>> and non-functional tests like using cloud connectors, but a bad fit for
>> functional / integration tests.
>> A similar docker-based system could probably improve on the configuration
>> complexity and restart times, but could still not be as fast as the
>> very stripped down minicluster.
>>
>> IUC SFDC is also working on something similar, but I don't know their
>> exact architecture for their internal tests.
>>
>> A significant amount of tests require very specifically configured HBase
>> instances with custom code and internal Hbase state management, which would
>> - take a huge amount of time and resources to start and run (as opposed
>> to the VERY optimized miniclusters)
>> - require bridging the different environments to manipulate and verify
>> the internal private state of the HBase cluster.
>> - require implementing a separate configuration management framework for
>> the stack. (something like Ambari)
>>
>> Phoenix already needs to be built for specific HBase versions, with each
>> branch only supporting a limited number of
>> HBase versions, for which we can run the tests (and we are already bad at
>> keeping the tests stable, even for our current coverage).
>>
>> On the other hand, Phoenix uses very few Hadoop features directly, it
>> mostly goes through HBase APIs.
>> All the heavy lifting for Hadoop compatibility is done in HBase, there is
>> very little to test from Phoenix itself in that regard.
>>
>> In short, this would take amount of resources - mostly in developers, but
>> also in computing resources for tests - that project does not have, for a
>> rather limited payoff.
>>
>> Istvan
>>
>> On Tue, Sep 10, 2024 at 11:17 AM Grzegorz Kokosiński <
>> g.kokosinski@gmail.com> wrote:
>>
>>> Hey,
>>>
>>> I am sorry I have joined the discussion late.
>>>
>>> If possible I would suggest to stop using mini clusters. I believe it is
>>> way more convenient to use docker with hadoop or hbase services. Notice
>>> that it is easy to test against multiple different versions, the
>>> environment is separated from the project classpath. In Trino that
>>> approach
>>> made testing easier.
>>>
>>> That way we could verify compatibility with old software even if we would
>>> be using newer libraries (hadoop and others).
>>>
>>
>>
>> --
>> *István Tóth* | Sr. Staff Software Engineer
>> *Email*: stoty@cloudera.com
>> cloudera.com <https://www.cloudera.com>
>> [image: Cloudera] <https://www.cloudera.com/>
>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> ------------------------------
>> ------------------------------
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Hi!
I have checked the OWASP output with Hadoop 3.3.6 and managed avro,
woodstox and jettison versions.
(It's too big to attach, but BUILDING.md includes the command line to
generate it)
While about a third of them are false positives, there are a number of real
ones.
Opened https://issues.apache.org/jira/browse/PHOENIX-7405 for Jetty.
We could bump jackson, though the CVE for it is disputed.
htrace is also a problem, but I would not hold up 5.2.1 for removing it.
For the others, we really need to think about where we draw the line.
Let me know if there are other transitive dependencies we should meddle
with.
On Tue, Sep 10, 2024 at 1:54 PM Istvan Toth <st...@cloudera.com> wrote:
> Thanks for the suggestion, but that is not realistic, HBase and Phoenix
> are WAY too tightly coupled for that.
> The Phoenix client JVM doesn't do that much, the majority of work and
> interesting stuff happens in the HBase cluster.
>
> There is a subset of tests which CAN be run on separate clusters, and
> there has been recent work done to improve on that.
> Your suggestion of using Docker would make sense for those, for example it
> could be used for testing interoperability of different Phoenix client and
> server
> versions, and upgrades (though we already have a - somewhat clumsy - test
> framework for that)
>
> CLDR does have an internal test suite running on real clusters, 90% effort
> and time is spent on configuring the clusters and re-starting them.
> It is a good fit for some tests, mostly involving larger amounts of data,
> and non-functional tests like using cloud connectors, but a bad fit for
> functional / integration tests.
> A similar docker-based system could probably improve on the configuration
> complexity and restart times, but could still not be as fast as the
> very stripped down minicluster.
>
> IUC SFDC is also working on something similar, but I don't know their
> exact architecture for their internal tests.
>
> A significant amount of tests require very specifically configured HBase
> instances with custom code and internal Hbase state management, which would
> - take a huge amount of time and resources to start and run (as opposed to
> the VERY optimized miniclusters)
> - require bridging the different environments to manipulate and verify the
> internal private state of the HBase cluster.
> - require implementing a separate configuration management framework for
> the stack. (something like Ambari)
>
> Phoenix already needs to be built for specific HBase versions, with each
> branch only supporting a limited number of
> HBase versions, for which we can run the tests (and we are already bad at
> keeping the tests stable, even for our current coverage).
>
> On the other hand, Phoenix uses very few Hadoop features directly, it
> mostly goes through HBase APIs.
> All the heavy lifting for Hadoop compatibility is done in HBase, there is
> very little to test from Phoenix itself in that regard.
>
> In short, this would take amount of resources - mostly in developers, but
> also in computing resources for tests - that project does not have, for a
> rather limited payoff.
>
> Istvan
>
> On Tue, Sep 10, 2024 at 11:17 AM Grzegorz Kokosiński <
> g.kokosinski@gmail.com> wrote:
>
>> Hey,
>>
>> I am sorry I have joined the discussion late.
>>
>> If possible I would suggest to stop using mini clusters. I believe it is
>> way more convenient to use docker with hadoop or hbase services. Notice
>> that it is easy to test against multiple different versions, the
>> environment is separated from the project classpath. In Trino that
>> approach
>> made testing easier.
>>
>> That way we could verify compatibility with old software even if we would
>> be using newer libraries (hadoop and others).
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Thanks for the suggestion, but that is not realistic, HBase and Phoenix are
WAY too tightly coupled for that.
The Phoenix client JVM doesn't do that much, the majority of work and
interesting stuff happens in the HBase cluster.
There is a subset of tests which CAN be run on separate clusters, and there
has been recent work done to improve on that.
Your suggestion of using Docker would make sense for those, for example it
could be used for testing interoperability of different Phoenix client and
server
versions, and upgrades (though we already have a - somewhat clumsy - test
framework for that)
CLDR does have an internal test suite running on real clusters, 90% effort
and time is spent on configuring the clusters and re-starting them.
It is a good fit for some tests, mostly involving larger amounts of data,
and non-functional tests like using cloud connectors, but a bad fit for
functional / integration tests.
A similar docker-based system could probably improve on the configuration
complexity and restart times, but could still not be as fast as the
very stripped down minicluster.
IUC SFDC is also working on something similar, but I don't know their exact
architecture for their internal tests.
A significant amount of tests require very specifically configured HBase
instances with custom code and internal Hbase state management, which would
- take a huge amount of time and resources to start and run (as opposed to
the VERY optimized miniclusters)
- require bridging the different environments to manipulate and verify the
internal private state of the HBase cluster.
- require implementing a separate configuration management framework for
the stack. (something like Ambari)
Phoenix already needs to be built for specific HBase versions, with each
branch only supporting a limited number of
HBase versions, for which we can run the tests (and we are already bad at
keeping the tests stable, even for our current coverage).
On the other hand, Phoenix uses very few Hadoop features directly, it
mostly goes through HBase APIs.
All the heavy lifting for Hadoop compatibility is done in HBase, there is
very little to test from Phoenix itself in that regard.
In short, this would take amount of resources - mostly in developers, but
also in computing resources for tests - that project does not have, for a
rather limited payoff.
Istvan
On Tue, Sep 10, 2024 at 11:17 AM Grzegorz Kokosiński <g....@gmail.com>
wrote:
> Hey,
>
> I am sorry I have joined the discussion late.
>
> If possible I would suggest to stop using mini clusters. I believe it is
> way more convenient to use docker with hadoop or hbase services. Notice
> that it is easy to test against multiple different versions, the
> environment is separated from the project classpath. In Trino that approach
> made testing easier.
>
> That way we could verify compatibility with old software even if we would
> be using newer libraries (hadoop and others).
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Grzegorz Kokosiński <g....@gmail.com>.
Hey,
I am sorry I have joined the discussion late.
If possible I would suggest to stop using mini clusters. I believe it is
way more convenient to use docker with hadoop or hbase services. Notice
that it is easy to test against multiple different versions, the
environment is separated from the project classpath. In Trino that approach
made testing easier.
That way we could verify compatibility with old software even if we would
be using newer libraries (hadoop and others).
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Viraj Jasani <vj...@apache.org>.
Yes that is correct, miniCluster can cause issues with mixed dependencies.
On Mon, Sep 9, 2024 at 4:50 AM Istvan Toth <st...@cloudera.com.invalid>
wrote:
> Now I remember that an important reason for sticking to the same Hadoop
> minor version as HBase was because the Hadoop minicluster built with older
> Hadoop versions has failed in some instances with newer Hadoop libraries.
> I also seem to recall I have not seen these failures when mixing Hadoop 3.2
> and 3.3, only for older releases.
>
>
> On Mon, Sep 9, 2024 at 10:40 AM Istvan Toth <st...@cloudera.com> wrote:
>
> > BTW does anyone have an idea why HBase is still on 3.3.5 instead of
> 3.3.6 ?
> > I found some mention that HBase does not want to use the latest Hadoop,
> > but I could not find the rationale for that.
> >
> > On Mon, Sep 9, 2024 at 10:30 AM Istvan Toth <st...@cloudera.com> wrote:
> >
> >> Sounds good.
> >>
> >> Let me put up a PR for the Hadoop upgrade, we can re-run OWASP, or
> >> whatever CVE checker after that.
> >>
> >> On Sat, Sep 7, 2024 at 1:28 AM Viraj Jasani <vj...@apache.org> wrote:
> >>
> >>> Sounds good, I agree with this solution. We can keep them in profile
> and
> >>> disable profile once Hadoop resolves CVEs.
> >>> I feel we should resolve Avro and Jettison CVEs and rollout 5.2.1
> unless
> >>> newer released version of Hadoop already has them resolved. WDYT?
> >>>
> >>>
> >>> On Thu, Sep 5, 2024 at 9:46 PM Istvan Toth <stoty@cloudera.com.invalid
> >
> >>> wrote:
> >>>
> >>> > Thanks for your response, Viraj.
> >>> >
> >>> > Sure, managing transitive dependencies and using a newer Hadoop are
> not
> >>> > mutually exclusive.
> >>> >
> >>> > For old HBase versions like 2.1 even the newest officially supported
> >>> Hadoop
> >>> > is going to be old.
> >>> >
> >>> > What do you think about my proposals on how to handle the transitive
> >>> > dependency management ?
> >>> > I quite like the idea of putting those into a profile (which is
> >>> enabled by
> >>> > default) so that they can be disabled if a newer hadoop is used for
> >>> > building.
> >>> >
> >>> > Istvan
> >>> >
> >>> > On Thu, Sep 5, 2024 at 5:37 PM Viraj Jasani <vj...@apache.org>
> >>> wrote:
> >>> >
> >>> > > I agree with upgrading Hadoop version but it is helpful only if the
> >>> CVEs
> >>> > > from indirect dependencies have been resolved by Hadoop.
> >>> > > The release frequency is too slow for Hadoop (similar to us) so
> some
> >>> CVE
> >>> > > resolution might be WIP and take considerable time to get released.
> >>> In
> >>> > such
> >>> > > cases, maybe we can also try to manage dependency within Phoenix
> pom.
> >>> > WDYT?
> >>> > >
> >>> > >
> >>> > > On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org>
> wrote:
> >>> > >
> >>> > > > Hi!
> >>> > > >
> >>> > > > In light of the recent discussions with the Trino team, we should
> >>> think
> >>> > > > about improving our CVE stance.
> >>> > > >
> >>> > > > We are doing a good job of updating our direct dependencies, but
> we
> >>> > don't
> >>> > > > have a good solution for our indirect dependencies, which are
> >>> included
> >>> > in
> >>> > > > our official shaded binary artifacts, and show up in CVE scans.
> >>> > > >
> >>> > > > These are basically all coming from Hadoop, as HBase is also good
> >>> at
> >>> > > > keeping its direct dependencies in order.
> >>> > > >
> >>> > > > The main problem is that in order to maximize compatibility, we
> >>> build
> >>> > our
> >>> > > > binaries with the oldest supported Hadoop major.minor version,
> >>> which is
> >>> > > > often long EOL.
> >>> > > >
> >>> > > > I don't have experience with mixing Hadoop server and client
> >>> versions,
> >>> > > but
> >>> > > > I recall that Hadoop major versions are supposed to be backwards
> >>> wire
> >>> > > > compatible.
> >>> > > >
> >>> > > > The only binaries that include the Hadoop (and HBase) code are
> the
> >>> > > > phoenix-client-embedded and phoenix-client-lite uberjars, which
> are
> >>> > > > unfortunately the most widely used artifacts.
> >>> > > >
> >>> > > > My first proposal is to use the latest supported Hadoop version
> >>> instead
> >>> > > of
> >>> > > > the oldest one for each HBase profile. This should remove many of
> >>> the
> >>> > > > existing CVEs from the binaries.
> >>> > > >
> >>> > > > Do you think this would cause backwards compatibility issues ?
> >>> > > >
> >>> > > > If not, do you think that we should do this for 5.2.1 ?
> >>> > > >
> >>> > > > My second discussion point is dependencyManage-ing transitive
> >>> > dependency
> >>> > > > versions for the CVEs that present in whatever Hadoop version we
> >>> build
> >>> > > > with.
> >>> > > > This is a slippery slope, and I personally would like to minimize
> >>> and
> >>> > > > compartmentalize it as much as possible.
> >>> > > >
> >>> > > > What is your opinion on this ?
> >>> > > > Is this worth doing ?
> >>> > > > Should we do it in the main code, or only for the uberjars (which
> >>> would
> >>> > > be
> >>> > > > big a testing problem)
> >>> > > > Should we put those dependencyManage entries in a profile, so
> that
> >>> they
> >>> > > can
> >>> > > > be turned off when building with newer Hadoop versions ?
> >>> > > >
> >>> > > > looking forward to hearing your thoughts on this
> >>> > > >
> >>> > > > Istvan
> >>> > > >
> >>> > >
> >>> >
> >>> >
> >>> > --
> >>> > *István Tóth* | Sr. Staff Software Engineer
> >>> > *Email*: stoty@cloudera.com
> >>> > cloudera.com <https://www.cloudera.com>
> >>> > [image: Cloudera] <https://www.cloudera.com/>
> >>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> >>> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> >>> Cloudera
> >>> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >>> > ------------------------------
> >>> > ------------------------------
> >>> >
> >>>
> >>
> >>
> >> --
> >> *István Tóth* | Sr. Staff Software Engineer
> >> *Email*: stoty@cloudera.com
> >> cloudera.com <https://www.cloudera.com>
> >> [image: Cloudera] <https://www.cloudera.com/>
> >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >> ------------------------------
> >> ------------------------------
> >>
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Now I remember that an important reason for sticking to the same Hadoop
minor version as HBase was because the Hadoop minicluster built with older
Hadoop versions has failed in some instances with newer Hadoop libraries.
I also seem to recall I have not seen these failures when mixing Hadoop 3.2
and 3.3, only for older releases.
On Mon, Sep 9, 2024 at 10:40 AM Istvan Toth <st...@cloudera.com> wrote:
> BTW does anyone have an idea why HBase is still on 3.3.5 instead of 3.3.6 ?
> I found some mention that HBase does not want to use the latest Hadoop,
> but I could not find the rationale for that.
>
> On Mon, Sep 9, 2024 at 10:30 AM Istvan Toth <st...@cloudera.com> wrote:
>
>> Sounds good.
>>
>> Let me put up a PR for the Hadoop upgrade, we can re-run OWASP, or
>> whatever CVE checker after that.
>>
>> On Sat, Sep 7, 2024 at 1:28 AM Viraj Jasani <vj...@apache.org> wrote:
>>
>>> Sounds good, I agree with this solution. We can keep them in profile and
>>> disable profile once Hadoop resolves CVEs.
>>> I feel we should resolve Avro and Jettison CVEs and rollout 5.2.1 unless
>>> newer released version of Hadoop already has them resolved. WDYT?
>>>
>>>
>>> On Thu, Sep 5, 2024 at 9:46 PM Istvan Toth <st...@cloudera.com.invalid>
>>> wrote:
>>>
>>> > Thanks for your response, Viraj.
>>> >
>>> > Sure, managing transitive dependencies and using a newer Hadoop are not
>>> > mutually exclusive.
>>> >
>>> > For old HBase versions like 2.1 even the newest officially supported
>>> Hadoop
>>> > is going to be old.
>>> >
>>> > What do you think about my proposals on how to handle the transitive
>>> > dependency management ?
>>> > I quite like the idea of putting those into a profile (which is
>>> enabled by
>>> > default) so that they can be disabled if a newer hadoop is used for
>>> > building.
>>> >
>>> > Istvan
>>> >
>>> > On Thu, Sep 5, 2024 at 5:37 PM Viraj Jasani <vj...@apache.org>
>>> wrote:
>>> >
>>> > > I agree with upgrading Hadoop version but it is helpful only if the
>>> CVEs
>>> > > from indirect dependencies have been resolved by Hadoop.
>>> > > The release frequency is too slow for Hadoop (similar to us) so some
>>> CVE
>>> > > resolution might be WIP and take considerable time to get released.
>>> In
>>> > such
>>> > > cases, maybe we can also try to manage dependency within Phoenix pom.
>>> > WDYT?
>>> > >
>>> > >
>>> > > On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org> wrote:
>>> > >
>>> > > > Hi!
>>> > > >
>>> > > > In light of the recent discussions with the Trino team, we should
>>> think
>>> > > > about improving our CVE stance.
>>> > > >
>>> > > > We are doing a good job of updating our direct dependencies, but we
>>> > don't
>>> > > > have a good solution for our indirect dependencies, which are
>>> included
>>> > in
>>> > > > our official shaded binary artifacts, and show up in CVE scans.
>>> > > >
>>> > > > These are basically all coming from Hadoop, as HBase is also good
>>> at
>>> > > > keeping its direct dependencies in order.
>>> > > >
>>> > > > The main problem is that in order to maximize compatibility, we
>>> build
>>> > our
>>> > > > binaries with the oldest supported Hadoop major.minor version,
>>> which is
>>> > > > often long EOL.
>>> > > >
>>> > > > I don't have experience with mixing Hadoop server and client
>>> versions,
>>> > > but
>>> > > > I recall that Hadoop major versions are supposed to be backwards
>>> wire
>>> > > > compatible.
>>> > > >
>>> > > > The only binaries that include the Hadoop (and HBase) code are the
>>> > > > phoenix-client-embedded and phoenix-client-lite uberjars, which are
>>> > > > unfortunately the most widely used artifacts.
>>> > > >
>>> > > > My first proposal is to use the latest supported Hadoop version
>>> instead
>>> > > of
>>> > > > the oldest one for each HBase profile. This should remove many of
>>> the
>>> > > > existing CVEs from the binaries.
>>> > > >
>>> > > > Do you think this would cause backwards compatibility issues ?
>>> > > >
>>> > > > If not, do you think that we should do this for 5.2.1 ?
>>> > > >
>>> > > > My second discussion point is dependencyManage-ing transitive
>>> > dependency
>>> > > > versions for the CVEs that present in whatever Hadoop version we
>>> build
>>> > > > with.
>>> > > > This is a slippery slope, and I personally would like to minimize
>>> and
>>> > > > compartmentalize it as much as possible.
>>> > > >
>>> > > > What is your opinion on this ?
>>> > > > Is this worth doing ?
>>> > > > Should we do it in the main code, or only for the uberjars (which
>>> would
>>> > > be
>>> > > > big a testing problem)
>>> > > > Should we put those dependencyManage entries in a profile, so that
>>> they
>>> > > can
>>> > > > be turned off when building with newer Hadoop versions ?
>>> > > >
>>> > > > looking forward to hearing your thoughts on this
>>> > > >
>>> > > > Istvan
>>> > > >
>>> > >
>>> >
>>> >
>>> > --
>>> > *István Tóth* | Sr. Staff Software Engineer
>>> > *Email*: stoty@cloudera.com
>>> > cloudera.com <https://www.cloudera.com>
>>> > [image: Cloudera] <https://www.cloudera.com/>
>>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>>> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>>> Cloudera
>>> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>> > ------------------------------
>>> > ------------------------------
>>> >
>>>
>>
>>
>> --
>> *István Tóth* | Sr. Staff Software Engineer
>> *Email*: stoty@cloudera.com
>> cloudera.com <https://www.cloudera.com>
>> [image: Cloudera] <https://www.cloudera.com/>
>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> ------------------------------
>> ------------------------------
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
BTW does anyone have an idea why HBase is still on 3.3.5 instead of 3.3.6 ?
I found some mention that HBase does not want to use the latest Hadoop, but
I could not find the rationale for that.
On Mon, Sep 9, 2024 at 10:30 AM Istvan Toth <st...@cloudera.com> wrote:
> Sounds good.
>
> Let me put up a PR for the Hadoop upgrade, we can re-run OWASP, or
> whatever CVE checker after that.
>
> On Sat, Sep 7, 2024 at 1:28 AM Viraj Jasani <vj...@apache.org> wrote:
>
>> Sounds good, I agree with this solution. We can keep them in profile and
>> disable profile once Hadoop resolves CVEs.
>> I feel we should resolve Avro and Jettison CVEs and rollout 5.2.1 unless
>> newer released version of Hadoop already has them resolved. WDYT?
>>
>>
>> On Thu, Sep 5, 2024 at 9:46 PM Istvan Toth <st...@cloudera.com.invalid>
>> wrote:
>>
>> > Thanks for your response, Viraj.
>> >
>> > Sure, managing transitive dependencies and using a newer Hadoop are not
>> > mutually exclusive.
>> >
>> > For old HBase versions like 2.1 even the newest officially supported
>> Hadoop
>> > is going to be old.
>> >
>> > What do you think about my proposals on how to handle the transitive
>> > dependency management ?
>> > I quite like the idea of putting those into a profile (which is enabled
>> by
>> > default) so that they can be disabled if a newer hadoop is used for
>> > building.
>> >
>> > Istvan
>> >
>> > On Thu, Sep 5, 2024 at 5:37 PM Viraj Jasani <vj...@apache.org> wrote:
>> >
>> > > I agree with upgrading Hadoop version but it is helpful only if the
>> CVEs
>> > > from indirect dependencies have been resolved by Hadoop.
>> > > The release frequency is too slow for Hadoop (similar to us) so some
>> CVE
>> > > resolution might be WIP and take considerable time to get released. In
>> > such
>> > > cases, maybe we can also try to manage dependency within Phoenix pom.
>> > WDYT?
>> > >
>> > >
>> > > On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org> wrote:
>> > >
>> > > > Hi!
>> > > >
>> > > > In light of the recent discussions with the Trino team, we should
>> think
>> > > > about improving our CVE stance.
>> > > >
>> > > > We are doing a good job of updating our direct dependencies, but we
>> > don't
>> > > > have a good solution for our indirect dependencies, which are
>> included
>> > in
>> > > > our official shaded binary artifacts, and show up in CVE scans.
>> > > >
>> > > > These are basically all coming from Hadoop, as HBase is also good at
>> > > > keeping its direct dependencies in order.
>> > > >
>> > > > The main problem is that in order to maximize compatibility, we
>> build
>> > our
>> > > > binaries with the oldest supported Hadoop major.minor version,
>> which is
>> > > > often long EOL.
>> > > >
>> > > > I don't have experience with mixing Hadoop server and client
>> versions,
>> > > but
>> > > > I recall that Hadoop major versions are supposed to be backwards
>> wire
>> > > > compatible.
>> > > >
>> > > > The only binaries that include the Hadoop (and HBase) code are the
>> > > > phoenix-client-embedded and phoenix-client-lite uberjars, which are
>> > > > unfortunately the most widely used artifacts.
>> > > >
>> > > > My first proposal is to use the latest supported Hadoop version
>> instead
>> > > of
>> > > > the oldest one for each HBase profile. This should remove many of
>> the
>> > > > existing CVEs from the binaries.
>> > > >
>> > > > Do you think this would cause backwards compatibility issues ?
>> > > >
>> > > > If not, do you think that we should do this for 5.2.1 ?
>> > > >
>> > > > My second discussion point is dependencyManage-ing transitive
>> > dependency
>> > > > versions for the CVEs that present in whatever Hadoop version we
>> build
>> > > > with.
>> > > > This is a slippery slope, and I personally would like to minimize
>> and
>> > > > compartmentalize it as much as possible.
>> > > >
>> > > > What is your opinion on this ?
>> > > > Is this worth doing ?
>> > > > Should we do it in the main code, or only for the uberjars (which
>> would
>> > > be
>> > > > big a testing problem)
>> > > > Should we put those dependencyManage entries in a profile, so that
>> they
>> > > can
>> > > > be turned off when building with newer Hadoop versions ?
>> > > >
>> > > > looking forward to hearing your thoughts on this
>> > > >
>> > > > Istvan
>> > > >
>> > >
>> >
>> >
>> > --
>> > *István Tóth* | Sr. Staff Software Engineer
>> > *Email*: stoty@cloudera.com
>> > cloudera.com <https://www.cloudera.com>
>> > [image: Cloudera] <https://www.cloudera.com/>
>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> Cloudera
>> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> > ------------------------------
>> > ------------------------------
>> >
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Sounds good.
Let me put up a PR for the Hadoop upgrade, we can re-run OWASP, or whatever
CVE checker after that.
On Sat, Sep 7, 2024 at 1:28 AM Viraj Jasani <vj...@apache.org> wrote:
> Sounds good, I agree with this solution. We can keep them in profile and
> disable profile once Hadoop resolves CVEs.
> I feel we should resolve Avro and Jettison CVEs and rollout 5.2.1 unless
> newer released version of Hadoop already has them resolved. WDYT?
>
>
> On Thu, Sep 5, 2024 at 9:46 PM Istvan Toth <st...@cloudera.com.invalid>
> wrote:
>
> > Thanks for your response, Viraj.
> >
> > Sure, managing transitive dependencies and using a newer Hadoop are not
> > mutually exclusive.
> >
> > For old HBase versions like 2.1 even the newest officially supported
> Hadoop
> > is going to be old.
> >
> > What do you think about my proposals on how to handle the transitive
> > dependency management ?
> > I quite like the idea of putting those into a profile (which is enabled
> by
> > default) so that they can be disabled if a newer hadoop is used for
> > building.
> >
> > Istvan
> >
> > On Thu, Sep 5, 2024 at 5:37 PM Viraj Jasani <vj...@apache.org> wrote:
> >
> > > I agree with upgrading Hadoop version but it is helpful only if the
> CVEs
> > > from indirect dependencies have been resolved by Hadoop.
> > > The release frequency is too slow for Hadoop (similar to us) so some
> CVE
> > > resolution might be WIP and take considerable time to get released. In
> > such
> > > cases, maybe we can also try to manage dependency within Phoenix pom.
> > WDYT?
> > >
> > >
> > > On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org> wrote:
> > >
> > > > Hi!
> > > >
> > > > In light of the recent discussions with the Trino team, we should
> think
> > > > about improving our CVE stance.
> > > >
> > > > We are doing a good job of updating our direct dependencies, but we
> > don't
> > > > have a good solution for our indirect dependencies, which are
> included
> > in
> > > > our official shaded binary artifacts, and show up in CVE scans.
> > > >
> > > > These are basically all coming from Hadoop, as HBase is also good at
> > > > keeping its direct dependencies in order.
> > > >
> > > > The main problem is that in order to maximize compatibility, we build
> > our
> > > > binaries with the oldest supported Hadoop major.minor version, which
> is
> > > > often long EOL.
> > > >
> > > > I don't have experience with mixing Hadoop server and client
> versions,
> > > but
> > > > I recall that Hadoop major versions are supposed to be backwards wire
> > > > compatible.
> > > >
> > > > The only binaries that include the Hadoop (and HBase) code are the
> > > > phoenix-client-embedded and phoenix-client-lite uberjars, which are
> > > > unfortunately the most widely used artifacts.
> > > >
> > > > My first proposal is to use the latest supported Hadoop version
> instead
> > > of
> > > > the oldest one for each HBase profile. This should remove many of the
> > > > existing CVEs from the binaries.
> > > >
> > > > Do you think this would cause backwards compatibility issues ?
> > > >
> > > > If not, do you think that we should do this for 5.2.1 ?
> > > >
> > > > My second discussion point is dependencyManage-ing transitive
> > dependency
> > > > versions for the CVEs that present in whatever Hadoop version we
> build
> > > > with.
> > > > This is a slippery slope, and I personally would like to minimize and
> > > > compartmentalize it as much as possible.
> > > >
> > > > What is your opinion on this ?
> > > > Is this worth doing ?
> > > > Should we do it in the main code, or only for the uberjars (which
> would
> > > be
> > > > big a testing problem)
> > > > Should we put those dependencyManage entries in a profile, so that
> they
> > > can
> > > > be turned off when building with newer Hadoop versions ?
> > > >
> > > > looking forward to hearing your thoughts on this
> > > >
> > > > Istvan
> > > >
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
> >
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Viraj Jasani <vj...@apache.org>.
Sounds good, I agree with this solution. We can keep them in profile and
disable profile once Hadoop resolves CVEs.
I feel we should resolve Avro and Jettison CVEs and rollout 5.2.1 unless
newer released version of Hadoop already has them resolved. WDYT?
On Thu, Sep 5, 2024 at 9:46 PM Istvan Toth <st...@cloudera.com.invalid>
wrote:
> Thanks for your response, Viraj.
>
> Sure, managing transitive dependencies and using a newer Hadoop are not
> mutually exclusive.
>
> For old HBase versions like 2.1 even the newest officially supported Hadoop
> is going to be old.
>
> What do you think about my proposals on how to handle the transitive
> dependency management ?
> I quite like the idea of putting those into a profile (which is enabled by
> default) so that they can be disabled if a newer hadoop is used for
> building.
>
> Istvan
>
> On Thu, Sep 5, 2024 at 5:37 PM Viraj Jasani <vj...@apache.org> wrote:
>
> > I agree with upgrading Hadoop version but it is helpful only if the CVEs
> > from indirect dependencies have been resolved by Hadoop.
> > The release frequency is too slow for Hadoop (similar to us) so some CVE
> > resolution might be WIP and take considerable time to get released. In
> such
> > cases, maybe we can also try to manage dependency within Phoenix pom.
> WDYT?
> >
> >
> > On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org> wrote:
> >
> > > Hi!
> > >
> > > In light of the recent discussions with the Trino team, we should think
> > > about improving our CVE stance.
> > >
> > > We are doing a good job of updating our direct dependencies, but we
> don't
> > > have a good solution for our indirect dependencies, which are included
> in
> > > our official shaded binary artifacts, and show up in CVE scans.
> > >
> > > These are basically all coming from Hadoop, as HBase is also good at
> > > keeping its direct dependencies in order.
> > >
> > > The main problem is that in order to maximize compatibility, we build
> our
> > > binaries with the oldest supported Hadoop major.minor version, which is
> > > often long EOL.
> > >
> > > I don't have experience with mixing Hadoop server and client versions,
> > but
> > > I recall that Hadoop major versions are supposed to be backwards wire
> > > compatible.
> > >
> > > The only binaries that include the Hadoop (and HBase) code are the
> > > phoenix-client-embedded and phoenix-client-lite uberjars, which are
> > > unfortunately the most widely used artifacts.
> > >
> > > My first proposal is to use the latest supported Hadoop version instead
> > of
> > > the oldest one for each HBase profile. This should remove many of the
> > > existing CVEs from the binaries.
> > >
> > > Do you think this would cause backwards compatibility issues ?
> > >
> > > If not, do you think that we should do this for 5.2.1 ?
> > >
> > > My second discussion point is dependencyManage-ing transitive
> dependency
> > > versions for the CVEs that present in whatever Hadoop version we build
> > > with.
> > > This is a slippery slope, and I personally would like to minimize and
> > > compartmentalize it as much as possible.
> > >
> > > What is your opinion on this ?
> > > Is this worth doing ?
> > > Should we do it in the main code, or only for the uberjars (which would
> > be
> > > big a testing problem)
> > > Should we put those dependencyManage entries in a profile, so that they
> > can
> > > be turned off when building with newer Hadoop versions ?
> > >
> > > looking forward to hearing your thoughts on this
> > >
> > > Istvan
> > >
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Thanks for your response, Viraj.
Sure, managing transitive dependencies and using a newer Hadoop are not
mutually exclusive.
For old HBase versions like 2.1 even the newest officially supported Hadoop
is going to be old.
What do you think about my proposals on how to handle the transitive
dependency management ?
I quite like the idea of putting those into a profile (which is enabled by
default) so that they can be disabled if a newer hadoop is used for
building.
Istvan
On Thu, Sep 5, 2024 at 5:37 PM Viraj Jasani <vj...@apache.org> wrote:
> I agree with upgrading Hadoop version but it is helpful only if the CVEs
> from indirect dependencies have been resolved by Hadoop.
> The release frequency is too slow for Hadoop (similar to us) so some CVE
> resolution might be WIP and take considerable time to get released. In such
> cases, maybe we can also try to manage dependency within Phoenix pom. WDYT?
>
>
> On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org> wrote:
>
> > Hi!
> >
> > In light of the recent discussions with the Trino team, we should think
> > about improving our CVE stance.
> >
> > We are doing a good job of updating our direct dependencies, but we don't
> > have a good solution for our indirect dependencies, which are included in
> > our official shaded binary artifacts, and show up in CVE scans.
> >
> > These are basically all coming from Hadoop, as HBase is also good at
> > keeping its direct dependencies in order.
> >
> > The main problem is that in order to maximize compatibility, we build our
> > binaries with the oldest supported Hadoop major.minor version, which is
> > often long EOL.
> >
> > I don't have experience with mixing Hadoop server and client versions,
> but
> > I recall that Hadoop major versions are supposed to be backwards wire
> > compatible.
> >
> > The only binaries that include the Hadoop (and HBase) code are the
> > phoenix-client-embedded and phoenix-client-lite uberjars, which are
> > unfortunately the most widely used artifacts.
> >
> > My first proposal is to use the latest supported Hadoop version instead
> of
> > the oldest one for each HBase profile. This should remove many of the
> > existing CVEs from the binaries.
> >
> > Do you think this would cause backwards compatibility issues ?
> >
> > If not, do you think that we should do this for 5.2.1 ?
> >
> > My second discussion point is dependencyManage-ing transitive dependency
> > versions for the CVEs that present in whatever Hadoop version we build
> > with.
> > This is a slippery slope, and I personally would like to minimize and
> > compartmentalize it as much as possible.
> >
> > What is your opinion on this ?
> > Is this worth doing ?
> > Should we do it in the main code, or only for the uberjars (which would
> be
> > big a testing problem)
> > Should we put those dependencyManage entries in a profile, so that they
> can
> > be turned off when building with newer Hadoop versions ?
> >
> > looking forward to hearing your thoughts on this
> >
> > Istvan
> >
>
--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------
Re: [DISCUSS] Improving the official binary CVE situation
Posted by Viraj Jasani <vj...@apache.org>.
I agree with upgrading Hadoop version but it is helpful only if the CVEs
from indirect dependencies have been resolved by Hadoop.
The release frequency is too slow for Hadoop (similar to us) so some CVE
resolution might be WIP and take considerable time to get released. In such
cases, maybe we can also try to manage dependency within Phoenix pom. WDYT?
On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org> wrote:
> Hi!
>
> In light of the recent discussions with the Trino team, we should think
> about improving our CVE stance.
>
> We are doing a good job of updating our direct dependencies, but we don't
> have a good solution for our indirect dependencies, which are included in
> our official shaded binary artifacts, and show up in CVE scans.
>
> These are basically all coming from Hadoop, as HBase is also good at
> keeping its direct dependencies in order.
>
> The main problem is that in order to maximize compatibility, we build our
> binaries with the oldest supported Hadoop major.minor version, which is
> often long EOL.
>
> I don't have experience with mixing Hadoop server and client versions, but
> I recall that Hadoop major versions are supposed to be backwards wire
> compatible.
>
> The only binaries that include the Hadoop (and HBase) code are the
> phoenix-client-embedded and phoenix-client-lite uberjars, which are
> unfortunately the most widely used artifacts.
>
> My first proposal is to use the latest supported Hadoop version instead of
> the oldest one for each HBase profile. This should remove many of the
> existing CVEs from the binaries.
>
> Do you think this would cause backwards compatibility issues ?
>
> If not, do you think that we should do this for 5.2.1 ?
>
> My second discussion point is dependencyManage-ing transitive dependency
> versions for the CVEs that present in whatever Hadoop version we build
> with.
> This is a slippery slope, and I personally would like to minimize and
> compartmentalize it as much as possible.
>
> What is your opinion on this ?
> Is this worth doing ?
> Should we do it in the main code, or only for the uberjars (which would be
> big a testing problem)
> Should we put those dependencyManage entries in a profile, so that they can
> be turned off when building with newer Hadoop versions ?
>
> looking forward to hearing your thoughts on this
>
> Istvan
>