You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Martijn Visser <ma...@apache.org> on 2022/10/05 03:01:56 UTC

[DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Hi everyone,

Little over a year ago a discussion thread was opened on changing the
minimal supported version of Hadoop and bringing that to 2.8.5. [1] In this
discussion thread, I would like to propose to bring that minimal supported
version of Hadoop to 2.10.2.

Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified as
Critical. [2] [3]. While Flink is not directly impacted by those, we do see
vulnerability scanners flag Flink as being vulnerable. We could easily
mitigate that by bumping the minimal supported version of Hadoop to 2.10.2.

I'm looking forward to your opinions on this topic.

Best regards,

Martijn
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser

[1] https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
[2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
[3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by Martijn Visser <ma...@apache.org>.
For those that are interested, I have a draft PR open for this
https://github.com/apache/flink/pull/21128 - Feel free to have a look.

I'm not sure yet why the Flink CI fails, since these tests are passing
locally.

Thanks,

Martijn

On Tue, Oct 25, 2022 at 10:21 AM Matthias Pohl
<ma...@aiven.io.invalid> wrote:

> > Additionally, having code that hasn't been touch for a while increases
> the risk of it
> Sorry about this incomplete confusing sentence. I was about to remove it
> when accidentally pushing the shortcut for sending the message out to the
> mailing list.
>
> On Tue, Oct 25, 2022 at 10:18 AM Matthias Pohl <ma...@aiven.io>
> wrote:
>
> > I guess upgrading the minimal version should also mean cleaning up the
> > codebase, i.e. removing code segments that have been around to allow
> > support for older versions. The overall goal should be to improve the
> Flink
> > codebase in my opinion. Considering what David said in the old thread
> about
> > Hadoop users usually lacking behind with version upgrades [1], would we
> do
> > this version bump in two phases, i.e. adding some deprecation notes and
> > doing the actual cleanup later on? I think Gabor has a point with it not
> > being really mentioned anywhere in the docs (the only location I could
> find
> > in the docs about Hadoop version is [2]). In this sense, the support for
> > older Hadoop versions was kind of implicit: We're talking about compiling
> > Flink with Hadoop 2.8.5 but also mention older Hadoop versions which
> leaves
> > room for interpretation.
> >
> > Additionally, having code that hasn't been touch for a while increases
> the
> > risk of it
> >
> > Matthias
> >
> > [1] https://lists.apache.org/thread/w7www13tossxrxo1mttgb68v81rf6fks
> > [2]
> >
> https://nightlies.a1pache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions
> > <
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions
> >
> >
> > On Fri, Oct 21, 2022 at 4:13 AM Xintong Song <to...@gmail.com>
> > wrote:
> >
> >> I believe there are some reflection based approaches in the `flink-yarn`
> >> module, for supporting outdated APIs in early Hadoop versions.
> >>
> >> I haven't done a thorough check, and these are what I get.
> >> - AMRMClientAsyncReflector
> >> - ApplicationSubmissionContextReflector
> >> - ContainerRequestReflector
> >> - RegisterApplicationMasterResponseReflector
> >> - ResourceInformationReflector
> >>
> >> Are we removing these as well? If yes, then Flink can no longer work
> with
> >> the old hadoop versions. (That's how I understand "bumping the minimal
> >> supported hadoop version".) I personally am not super eager to get rid
> of
> >> theses, because the relevant parts of codes are no longer frequently
> >> changing, thus the maintenance overhead is low.
> >>
> >> Best,
> >>
> >> Xintong
> >>
> >>
> >>
> >> On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <da...@gmail.com>
> wrote:
> >>
> >> > Given that we do not bundle any hadoop classes in the Flink binary, do
> >> you
> >> > mean simply bump the hadoop version in the parent pom?
> >> > If it is, why do not we use the latest stable hadoop version 3.3.4? It
> >> > seems that our cron build has verified that hadoop3 could work.
> >> >
> >> > Best,
> >> > Yang
> >> >
> >> > David Morávek <da...@gmail.com> 于2022年10月19日周三 16:29写道：
> >> >
> >> > > +1; anything below 2.10.x seems to be EOL
> >> > >
> >> > > Best,
> >> > > D.
> >> > >
> >> > > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi <
> >> > balassi.marton@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Martjin,
> >> > > >
> >> > > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to
> >> > > > implement the bump?
> >> > > >
> >> > > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <
> >> > gabor.g.somogyi@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Martin,
> >> > > > >
> >> > > > > Thanks for bringing this up! Lately I was thinking about to bump
> >> the
> >> > > > hadoop
> >> > > > > version to at least 2.6.1 to clean up issues like this:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
> >> > > > >
> >> > > > > All in all +1 from my perspective.
> >> > > > >
> >> > > > > Just a question here. Are we stating the minimum Hadoop version
> >> for
> >> > > users
> >> > > > > somewhere in the doc or they need to find it out from source
> code
> >> > like
> >> > > > > this?
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
> >> > > > >
> >> > > > > BR,
> >> > > > > G
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <
> >> > > martijnvisser@apache.org>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi everyone,
> >> > > > > >
> >> > > > > > Little over a year ago a discussion thread was opened on
> >> changing
> >> > the
> >> > > > > > minimal supported version of Hadoop and bringing that to
> 2.8.5.
> >> [1]
> >> > > In
> >> > > > > this
> >> > > > > > discussion thread, I would like to propose to bring that
> minimal
> >> > > > > supported
> >> > > > > > version of Hadoop to 2.10.2.
> >> > > > > >
> >> > > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are
> >> classified
> >> > as
> >> > > > > > Critical. [2] [3]. While Flink is not directly impacted by
> >> those,
> >> > we
> >> > > do
> >> > > > > see
> >> > > > > > vulnerability scanners flag Flink as being vulnerable. We
> could
> >> > > easily
> >> > > > > > mitigate that by bumping the minimal supported version of
> >> Hadoop to
> >> > > > > 2.10.2.
> >> > > > > >
> >> > > > > > I'm looking forward to your opinions on this topic.
> >> > > > > >
> >> > > > > > Best regards,
> >> > > > > >
> >> > > > > > Martijn
> >> > > > > > https://twitter.com/MartijnVisser82
> >> > > > > > https://github.com/MartijnVisser
> >> > > > > >
> >> > > > > > [1]
> >> > https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> >> > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> >> > > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by Matthias Pohl <ma...@aiven.io.INVALID>.
> Additionally, having code that hasn't been touch for a while increases
the risk of it
Sorry about this incomplete confusing sentence. I was about to remove it
when accidentally pushing the shortcut for sending the message out to the
mailing list.

On Tue, Oct 25, 2022 at 10:18 AM Matthias Pohl <ma...@aiven.io>
wrote:

> I guess upgrading the minimal version should also mean cleaning up the
> codebase, i.e. removing code segments that have been around to allow
> support for older versions. The overall goal should be to improve the Flink
> codebase in my opinion. Considering what David said in the old thread about
> Hadoop users usually lacking behind with version upgrades [1], would we do
> this version bump in two phases, i.e. adding some deprecation notes and
> doing the actual cleanup later on? I think Gabor has a point with it not
> being really mentioned anywhere in the docs (the only location I could find
> in the docs about Hadoop version is [2]). In this sense, the support for
> older Hadoop versions was kind of implicit: We're talking about compiling
> Flink with Hadoop 2.8.5 but also mention older Hadoop versions which leaves
> room for interpretation.
>
> Additionally, having code that hasn't been touch for a while increases the
> risk of it
>
> Matthias
>
> [1] https://lists.apache.org/thread/w7www13tossxrxo1mttgb68v81rf6fks
> [2]
> https://nightlies.a1pache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions
> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions>
>
> On Fri, Oct 21, 2022 at 4:13 AM Xintong Song <to...@gmail.com>
> wrote:
>
>> I believe there are some reflection based approaches in the `flink-yarn`
>> module, for supporting outdated APIs in early Hadoop versions.
>>
>> I haven't done a thorough check, and these are what I get.
>> - AMRMClientAsyncReflector
>> - ApplicationSubmissionContextReflector
>> - ContainerRequestReflector
>> - RegisterApplicationMasterResponseReflector
>> - ResourceInformationReflector
>>
>> Are we removing these as well? If yes, then Flink can no longer work with
>> the old hadoop versions. (That's how I understand "bumping the minimal
>> supported hadoop version".) I personally am not super eager to get rid of
>> theses, because the relevant parts of codes are no longer frequently
>> changing, thus the maintenance overhead is low.
>>
>> Best,
>>
>> Xintong
>>
>>
>>
>> On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <da...@gmail.com> wrote:
>>
>> > Given that we do not bundle any hadoop classes in the Flink binary, do
>> you
>> > mean simply bump the hadoop version in the parent pom?
>> > If it is, why do not we use the latest stable hadoop version 3.3.4? It
>> > seems that our cron build has verified that hadoop3 could work.
>> >
>> > Best,
>> > Yang
>> >
>> > David Morávek <da...@gmail.com> 于2022年10月19日周三 16:29写道：
>> >
>> > > +1; anything below 2.10.x seems to be EOL
>> > >
>> > > Best,
>> > > D.
>> > >
>> > > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi <
>> > balassi.marton@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Martjin,
>> > > >
>> > > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to
>> > > > implement the bump?
>> > > >
>> > > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <
>> > gabor.g.somogyi@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Martin,
>> > > > >
>> > > > > Thanks for bringing this up! Lately I was thinking about to bump
>> the
>> > > > hadoop
>> > > > > version to at least 2.6.1 to clean up issues like this:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
>> > > > >
>> > > > > All in all +1 from my perspective.
>> > > > >
>> > > > > Just a question here. Are we stating the minimum Hadoop version
>> for
>> > > users
>> > > > > somewhere in the doc or they need to find it out from source code
>> > like
>> > > > > this?
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
>> > > > >
>> > > > > BR,
>> > > > > G
>> > > > >
>> > > > >
>> > > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <
>> > > martijnvisser@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi everyone,
>> > > > > >
>> > > > > > Little over a year ago a discussion thread was opened on
>> changing
>> > the
>> > > > > > minimal supported version of Hadoop and bringing that to 2.8.5.
>> [1]
>> > > In
>> > > > > this
>> > > > > > discussion thread, I would like to propose to bring that minimal
>> > > > > supported
>> > > > > > version of Hadoop to 2.10.2.
>> > > > > >
>> > > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are
>> classified
>> > as
>> > > > > > Critical. [2] [3]. While Flink is not directly impacted by
>> those,
>> > we
>> > > do
>> > > > > see
>> > > > > > vulnerability scanners flag Flink as being vulnerable. We could
>> > > easily
>> > > > > > mitigate that by bumping the minimal supported version of
>> Hadoop to
>> > > > > 2.10.2.
>> > > > > >
>> > > > > > I'm looking forward to your opinions on this topic.
>> > > > > >
>> > > > > > Best regards,
>> > > > > >
>> > > > > > Martijn
>> > > > > > https://twitter.com/MartijnVisser82
>> > > > > > https://github.com/MartijnVisser
>> > > > > >
>> > > > > > [1]
>> > https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
>> > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
>> > > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by Matthias Pohl <ma...@aiven.io.INVALID>.
I guess upgrading the minimal version should also mean cleaning up the
codebase, i.e. removing code segments that have been around to allow
support for older versions. The overall goal should be to improve the Flink
codebase in my opinion. Considering what David said in the old thread about
Hadoop users usually lacking behind with version upgrades [1], would we do
this version bump in two phases, i.e. adding some deprecation notes and
doing the actual cleanup later on? I think Gabor has a point with it not
being really mentioned anywhere in the docs (the only location I could find
in the docs about Hadoop version is [2]). In this sense, the support for
older Hadoop versions was kind of implicit: We're talking about compiling
Flink with Hadoop 2.8.5 but also mention older Hadoop versions which leaves
room for interpretation.

Additionally, having code that hasn't been touch for a while increases the
risk of it

Matthias

[1] https://lists.apache.org/thread/w7www13tossxrxo1mttgb68v81rf6fks
[2]
https://nightlies.a1pache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions>

On Fri, Oct 21, 2022 at 4:13 AM Xintong Song <to...@gmail.com> wrote:

> I believe there are some reflection based approaches in the `flink-yarn`
> module, for supporting outdated APIs in early Hadoop versions.
>
> I haven't done a thorough check, and these are what I get.
> - AMRMClientAsyncReflector
> - ApplicationSubmissionContextReflector
> - ContainerRequestReflector
> - RegisterApplicationMasterResponseReflector
> - ResourceInformationReflector
>
> Are we removing these as well? If yes, then Flink can no longer work with
> the old hadoop versions. (That's how I understand "bumping the minimal
> supported hadoop version".) I personally am not super eager to get rid of
> theses, because the relevant parts of codes are no longer frequently
> changing, thus the maintenance overhead is low.
>
> Best,
>
> Xintong
>
>
>
> On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <da...@gmail.com> wrote:
>
> > Given that we do not bundle any hadoop classes in the Flink binary, do
> you
> > mean simply bump the hadoop version in the parent pom?
> > If it is, why do not we use the latest stable hadoop version 3.3.4? It
> > seems that our cron build has verified that hadoop3 could work.
> >
> > Best,
> > Yang
> >
> > David Morávek <da...@gmail.com> 于2022年10月19日周三 16:29写道：
> >
> > > +1; anything below 2.10.x seems to be EOL
> > >
> > > Best,
> > > D.
> > >
> > > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi <
> > balassi.marton@gmail.com>
> > > wrote:
> > >
> > > > Hi Martjin,
> > > >
> > > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to
> > > > implement the bump?
> > > >
> > > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <
> > gabor.g.somogyi@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Martin,
> > > > >
> > > > > Thanks for bringing this up! Lately I was thinking about to bump
> the
> > > > hadoop
> > > > > version to at least 2.6.1 to clean up issues like this:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
> > > > >
> > > > > All in all +1 from my perspective.
> > > > >
> > > > > Just a question here. Are we stating the minimum Hadoop version for
> > > users
> > > > > somewhere in the doc or they need to find it out from source code
> > like
> > > > > this?
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
> > > > >
> > > > > BR,
> > > > > G
> > > > >
> > > > >
> > > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <
> > > martijnvisser@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Little over a year ago a discussion thread was opened on changing
> > the
> > > > > > minimal supported version of Hadoop and bringing that to 2.8.5.
> [1]
> > > In
> > > > > this
> > > > > > discussion thread, I would like to propose to bring that minimal
> > > > > supported
> > > > > > version of Hadoop to 2.10.2.
> > > > > >
> > > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified
> > as
> > > > > > Critical. [2] [3]. While Flink is not directly impacted by those,
> > we
> > > do
> > > > > see
> > > > > > vulnerability scanners flag Flink as being vulnerable. We could
> > > easily
> > > > > > mitigate that by bumping the minimal supported version of Hadoop
> to
> > > > > 2.10.2.
> > > > > >
> > > > > > I'm looking forward to your opinions on this topic.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > > https://twitter.com/MartijnVisser82
> > > > > > https://github.com/MartijnVisser
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> > > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by Xintong Song <to...@gmail.com>.
I believe there are some reflection based approaches in the `flink-yarn`
module, for supporting outdated APIs in early Hadoop versions.

I haven't done a thorough check, and these are what I get.
- AMRMClientAsyncReflector
- ApplicationSubmissionContextReflector
- ContainerRequestReflector
- RegisterApplicationMasterResponseReflector
- ResourceInformationReflector

Are we removing these as well? If yes, then Flink can no longer work with
the old hadoop versions. (That's how I understand "bumping the minimal
supported hadoop version".) I personally am not super eager to get rid of
theses, because the relevant parts of codes are no longer frequently
changing, thus the maintenance overhead is low.

Best,

Xintong



On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <da...@gmail.com> wrote:

> Given that we do not bundle any hadoop classes in the Flink binary, do you
> mean simply bump the hadoop version in the parent pom?
> If it is, why do not we use the latest stable hadoop version 3.3.4? It
> seems that our cron build has verified that hadoop3 could work.
>
> Best,
> Yang
>
> David Morávek <da...@gmail.com> 于2022年10月19日周三 16:29写道：
>
> > +1; anything below 2.10.x seems to be EOL
> >
> > Best,
> > D.
> >
> > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi <
> balassi.marton@gmail.com>
> > wrote:
> >
> > > Hi Martjin,
> > >
> > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to
> > > implement the bump?
> > >
> > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <
> gabor.g.somogyi@gmail.com>
> > > wrote:
> > >
> > > > Hi Martin,
> > > >
> > > > Thanks for bringing this up! Lately I was thinking about to bump the
> > > hadoop
> > > > version to at least 2.6.1 to clean up issues like this:
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
> > > >
> > > > All in all +1 from my perspective.
> > > >
> > > > Just a question here. Are we stating the minimum Hadoop version for
> > users
> > > > somewhere in the doc or they need to find it out from source code
> like
> > > > this?
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
> > > >
> > > > BR,
> > > > G
> > > >
> > > >
> > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <
> > martijnvisser@apache.org>
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Little over a year ago a discussion thread was opened on changing
> the
> > > > > minimal supported version of Hadoop and bringing that to 2.8.5. [1]
> > In
> > > > this
> > > > > discussion thread, I would like to propose to bring that minimal
> > > > supported
> > > > > version of Hadoop to 2.10.2.
> > > > >
> > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified
> as
> > > > > Critical. [2] [3]. While Flink is not directly impacted by those,
> we
> > do
> > > > see
> > > > > vulnerability scanners flag Flink as being vulnerable. We could
> > easily
> > > > > mitigate that by bumping the minimal supported version of Hadoop to
> > > > 2.10.2.
> > > > >
> > > > > I'm looking forward to your opinions on this topic.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > > https://twitter.com/MartijnVisser82
> > > > > https://github.com/MartijnVisser
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by Yang Wang <da...@gmail.com>.
Given that we do not bundle any hadoop classes in the Flink binary, do you
mean simply bump the hadoop version in the parent pom?
If it is, why do not we use the latest stable hadoop version 3.3.4? It
seems that our cron build has verified that hadoop3 could work.

Best,
Yang

David Morávek <da...@gmail.com> 于2022年10月19日周三 16:29写道：

> +1; anything below 2.10.x seems to be EOL
>
> Best,
> D.
>
> On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hi Martjin,
> >
> > +1 for 2.10.2. Do you expect to have bandwidth in the near term to
> > implement the bump?
> >
> > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <ga...@gmail.com>
> > wrote:
> >
> > > Hi Martin,
> > >
> > > Thanks for bringing this up! Lately I was thinking about to bump the
> > hadoop
> > > version to at least 2.6.1 to clean up issues like this:
> > >
> > >
> >
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
> > >
> > > All in all +1 from my perspective.
> > >
> > > Just a question here. Are we stating the minimum Hadoop version for
> users
> > > somewhere in the doc or they need to find it out from source code like
> > > this?
> > >
> > >
> >
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
> > >
> > > BR,
> > > G
> > >
> > >
> > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <
> martijnvisser@apache.org>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Little over a year ago a discussion thread was opened on changing the
> > > > minimal supported version of Hadoop and bringing that to 2.8.5. [1]
> In
> > > this
> > > > discussion thread, I would like to propose to bring that minimal
> > > supported
> > > > version of Hadoop to 2.10.2.
> > > >
> > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified as
> > > > Critical. [2] [3]. While Flink is not directly impacted by those, we
> do
> > > see
> > > > vulnerability scanners flag Flink as being vulnerable. We could
> easily
> > > > mitigate that by bumping the minimal supported version of Hadoop to
> > > 2.10.2.
> > > >
> > > > I'm looking forward to your opinions on this topic.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > > https://twitter.com/MartijnVisser82
> > > > https://github.com/MartijnVisser
> > > >
> > > > [1] https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> > > >
> > >
> >
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by David Morávek <da...@gmail.com>.
+1; anything below 2.10.x seems to be EOL

Best,
D.

On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi <ba...@gmail.com>
wrote:

> Hi Martjin,
>
> +1 for 2.10.2. Do you expect to have bandwidth in the near term to
> implement the bump?
>
> On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
> > Hi Martin,
> >
> > Thanks for bringing this up! Lately I was thinking about to bump the
> hadoop
> > version to at least 2.6.1 to clean up issues like this:
> >
> >
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
> >
> > All in all +1 from my perspective.
> >
> > Just a question here. Are we stating the minimum Hadoop version for users
> > somewhere in the doc or they need to find it out from source code like
> > this?
> >
> >
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
> >
> > BR,
> > G
> >
> >
> > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <ma...@apache.org>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Little over a year ago a discussion thread was opened on changing the
> > > minimal supported version of Hadoop and bringing that to 2.8.5. [1] In
> > this
> > > discussion thread, I would like to propose to bring that minimal
> > supported
> > > version of Hadoop to 2.10.2.
> > >
> > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified as
> > > Critical. [2] [3]. While Flink is not directly impacted by those, we do
> > see
> > > vulnerability scanners flag Flink as being vulnerable. We could easily
> > > mitigate that by bumping the minimal supported version of Hadoop to
> > 2.10.2.
> > >
> > > I'm looking forward to your opinions on this topic.
> > >
> > > Best regards,
> > >
> > > Martijn
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
> > >
> > > [1] https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> > >
> >
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by Márton Balassi <ba...@gmail.com>.
Hi Martjin,

+1 for 2.10.2. Do you expect to have bandwidth in the near term to
implement the bump?

On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <ga...@gmail.com>
wrote:

> Hi Martin,
>
> Thanks for bringing this up! Lately I was thinking about to bump the hadoop
> version to at least 2.6.1 to clean up issues like this:
>
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
>
> All in all +1 from my perspective.
>
> Just a question here. Are we stating the minimum Hadoop version for users
> somewhere in the doc or they need to find it out from source code like
> this?
>
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
>
> BR,
> G
>
>
> On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <ma...@apache.org>
> wrote:
>
> > Hi everyone,
> >
> > Little over a year ago a discussion thread was opened on changing the
> > minimal supported version of Hadoop and bringing that to 2.8.5. [1] In
> this
> > discussion thread, I would like to propose to bring that minimal
> supported
> > version of Hadoop to 2.10.2.
> >
> > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified as
> > Critical. [2] [3]. While Flink is not directly impacted by those, we do
> see
> > vulnerability scanners flag Flink as being vulnerable. We could easily
> > mitigate that by bumping the minimal supported version of Hadoop to
> 2.10.2.
> >
> > I'm looking forward to your opinions on this topic.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> > [1] https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> >
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Martin,

Thanks for bringing this up! Lately I was thinking about to bump the hadoop
version to at least 2.6.1 to clean up issues like this:
https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159

All in all +1 from my perspective.

Just a question here. Are we stating the minimum Hadoop version for users
somewhere in the doc or they need to find it out from source code like this?
https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113

BR,
G


On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <ma...@apache.org>
wrote:

> Hi everyone,
>
> Little over a year ago a discussion thread was opened on changing the
> minimal supported version of Hadoop and bringing that to 2.8.5. [1] In this
> discussion thread, I would like to propose to bring that minimal supported
> version of Hadoop to 2.10.2.
>
> Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified as
> Critical. [2] [3]. While Flink is not directly impacted by those, we do see
> vulnerability scanners flag Flink as being vulnerable. We could easily
> mitigate that by bumping the minimal supported version of Hadoop to 2.10.2.
>
> I'm looking forward to your opinions on this topic.
>
> Best regards,
>
> Martijn
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
> [1] https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
>