You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by slfan1989 <sl...@apache.org> on 2025/06/03 23:23:52 UTC
Re Hadoop 3.5.0/Java 17 Release Planning
Hi, Chris
Thank you very much for your efforts in driving the JDK17 upgrade. I will
continue to follow up on the JUnit5 migration, and I hope we can
successfully complete this upgrade together.
Best Regards,
Shilun Fan.
On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
> Hello everyone,
> I've mentioned in other forums that I'd volunteer as release manager for
> Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
> get more specific about release planning and a potential target date.
> It can be easy to get lost in the nested historical JIRAs for Java 17
>
support [1]. I'd like to start by summarizing my understanding of work done
> and work remaining for Java 17 as well as other significant features the
> community wants to ship in 3.5.0:
> - We've completed the Jersey upgrade [2].
> - We've made significant progress on the JUnit 5 upgrade [3], and we
> have a list of several PRs remaining to be reviewed and committed [4].
> - There is some sun.misc usage that needs alternate implementations [5]
> [6]. There are PRs, but they haven't had any recent activity.
> - Once these issues are settled, we need to (re-)introduce a Java 17
> build profile [7].
> - I and a few others are also interested in trying to target the GCS
> FileSystem implementation for 3.5.0 [8].
> Does this look accurate? Did I miss anything important for 3.5.0, either
> for Java 17 or other features?
> Chris Nauroth
> [1] https://issues.apache.org/jira/browse/HADOOP-17177
> [2] https://issues.apache.org/jira/browse/HADOOP-15984
> [3] https://issues.apache.org/jira/browse/HADOOP-14693
> [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> [5] https://issues.apache.org/jira/browse/HADOOP-19329
> [6] https://issues.apache.org/jira/browse/HADOOP-19334
> [7] https://issues.apache.org/jira/browse/HADOOP-19298
> [8] https://issues.apache.org/jira/browse/HADOOP-19343
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Chris Nauroth <cn...@apache.org>.
Hi everyone,
This is indeed great news, and thank you to everyone!
I have one more GCS patch in mind for 3.5.0. I'd like to move hadoop-gcp
from hadoop-tools to hadoop-cloud-storage. I think we confused ourselves a
bit in some of the early discussions and made a mistake picking
hadoop-tools. It's important to make the move to hadoop-cloud-storage
before the 3.5.0 release so that we don't get stuck with compatibility
issues trying to make the move later. I'll aim to get a patch together in
the next few days.
Aside from that, it seems that JDiff is the only other open item. That's
getting close to a +1 here:
https://github.com/apache/hadoop/pull/8038
I had volunteered to be release manager for 3.5.0. Is it safe to say that
after inclusion of these 2 remaining items, I can kick off the release
process? Please let me know if there are any other pending items.
Chris Nauroth
On Mon, Nov 3, 2025 at 6:03 PM slfan1989 <sl...@apache.org> wrote:
> Many thanks to Steve for the feedback. We still have a few upgrade details
> to track to ensure a smooth Hadoop 3.5.0 release.
>
> Best Regards,
> Shilun Fan
>
> On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com>
> wrote:
>
> >
> > This is truly awesome. Great work by everyone involved!
> >
> > It does mean that
> >
> >
> > 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> > things are still all good on java8.
> > 2. People can start to learn and play with java17 language and API
> > changes. I'd recommend starting with the tests there. And I have no
> idea
> > what the changes are having been stuck in java8...which at least added
> > closures.
> >
> > Again, well done for a massive effort
> >
> >
> >
> > On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> >
> >> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> >> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> >> results daily*, and we will continue to monitor the outcomes over the
> >> next few days.
> >>
> >> The following issues and follow-up actions have been identified:
> >>
> >> 1.
> >>
> >> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> >> verification rules have become stricter, resulting in new SpotBugs
> >> warnings. We plan to start refactoring and fixing these issues *early
> >> next week*.
> >> 2.
> >>
> >> *Some modules are reporting Javadoc errors*, which will also be
> >> addressed *early next week*.
> >> 3.
> >>
> >> The *mvn site build is currently failing*. We will submit a *separate
> >> PR* to resolve this issue. *Hualong Zhang* has already completed
> *JDiff
> >> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> >> is merged, the mvn site build is expected to return to normal.
> >>
> >> Overall, the mainline compilation upgrade has been successfully
> >> completed, and the next focus will be on improving code quality and
> >> stabilizing the build pipeline.
> >>
> >> Best Regards, Shilun Fan
> >>
> >>
> >>>
> >>>
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Chris Nauroth <cn...@apache.org>.
Hi everyone,
This is indeed great news, and thank you to everyone!
I have one more GCS patch in mind for 3.5.0. I'd like to move hadoop-gcp
from hadoop-tools to hadoop-cloud-storage. I think we confused ourselves a
bit in some of the early discussions and made a mistake picking
hadoop-tools. It's important to make the move to hadoop-cloud-storage
before the 3.5.0 release so that we don't get stuck with compatibility
issues trying to make the move later. I'll aim to get a patch together in
the next few days.
Aside from that, it seems that JDiff is the only other open item. That's
getting close to a +1 here:
https://github.com/apache/hadoop/pull/8038
I had volunteered to be release manager for 3.5.0. Is it safe to say that
after inclusion of these 2 remaining items, I can kick off the release
process? Please let me know if there are any other pending items.
Chris Nauroth
On Mon, Nov 3, 2025 at 6:03 PM slfan1989 <sl...@apache.org> wrote:
> Many thanks to Steve for the feedback. We still have a few upgrade details
> to track to ensure a smooth Hadoop 3.5.0 release.
>
> Best Regards,
> Shilun Fan
>
> On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com>
> wrote:
>
> >
> > This is truly awesome. Great work by everyone involved!
> >
> > It does mean that
> >
> >
> > 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> > things are still all good on java8.
> > 2. People can start to learn and play with java17 language and API
> > changes. I'd recommend starting with the tests there. And I have no
> idea
> > what the changes are having been stuck in java8...which at least added
> > closures.
> >
> > Again, well done for a massive effort
> >
> >
> >
> > On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> >
> >> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> >> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> >> results daily*, and we will continue to monitor the outcomes over the
> >> next few days.
> >>
> >> The following issues and follow-up actions have been identified:
> >>
> >> 1.
> >>
> >> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> >> verification rules have become stricter, resulting in new SpotBugs
> >> warnings. We plan to start refactoring and fixing these issues *early
> >> next week*.
> >> 2.
> >>
> >> *Some modules are reporting Javadoc errors*, which will also be
> >> addressed *early next week*.
> >> 3.
> >>
> >> The *mvn site build is currently failing*. We will submit a *separate
> >> PR* to resolve this issue. *Hualong Zhang* has already completed
> *JDiff
> >> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> >> is merged, the mvn site build is expected to return to normal.
> >>
> >> Overall, the mainline compilation upgrade has been successfully
> >> completed, and the next focus will be on improving code quality and
> >> stabilizing the build pipeline.
> >>
> >> Best Regards, Shilun Fan
> >>
> >>
> >>>
> >>>
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Chris Nauroth <cn...@apache.org>.
Hi everyone,
This is indeed great news, and thank you to everyone!
I have one more GCS patch in mind for 3.5.0. I'd like to move hadoop-gcp
from hadoop-tools to hadoop-cloud-storage. I think we confused ourselves a
bit in some of the early discussions and made a mistake picking
hadoop-tools. It's important to make the move to hadoop-cloud-storage
before the 3.5.0 release so that we don't get stuck with compatibility
issues trying to make the move later. I'll aim to get a patch together in
the next few days.
Aside from that, it seems that JDiff is the only other open item. That's
getting close to a +1 here:
https://github.com/apache/hadoop/pull/8038
I had volunteered to be release manager for 3.5.0. Is it safe to say that
after inclusion of these 2 remaining items, I can kick off the release
process? Please let me know if there are any other pending items.
Chris Nauroth
On Mon, Nov 3, 2025 at 6:03 PM slfan1989 <sl...@apache.org> wrote:
> Many thanks to Steve for the feedback. We still have a few upgrade details
> to track to ensure a smooth Hadoop 3.5.0 release.
>
> Best Regards,
> Shilun Fan
>
> On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com>
> wrote:
>
> >
> > This is truly awesome. Great work by everyone involved!
> >
> > It does mean that
> >
> >
> > 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> > things are still all good on java8.
> > 2. People can start to learn and play with java17 language and API
> > changes. I'd recommend starting with the tests there. And I have no
> idea
> > what the changes are having been stuck in java8...which at least added
> > closures.
> >
> > Again, well done for a massive effort
> >
> >
> >
> > On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> >
> >> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> >> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> >> results daily*, and we will continue to monitor the outcomes over the
> >> next few days.
> >>
> >> The following issues and follow-up actions have been identified:
> >>
> >> 1.
> >>
> >> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> >> verification rules have become stricter, resulting in new SpotBugs
> >> warnings. We plan to start refactoring and fixing these issues *early
> >> next week*.
> >> 2.
> >>
> >> *Some modules are reporting Javadoc errors*, which will also be
> >> addressed *early next week*.
> >> 3.
> >>
> >> The *mvn site build is currently failing*. We will submit a *separate
> >> PR* to resolve this issue. *Hualong Zhang* has already completed
> *JDiff
> >> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> >> is merged, the mvn site build is expected to return to normal.
> >>
> >> Overall, the mainline compilation upgrade has been successfully
> >> completed, and the next focus will be on improving code quality and
> >> stabilizing the build pipeline.
> >>
> >> Best Regards, Shilun Fan
> >>
> >>
> >>>
> >>>
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Chris Nauroth <cn...@apache.org>.
Hi everyone,
This is indeed great news, and thank you to everyone!
I have one more GCS patch in mind for 3.5.0. I'd like to move hadoop-gcp
from hadoop-tools to hadoop-cloud-storage. I think we confused ourselves a
bit in some of the early discussions and made a mistake picking
hadoop-tools. It's important to make the move to hadoop-cloud-storage
before the 3.5.0 release so that we don't get stuck with compatibility
issues trying to make the move later. I'll aim to get a patch together in
the next few days.
Aside from that, it seems that JDiff is the only other open item. That's
getting close to a +1 here:
https://github.com/apache/hadoop/pull/8038
I had volunteered to be release manager for 3.5.0. Is it safe to say that
after inclusion of these 2 remaining items, I can kick off the release
process? Please let me know if there are any other pending items.
Chris Nauroth
On Mon, Nov 3, 2025 at 6:03 PM slfan1989 <sl...@apache.org> wrote:
> Many thanks to Steve for the feedback. We still have a few upgrade details
> to track to ensure a smooth Hadoop 3.5.0 release.
>
> Best Regards,
> Shilun Fan
>
> On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com>
> wrote:
>
> >
> > This is truly awesome. Great work by everyone involved!
> >
> > It does mean that
> >
> >
> > 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> > things are still all good on java8.
> > 2. People can start to learn and play with java17 language and API
> > changes. I'd recommend starting with the tests there. And I have no
> idea
> > what the changes are having been stuck in java8...which at least added
> > closures.
> >
> > Again, well done for a massive effort
> >
> >
> >
> > On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> >
> >> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> >> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> >> results daily*, and we will continue to monitor the outcomes over the
> >> next few days.
> >>
> >> The following issues and follow-up actions have been identified:
> >>
> >> 1.
> >>
> >> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> >> verification rules have become stricter, resulting in new SpotBugs
> >> warnings. We plan to start refactoring and fixing these issues *early
> >> next week*.
> >> 2.
> >>
> >> *Some modules are reporting Javadoc errors*, which will also be
> >> addressed *early next week*.
> >> 3.
> >>
> >> The *mvn site build is currently failing*. We will submit a *separate
> >> PR* to resolve this issue. *Hualong Zhang* has already completed
> *JDiff
> >> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> >> is merged, the mvn site build is expected to return to normal.
> >>
> >> Overall, the mainline compilation upgrade has been successfully
> >> completed, and the next focus will be on improving code quality and
> >> stabilizing the build pipeline.
> >>
> >> Best Regards, Shilun Fan
> >>
> >>
> >>>
> >>>
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Many thanks to Steve for the feedback. We still have a few upgrade details
to track to ensure a smooth Hadoop 3.5.0 release.
Best Regards,
Shilun Fan
On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com> wrote:
>
> This is truly awesome. Great work by everyone involved!
>
> It does mean that
>
>
> 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> things are still all good on java8.
> 2. People can start to learn and play with java17 language and API
> changes. I'd recommend starting with the tests there. And I have no idea
> what the changes are having been stuck in java8...which at least added
> closures.
>
> Again, well done for a massive effort
>
>
>
> On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
>
>> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
>> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
>> results daily*, and we will continue to monitor the outcomes over the
>> next few days.
>>
>> The following issues and follow-up actions have been identified:
>>
>> 1.
>>
>> After upgrading the *SpotBugs plugin to version 4.9.7*, the
>> verification rules have become stricter, resulting in new SpotBugs
>> warnings. We plan to start refactoring and fixing these issues *early
>> next week*.
>> 2.
>>
>> *Some modules are reporting Javadoc errors*, which will also be
>> addressed *early next week*.
>> 3.
>>
>> The *mvn site build is currently failing*. We will submit a *separate
>> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
>> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
>> is merged, the mvn site build is expected to return to normal.
>>
>> Overall, the mainline compilation upgrade has been successfully
>> completed, and the next focus will be on improving code quality and
>> stabilizing the build pipeline.
>>
>> Best Regards, Shilun Fan
>>
>>
>>>
>>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Many thanks to Steve for the feedback. We still have a few upgrade details
to track to ensure a smooth Hadoop 3.5.0 release.
Best Regards,
Shilun Fan
On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com> wrote:
>
> This is truly awesome. Great work by everyone involved!
>
> It does mean that
>
>
> 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> things are still all good on java8.
> 2. People can start to learn and play with java17 language and API
> changes. I'd recommend starting with the tests there. And I have no idea
> what the changes are having been stuck in java8...which at least added
> closures.
>
> Again, well done for a massive effort
>
>
>
> On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
>
>> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
>> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
>> results daily*, and we will continue to monitor the outcomes over the
>> next few days.
>>
>> The following issues and follow-up actions have been identified:
>>
>> 1.
>>
>> After upgrading the *SpotBugs plugin to version 4.9.7*, the
>> verification rules have become stricter, resulting in new SpotBugs
>> warnings. We plan to start refactoring and fixing these issues *early
>> next week*.
>> 2.
>>
>> *Some modules are reporting Javadoc errors*, which will also be
>> addressed *early next week*.
>> 3.
>>
>> The *mvn site build is currently failing*. We will submit a *separate
>> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
>> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
>> is merged, the mvn site build is expected to return to normal.
>>
>> Overall, the mainline compilation upgrade has been successfully
>> completed, and the next focus will be on improving code quality and
>> stabilizing the build pipeline.
>>
>> Best Regards, Shilun Fan
>>
>>
>>>
>>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Many thanks to Steve for the feedback. We still have a few upgrade details
to track to ensure a smooth Hadoop 3.5.0 release.
Best Regards,
Shilun Fan
On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com> wrote:
>
> This is truly awesome. Great work by everyone involved!
>
> It does mean that
>
>
> 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> things are still all good on java8.
> 2. People can start to learn and play with java17 language and API
> changes. I'd recommend starting with the tests there. And I have no idea
> what the changes are having been stuck in java8...which at least added
> closures.
>
> Again, well done for a massive effort
>
>
>
> On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
>
>> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
>> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
>> results daily*, and we will continue to monitor the outcomes over the
>> next few days.
>>
>> The following issues and follow-up actions have been identified:
>>
>> 1.
>>
>> After upgrading the *SpotBugs plugin to version 4.9.7*, the
>> verification rules have become stricter, resulting in new SpotBugs
>> warnings. We plan to start refactoring and fixing these issues *early
>> next week*.
>> 2.
>>
>> *Some modules are reporting Javadoc errors*, which will also be
>> addressed *early next week*.
>> 3.
>>
>> The *mvn site build is currently failing*. We will submit a *separate
>> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
>> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
>> is merged, the mvn site build is expected to return to normal.
>>
>> Overall, the mainline compilation upgrade has been successfully
>> completed, and the next focus will be on improving code quality and
>> stabilizing the build pipeline.
>>
>> Best Regards, Shilun Fan
>>
>>
>>>
>>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Many thanks to Steve for the feedback. We still have a few upgrade details
to track to ensure a smooth Hadoop 3.5.0 release.
Best Regards,
Shilun Fan
On Thu, Oct 23, 2025 at 3:52 AM Steve Loughran <st...@cloudera.com> wrote:
>
> This is truly awesome. Great work by everyone involved!
>
> It does mean that
>
>
> 1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
> things are still all good on java8.
> 2. People can start to learn and play with java17 language and API
> changes. I'd recommend starting with the tests there. And I have no idea
> what the changes are having been stuck in java8...which at least added
> closures.
>
> Again, well done for a massive effort
>
>
>
> On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
>
>> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
>> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
>> results daily*, and we will continue to monitor the outcomes over the
>> next few days.
>>
>> The following issues and follow-up actions have been identified:
>>
>> 1.
>>
>> After upgrading the *SpotBugs plugin to version 4.9.7*, the
>> verification rules have become stricter, resulting in new SpotBugs
>> warnings. We plan to start refactoring and fixing these issues *early
>> next week*.
>> 2.
>>
>> *Some modules are reporting Javadoc errors*, which will also be
>> addressed *early next week*.
>> 3.
>>
>> The *mvn site build is currently failing*. We will submit a *separate
>> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
>> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
>> is merged, the mvn site build is expected to return to normal.
>>
>> Overall, the mainline compilation upgrade has been successfully
>> completed, and the next focus will be on improving code quality and
>> stabilizing the build pipeline.
>>
>> Best Regards, Shilun Fan
>>
>>
>>>
>>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
This is truly awesome. Great work by everyone involved!
It does mean that
1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
things are still all good on java8.
2. People can start to learn and play with java17 language and API
changes. I'd recommend starting with the tests there. And I have no idea
what the changes are having been stuck in java8...which at least added
closures.
Again, well done for a massive effort
On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> results daily*, and we will continue to monitor the outcomes over the
> next few days.
>
> The following issues and follow-up actions have been identified:
>
> 1.
>
> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> verification rules have become stricter, resulting in new SpotBugs
> warnings. We plan to start refactoring and fixing these issues *early
> next week*.
> 2.
>
> *Some modules are reporting Javadoc errors*, which will also be
> addressed *early next week*.
> 3.
>
> The *mvn site build is currently failing*. We will submit a *separate
> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> is merged, the mvn site build is expected to return to normal.
>
> Overall, the mainline compilation upgrade has been successfully completed,
> and the next focus will be on improving code quality and stabilizing the
> build pipeline.
>
> Best Regards, Shilun Fan
>
>
>>
>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
This is truly awesome. Great work by everyone involved!
It does mean that
1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
things are still all good on java8.
2. People can start to learn and play with java17 language and API
changes. I'd recommend starting with the tests there. And I have no idea
what the changes are having been stuck in java8...which at least added
closures.
Again, well done for a massive effort
On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> results daily*, and we will continue to monitor the outcomes over the
> next few days.
>
> The following issues and follow-up actions have been identified:
>
> 1.
>
> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> verification rules have become stricter, resulting in new SpotBugs
> warnings. We plan to start refactoring and fixing these issues *early
> next week*.
> 2.
>
> *Some modules are reporting Javadoc errors*, which will also be
> addressed *early next week*.
> 3.
>
> The *mvn site build is currently failing*. We will submit a *separate
> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> is merged, the mvn site build is expected to return to normal.
>
> Overall, the mainline compilation upgrade has been successfully completed,
> and the next focus will be on improving code quality and stabilizing the
> build pipeline.
>
> Best Regards, Shilun Fan
>
>
>>
>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
This is truly awesome. Great work by everyone involved!
It does mean that
1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
things are still all good on java8.
2. People can start to learn and play with java17 language and API
changes. I'd recommend starting with the tests there. And I have no idea
what the changes are having been stuck in java8...which at least added
closures.
Again, well done for a massive effort
On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> results daily*, and we will continue to monitor the outcomes over the
> next few days.
>
> The following issues and follow-up actions have been identified:
>
> 1.
>
> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> verification rules have become stricter, resulting in new SpotBugs
> warnings. We plan to start refactoring and fixing these issues *early
> next week*.
> 2.
>
> *Some modules are reporting Javadoc errors*, which will also be
> addressed *early next week*.
> 3.
>
> The *mvn site build is currently failing*. We will submit a *separate
> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> is merged, the mvn site build is expected to return to normal.
>
> Overall, the mainline compilation upgrade has been successfully completed,
> and the next focus will be on improving code quality and stabilizing the
> build pipeline.
>
> Best Regards, Shilun Fan
>
>
>>
>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
This is truly awesome. Great work by everyone involved!
It does mean that
1. *anyone* backporting to branch-3.4 MUST do a yetus run to make sure
things are still all good on java8.
2. People can start to learn and play with java17 language and API
changes. I'd recommend starting with the tests there. And I have no idea
what the changes are having been stuck in java8...which at least added
closures.
Again, well done for a massive effort
On Fri, 17 Oct 2025 at 01:09, slfan1989 <sl...@apache.org> wrote:
> The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
> have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build
> results daily*, and we will continue to monitor the outcomes over the
> next few days.
>
> The following issues and follow-up actions have been identified:
>
> 1.
>
> After upgrading the *SpotBugs plugin to version 4.9.7*, the
> verification rules have become stricter, resulting in new SpotBugs
> warnings. We plan to start refactoring and fixing these issues *early
> next week*.
> 2.
>
> *Some modules are reporting Javadoc errors*, which will also be
> addressed *early next week*.
> 3.
>
> The *mvn site build is currently failing*. We will submit a *separate
> PR* to resolve this issue. *Hualong Zhang* has already completed *JDiff
> and hadoop-annotation compatibility updates for JDK 17*. Once his PR
> is merged, the mvn site build is expected to return to normal.
>
> Overall, the mainline compilation upgrade has been successfully completed,
> and the next focus will be on improving code quality and stabilizing the
> build pipeline.
>
> Best Regards, Shilun Fan
>
>
>>
>>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build results
daily*, and we will continue to monitor the outcomes over the next few days.
The following issues and follow-up actions have been identified:
1.
After upgrading the *SpotBugs plugin to version 4.9.7*, the verification
rules have become stricter, resulting in new SpotBugs warnings. We plan to
start refactoring and fixing these issues *early next week*.
2.
*Some modules are reporting Javadoc errors*, which will also be
addressed *early next week*.
3.
The *mvn site build is currently failing*. We will submit a *separate PR*
to resolve this issue. *Hualong Zhang* has already completed *JDiff and
hadoop-annotation compatibility updates for JDK 17*. Once his PR is
merged, the mvn site build is expected to return to normal.
Overall, the mainline compilation upgrade has been successfully completed,
and the next focus will be on improving code quality and stabilizing the
build pipeline.
Best Regards, Shilun Fan
On Wed, Oct 8, 2025 at 3:08 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
>
> > I would like to share the current progress on Hadoop JDK17.
> >
> > 1. Junit5 Upgrade
> >
> > We have completed the migration of unit tests to Junit5 across the
> project
> > and removed Junit4 dependencies wherever possible.
> > In addition, we configured an Import Rule to ensure that Junit4
> references
> > are not reintroduced.
> > However, some Junit4 dependencies still remain because certain
> third-party
> > plugins continue to rely on Junit4 components (
> > see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> > dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> > We will need to wait until these plugins are upgraded before fully
> > removing Junit4.
> >
>
> yeah, that was tough.
>
>
> >
> > 2. Unit Test Issues
> >
> > Several failing unit tests remain, mainly in the YARN module, including
> RM
> > crash tests and other errors.
> > There are also issues reported with the Jesery2 Client (see mailing list
> > thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo
> >
> > ).
> > I have set up a complete local environment to run the full test suite and
> > will continue to follow up on these issues with high priority.
> >
> > hadoop-aws and hadoop-azure should be happy now.
>
>
> > 3. JDIFF Issues and Progress
> >
> > JDIFF is a long-standing plugin that has not been updated since JDK8.
> > Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> > compiled under JDK17.
> > He also updated the custom annotations in hadoop-annotations, enabling
> > jdiff to generate difference files under JDK17.
> > I assisted in reviewing part of the code changes and consider them
> > reasonable.
> > Hualong is currently on leave and is expected to submit these changes to
> > the community next week.
> >
> > Additionally, we identified missing files in some versions:
> >
> > hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> > hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to
> the
> > trunk branch.
> >
> > 4. JDK17 Validation Pipeline
> >
> > With the assistance of Gautham, we have configured a Jenkins JDK17
> > validation pipeline.
> > I will run a full validation on the trunk branch as soon as possible, and
> > if no issues are found,
> > we plan to officially switch the trunk branch build to JDK17 within the
> > next 1–2 weeks.
> >
>
> lovely!
>
> I'm really looking forward to not having to do java8 stuff, though it'll
> take a long time to move away from it, at least as far as bug fixes go.
>
> If we get a feature complete version of 3.4.3 out (JAR updates and more)
> then that can be the official "last 3.4.x you can expect" release other
> than for cves/critical fixes.
>
> Note also I've been working on cloud store packaging. Still WiP but I want
> * all cloud connectors to be in common/lib
> * azure, gcs and s3a *excluding bundle.jar* in by default
> * profile options for the others, and to include bundle.jar
>
> This'll put these on the classpath by default, the way the vendor distros
> do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
>
>
> https://github.com/apache/hadoop/pull/7980
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build results
daily*, and we will continue to monitor the outcomes over the next few days.
The following issues and follow-up actions have been identified:
1.
After upgrading the *SpotBugs plugin to version 4.9.7*, the verification
rules have become stricter, resulting in new SpotBugs warnings. We plan to
start refactoring and fixing these issues *early next week*.
2.
*Some modules are reporting Javadoc errors*, which will also be
addressed *early next week*.
3.
The *mvn site build is currently failing*. We will submit a *separate PR*
to resolve this issue. *Hualong Zhang* has already completed *JDiff and
hadoop-annotation compatibility updates for JDK 17*. Once his PR is
merged, the mvn site build is expected to return to normal.
Overall, the mainline compilation upgrade has been successfully completed,
and the next focus will be on improving code quality and stabilizing the
build pipeline.
Best Regards, Shilun Fan
On Wed, Oct 8, 2025 at 3:08 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
>
> > I would like to share the current progress on Hadoop JDK17.
> >
> > 1. Junit5 Upgrade
> >
> > We have completed the migration of unit tests to Junit5 across the
> project
> > and removed Junit4 dependencies wherever possible.
> > In addition, we configured an Import Rule to ensure that Junit4
> references
> > are not reintroduced.
> > However, some Junit4 dependencies still remain because certain
> third-party
> > plugins continue to rely on Junit4 components (
> > see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> > dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> > We will need to wait until these plugins are upgraded before fully
> > removing Junit4.
> >
>
> yeah, that was tough.
>
>
> >
> > 2. Unit Test Issues
> >
> > Several failing unit tests remain, mainly in the YARN module, including
> RM
> > crash tests and other errors.
> > There are also issues reported with the Jesery2 Client (see mailing list
> > thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo
> >
> > ).
> > I have set up a complete local environment to run the full test suite and
> > will continue to follow up on these issues with high priority.
> >
> > hadoop-aws and hadoop-azure should be happy now.
>
>
> > 3. JDIFF Issues and Progress
> >
> > JDIFF is a long-standing plugin that has not been updated since JDK8.
> > Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> > compiled under JDK17.
> > He also updated the custom annotations in hadoop-annotations, enabling
> > jdiff to generate difference files under JDK17.
> > I assisted in reviewing part of the code changes and consider them
> > reasonable.
> > Hualong is currently on leave and is expected to submit these changes to
> > the community next week.
> >
> > Additionally, we identified missing files in some versions:
> >
> > hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> > hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to
> the
> > trunk branch.
> >
> > 4. JDK17 Validation Pipeline
> >
> > With the assistance of Gautham, we have configured a Jenkins JDK17
> > validation pipeline.
> > I will run a full validation on the trunk branch as soon as possible, and
> > if no issues are found,
> > we plan to officially switch the trunk branch build to JDK17 within the
> > next 1–2 weeks.
> >
>
> lovely!
>
> I'm really looking forward to not having to do java8 stuff, though it'll
> take a long time to move away from it, at least as far as bug fixes go.
>
> If we get a feature complete version of 3.4.3 out (JAR updates and more)
> then that can be the official "last 3.4.x you can expect" release other
> than for cves/critical fixes.
>
> Note also I've been working on cloud store packaging. Still WiP but I want
> * all cloud connectors to be in common/lib
> * azure, gcs and s3a *excluding bundle.jar* in by default
> * profile options for the others, and to include bundle.jar
>
> This'll put these on the classpath by default, the way the vendor distros
> do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
>
>
> https://github.com/apache/hadoop/pull/7980
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build results
daily*, and we will continue to monitor the outcomes over the next few days.
The following issues and follow-up actions have been identified:
1.
After upgrading the *SpotBugs plugin to version 4.9.7*, the verification
rules have become stricter, resulting in new SpotBugs warnings. We plan to
start refactoring and fixing these issues *early next week*.
2.
*Some modules are reporting Javadoc errors*, which will also be
addressed *early next week*.
3.
The *mvn site build is currently failing*. We will submit a *separate PR*
to resolve this issue. *Hualong Zhang* has already completed *JDiff and
hadoop-annotation compatibility updates for JDK 17*. Once his PR is
merged, the mvn site build is expected to return to normal.
Overall, the mainline compilation upgrade has been successfully completed,
and the next focus will be on improving code quality and stabilizing the
build pipeline.
Best Regards, Shilun Fan
On Wed, Oct 8, 2025 at 3:08 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
>
> > I would like to share the current progress on Hadoop JDK17.
> >
> > 1. Junit5 Upgrade
> >
> > We have completed the migration of unit tests to Junit5 across the
> project
> > and removed Junit4 dependencies wherever possible.
> > In addition, we configured an Import Rule to ensure that Junit4
> references
> > are not reintroduced.
> > However, some Junit4 dependencies still remain because certain
> third-party
> > plugins continue to rely on Junit4 components (
> > see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> > dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> > We will need to wait until these plugins are upgraded before fully
> > removing Junit4.
> >
>
> yeah, that was tough.
>
>
> >
> > 2. Unit Test Issues
> >
> > Several failing unit tests remain, mainly in the YARN module, including
> RM
> > crash tests and other errors.
> > There are also issues reported with the Jesery2 Client (see mailing list
> > thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo
> >
> > ).
> > I have set up a complete local environment to run the full test suite and
> > will continue to follow up on these issues with high priority.
> >
> > hadoop-aws and hadoop-azure should be happy now.
>
>
> > 3. JDIFF Issues and Progress
> >
> > JDIFF is a long-standing plugin that has not been updated since JDK8.
> > Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> > compiled under JDK17.
> > He also updated the custom annotations in hadoop-annotations, enabling
> > jdiff to generate difference files under JDK17.
> > I assisted in reviewing part of the code changes and consider them
> > reasonable.
> > Hualong is currently on leave and is expected to submit these changes to
> > the community next week.
> >
> > Additionally, we identified missing files in some versions:
> >
> > hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> > hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to
> the
> > trunk branch.
> >
> > 4. JDK17 Validation Pipeline
> >
> > With the assistance of Gautham, we have configured a Jenkins JDK17
> > validation pipeline.
> > I will run a full validation on the trunk branch as soon as possible, and
> > if no issues are found,
> > we plan to officially switch the trunk branch build to JDK17 within the
> > next 1–2 weeks.
> >
>
> lovely!
>
> I'm really looking forward to not having to do java8 stuff, though it'll
> take a long time to move away from it, at least as far as bug fixes go.
>
> If we get a feature complete version of 3.4.3 out (JAR updates and more)
> then that can be the official "last 3.4.x you can expect" release other
> than for cves/critical fixes.
>
> Note also I've been working on cloud store packaging. Still WiP but I want
> * all cloud connectors to be in common/lib
> * azure, gcs and s3a *excluding bundle.jar* in by default
> * profile options for the others, and to include bundle.jar
>
> This'll put these on the classpath by default, the way the vendor distros
> do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
>
>
> https://github.com/apache/hadoop/pull/7980
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
The *trunk branch* has now been fully upgraded to *JDK 17 and JDK 21*. I
have updated *hadoop-qbt-trunk-java17-linux-x86_64* to *push build results
daily*, and we will continue to monitor the outcomes over the next few days.
The following issues and follow-up actions have been identified:
1.
After upgrading the *SpotBugs plugin to version 4.9.7*, the verification
rules have become stricter, resulting in new SpotBugs warnings. We plan to
start refactoring and fixing these issues *early next week*.
2.
*Some modules are reporting Javadoc errors*, which will also be
addressed *early next week*.
3.
The *mvn site build is currently failing*. We will submit a *separate PR*
to resolve this issue. *Hualong Zhang* has already completed *JDiff and
hadoop-annotation compatibility updates for JDK 17*. Once his PR is
merged, the mvn site build is expected to return to normal.
Overall, the mainline compilation upgrade has been successfully completed,
and the next focus will be on improving code quality and stabilizing the
build pipeline.
Best Regards, Shilun Fan
On Wed, Oct 8, 2025 at 3:08 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
>
> > I would like to share the current progress on Hadoop JDK17.
> >
> > 1. Junit5 Upgrade
> >
> > We have completed the migration of unit tests to Junit5 across the
> project
> > and removed Junit4 dependencies wherever possible.
> > In addition, we configured an Import Rule to ensure that Junit4
> references
> > are not reintroduced.
> > However, some Junit4 dependencies still remain because certain
> third-party
> > plugins continue to rely on Junit4 components (
> > see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> > dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> > We will need to wait until these plugins are upgraded before fully
> > removing Junit4.
> >
>
> yeah, that was tough.
>
>
> >
> > 2. Unit Test Issues
> >
> > Several failing unit tests remain, mainly in the YARN module, including
> RM
> > crash tests and other errors.
> > There are also issues reported with the Jesery2 Client (see mailing list
> > thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo
> >
> > ).
> > I have set up a complete local environment to run the full test suite and
> > will continue to follow up on these issues with high priority.
> >
> > hadoop-aws and hadoop-azure should be happy now.
>
>
> > 3. JDIFF Issues and Progress
> >
> > JDIFF is a long-standing plugin that has not been updated since JDK8.
> > Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> > compiled under JDK17.
> > He also updated the custom annotations in hadoop-annotations, enabling
> > jdiff to generate difference files under JDK17.
> > I assisted in reviewing part of the code changes and consider them
> > reasonable.
> > Hualong is currently on leave and is expected to submit these changes to
> > the community next week.
> >
> > Additionally, we identified missing files in some versions:
> >
> > hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> > hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to
> the
> > trunk branch.
> >
> > 4. JDK17 Validation Pipeline
> >
> > With the assistance of Gautham, we have configured a Jenkins JDK17
> > validation pipeline.
> > I will run a full validation on the trunk branch as soon as possible, and
> > if no issues are found,
> > we plan to officially switch the trunk branch build to JDK17 within the
> > next 1–2 weeks.
> >
>
> lovely!
>
> I'm really looking forward to not having to do java8 stuff, though it'll
> take a long time to move away from it, at least as far as bug fixes go.
>
> If we get a feature complete version of 3.4.3 out (JAR updates and more)
> then that can be the official "last 3.4.x you can expect" release other
> than for cves/critical fixes.
>
> Note also I've been working on cloud store packaging. Still WiP but I want
> * all cloud connectors to be in common/lib
> * azure, gcs and s3a *excluding bundle.jar* in by default
> * profile options for the others, and to include bundle.jar
>
> This'll put these on the classpath by default, the way the vendor distros
> do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
>
>
> https://github.com/apache/hadoop/pull/7980
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
> I would like to share the current progress on Hadoop JDK17.
>
> 1. Junit5 Upgrade
>
> We have completed the migration of unit tests to Junit5 across the project
> and removed Junit4 dependencies wherever possible.
> In addition, we configured an Import Rule to ensure that Junit4 references
> are not reintroduced.
> However, some Junit4 dependencies still remain because certain third-party
> plugins continue to rely on Junit4 components (
> see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> We will need to wait until these plugins are upgraded before fully
> removing Junit4.
>
yeah, that was tough.
>
> 2. Unit Test Issues
>
> Several failing unit tests remain, mainly in the YARN module, including RM
> crash tests and other errors.
> There are also issues reported with the Jesery2 Client (see mailing list
> thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>
> ).
> I have set up a complete local environment to run the full test suite and
> will continue to follow up on these issues with high priority.
>
> hadoop-aws and hadoop-azure should be happy now.
> 3. JDIFF Issues and Progress
>
> JDIFF is a long-standing plugin that has not been updated since JDK8.
> Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> compiled under JDK17.
> He also updated the custom annotations in hadoop-annotations, enabling
> jdiff to generate difference files under JDK17.
> I assisted in reviewing part of the code changes and consider them
> reasonable.
> Hualong is currently on leave and is expected to submit these changes to
> the community next week.
>
> Additionally, we identified missing files in some versions:
>
> hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
> trunk branch.
>
> 4. JDK17 Validation Pipeline
>
> With the assistance of Gautham, we have configured a Jenkins JDK17
> validation pipeline.
> I will run a full validation on the trunk branch as soon as possible, and
> if no issues are found,
> we plan to officially switch the trunk branch build to JDK17 within the
> next 1–2 weeks.
>
lovely!
I'm really looking forward to not having to do java8 stuff, though it'll
take a long time to move away from it, at least as far as bug fixes go.
If we get a feature complete version of 3.4.3 out (JAR updates and more)
then that can be the official "last 3.4.x you can expect" release other
than for cves/critical fixes.
Note also I've been working on cloud store packaging. Still WiP but I want
* all cloud connectors to be in common/lib
* azure, gcs and s3a *excluding bundle.jar* in by default
* profile options for the others, and to include bundle.jar
This'll put these on the classpath by default, the way the vendor distros
do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
https://github.com/apache/hadoop/pull/7980
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
> I would like to share the current progress on Hadoop JDK17.
>
> 1. Junit5 Upgrade
>
> We have completed the migration of unit tests to Junit5 across the project
> and removed Junit4 dependencies wherever possible.
> In addition, we configured an Import Rule to ensure that Junit4 references
> are not reintroduced.
> However, some Junit4 dependencies still remain because certain third-party
> plugins continue to rely on Junit4 components (
> see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> We will need to wait until these plugins are upgraded before fully
> removing Junit4.
>
yeah, that was tough.
>
> 2. Unit Test Issues
>
> Several failing unit tests remain, mainly in the YARN module, including RM
> crash tests and other errors.
> There are also issues reported with the Jesery2 Client (see mailing list
> thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>
> ).
> I have set up a complete local environment to run the full test suite and
> will continue to follow up on these issues with high priority.
>
> hadoop-aws and hadoop-azure should be happy now.
> 3. JDIFF Issues and Progress
>
> JDIFF is a long-standing plugin that has not been updated since JDK8.
> Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> compiled under JDK17.
> He also updated the custom annotations in hadoop-annotations, enabling
> jdiff to generate difference files under JDK17.
> I assisted in reviewing part of the code changes and consider them
> reasonable.
> Hualong is currently on leave and is expected to submit these changes to
> the community next week.
>
> Additionally, we identified missing files in some versions:
>
> hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
> trunk branch.
>
> 4. JDK17 Validation Pipeline
>
> With the assistance of Gautham, we have configured a Jenkins JDK17
> validation pipeline.
> I will run a full validation on the trunk branch as soon as possible, and
> if no issues are found,
> we plan to officially switch the trunk branch build to JDK17 within the
> next 1–2 weeks.
>
lovely!
I'm really looking forward to not having to do java8 stuff, though it'll
take a long time to move away from it, at least as far as bug fixes go.
If we get a feature complete version of 3.4.3 out (JAR updates and more)
then that can be the official "last 3.4.x you can expect" release other
than for cves/critical fixes.
Note also I've been working on cloud store packaging. Still WiP but I want
* all cloud connectors to be in common/lib
* azure, gcs and s3a *excluding bundle.jar* in by default
* profile options for the others, and to include bundle.jar
This'll put these on the classpath by default, the way the vendor distros
do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
https://github.com/apache/hadoop/pull/7980
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
> I would like to share the current progress on Hadoop JDK17.
>
> 1. Junit5 Upgrade
>
> We have completed the migration of unit tests to Junit5 across the project
> and removed Junit4 dependencies wherever possible.
> In addition, we configured an Import Rule to ensure that Junit4 references
> are not reintroduced.
> However, some Junit4 dependencies still remain because certain third-party
> plugins continue to rely on Junit4 components (
> see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> We will need to wait until these plugins are upgraded before fully
> removing Junit4.
>
yeah, that was tough.
>
> 2. Unit Test Issues
>
> Several failing unit tests remain, mainly in the YARN module, including RM
> crash tests and other errors.
> There are also issues reported with the Jesery2 Client (see mailing list
> thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>
> ).
> I have set up a complete local environment to run the full test suite and
> will continue to follow up on these issues with high priority.
>
> hadoop-aws and hadoop-azure should be happy now.
> 3. JDIFF Issues and Progress
>
> JDIFF is a long-standing plugin that has not been updated since JDK8.
> Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> compiled under JDK17.
> He also updated the custom annotations in hadoop-annotations, enabling
> jdiff to generate difference files under JDK17.
> I assisted in reviewing part of the code changes and consider them
> reasonable.
> Hualong is currently on leave and is expected to submit these changes to
> the community next week.
>
> Additionally, we identified missing files in some versions:
>
> hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
> trunk branch.
>
> 4. JDK17 Validation Pipeline
>
> With the assistance of Gautham, we have configured a Jenkins JDK17
> validation pipeline.
> I will run a full validation on the trunk branch as soon as possible, and
> if no issues are found,
> we plan to officially switch the trunk branch build to JDK17 within the
> next 1–2 weeks.
>
lovely!
I'm really looking forward to not having to do java8 stuff, though it'll
take a long time to move away from it, at least as far as bug fixes go.
If we get a feature complete version of 3.4.3 out (JAR updates and more)
then that can be the official "last 3.4.x you can expect" release other
than for cves/critical fixes.
Note also I've been working on cloud store packaging. Still WiP but I want
* all cloud connectors to be in common/lib
* azure, gcs and s3a *excluding bundle.jar* in by default
* profile options for the others, and to include bundle.jar
This'll put these on the classpath by default, the way the vendor distros
do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
https://github.com/apache/hadoop/pull/7980
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Oct 2025 at 09:53, slfan1989 <sl...@apache.org> wrote:
> I would like to share the current progress on Hadoop JDK17.
>
> 1. Junit5 Upgrade
>
> We have completed the migration of unit tests to Junit5 across the project
> and removed Junit4 dependencies wherever possible.
> In addition, we configured an Import Rule to ensure that Junit4 references
> are not reintroduced.
> However, some Junit4 dependencies still remain because certain third-party
> plugins continue to rely on Junit4 components (
> see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive
> dependency <https://issues.apache.org/jira/browse/YARN-11872>).
> We will need to wait until these plugins are upgraded before fully
> removing Junit4.
>
yeah, that was tough.
>
> 2. Unit Test Issues
>
> Several failing unit tests remain, mainly in the YARN module, including RM
> crash tests and other errors.
> There are also issues reported with the Jesery2 Client (see mailing list
> thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>
> ).
> I have set up a complete local environment to run the full test suite and
> will continue to follow up on these issues with high priority.
>
> hadoop-aws and hadoop-azure should be happy now.
> 3. JDIFF Issues and Progress
>
> JDIFF is a long-standing plugin that has not been updated since JDK8.
> Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
> compiled under JDK17.
> He also updated the custom annotations in hadoop-annotations, enabling
> jdiff to generate difference files under JDK17.
> I assisted in reviewing part of the code changes and consider them
> reasonable.
> Hualong is currently on leave and is expected to submit these changes to
> the community next week.
>
> Additionally, we identified missing files in some versions:
>
> hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
> hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
> trunk branch.
>
> 4. JDK17 Validation Pipeline
>
> With the assistance of Gautham, we have configured a Jenkins JDK17
> validation pipeline.
> I will run a full validation on the trunk branch as soon as possible, and
> if no issues are found,
> we plan to officially switch the trunk branch build to JDK17 within the
> next 1–2 weeks.
>
lovely!
I'm really looking forward to not having to do java8 stuff, though it'll
take a long time to move away from it, at least as far as bug fixes go.
If we get a feature complete version of 3.4.3 out (JAR updates and more)
then that can be the official "last 3.4.x you can expect" release other
than for cves/critical fixes.
Note also I've been working on cloud store packaging. Still WiP but I want
* all cloud connectors to be in common/lib
* azure, gcs and s3a *excluding bundle.jar* in by default
* profile options for the others, and to include bundle.jar
This'll put these on the classpath by default, the way the vendor distros
do, and avoid the games with stripping bundle-X.Y.jar out of tools/lib
https://github.com/apache/hadoop/pull/7980
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
I would like to share the current progress on Hadoop JDK17.
1. Junit5 Upgrade
We have completed the migration of unit tests to Junit5 across the project
and removed Junit4 dependencies wherever possible.
In addition, we configured an Import Rule to ensure that Junit4 references
are not reintroduced.
However, some Junit4 dependencies still remain because certain third-party
plugins continue to rely on Junit4 components (
see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive dependency
<https://issues.apache.org/jira/browse/YARN-11872>).
We will need to wait until these plugins are upgraded before fully removing
Junit4.
2. Unit Test Issues
Several failing unit tests remain, mainly in the YARN module, including RM
crash tests and other errors.
There are also issues reported with the Jesery2 Client (see mailing list
thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>).
I have set up a complete local environment to run the full test suite and
will continue to follow up on these issues with high priority.
3. JDIFF Issues and Progress
JDIFF is a long-standing plugin that has not been updated since JDK8.
Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
compiled under JDK17.
He also updated the custom annotations in hadoop-annotations, enabling
jdiff to generate difference files under JDK17.
I assisted in reviewing part of the code changes and consider them
reasonable.
Hualong is currently on leave and is expected to submit these changes to
the community next week.
Additionally, we identified missing files in some versions:
hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
trunk branch.
4. JDK17 Validation Pipeline
With the assistance of Gautham, we have configured a Jenkins JDK17
validation pipeline.
I will run a full validation on the trunk branch as soon as possible, and
if no issues are found,
we plan to officially switch the trunk branch build to JDK17 within the
next 1–2 weeks.
Best Regards, Shilun Fan
On Fri, Jul 11, 2025 at 3:16 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hi Shilun,
>
> Thanks for initiating this discussion and for all the efforts you're
> putting into the 3.5.0 release — much appreciated!
>
> > Ongoing Support for the Hadoop 3.4 Branch
>
> It’s not just 3.4 — we currently have active release lines for 2.10,
> 3.2, and 3.3 as well. The officially designated EOL release lines are
> listed here [1]. Personally, I think it’s time we consider marking 3.2
> and 3.3 as EOL. However, I would be cautious about doing the same for
> 3.4 immediately after the 3.5.0 release.
>
> 3.5.0 brings substantial changes, including the JDK migration, which
> might make immediate adoption difficult for some users. I suggest we
> keep 3.4.x as the sole active JDK 8-compatible line for a while
> longer. We could maintain it with selective cherry-picks for critical
> issues and only roll a release if there’s significant demand. We can
> always revisit this decision a few months after 3.5.0 is out.
>
> While many projects have moved to newer JDK versions, a few downstream
> consumers still rely on JDK 8 — that’s something we should factor in.
>
> > Preparation for Hadoop 3.5 Release
>
> Great to see progress on this front. My responses inline:
>
> > Should we set up a new CI pipeline for JDK 17 on the trunk branch?
>
> Yes, since we're bumping the compilation version to 17, our regular CI
> should move entirely to JDK 17.
>
> > Should we also add support for JDK 21 in CI?
>
> I don’t think running full test suites on both JDK 17 and 21 is
> practical. I propose we follow a similar approach as we did for JDK
> 11:
>
> As of the current code compilation targets JDK 17 only, the --release
> 17 flag is already hardcoded [2], so the generated bytecode is always
> JDK 17, regardless of whether you build with JDK 21, So we can
> configure pre-commit jobs via Yetus ensure multi-JDK compilation
> doesn’t break and a separate daily CI job can verify runtime
> compatibility on JDK 21 running tests.
>
> > If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
> Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
> JDK 11 can be dropped. And moreover if you bump the version at [3] to
> 17, JDK-11 won't compile only.
>
> > Operating System Support Strategy
>
> +1 to all three of your proposals. Maintaining outdated OS images is
> tedious and brings little value. It’s better to validate against
> newer, actively supported distros and ensure a smooth experience
> there.
>
> > Release Strategy Under Multi-JDK Builds
>
> I don't recall any active discussions around multi-JDK release
> variants. As mentioned earlier, we would be compiling with JDK 17 as
> the version at [3] would change to 17, and that should be the only
> binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
> We can document how users can build from source with higher JDKs if
> needed.
>
> In essence, the official release is the source. Binaries are
> convenience artifacts — and should reflect our declared compilation
> baseline.
>
> The other sub-questions under this section become moot in light of the
> above, so skipping them for brevity.
>
>
> Just my thoughts — I’m open to adjusting the plan based on what the
> broader community agrees upon.
>
> Best,
> Ayush
>
> [1]
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> [2]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
> [3]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
>
>
>
> On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
> >
> > Dear community,
> >
> > I'd like to initiate a discussion on the future release and support
> strategy for Hadoop. This includes a few important topics that I believe we
> should align on:
> >
> > 1. Ongoing Support for the Hadoop 3.4 Branch
> >
> > As it stands, Hadoop 3.4 is the last release line that supports Java 8.
> It is expected that Java 8 compilation support will be officially removed
> from the trunk branch after a community vote.
> > At the same time, the trunk branch is increasingly diverging from the
> 3.4 line due to the adoption of JUnit 5 and new features based on JDK17
> syntax. As a result, future bug fixes and backports will become
> significantly more difficult.
> > Given this, I’d like to ask: after the release of hadoop-3.4.2, does the
> community plan to continue maintaining and releasing hadoop-3.4.3 or other
> 3.4-series versions?
> >
> > 2. Preparation for Hadoop 3.5 Release
> >
> > With the JDK17 support work nearing completion, it’s time to start
> planning for the Hadoop 3.5 release. Specific questions for discussion
> include:
> >
> > Should we set up a new CI pipeline for JDK17 on the trunk branch?
> >
> > Should we also add support for JDK21 in CI?
> >
> > If JDK21 support is added, can we remove the existing JDK11 pipeline?
> >
> > 3. Operating System Support Strategy
> >
> > Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
> already reached or are approaching end-of-life (EOL).
> > Should we adjust our support policy for these systems? For example:
> >
> > Officially deprecate EOL systems;
> >
> > Provide documentation-only support for compilation;
> >
> > Remove outdated base images from Dockerfiles.
> >
> > 4. Release Strategy Under Multi-JDK Builds
> >
> > As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
> strategy for future releases. Key questions include:
> >
> > Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
> hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
> >
> > What naming convention should we adopt for these variants?
> >
> > For Maven Central artifacts, should we distinguish by JDK version or
> consider a unified approach?
> >
> > Additional Context and Suggestions from My Side:
> >
> > The JDK17 compatibility work is being tracked under HADOOP-17177. I am
> in the process of reviewing and linking all relevant JIRAs—both completed
> and pending—to this umbrella JIRA, and will share an overview in the coming
> week.
> >
> > Regarding the JUnit 5 migration:
> > Based on our experience, JUnit 5 introduces significant differences
> compared to JUnit 4. It fundamentally changes how tests are written and
> run. My proposed upgrade path is as follows:
> >
> > Complete the migration to JUnit 5.8.2 (we are still working on HDFS and
> Hadoop-Azure and aim to finish in the next two weeks);
> >
> > Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
> Common, etc.);
> >
> > Upgrade to JUnit 5.13.2 (as Steve suggested);
> >
> > Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed
> use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is
> fully adopted;
> >
> > Upgrade the Maven version to 3.9.0 for better compatibility and plugin
> support.
> >
> > Once the JUnit 5 migration is complete, we will conduct a thorough test
> audit to ensure everything runs correctly and mark unstable tests for
> future attention.
> >
> > Regarding legacy OS support, I recommend a "lightweight" approach:
> >
> > Remove unsupported base images from our Dockerfiles or upgrade them to
> newer versions;
> >
> > Provide documentation to help users build on older systems;
> >
> > But formally discontinue full support for EOL systems.
> >
> > I’d appreciate hearing your thoughts on these topics. Looking forward to
> a fruitful discussion!
> >
> > Best regards,
> > Shilun Fan.
> >
> >
> > On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> >>
> >> Hi, Chris
> >>
> >> Thank you very much for your efforts in driving the JDK17 upgrade. I
> will continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
> >>
> >> Best Regards,
> >> Shilun Fan.
> >>
> >> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
> >>
> >> > Hello everyone,
> >>
> >> > I've mentioned in other forums that I'd volunteer as release manager
> for
> >> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd
> like to
> >> > get more specific about release planning and a potential target date.
> >>
> >> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >> > support [1]. I'd like to start by summarizing my understanding of
> work done
> >> > and work remaining for Java 17 as well as other significant features
> the
> >> > community wants to ship in 3.5.0:
> >>
> >> > - We've completed the Jersey upgrade [2].
> >> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> >> > have a list of several PRs remaining to be reviewed and committed
> [4].
> >> > - There is some sun.misc usage that needs alternate implementations
> [5]
> >> > [6]. There are PRs, but they haven't had any recent activity.
> >> > - Once these issues are settled, we need to (re-)introduce a Java 17
> >> > build profile [7].
> >> > - I and a few others are also interested in trying to target the GCS
> >> > FileSystem implementation for 3.5.0 [8].
> >>
> >> > Does this look accurate? Did I miss anything important for 3.5.0,
> either
> >> > for Java 17 or other features?
> >>
> >> > Chris Nauroth
> >>
> >> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> >> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> >> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> >> > [4]
> https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> >> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> >> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> >> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> >> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
I would like to share the current progress on Hadoop JDK17.
1. Junit5 Upgrade
We have completed the migration of unit tests to Junit5 across the project
and removed Junit4 dependencies wherever possible.
In addition, we configured an Import Rule to ensure that Junit4 references
are not reintroduced.
However, some Junit4 dependencies still remain because certain third-party
plugins continue to rely on Junit4 components (
see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive dependency
<https://issues.apache.org/jira/browse/YARN-11872>).
We will need to wait until these plugins are upgraded before fully removing
Junit4.
2. Unit Test Issues
Several failing unit tests remain, mainly in the YARN module, including RM
crash tests and other errors.
There are also issues reported with the Jesery2 Client (see mailing list
thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>).
I have set up a complete local environment to run the full test suite and
will continue to follow up on these issues with high priority.
3. JDIFF Issues and Progress
JDIFF is a long-standing plugin that has not been updated since JDK8.
Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
compiled under JDK17.
He also updated the custom annotations in hadoop-annotations, enabling
jdiff to generate difference files under JDK17.
I assisted in reviewing part of the code changes and consider them
reasonable.
Hualong is currently on leave and is expected to submit these changes to
the community next week.
Additionally, we identified missing files in some versions:
hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
trunk branch.
4. JDK17 Validation Pipeline
With the assistance of Gautham, we have configured a Jenkins JDK17
validation pipeline.
I will run a full validation on the trunk branch as soon as possible, and
if no issues are found,
we plan to officially switch the trunk branch build to JDK17 within the
next 1–2 weeks.
Best Regards, Shilun Fan
On Fri, Jul 11, 2025 at 3:16 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hi Shilun,
>
> Thanks for initiating this discussion and for all the efforts you're
> putting into the 3.5.0 release — much appreciated!
>
> > Ongoing Support for the Hadoop 3.4 Branch
>
> It’s not just 3.4 — we currently have active release lines for 2.10,
> 3.2, and 3.3 as well. The officially designated EOL release lines are
> listed here [1]. Personally, I think it’s time we consider marking 3.2
> and 3.3 as EOL. However, I would be cautious about doing the same for
> 3.4 immediately after the 3.5.0 release.
>
> 3.5.0 brings substantial changes, including the JDK migration, which
> might make immediate adoption difficult for some users. I suggest we
> keep 3.4.x as the sole active JDK 8-compatible line for a while
> longer. We could maintain it with selective cherry-picks for critical
> issues and only roll a release if there’s significant demand. We can
> always revisit this decision a few months after 3.5.0 is out.
>
> While many projects have moved to newer JDK versions, a few downstream
> consumers still rely on JDK 8 — that’s something we should factor in.
>
> > Preparation for Hadoop 3.5 Release
>
> Great to see progress on this front. My responses inline:
>
> > Should we set up a new CI pipeline for JDK 17 on the trunk branch?
>
> Yes, since we're bumping the compilation version to 17, our regular CI
> should move entirely to JDK 17.
>
> > Should we also add support for JDK 21 in CI?
>
> I don’t think running full test suites on both JDK 17 and 21 is
> practical. I propose we follow a similar approach as we did for JDK
> 11:
>
> As of the current code compilation targets JDK 17 only, the --release
> 17 flag is already hardcoded [2], so the generated bytecode is always
> JDK 17, regardless of whether you build with JDK 21, So we can
> configure pre-commit jobs via Yetus ensure multi-JDK compilation
> doesn’t break and a separate daily CI job can verify runtime
> compatibility on JDK 21 running tests.
>
> > If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
> Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
> JDK 11 can be dropped. And moreover if you bump the version at [3] to
> 17, JDK-11 won't compile only.
>
> > Operating System Support Strategy
>
> +1 to all three of your proposals. Maintaining outdated OS images is
> tedious and brings little value. It’s better to validate against
> newer, actively supported distros and ensure a smooth experience
> there.
>
> > Release Strategy Under Multi-JDK Builds
>
> I don't recall any active discussions around multi-JDK release
> variants. As mentioned earlier, we would be compiling with JDK 17 as
> the version at [3] would change to 17, and that should be the only
> binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
> We can document how users can build from source with higher JDKs if
> needed.
>
> In essence, the official release is the source. Binaries are
> convenience artifacts — and should reflect our declared compilation
> baseline.
>
> The other sub-questions under this section become moot in light of the
> above, so skipping them for brevity.
>
>
> Just my thoughts — I’m open to adjusting the plan based on what the
> broader community agrees upon.
>
> Best,
> Ayush
>
> [1]
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> [2]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
> [3]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
>
>
>
> On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
> >
> > Dear community,
> >
> > I'd like to initiate a discussion on the future release and support
> strategy for Hadoop. This includes a few important topics that I believe we
> should align on:
> >
> > 1. Ongoing Support for the Hadoop 3.4 Branch
> >
> > As it stands, Hadoop 3.4 is the last release line that supports Java 8.
> It is expected that Java 8 compilation support will be officially removed
> from the trunk branch after a community vote.
> > At the same time, the trunk branch is increasingly diverging from the
> 3.4 line due to the adoption of JUnit 5 and new features based on JDK17
> syntax. As a result, future bug fixes and backports will become
> significantly more difficult.
> > Given this, I’d like to ask: after the release of hadoop-3.4.2, does the
> community plan to continue maintaining and releasing hadoop-3.4.3 or other
> 3.4-series versions?
> >
> > 2. Preparation for Hadoop 3.5 Release
> >
> > With the JDK17 support work nearing completion, it’s time to start
> planning for the Hadoop 3.5 release. Specific questions for discussion
> include:
> >
> > Should we set up a new CI pipeline for JDK17 on the trunk branch?
> >
> > Should we also add support for JDK21 in CI?
> >
> > If JDK21 support is added, can we remove the existing JDK11 pipeline?
> >
> > 3. Operating System Support Strategy
> >
> > Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
> already reached or are approaching end-of-life (EOL).
> > Should we adjust our support policy for these systems? For example:
> >
> > Officially deprecate EOL systems;
> >
> > Provide documentation-only support for compilation;
> >
> > Remove outdated base images from Dockerfiles.
> >
> > 4. Release Strategy Under Multi-JDK Builds
> >
> > As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
> strategy for future releases. Key questions include:
> >
> > Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
> hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
> >
> > What naming convention should we adopt for these variants?
> >
> > For Maven Central artifacts, should we distinguish by JDK version or
> consider a unified approach?
> >
> > Additional Context and Suggestions from My Side:
> >
> > The JDK17 compatibility work is being tracked under HADOOP-17177. I am
> in the process of reviewing and linking all relevant JIRAs—both completed
> and pending—to this umbrella JIRA, and will share an overview in the coming
> week.
> >
> > Regarding the JUnit 5 migration:
> > Based on our experience, JUnit 5 introduces significant differences
> compared to JUnit 4. It fundamentally changes how tests are written and
> run. My proposed upgrade path is as follows:
> >
> > Complete the migration to JUnit 5.8.2 (we are still working on HDFS and
> Hadoop-Azure and aim to finish in the next two weeks);
> >
> > Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
> Common, etc.);
> >
> > Upgrade to JUnit 5.13.2 (as Steve suggested);
> >
> > Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed
> use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is
> fully adopted;
> >
> > Upgrade the Maven version to 3.9.0 for better compatibility and plugin
> support.
> >
> > Once the JUnit 5 migration is complete, we will conduct a thorough test
> audit to ensure everything runs correctly and mark unstable tests for
> future attention.
> >
> > Regarding legacy OS support, I recommend a "lightweight" approach:
> >
> > Remove unsupported base images from our Dockerfiles or upgrade them to
> newer versions;
> >
> > Provide documentation to help users build on older systems;
> >
> > But formally discontinue full support for EOL systems.
> >
> > I’d appreciate hearing your thoughts on these topics. Looking forward to
> a fruitful discussion!
> >
> > Best regards,
> > Shilun Fan.
> >
> >
> > On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> >>
> >> Hi, Chris
> >>
> >> Thank you very much for your efforts in driving the JDK17 upgrade. I
> will continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
> >>
> >> Best Regards,
> >> Shilun Fan.
> >>
> >> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
> >>
> >> > Hello everyone,
> >>
> >> > I've mentioned in other forums that I'd volunteer as release manager
> for
> >> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd
> like to
> >> > get more specific about release planning and a potential target date.
> >>
> >> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >> > support [1]. I'd like to start by summarizing my understanding of
> work done
> >> > and work remaining for Java 17 as well as other significant features
> the
> >> > community wants to ship in 3.5.0:
> >>
> >> > - We've completed the Jersey upgrade [2].
> >> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> >> > have a list of several PRs remaining to be reviewed and committed
> [4].
> >> > - There is some sun.misc usage that needs alternate implementations
> [5]
> >> > [6]. There are PRs, but they haven't had any recent activity.
> >> > - Once these issues are settled, we need to (re-)introduce a Java 17
> >> > build profile [7].
> >> > - I and a few others are also interested in trying to target the GCS
> >> > FileSystem implementation for 3.5.0 [8].
> >>
> >> > Does this look accurate? Did I miss anything important for 3.5.0,
> either
> >> > for Java 17 or other features?
> >>
> >> > Chris Nauroth
> >>
> >> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> >> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> >> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> >> > [4]
> https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> >> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> >> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> >> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> >> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
I would like to share the current progress on Hadoop JDK17.
1. Junit5 Upgrade
We have completed the migration of unit tests to Junit5 across the project
and removed Junit4 dependencies wherever possible.
In addition, we configured an Import Rule to ensure that Junit4 references
are not reintroduced.
However, some Junit4 dependencies still remain because certain third-party
plugins continue to rely on Junit4 components (
see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive dependency
<https://issues.apache.org/jira/browse/YARN-11872>).
We will need to wait until these plugins are upgraded before fully removing
Junit4.
2. Unit Test Issues
Several failing unit tests remain, mainly in the YARN module, including RM
crash tests and other errors.
There are also issues reported with the Jesery2 Client (see mailing list
thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>).
I have set up a complete local environment to run the full test suite and
will continue to follow up on these issues with high priority.
3. JDIFF Issues and Progress
JDIFF is a long-standing plugin that has not been updated since JDK8.
Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
compiled under JDK17.
He also updated the custom annotations in hadoop-annotations, enabling
jdiff to generate difference files under JDK17.
I assisted in reviewing part of the code changes and consider them
reasonable.
Hualong is currently on leave and is expected to submit these changes to
the community next week.
Additionally, we identified missing files in some versions:
hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
trunk branch.
4. JDK17 Validation Pipeline
With the assistance of Gautham, we have configured a Jenkins JDK17
validation pipeline.
I will run a full validation on the trunk branch as soon as possible, and
if no issues are found,
we plan to officially switch the trunk branch build to JDK17 within the
next 1–2 weeks.
Best Regards, Shilun Fan
On Fri, Jul 11, 2025 at 3:16 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hi Shilun,
>
> Thanks for initiating this discussion and for all the efforts you're
> putting into the 3.5.0 release — much appreciated!
>
> > Ongoing Support for the Hadoop 3.4 Branch
>
> It’s not just 3.4 — we currently have active release lines for 2.10,
> 3.2, and 3.3 as well. The officially designated EOL release lines are
> listed here [1]. Personally, I think it’s time we consider marking 3.2
> and 3.3 as EOL. However, I would be cautious about doing the same for
> 3.4 immediately after the 3.5.0 release.
>
> 3.5.0 brings substantial changes, including the JDK migration, which
> might make immediate adoption difficult for some users. I suggest we
> keep 3.4.x as the sole active JDK 8-compatible line for a while
> longer. We could maintain it with selective cherry-picks for critical
> issues and only roll a release if there’s significant demand. We can
> always revisit this decision a few months after 3.5.0 is out.
>
> While many projects have moved to newer JDK versions, a few downstream
> consumers still rely on JDK 8 — that’s something we should factor in.
>
> > Preparation for Hadoop 3.5 Release
>
> Great to see progress on this front. My responses inline:
>
> > Should we set up a new CI pipeline for JDK 17 on the trunk branch?
>
> Yes, since we're bumping the compilation version to 17, our regular CI
> should move entirely to JDK 17.
>
> > Should we also add support for JDK 21 in CI?
>
> I don’t think running full test suites on both JDK 17 and 21 is
> practical. I propose we follow a similar approach as we did for JDK
> 11:
>
> As of the current code compilation targets JDK 17 only, the --release
> 17 flag is already hardcoded [2], so the generated bytecode is always
> JDK 17, regardless of whether you build with JDK 21, So we can
> configure pre-commit jobs via Yetus ensure multi-JDK compilation
> doesn’t break and a separate daily CI job can verify runtime
> compatibility on JDK 21 running tests.
>
> > If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
> Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
> JDK 11 can be dropped. And moreover if you bump the version at [3] to
> 17, JDK-11 won't compile only.
>
> > Operating System Support Strategy
>
> +1 to all three of your proposals. Maintaining outdated OS images is
> tedious and brings little value. It’s better to validate against
> newer, actively supported distros and ensure a smooth experience
> there.
>
> > Release Strategy Under Multi-JDK Builds
>
> I don't recall any active discussions around multi-JDK release
> variants. As mentioned earlier, we would be compiling with JDK 17 as
> the version at [3] would change to 17, and that should be the only
> binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
> We can document how users can build from source with higher JDKs if
> needed.
>
> In essence, the official release is the source. Binaries are
> convenience artifacts — and should reflect our declared compilation
> baseline.
>
> The other sub-questions under this section become moot in light of the
> above, so skipping them for brevity.
>
>
> Just my thoughts — I’m open to adjusting the plan based on what the
> broader community agrees upon.
>
> Best,
> Ayush
>
> [1]
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> [2]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
> [3]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
>
>
>
> On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
> >
> > Dear community,
> >
> > I'd like to initiate a discussion on the future release and support
> strategy for Hadoop. This includes a few important topics that I believe we
> should align on:
> >
> > 1. Ongoing Support for the Hadoop 3.4 Branch
> >
> > As it stands, Hadoop 3.4 is the last release line that supports Java 8.
> It is expected that Java 8 compilation support will be officially removed
> from the trunk branch after a community vote.
> > At the same time, the trunk branch is increasingly diverging from the
> 3.4 line due to the adoption of JUnit 5 and new features based on JDK17
> syntax. As a result, future bug fixes and backports will become
> significantly more difficult.
> > Given this, I’d like to ask: after the release of hadoop-3.4.2, does the
> community plan to continue maintaining and releasing hadoop-3.4.3 or other
> 3.4-series versions?
> >
> > 2. Preparation for Hadoop 3.5 Release
> >
> > With the JDK17 support work nearing completion, it’s time to start
> planning for the Hadoop 3.5 release. Specific questions for discussion
> include:
> >
> > Should we set up a new CI pipeline for JDK17 on the trunk branch?
> >
> > Should we also add support for JDK21 in CI?
> >
> > If JDK21 support is added, can we remove the existing JDK11 pipeline?
> >
> > 3. Operating System Support Strategy
> >
> > Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
> already reached or are approaching end-of-life (EOL).
> > Should we adjust our support policy for these systems? For example:
> >
> > Officially deprecate EOL systems;
> >
> > Provide documentation-only support for compilation;
> >
> > Remove outdated base images from Dockerfiles.
> >
> > 4. Release Strategy Under Multi-JDK Builds
> >
> > As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
> strategy for future releases. Key questions include:
> >
> > Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
> hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
> >
> > What naming convention should we adopt for these variants?
> >
> > For Maven Central artifacts, should we distinguish by JDK version or
> consider a unified approach?
> >
> > Additional Context and Suggestions from My Side:
> >
> > The JDK17 compatibility work is being tracked under HADOOP-17177. I am
> in the process of reviewing and linking all relevant JIRAs—both completed
> and pending—to this umbrella JIRA, and will share an overview in the coming
> week.
> >
> > Regarding the JUnit 5 migration:
> > Based on our experience, JUnit 5 introduces significant differences
> compared to JUnit 4. It fundamentally changes how tests are written and
> run. My proposed upgrade path is as follows:
> >
> > Complete the migration to JUnit 5.8.2 (we are still working on HDFS and
> Hadoop-Azure and aim to finish in the next two weeks);
> >
> > Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
> Common, etc.);
> >
> > Upgrade to JUnit 5.13.2 (as Steve suggested);
> >
> > Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed
> use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is
> fully adopted;
> >
> > Upgrade the Maven version to 3.9.0 for better compatibility and plugin
> support.
> >
> > Once the JUnit 5 migration is complete, we will conduct a thorough test
> audit to ensure everything runs correctly and mark unstable tests for
> future attention.
> >
> > Regarding legacy OS support, I recommend a "lightweight" approach:
> >
> > Remove unsupported base images from our Dockerfiles or upgrade them to
> newer versions;
> >
> > Provide documentation to help users build on older systems;
> >
> > But formally discontinue full support for EOL systems.
> >
> > I’d appreciate hearing your thoughts on these topics. Looking forward to
> a fruitful discussion!
> >
> > Best regards,
> > Shilun Fan.
> >
> >
> > On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> >>
> >> Hi, Chris
> >>
> >> Thank you very much for your efforts in driving the JDK17 upgrade. I
> will continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
> >>
> >> Best Regards,
> >> Shilun Fan.
> >>
> >> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
> >>
> >> > Hello everyone,
> >>
> >> > I've mentioned in other forums that I'd volunteer as release manager
> for
> >> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd
> like to
> >> > get more specific about release planning and a potential target date.
> >>
> >> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >> > support [1]. I'd like to start by summarizing my understanding of
> work done
> >> > and work remaining for Java 17 as well as other significant features
> the
> >> > community wants to ship in 3.5.0:
> >>
> >> > - We've completed the Jersey upgrade [2].
> >> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> >> > have a list of several PRs remaining to be reviewed and committed
> [4].
> >> > - There is some sun.misc usage that needs alternate implementations
> [5]
> >> > [6]. There are PRs, but they haven't had any recent activity.
> >> > - Once these issues are settled, we need to (re-)introduce a Java 17
> >> > build profile [7].
> >> > - I and a few others are also interested in trying to target the GCS
> >> > FileSystem implementation for 3.5.0 [8].
> >>
> >> > Does this look accurate? Did I miss anything important for 3.5.0,
> either
> >> > for Java 17 or other features?
> >>
> >> > Chris Nauroth
> >>
> >> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> >> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> >> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> >> > [4]
> https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> >> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> >> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> >> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> >> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
I would like to share the current progress on Hadoop JDK17.
1. Junit5 Upgrade
We have completed the migration of unit tests to Junit5 across the project
and removed Junit4 dependencies wherever possible.
In addition, we configured an Import Rule to ensure that Junit4 references
are not reintroduced.
However, some Junit4 dependencies still remain because certain third-party
plugins continue to rely on Junit4 components (
see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive dependency
<https://issues.apache.org/jira/browse/YARN-11872>).
We will need to wait until these plugins are upgraded before fully removing
Junit4.
2. Unit Test Issues
Several failing unit tests remain, mainly in the YARN module, including RM
crash tests and other errors.
There are also issues reported with the Jesery2 Client (see mailing list
thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>).
I have set up a complete local environment to run the full test suite and
will continue to follow up on these issues with high priority.
3. JDIFF Issues and Progress
JDIFF is a long-standing plugin that has not been updated since JDK8.
Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully
compiled under JDK17.
He also updated the custom annotations in hadoop-annotations, enabling
jdiff to generate difference files under JDK17.
I assisted in reviewing part of the code changes and consider them
reasonable.
Hualong is currently on leave and is expected to submit these changes to
the community next week.
Additionally, we identified missing files in some versions:
hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml;
hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the
trunk branch.
4. JDK17 Validation Pipeline
With the assistance of Gautham, we have configured a Jenkins JDK17
validation pipeline.
I will run a full validation on the trunk branch as soon as possible, and
if no issues are found,
we plan to officially switch the trunk branch build to JDK17 within the
next 1–2 weeks.
Best Regards, Shilun Fan
On Fri, Jul 11, 2025 at 3:16 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hi Shilun,
>
> Thanks for initiating this discussion and for all the efforts you're
> putting into the 3.5.0 release — much appreciated!
>
> > Ongoing Support for the Hadoop 3.4 Branch
>
> It’s not just 3.4 — we currently have active release lines for 2.10,
> 3.2, and 3.3 as well. The officially designated EOL release lines are
> listed here [1]. Personally, I think it’s time we consider marking 3.2
> and 3.3 as EOL. However, I would be cautious about doing the same for
> 3.4 immediately after the 3.5.0 release.
>
> 3.5.0 brings substantial changes, including the JDK migration, which
> might make immediate adoption difficult for some users. I suggest we
> keep 3.4.x as the sole active JDK 8-compatible line for a while
> longer. We could maintain it with selective cherry-picks for critical
> issues and only roll a release if there’s significant demand. We can
> always revisit this decision a few months after 3.5.0 is out.
>
> While many projects have moved to newer JDK versions, a few downstream
> consumers still rely on JDK 8 — that’s something we should factor in.
>
> > Preparation for Hadoop 3.5 Release
>
> Great to see progress on this front. My responses inline:
>
> > Should we set up a new CI pipeline for JDK 17 on the trunk branch?
>
> Yes, since we're bumping the compilation version to 17, our regular CI
> should move entirely to JDK 17.
>
> > Should we also add support for JDK 21 in CI?
>
> I don’t think running full test suites on both JDK 17 and 21 is
> practical. I propose we follow a similar approach as we did for JDK
> 11:
>
> As of the current code compilation targets JDK 17 only, the --release
> 17 flag is already hardcoded [2], so the generated bytecode is always
> JDK 17, regardless of whether you build with JDK 21, So we can
> configure pre-commit jobs via Yetus ensure multi-JDK compilation
> doesn’t break and a separate daily CI job can verify runtime
> compatibility on JDK 21 running tests.
>
> > If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
> Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
> JDK 11 can be dropped. And moreover if you bump the version at [3] to
> 17, JDK-11 won't compile only.
>
> > Operating System Support Strategy
>
> +1 to all three of your proposals. Maintaining outdated OS images is
> tedious and brings little value. It’s better to validate against
> newer, actively supported distros and ensure a smooth experience
> there.
>
> > Release Strategy Under Multi-JDK Builds
>
> I don't recall any active discussions around multi-JDK release
> variants. As mentioned earlier, we would be compiling with JDK 17 as
> the version at [3] would change to 17, and that should be the only
> binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
> We can document how users can build from source with higher JDKs if
> needed.
>
> In essence, the official release is the source. Binaries are
> convenience artifacts — and should reflect our declared compilation
> baseline.
>
> The other sub-questions under this section become moot in light of the
> above, so skipping them for brevity.
>
>
> Just my thoughts — I’m open to adjusting the plan based on what the
> broader community agrees upon.
>
> Best,
> Ayush
>
> [1]
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> [2]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
> [3]
> https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
>
>
>
> On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
> >
> > Dear community,
> >
> > I'd like to initiate a discussion on the future release and support
> strategy for Hadoop. This includes a few important topics that I believe we
> should align on:
> >
> > 1. Ongoing Support for the Hadoop 3.4 Branch
> >
> > As it stands, Hadoop 3.4 is the last release line that supports Java 8.
> It is expected that Java 8 compilation support will be officially removed
> from the trunk branch after a community vote.
> > At the same time, the trunk branch is increasingly diverging from the
> 3.4 line due to the adoption of JUnit 5 and new features based on JDK17
> syntax. As a result, future bug fixes and backports will become
> significantly more difficult.
> > Given this, I’d like to ask: after the release of hadoop-3.4.2, does the
> community plan to continue maintaining and releasing hadoop-3.4.3 or other
> 3.4-series versions?
> >
> > 2. Preparation for Hadoop 3.5 Release
> >
> > With the JDK17 support work nearing completion, it’s time to start
> planning for the Hadoop 3.5 release. Specific questions for discussion
> include:
> >
> > Should we set up a new CI pipeline for JDK17 on the trunk branch?
> >
> > Should we also add support for JDK21 in CI?
> >
> > If JDK21 support is added, can we remove the existing JDK11 pipeline?
> >
> > 3. Operating System Support Strategy
> >
> > Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
> already reached or are approaching end-of-life (EOL).
> > Should we adjust our support policy for these systems? For example:
> >
> > Officially deprecate EOL systems;
> >
> > Provide documentation-only support for compilation;
> >
> > Remove outdated base images from Dockerfiles.
> >
> > 4. Release Strategy Under Multi-JDK Builds
> >
> > As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
> strategy for future releases. Key questions include:
> >
> > Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
> hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
> >
> > What naming convention should we adopt for these variants?
> >
> > For Maven Central artifacts, should we distinguish by JDK version or
> consider a unified approach?
> >
> > Additional Context and Suggestions from My Side:
> >
> > The JDK17 compatibility work is being tracked under HADOOP-17177. I am
> in the process of reviewing and linking all relevant JIRAs—both completed
> and pending—to this umbrella JIRA, and will share an overview in the coming
> week.
> >
> > Regarding the JUnit 5 migration:
> > Based on our experience, JUnit 5 introduces significant differences
> compared to JUnit 4. It fundamentally changes how tests are written and
> run. My proposed upgrade path is as follows:
> >
> > Complete the migration to JUnit 5.8.2 (we are still working on HDFS and
> Hadoop-Azure and aim to finish in the next two weeks);
> >
> > Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
> Common, etc.);
> >
> > Upgrade to JUnit 5.13.2 (as Steve suggested);
> >
> > Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed
> use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is
> fully adopted;
> >
> > Upgrade the Maven version to 3.9.0 for better compatibility and plugin
> support.
> >
> > Once the JUnit 5 migration is complete, we will conduct a thorough test
> audit to ensure everything runs correctly and mark unstable tests for
> future attention.
> >
> > Regarding legacy OS support, I recommend a "lightweight" approach:
> >
> > Remove unsupported base images from our Dockerfiles or upgrade them to
> newer versions;
> >
> > Provide documentation to help users build on older systems;
> >
> > But formally discontinue full support for EOL systems.
> >
> > I’d appreciate hearing your thoughts on these topics. Looking forward to
> a fruitful discussion!
> >
> > Best regards,
> > Shilun Fan.
> >
> >
> > On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> >>
> >> Hi, Chris
> >>
> >> Thank you very much for your efforts in driving the JDK17 upgrade. I
> will continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
> >>
> >> Best Regards,
> >> Shilun Fan.
> >>
> >> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
> >>
> >> > Hello everyone,
> >>
> >> > I've mentioned in other forums that I'd volunteer as release manager
> for
> >> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd
> like to
> >> > get more specific about release planning and a potential target date.
> >>
> >> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >> > support [1]. I'd like to start by summarizing my understanding of
> work done
> >> > and work remaining for Java 17 as well as other significant features
> the
> >> > community wants to ship in 3.5.0:
> >>
> >> > - We've completed the Jersey upgrade [2].
> >> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> >> > have a list of several PRs remaining to be reviewed and committed
> [4].
> >> > - There is some sun.misc usage that needs alternate implementations
> [5]
> >> > [6]. There are PRs, but they haven't had any recent activity.
> >> > - Once these issues are settled, we need to (re-)introduce a Java 17
> >> > build profile [7].
> >> > - I and a few others are also interested in trying to target the GCS
> >> > FileSystem implementation for 3.5.0 [8].
> >>
> >> > Does this look accurate? Did I miss anything important for 3.5.0,
> either
> >> > for Java 17 or other features?
> >>
> >> > Chris Nauroth
> >>
> >> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> >> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> >> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> >> > [4]
> https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> >> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> >> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> >> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> >> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Shilun,
Thanks for initiating this discussion and for all the efforts you're
putting into the 3.5.0 release — much appreciated!
> Ongoing Support for the Hadoop 3.4 Branch
It’s not just 3.4 — we currently have active release lines for 2.10,
3.2, and 3.3 as well. The officially designated EOL release lines are
listed here [1]. Personally, I think it’s time we consider marking 3.2
and 3.3 as EOL. However, I would be cautious about doing the same for
3.4 immediately after the 3.5.0 release.
3.5.0 brings substantial changes, including the JDK migration, which
might make immediate adoption difficult for some users. I suggest we
keep 3.4.x as the sole active JDK 8-compatible line for a while
longer. We could maintain it with selective cherry-picks for critical
issues and only roll a release if there’s significant demand. We can
always revisit this decision a few months after 3.5.0 is out.
While many projects have moved to newer JDK versions, a few downstream
consumers still rely on JDK 8 — that’s something we should factor in.
> Preparation for Hadoop 3.5 Release
Great to see progress on this front. My responses inline:
> Should we set up a new CI pipeline for JDK 17 on the trunk branch?
Yes, since we're bumping the compilation version to 17, our regular CI
should move entirely to JDK 17.
> Should we also add support for JDK 21 in CI?
I don’t think running full test suites on both JDK 17 and 21 is
practical. I propose we follow a similar approach as we did for JDK
11:
As of the current code compilation targets JDK 17 only, the --release
17 flag is already hardcoded [2], so the generated bytecode is always
JDK 17, regardless of whether you build with JDK 21, So we can
configure pre-commit jobs via Yetus ensure multi-JDK compilation
doesn’t break and a separate daily CI job can verify runtime
compatibility on JDK 21 running tests.
> If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
JDK 11 can be dropped. And moreover if you bump the version at [3] to
17, JDK-11 won't compile only.
> Operating System Support Strategy
+1 to all three of your proposals. Maintaining outdated OS images is
tedious and brings little value. It’s better to validate against
newer, actively supported distros and ensure a smooth experience
there.
> Release Strategy Under Multi-JDK Builds
I don't recall any active discussions around multi-JDK release
variants. As mentioned earlier, we would be compiling with JDK 17 as
the version at [3] would change to 17, and that should be the only
binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
We can document how users can build from source with higher JDKs if
needed.
In essence, the official release is the source. Binaries are
convenience artifacts — and should reflect our declared compilation
baseline.
The other sub-questions under this section become moot in light of the
above, so skipping them for brevity.
Just my thoughts — I’m open to adjusting the plan based on what the
broader community agrees upon.
Best,
Ayush
[1] https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
[2] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
[3] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
>
> Dear community,
>
> I'd like to initiate a discussion on the future release and support strategy for Hadoop. This includes a few important topics that I believe we should align on:
>
> 1. Ongoing Support for the Hadoop 3.4 Branch
>
> As it stands, Hadoop 3.4 is the last release line that supports Java 8. It is expected that Java 8 compilation support will be officially removed from the trunk branch after a community vote.
> At the same time, the trunk branch is increasingly diverging from the 3.4 line due to the adoption of JUnit 5 and new features based on JDK17 syntax. As a result, future bug fixes and backports will become significantly more difficult.
> Given this, I’d like to ask: after the release of hadoop-3.4.2, does the community plan to continue maintaining and releasing hadoop-3.4.3 or other 3.4-series versions?
>
> 2. Preparation for Hadoop 3.5 Release
>
> With the JDK17 support work nearing completion, it’s time to start planning for the Hadoop 3.5 release. Specific questions for discussion include:
>
> Should we set up a new CI pipeline for JDK17 on the trunk branch?
>
> Should we also add support for JDK21 in CI?
>
> If JDK21 support is added, can we remove the existing JDK11 pipeline?
>
> 3. Operating System Support Strategy
>
> Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have already reached or are approaching end-of-life (EOL).
> Should we adjust our support policy for these systems? For example:
>
> Officially deprecate EOL systems;
>
> Provide documentation-only support for compilation;
>
> Remove outdated base images from Dockerfiles.
>
> 4. Release Strategy Under Multi-JDK Builds
>
> As we prepare to support JDK11, JDK17, and JDK21, we also need a clear strategy for future releases. Key questions include:
>
> Should we publish multiple build variants like: hadoop-3.5.0-jdk11, hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
>
> What naming convention should we adopt for these variants?
>
> For Maven Central artifacts, should we distinguish by JDK version or consider a unified approach?
>
> Additional Context and Suggestions from My Side:
>
> The JDK17 compatibility work is being tracked under HADOOP-17177. I am in the process of reviewing and linking all relevant JIRAs—both completed and pending—to this umbrella JIRA, and will share an overview in the coming week.
>
> Regarding the JUnit 5 migration:
> Based on our experience, JUnit 5 introduces significant differences compared to JUnit 4. It fundamentally changes how tests are written and run. My proposed upgrade path is as follows:
>
> Complete the migration to JUnit 5.8.2 (we are still working on HDFS and Hadoop-Azure and aim to finish in the next two weeks);
>
> Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS, Common, etc.);
>
> Upgrade to JUnit 5.13.2 (as Steve suggested);
>
> Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is fully adopted;
>
> Upgrade the Maven version to 3.9.0 for better compatibility and plugin support.
>
> Once the JUnit 5 migration is complete, we will conduct a thorough test audit to ensure everything runs correctly and mark unstable tests for future attention.
>
> Regarding legacy OS support, I recommend a "lightweight" approach:
>
> Remove unsupported base images from our Dockerfiles or upgrade them to newer versions;
>
> Provide documentation to help users build on older systems;
>
> But formally discontinue full support for EOL systems.
>
> I’d appreciate hearing your thoughts on these topics. Looking forward to a fruitful discussion!
>
> Best regards,
> Shilun Fan.
>
>
> On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>> Hi, Chris
>>
>> Thank you very much for your efforts in driving the JDK17 upgrade. I will continue to follow up on the JUnit5 migration, and I hope we can successfully complete this upgrade together.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hello everyone,
>>
>> > I've mentioned in other forums that I'd volunteer as release manager for
>> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
>> > get more specific about release planning and a potential target date.
>>
>> > It can be easy to get lost in the nested historical JIRAs for Java 17
>> > support [1]. I'd like to start by summarizing my understanding of work done
>> > and work remaining for Java 17 as well as other significant features the
>> > community wants to ship in 3.5.0:
>>
>> > - We've completed the Jersey upgrade [2].
>> > - We've made significant progress on the JUnit 5 upgrade [3], and we
>> > have a list of several PRs remaining to be reviewed and committed [4].
>> > - There is some sun.misc usage that needs alternate implementations [5]
>> > [6]. There are PRs, but they haven't had any recent activity.
>> > - Once these issues are settled, we need to (re-)introduce a Java 17
>> > build profile [7].
>> > - I and a few others are also interested in trying to target the GCS
>> > FileSystem implementation for 3.5.0 [8].
>>
>> > Does this look accurate? Did I miss anything important for 3.5.0, either
>> > for Java 17 or other features?
>>
>> > Chris Nauroth
>>
>> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
>> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
>> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
>> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
>> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
>> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
>> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
>> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Shilun,
Thanks for initiating this discussion and for all the efforts you're
putting into the 3.5.0 release — much appreciated!
> Ongoing Support for the Hadoop 3.4 Branch
It’s not just 3.4 — we currently have active release lines for 2.10,
3.2, and 3.3 as well. The officially designated EOL release lines are
listed here [1]. Personally, I think it’s time we consider marking 3.2
and 3.3 as EOL. However, I would be cautious about doing the same for
3.4 immediately after the 3.5.0 release.
3.5.0 brings substantial changes, including the JDK migration, which
might make immediate adoption difficult for some users. I suggest we
keep 3.4.x as the sole active JDK 8-compatible line for a while
longer. We could maintain it with selective cherry-picks for critical
issues and only roll a release if there’s significant demand. We can
always revisit this decision a few months after 3.5.0 is out.
While many projects have moved to newer JDK versions, a few downstream
consumers still rely on JDK 8 — that’s something we should factor in.
> Preparation for Hadoop 3.5 Release
Great to see progress on this front. My responses inline:
> Should we set up a new CI pipeline for JDK 17 on the trunk branch?
Yes, since we're bumping the compilation version to 17, our regular CI
should move entirely to JDK 17.
> Should we also add support for JDK 21 in CI?
I don’t think running full test suites on both JDK 17 and 21 is
practical. I propose we follow a similar approach as we did for JDK
11:
As of the current code compilation targets JDK 17 only, the --release
17 flag is already hardcoded [2], so the generated bytecode is always
JDK 17, regardless of whether you build with JDK 21, So we can
configure pre-commit jobs via Yetus ensure multi-JDK compilation
doesn’t break and a separate daily CI job can verify runtime
compatibility on JDK 21 running tests.
> If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
JDK 11 can be dropped. And moreover if you bump the version at [3] to
17, JDK-11 won't compile only.
> Operating System Support Strategy
+1 to all three of your proposals. Maintaining outdated OS images is
tedious and brings little value. It’s better to validate against
newer, actively supported distros and ensure a smooth experience
there.
> Release Strategy Under Multi-JDK Builds
I don't recall any active discussions around multi-JDK release
variants. As mentioned earlier, we would be compiling with JDK 17 as
the version at [3] would change to 17, and that should be the only
binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
We can document how users can build from source with higher JDKs if
needed.
In essence, the official release is the source. Binaries are
convenience artifacts — and should reflect our declared compilation
baseline.
The other sub-questions under this section become moot in light of the
above, so skipping them for brevity.
Just my thoughts — I’m open to adjusting the plan based on what the
broader community agrees upon.
Best,
Ayush
[1] https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
[2] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
[3] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
>
> Dear community,
>
> I'd like to initiate a discussion on the future release and support strategy for Hadoop. This includes a few important topics that I believe we should align on:
>
> 1. Ongoing Support for the Hadoop 3.4 Branch
>
> As it stands, Hadoop 3.4 is the last release line that supports Java 8. It is expected that Java 8 compilation support will be officially removed from the trunk branch after a community vote.
> At the same time, the trunk branch is increasingly diverging from the 3.4 line due to the adoption of JUnit 5 and new features based on JDK17 syntax. As a result, future bug fixes and backports will become significantly more difficult.
> Given this, I’d like to ask: after the release of hadoop-3.4.2, does the community plan to continue maintaining and releasing hadoop-3.4.3 or other 3.4-series versions?
>
> 2. Preparation for Hadoop 3.5 Release
>
> With the JDK17 support work nearing completion, it’s time to start planning for the Hadoop 3.5 release. Specific questions for discussion include:
>
> Should we set up a new CI pipeline for JDK17 on the trunk branch?
>
> Should we also add support for JDK21 in CI?
>
> If JDK21 support is added, can we remove the existing JDK11 pipeline?
>
> 3. Operating System Support Strategy
>
> Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have already reached or are approaching end-of-life (EOL).
> Should we adjust our support policy for these systems? For example:
>
> Officially deprecate EOL systems;
>
> Provide documentation-only support for compilation;
>
> Remove outdated base images from Dockerfiles.
>
> 4. Release Strategy Under Multi-JDK Builds
>
> As we prepare to support JDK11, JDK17, and JDK21, we also need a clear strategy for future releases. Key questions include:
>
> Should we publish multiple build variants like: hadoop-3.5.0-jdk11, hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
>
> What naming convention should we adopt for these variants?
>
> For Maven Central artifacts, should we distinguish by JDK version or consider a unified approach?
>
> Additional Context and Suggestions from My Side:
>
> The JDK17 compatibility work is being tracked under HADOOP-17177. I am in the process of reviewing and linking all relevant JIRAs—both completed and pending—to this umbrella JIRA, and will share an overview in the coming week.
>
> Regarding the JUnit 5 migration:
> Based on our experience, JUnit 5 introduces significant differences compared to JUnit 4. It fundamentally changes how tests are written and run. My proposed upgrade path is as follows:
>
> Complete the migration to JUnit 5.8.2 (we are still working on HDFS and Hadoop-Azure and aim to finish in the next two weeks);
>
> Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS, Common, etc.);
>
> Upgrade to JUnit 5.13.2 (as Steve suggested);
>
> Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is fully adopted;
>
> Upgrade the Maven version to 3.9.0 for better compatibility and plugin support.
>
> Once the JUnit 5 migration is complete, we will conduct a thorough test audit to ensure everything runs correctly and mark unstable tests for future attention.
>
> Regarding legacy OS support, I recommend a "lightweight" approach:
>
> Remove unsupported base images from our Dockerfiles or upgrade them to newer versions;
>
> Provide documentation to help users build on older systems;
>
> But formally discontinue full support for EOL systems.
>
> I’d appreciate hearing your thoughts on these topics. Looking forward to a fruitful discussion!
>
> Best regards,
> Shilun Fan.
>
>
> On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>> Hi, Chris
>>
>> Thank you very much for your efforts in driving the JDK17 upgrade. I will continue to follow up on the JUnit5 migration, and I hope we can successfully complete this upgrade together.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hello everyone,
>>
>> > I've mentioned in other forums that I'd volunteer as release manager for
>> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
>> > get more specific about release planning and a potential target date.
>>
>> > It can be easy to get lost in the nested historical JIRAs for Java 17
>> > support [1]. I'd like to start by summarizing my understanding of work done
>> > and work remaining for Java 17 as well as other significant features the
>> > community wants to ship in 3.5.0:
>>
>> > - We've completed the Jersey upgrade [2].
>> > - We've made significant progress on the JUnit 5 upgrade [3], and we
>> > have a list of several PRs remaining to be reviewed and committed [4].
>> > - There is some sun.misc usage that needs alternate implementations [5]
>> > [6]. There are PRs, but they haven't had any recent activity.
>> > - Once these issues are settled, we need to (re-)introduce a Java 17
>> > build profile [7].
>> > - I and a few others are also interested in trying to target the GCS
>> > FileSystem implementation for 3.5.0 [8].
>>
>> > Does this look accurate? Did I miss anything important for 3.5.0, either
>> > for Java 17 or other features?
>>
>> > Chris Nauroth
>>
>> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
>> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
>> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
>> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
>> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
>> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
>> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
>> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Shilun,
Thanks for initiating this discussion and for all the efforts you're
putting into the 3.5.0 release — much appreciated!
> Ongoing Support for the Hadoop 3.4 Branch
It’s not just 3.4 — we currently have active release lines for 2.10,
3.2, and 3.3 as well. The officially designated EOL release lines are
listed here [1]. Personally, I think it’s time we consider marking 3.2
and 3.3 as EOL. However, I would be cautious about doing the same for
3.4 immediately after the 3.5.0 release.
3.5.0 brings substantial changes, including the JDK migration, which
might make immediate adoption difficult for some users. I suggest we
keep 3.4.x as the sole active JDK 8-compatible line for a while
longer. We could maintain it with selective cherry-picks for critical
issues and only roll a release if there’s significant demand. We can
always revisit this decision a few months after 3.5.0 is out.
While many projects have moved to newer JDK versions, a few downstream
consumers still rely on JDK 8 — that’s something we should factor in.
> Preparation for Hadoop 3.5 Release
Great to see progress on this front. My responses inline:
> Should we set up a new CI pipeline for JDK 17 on the trunk branch?
Yes, since we're bumping the compilation version to 17, our regular CI
should move entirely to JDK 17.
> Should we also add support for JDK 21 in CI?
I don’t think running full test suites on both JDK 17 and 21 is
practical. I propose we follow a similar approach as we did for JDK
11:
As of the current code compilation targets JDK 17 only, the --release
17 flag is already hardcoded [2], so the generated bytecode is always
JDK 17, regardless of whether you build with JDK 21, So we can
configure pre-commit jobs via Yetus ensure multi-JDK compilation
doesn’t break and a separate daily CI job can verify runtime
compatibility on JDK 21 running tests.
> If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
JDK 11 can be dropped. And moreover if you bump the version at [3] to
17, JDK-11 won't compile only.
> Operating System Support Strategy
+1 to all three of your proposals. Maintaining outdated OS images is
tedious and brings little value. It’s better to validate against
newer, actively supported distros and ensure a smooth experience
there.
> Release Strategy Under Multi-JDK Builds
I don't recall any active discussions around multi-JDK release
variants. As mentioned earlier, we would be compiling with JDK 17 as
the version at [3] would change to 17, and that should be the only
binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
We can document how users can build from source with higher JDKs if
needed.
In essence, the official release is the source. Binaries are
convenience artifacts — and should reflect our declared compilation
baseline.
The other sub-questions under this section become moot in light of the
above, so skipping them for brevity.
Just my thoughts — I’m open to adjusting the plan based on what the
broader community agrees upon.
Best,
Ayush
[1] https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
[2] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
[3] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
>
> Dear community,
>
> I'd like to initiate a discussion on the future release and support strategy for Hadoop. This includes a few important topics that I believe we should align on:
>
> 1. Ongoing Support for the Hadoop 3.4 Branch
>
> As it stands, Hadoop 3.4 is the last release line that supports Java 8. It is expected that Java 8 compilation support will be officially removed from the trunk branch after a community vote.
> At the same time, the trunk branch is increasingly diverging from the 3.4 line due to the adoption of JUnit 5 and new features based on JDK17 syntax. As a result, future bug fixes and backports will become significantly more difficult.
> Given this, I’d like to ask: after the release of hadoop-3.4.2, does the community plan to continue maintaining and releasing hadoop-3.4.3 or other 3.4-series versions?
>
> 2. Preparation for Hadoop 3.5 Release
>
> With the JDK17 support work nearing completion, it’s time to start planning for the Hadoop 3.5 release. Specific questions for discussion include:
>
> Should we set up a new CI pipeline for JDK17 on the trunk branch?
>
> Should we also add support for JDK21 in CI?
>
> If JDK21 support is added, can we remove the existing JDK11 pipeline?
>
> 3. Operating System Support Strategy
>
> Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have already reached or are approaching end-of-life (EOL).
> Should we adjust our support policy for these systems? For example:
>
> Officially deprecate EOL systems;
>
> Provide documentation-only support for compilation;
>
> Remove outdated base images from Dockerfiles.
>
> 4. Release Strategy Under Multi-JDK Builds
>
> As we prepare to support JDK11, JDK17, and JDK21, we also need a clear strategy for future releases. Key questions include:
>
> Should we publish multiple build variants like: hadoop-3.5.0-jdk11, hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
>
> What naming convention should we adopt for these variants?
>
> For Maven Central artifacts, should we distinguish by JDK version or consider a unified approach?
>
> Additional Context and Suggestions from My Side:
>
> The JDK17 compatibility work is being tracked under HADOOP-17177. I am in the process of reviewing and linking all relevant JIRAs—both completed and pending—to this umbrella JIRA, and will share an overview in the coming week.
>
> Regarding the JUnit 5 migration:
> Based on our experience, JUnit 5 introduces significant differences compared to JUnit 4. It fundamentally changes how tests are written and run. My proposed upgrade path is as follows:
>
> Complete the migration to JUnit 5.8.2 (we are still working on HDFS and Hadoop-Azure and aim to finish in the next two weeks);
>
> Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS, Common, etc.);
>
> Upgrade to JUnit 5.13.2 (as Steve suggested);
>
> Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is fully adopted;
>
> Upgrade the Maven version to 3.9.0 for better compatibility and plugin support.
>
> Once the JUnit 5 migration is complete, we will conduct a thorough test audit to ensure everything runs correctly and mark unstable tests for future attention.
>
> Regarding legacy OS support, I recommend a "lightweight" approach:
>
> Remove unsupported base images from our Dockerfiles or upgrade them to newer versions;
>
> Provide documentation to help users build on older systems;
>
> But formally discontinue full support for EOL systems.
>
> I’d appreciate hearing your thoughts on these topics. Looking forward to a fruitful discussion!
>
> Best regards,
> Shilun Fan.
>
>
> On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>> Hi, Chris
>>
>> Thank you very much for your efforts in driving the JDK17 upgrade. I will continue to follow up on the JUnit5 migration, and I hope we can successfully complete this upgrade together.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hello everyone,
>>
>> > I've mentioned in other forums that I'd volunteer as release manager for
>> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
>> > get more specific about release planning and a potential target date.
>>
>> > It can be easy to get lost in the nested historical JIRAs for Java 17
>> > support [1]. I'd like to start by summarizing my understanding of work done
>> > and work remaining for Java 17 as well as other significant features the
>> > community wants to ship in 3.5.0:
>>
>> > - We've completed the Jersey upgrade [2].
>> > - We've made significant progress on the JUnit 5 upgrade [3], and we
>> > have a list of several PRs remaining to be reviewed and committed [4].
>> > - There is some sun.misc usage that needs alternate implementations [5]
>> > [6]. There are PRs, but they haven't had any recent activity.
>> > - Once these issues are settled, we need to (re-)introduce a Java 17
>> > build profile [7].
>> > - I and a few others are also interested in trying to target the GCS
>> > FileSystem implementation for 3.5.0 [8].
>>
>> > Does this look accurate? Did I miss anything important for 3.5.0, either
>> > for Java 17 or other features?
>>
>> > Chris Nauroth
>>
>> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
>> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
>> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
>> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
>> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
>> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
>> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
>> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Future Release and Support Strategy for Hadoop
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Shilun,
Thanks for initiating this discussion and for all the efforts you're
putting into the 3.5.0 release — much appreciated!
> Ongoing Support for the Hadoop 3.4 Branch
It’s not just 3.4 — we currently have active release lines for 2.10,
3.2, and 3.3 as well. The officially designated EOL release lines are
listed here [1]. Personally, I think it’s time we consider marking 3.2
and 3.3 as EOL. However, I would be cautious about doing the same for
3.4 immediately after the 3.5.0 release.
3.5.0 brings substantial changes, including the JDK migration, which
might make immediate adoption difficult for some users. I suggest we
keep 3.4.x as the sole active JDK 8-compatible line for a while
longer. We could maintain it with selective cherry-picks for critical
issues and only roll a release if there’s significant demand. We can
always revisit this decision a few months after 3.5.0 is out.
While many projects have moved to newer JDK versions, a few downstream
consumers still rely on JDK 8 — that’s something we should factor in.
> Preparation for Hadoop 3.5 Release
Great to see progress on this front. My responses inline:
> Should we set up a new CI pipeline for JDK 17 on the trunk branch?
Yes, since we're bumping the compilation version to 17, our regular CI
should move entirely to JDK 17.
> Should we also add support for JDK 21 in CI?
I don’t think running full test suites on both JDK 17 and 21 is
practical. I propose we follow a similar approach as we did for JDK
11:
As of the current code compilation targets JDK 17 only, the --release
17 flag is already hardcoded [2], so the generated bytecode is always
JDK 17, regardless of whether you build with JDK 21, So we can
configure pre-commit jobs via Yetus ensure multi-JDK compilation
doesn’t break and a separate daily CI job can verify runtime
compatibility on JDK 21 running tests.
> If JDK 21 support is added, can we remove the existing JDK 11 pipeline?
Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline.
JDK 11 can be dropped. And moreover if you bump the version at [3] to
17, JDK-11 won't compile only.
> Operating System Support Strategy
+1 to all three of your proposals. Maintaining outdated OS images is
tedious and brings little value. It’s better to validate against
newer, actively supported distros and ensure a smooth experience
there.
> Release Strategy Under Multi-JDK Builds
I don't recall any active discussions around multi-JDK release
variants. As mentioned earlier, we would be compiling with JDK 17 as
the version at [3] would change to 17, and that should be the only
binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17.
We can document how users can build from source with higher JDKs if
needed.
In essence, the official release is the source. Binaries are
convenience artifacts — and should reflect our declared compilation
baseline.
The other sub-questions under this section become moot in light of the
above, so skipping them for brevity.
Just my thoughts — I’m open to adjusting the plan based on what the
broader community agrees upon.
Best,
Ayush
[1] https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
[2] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769
[3] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205
On Fri, 11 Jul 2025 at 11:14, slfan1989 <sl...@apache.org> wrote:
>
> Dear community,
>
> I'd like to initiate a discussion on the future release and support strategy for Hadoop. This includes a few important topics that I believe we should align on:
>
> 1. Ongoing Support for the Hadoop 3.4 Branch
>
> As it stands, Hadoop 3.4 is the last release line that supports Java 8. It is expected that Java 8 compilation support will be officially removed from the trunk branch after a community vote.
> At the same time, the trunk branch is increasingly diverging from the 3.4 line due to the adoption of JUnit 5 and new features based on JDK17 syntax. As a result, future bug fixes and backports will become significantly more difficult.
> Given this, I’d like to ask: after the release of hadoop-3.4.2, does the community plan to continue maintaining and releasing hadoop-3.4.3 or other 3.4-series versions?
>
> 2. Preparation for Hadoop 3.5 Release
>
> With the JDK17 support work nearing completion, it’s time to start planning for the Hadoop 3.5 release. Specific questions for discussion include:
>
> Should we set up a new CI pipeline for JDK17 on the trunk branch?
>
> Should we also add support for JDK21 in CI?
>
> If JDK21 support is added, can we remove the existing JDK11 pipeline?
>
> 3. Operating System Support Strategy
>
> Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have already reached or are approaching end-of-life (EOL).
> Should we adjust our support policy for these systems? For example:
>
> Officially deprecate EOL systems;
>
> Provide documentation-only support for compilation;
>
> Remove outdated base images from Dockerfiles.
>
> 4. Release Strategy Under Multi-JDK Builds
>
> As we prepare to support JDK11, JDK17, and JDK21, we also need a clear strategy for future releases. Key questions include:
>
> Should we publish multiple build variants like: hadoop-3.5.0-jdk11, hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
>
> What naming convention should we adopt for these variants?
>
> For Maven Central artifacts, should we distinguish by JDK version or consider a unified approach?
>
> Additional Context and Suggestions from My Side:
>
> The JDK17 compatibility work is being tracked under HADOOP-17177. I am in the process of reviewing and linking all relevant JIRAs—both completed and pending—to this umbrella JIRA, and will share an overview in the coming week.
>
> Regarding the JUnit 5 migration:
> Based on our experience, JUnit 5 introduces significant differences compared to JUnit 4. It fundamentally changes how tests are written and run. My proposed upgrade path is as follows:
>
> Complete the migration to JUnit 5.8.2 (we are still working on HDFS and Hadoop-Azure and aim to finish in the next two weeks);
>
> Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS, Common, etc.);
>
> Upgrade to JUnit 5.13.2 (as Steve suggested);
>
> Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is fully adopted;
>
> Upgrade the Maven version to 3.9.0 for better compatibility and plugin support.
>
> Once the JUnit 5 migration is complete, we will conduct a thorough test audit to ensure everything runs correctly and mark unstable tests for future attention.
>
> Regarding legacy OS support, I recommend a "lightweight" approach:
>
> Remove unsupported base images from our Dockerfiles or upgrade them to newer versions;
>
> Provide documentation to help users build on older systems;
>
> But formally discontinue full support for EOL systems.
>
> I’d appreciate hearing your thoughts on these topics. Looking forward to a fruitful discussion!
>
> Best regards,
> Shilun Fan.
>
>
> On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>> Hi, Chris
>>
>> Thank you very much for your efforts in driving the JDK17 upgrade. I will continue to follow up on the JUnit5 migration, and I hope we can successfully complete this upgrade together.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hello everyone,
>>
>> > I've mentioned in other forums that I'd volunteer as release manager for
>> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
>> > get more specific about release planning and a potential target date.
>>
>> > It can be easy to get lost in the nested historical JIRAs for Java 17
>> > support [1]. I'd like to start by summarizing my understanding of work done
>> > and work remaining for Java 17 as well as other significant features the
>> > community wants to ship in 3.5.0:
>>
>> > - We've completed the Jersey upgrade [2].
>> > - We've made significant progress on the JUnit 5 upgrade [3], and we
>> > have a list of several PRs remaining to be reviewed and committed [4].
>> > - There is some sun.misc usage that needs alternate implementations [5]
>> > [6]. There are PRs, but they haven't had any recent activity.
>> > - Once these issues are settled, we need to (re-)introduce a Java 17
>> > build profile [7].
>> > - I and a few others are also interested in trying to target the GCS
>> > FileSystem implementation for 3.5.0 [8].
>>
>> > Does this look accurate? Did I miss anything important for 3.5.0, either
>> > for Java 17 or other features?
>>
>> > Chris Nauroth
>>
>> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
>> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
>> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
>> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
>> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
>> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
>> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
>> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
[DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Dear community,
I'd like to initiate a discussion on the future release and support
strategy for Hadoop. This includes a few important topics that I believe we
should align on:
1. Ongoing Support for the Hadoop 3.4 Branch
As it stands, Hadoop 3.4 is the last release line that supports Java 8. It
is expected that Java 8 compilation support will be officially removed from
the trunk branch after a community vote.
At the same time, the trunk branch is increasingly diverging from the 3.4
line due to the adoption of JUnit 5 and new features based on JDK17 syntax.
As a result, future bug fixes and backports will become significantly more
difficult.
Given this, I’d like to ask: *after the release of hadoop-3.4.2, does the
community plan to continue maintaining and releasing hadoop-3.4.3 or other
3.4-series versions?*
2. Preparation for Hadoop 3.5 Release
With the JDK17 support work nearing completion, it’s time to start planning
for the Hadoop 3.5 release. Specific questions for discussion include:
-
Should we set up a new CI pipeline for JDK17 on the trunk branch?
-
Should we also add support for JDK21 in CI?
-
If JDK21 support is added, can we remove the existing JDK11 pipeline?
3. Operating System Support Strategy
Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
already reached or are approaching end-of-life (EOL).
Should we adjust our support policy for these systems? For example:
-
Officially deprecate EOL systems;
-
Provide documentation-only support for compilation;
-
Remove outdated base images from Dockerfiles.
4. Release Strategy Under Multi-JDK Builds
As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
strategy for future releases. Key questions include:
-
Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
-
What naming convention should we adopt for these variants?
-
For Maven Central artifacts, should we distinguish by JDK version or
consider a unified approach?
Additional Context and Suggestions from My Side:
1.
The JDK17 compatibility work is being tracked under HADOOP-17177
<https://issues.apache.org/jira/browse/HADOOP-17177>. I am in the
process of reviewing and linking all relevant JIRAs—both completed and
pending—to this umbrella JIRA, and will share an overview in the coming
week.
2.
Regarding the JUnit 5 migration:
Based on our experience, JUnit 5 introduces significant differences
compared to JUnit 4. It fundamentally changes how tests are written and
run. My proposed upgrade path is as follows:
-
Complete the migration to *JUnit 5.8.2* (we are still working on HDFS
and Hadoop-Azure and aim to finish in the next two weeks);
-
Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
Common, etc.);
-
Upgrade to *JUnit 5.13.2* (as Steve suggested);
-
Upgrade the *Maven Surefire Plugin*. Earlier issues were due to the
mixed use of JUnit 4 and 5, which will no longer be a concern
once JUnit 5
is fully adopted;
-
Upgrade the *Maven version* to 3.9.0 for better compatibility and
plugin support.
3.
Once the JUnit 5 migration is complete, we will conduct a thorough test
audit to ensure everything runs correctly and mark unstable tests for
future attention.
4.
Regarding legacy OS support, I recommend a "lightweight" approach:
-
Remove unsupported base images from our Dockerfiles or upgrade them
to newer versions;
-
Provide documentation to help users build on older systems;
-
But *formally discontinue* full support for EOL systems.
I’d appreciate hearing your thoughts on these topics. Looking forward to a
fruitful discussion!
Best regards,
Shilun Fan.
On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> Hi, Chris
>
> Thank you very much for your efforts in driving the JDK17 upgrade. I will
> continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>
> > Hello everyone,
>
> > I've mentioned in other forums that I'd volunteer as release manager for
> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
> > get more specific about release planning and a potential target date.
>
> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >
> support [1]. I'd like to start by summarizing my understanding of work done
> > and work remaining for Java 17 as well as other significant features the
> > community wants to ship in 3.5.0:
>
> > - We've completed the Jersey upgrade [2].
> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> > have a list of several PRs remaining to be reviewed and committed [4].
> > - There is some sun.misc
> usage that needs alternate implementations [5]
> > [6]. There are PRs, but they haven't had any recent activity.
> > - Once these issues are settled, we need to (re-)introduce a Java 17
> > build profile [7].
> > - I and a few others are also interested in trying to target the GCS
> > FileSystem implementation for 3.5.0 [8].
>
> > Does this look accurate? Did I miss anything important for 3.5.0, either
> > for Java 17 or other features?
>
> > Chris Nauroth
>
> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>
[DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Dear community,
I'd like to initiate a discussion on the future release and support
strategy for Hadoop. This includes a few important topics that I believe we
should align on:
1. Ongoing Support for the Hadoop 3.4 Branch
As it stands, Hadoop 3.4 is the last release line that supports Java 8. It
is expected that Java 8 compilation support will be officially removed from
the trunk branch after a community vote.
At the same time, the trunk branch is increasingly diverging from the 3.4
line due to the adoption of JUnit 5 and new features based on JDK17 syntax.
As a result, future bug fixes and backports will become significantly more
difficult.
Given this, I’d like to ask: *after the release of hadoop-3.4.2, does the
community plan to continue maintaining and releasing hadoop-3.4.3 or other
3.4-series versions?*
2. Preparation for Hadoop 3.5 Release
With the JDK17 support work nearing completion, it’s time to start planning
for the Hadoop 3.5 release. Specific questions for discussion include:
-
Should we set up a new CI pipeline for JDK17 on the trunk branch?
-
Should we also add support for JDK21 in CI?
-
If JDK21 support is added, can we remove the existing JDK11 pipeline?
3. Operating System Support Strategy
Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
already reached or are approaching end-of-life (EOL).
Should we adjust our support policy for these systems? For example:
-
Officially deprecate EOL systems;
-
Provide documentation-only support for compilation;
-
Remove outdated base images from Dockerfiles.
4. Release Strategy Under Multi-JDK Builds
As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
strategy for future releases. Key questions include:
-
Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
-
What naming convention should we adopt for these variants?
-
For Maven Central artifacts, should we distinguish by JDK version or
consider a unified approach?
Additional Context and Suggestions from My Side:
1.
The JDK17 compatibility work is being tracked under HADOOP-17177
<https://issues.apache.org/jira/browse/HADOOP-17177>. I am in the
process of reviewing and linking all relevant JIRAs—both completed and
pending—to this umbrella JIRA, and will share an overview in the coming
week.
2.
Regarding the JUnit 5 migration:
Based on our experience, JUnit 5 introduces significant differences
compared to JUnit 4. It fundamentally changes how tests are written and
run. My proposed upgrade path is as follows:
-
Complete the migration to *JUnit 5.8.2* (we are still working on HDFS
and Hadoop-Azure and aim to finish in the next two weeks);
-
Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
Common, etc.);
-
Upgrade to *JUnit 5.13.2* (as Steve suggested);
-
Upgrade the *Maven Surefire Plugin*. Earlier issues were due to the
mixed use of JUnit 4 and 5, which will no longer be a concern
once JUnit 5
is fully adopted;
-
Upgrade the *Maven version* to 3.9.0 for better compatibility and
plugin support.
3.
Once the JUnit 5 migration is complete, we will conduct a thorough test
audit to ensure everything runs correctly and mark unstable tests for
future attention.
4.
Regarding legacy OS support, I recommend a "lightweight" approach:
-
Remove unsupported base images from our Dockerfiles or upgrade them
to newer versions;
-
Provide documentation to help users build on older systems;
-
But *formally discontinue* full support for EOL systems.
I’d appreciate hearing your thoughts on these topics. Looking forward to a
fruitful discussion!
Best regards,
Shilun Fan.
On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> Hi, Chris
>
> Thank you very much for your efforts in driving the JDK17 upgrade. I will
> continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>
> > Hello everyone,
>
> > I've mentioned in other forums that I'd volunteer as release manager for
> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
> > get more specific about release planning and a potential target date.
>
> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >
> support [1]. I'd like to start by summarizing my understanding of work done
> > and work remaining for Java 17 as well as other significant features the
> > community wants to ship in 3.5.0:
>
> > - We've completed the Jersey upgrade [2].
> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> > have a list of several PRs remaining to be reviewed and committed [4].
> > - There is some sun.misc
> usage that needs alternate implementations [5]
> > [6]. There are PRs, but they haven't had any recent activity.
> > - Once these issues are settled, we need to (re-)introduce a Java 17
> > build profile [7].
> > - I and a few others are also interested in trying to target the GCS
> > FileSystem implementation for 3.5.0 [8].
>
> > Does this look accurate? Did I miss anything important for 3.5.0, either
> > for Java 17 or other features?
>
> > Chris Nauroth
>
> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>
[DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Dear community,
I'd like to initiate a discussion on the future release and support
strategy for Hadoop. This includes a few important topics that I believe we
should align on:
1. Ongoing Support for the Hadoop 3.4 Branch
As it stands, Hadoop 3.4 is the last release line that supports Java 8. It
is expected that Java 8 compilation support will be officially removed from
the trunk branch after a community vote.
At the same time, the trunk branch is increasingly diverging from the 3.4
line due to the adoption of JUnit 5 and new features based on JDK17 syntax.
As a result, future bug fixes and backports will become significantly more
difficult.
Given this, I’d like to ask: *after the release of hadoop-3.4.2, does the
community plan to continue maintaining and releasing hadoop-3.4.3 or other
3.4-series versions?*
2. Preparation for Hadoop 3.5 Release
With the JDK17 support work nearing completion, it’s time to start planning
for the Hadoop 3.5 release. Specific questions for discussion include:
-
Should we set up a new CI pipeline for JDK17 on the trunk branch?
-
Should we also add support for JDK21 in CI?
-
If JDK21 support is added, can we remove the existing JDK11 pipeline?
3. Operating System Support Strategy
Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
already reached or are approaching end-of-life (EOL).
Should we adjust our support policy for these systems? For example:
-
Officially deprecate EOL systems;
-
Provide documentation-only support for compilation;
-
Remove outdated base images from Dockerfiles.
4. Release Strategy Under Multi-JDK Builds
As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
strategy for future releases. Key questions include:
-
Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
-
What naming convention should we adopt for these variants?
-
For Maven Central artifacts, should we distinguish by JDK version or
consider a unified approach?
Additional Context and Suggestions from My Side:
1.
The JDK17 compatibility work is being tracked under HADOOP-17177
<https://issues.apache.org/jira/browse/HADOOP-17177>. I am in the
process of reviewing and linking all relevant JIRAs—both completed and
pending—to this umbrella JIRA, and will share an overview in the coming
week.
2.
Regarding the JUnit 5 migration:
Based on our experience, JUnit 5 introduces significant differences
compared to JUnit 4. It fundamentally changes how tests are written and
run. My proposed upgrade path is as follows:
-
Complete the migration to *JUnit 5.8.2* (we are still working on HDFS
and Hadoop-Azure and aim to finish in the next two weeks);
-
Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
Common, etc.);
-
Upgrade to *JUnit 5.13.2* (as Steve suggested);
-
Upgrade the *Maven Surefire Plugin*. Earlier issues were due to the
mixed use of JUnit 4 and 5, which will no longer be a concern
once JUnit 5
is fully adopted;
-
Upgrade the *Maven version* to 3.9.0 for better compatibility and
plugin support.
3.
Once the JUnit 5 migration is complete, we will conduct a thorough test
audit to ensure everything runs correctly and mark unstable tests for
future attention.
4.
Regarding legacy OS support, I recommend a "lightweight" approach:
-
Remove unsupported base images from our Dockerfiles or upgrade them
to newer versions;
-
Provide documentation to help users build on older systems;
-
But *formally discontinue* full support for EOL systems.
I’d appreciate hearing your thoughts on these topics. Looking forward to a
fruitful discussion!
Best regards,
Shilun Fan.
On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> Hi, Chris
>
> Thank you very much for your efforts in driving the JDK17 upgrade. I will
> continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>
> > Hello everyone,
>
> > I've mentioned in other forums that I'd volunteer as release manager for
> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
> > get more specific about release planning and a potential target date.
>
> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >
> support [1]. I'd like to start by summarizing my understanding of work done
> > and work remaining for Java 17 as well as other significant features the
> > community wants to ship in 3.5.0:
>
> > - We've completed the Jersey upgrade [2].
> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> > have a list of several PRs remaining to be reviewed and committed [4].
> > - There is some sun.misc
> usage that needs alternate implementations [5]
> > [6]. There are PRs, but they haven't had any recent activity.
> > - Once these issues are settled, we need to (re-)introduce a Java 17
> > build profile [7].
> > - I and a few others are also interested in trying to target the GCS
> > FileSystem implementation for 3.5.0 [8].
>
> > Does this look accurate? Did I miss anything important for 3.5.0, either
> > for Java 17 or other features?
>
> > Chris Nauroth
>
> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>
[DISCUSS] Future Release and Support Strategy for Hadoop
Posted by slfan1989 <sl...@apache.org>.
Dear community,
I'd like to initiate a discussion on the future release and support
strategy for Hadoop. This includes a few important topics that I believe we
should align on:
1. Ongoing Support for the Hadoop 3.4 Branch
As it stands, Hadoop 3.4 is the last release line that supports Java 8. It
is expected that Java 8 compilation support will be officially removed from
the trunk branch after a community vote.
At the same time, the trunk branch is increasingly diverging from the 3.4
line due to the adoption of JUnit 5 and new features based on JDK17 syntax.
As a result, future bug fixes and backports will become significantly more
difficult.
Given this, I’d like to ask: *after the release of hadoop-3.4.2, does the
community plan to continue maintaining and releasing hadoop-3.4.3 or other
3.4-series versions?*
2. Preparation for Hadoop 3.5 Release
With the JDK17 support work nearing completion, it’s time to start planning
for the Hadoop 3.5 release. Specific questions for discussion include:
-
Should we set up a new CI pipeline for JDK17 on the trunk branch?
-
Should we also add support for JDK21 in CI?
-
If JDK21 support is added, can we remove the existing JDK11 pipeline?
3. Operating System Support Strategy
Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have
already reached or are approaching end-of-life (EOL).
Should we adjust our support policy for these systems? For example:
-
Officially deprecate EOL systems;
-
Provide documentation-only support for compilation;
-
Remove outdated base images from Dockerfiles.
4. Release Strategy Under Multi-JDK Builds
As we prepare to support JDK11, JDK17, and JDK21, we also need a clear
strategy for future releases. Key questions include:
-
Should we publish multiple build variants like: hadoop-3.5.0-jdk11,
hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21?
-
What naming convention should we adopt for these variants?
-
For Maven Central artifacts, should we distinguish by JDK version or
consider a unified approach?
Additional Context and Suggestions from My Side:
1.
The JDK17 compatibility work is being tracked under HADOOP-17177
<https://issues.apache.org/jira/browse/HADOOP-17177>. I am in the
process of reviewing and linking all relevant JIRAs—both completed and
pending—to this umbrella JIRA, and will share an overview in the coming
week.
2.
Regarding the JUnit 5 migration:
Based on our experience, JUnit 5 introduces significant differences
compared to JUnit 4. It fundamentally changes how tests are written and
run. My proposed upgrade path is as follows:
-
Complete the migration to *JUnit 5.8.2* (we are still working on HDFS
and Hadoop-Azure and aim to finish in the next two weeks);
-
Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS,
Common, etc.);
-
Upgrade to *JUnit 5.13.2* (as Steve suggested);
-
Upgrade the *Maven Surefire Plugin*. Earlier issues were due to the
mixed use of JUnit 4 and 5, which will no longer be a concern
once JUnit 5
is fully adopted;
-
Upgrade the *Maven version* to 3.9.0 for better compatibility and
plugin support.
3.
Once the JUnit 5 migration is complete, we will conduct a thorough test
audit to ensure everything runs correctly and mark unstable tests for
future attention.
4.
Regarding legacy OS support, I recommend a "lightweight" approach:
-
Remove unsupported base images from our Dockerfiles or upgrade them
to newer versions;
-
Provide documentation to help users build on older systems;
-
But *formally discontinue* full support for EOL systems.
I’d appreciate hearing your thoughts on these topics. Looking forward to a
fruitful discussion!
Best regards,
Shilun Fan.
On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> Hi, Chris
>
> Thank you very much for your efforts in driving the JDK17 upgrade. I will
> continue to follow up on the JUnit5 migration, and I hope we can
> successfully complete this upgrade together.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <sl...@apache.org> wrote:
>
> > Hello everyone,
>
> > I've mentioned in other forums that I'd volunteer as release manager for
> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to
> > get more specific about release planning and a potential target date.
>
> > It can be easy to get lost in the nested historical JIRAs for Java 17
> >
> support [1]. I'd like to start by summarizing my understanding of work done
> > and work remaining for Java 17 as well as other significant features the
> > community wants to ship in 3.5.0:
>
> > - We've completed the Jersey upgrade [2].
> > - We've made significant progress on the JUnit 5 upgrade [3], and we
> > have a list of several PRs remaining to be reviewed and committed [4].
> > - There is some sun.misc
> usage that needs alternate implementations [5]
> > [6]. There are PRs, but they haven't had any recent activity.
> > - Once these issues are settled, we need to (re-)introduce a Java 17
> > build profile [7].
> > - I and a few others are also interested in trying to target the GCS
> > FileSystem implementation for 3.5.0 [8].
>
> > Does this look accurate? Did I miss anything important for 3.5.0, either
> > for Java 17 or other features?
>
> > Chris Nauroth
>
> > [1] https://issues.apache.org/jira/browse/HADOOP-17177
> > [2] https://issues.apache.org/jira/browse/HADOOP-15984
> > [3] https://issues.apache.org/jira/browse/HADOOP-14693
> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756
> > [5] https://issues.apache.org/jira/browse/HADOOP-19329
> > [6] https://issues.apache.org/jira/browse/HADOOP-19334
> > [7] https://issues.apache.org/jira/browse/HADOOP-19298
> > [8] https://issues.apache.org/jira/browse/HADOOP-19343
>