You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Ahmar Suhail <ah...@apache.org> on 2025/05/28 12:24:52 UTC
[VOTE] Release Apache Hadoop 3.4.2
Hey all,
The first release candidate for Hadoop 3.4.2 is now available for voting.
There are a couple of things to note:
1/ No Arm64 artifacts. This is due to previously reported issues:
https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
<https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
mean that the build fails on arm64.
2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
For S3A, download SDK bundle v2.29.52 from:
https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
and drop it into /share/hadoop/common/lib. Release notes will be updated
with these instructions.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
The git tag is release-3.4.2-RC1, commit
09870840ec35b48cd107972eb24d25e8aece04c9
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1437
You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
This release has been created off of branch-3.4. Key changes include:
* S3A: Integration with S3 Analytics Accelerator input stream
* S3A: Support for S3 conditional writes
* ABFS: Deprecation of WASB driver
* ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
This is my first attempt at managing a release, please do test the release
and let me know in case of any issues.
Thanks,
Ahmar
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
regarding parquet, the jira there is :
https://github.com/apache/parquet-java/issues/3237
and while the change in that lib is "release the buffers", it hits a
problem with coalesced reads as there the releasing doesn't work.
my fix will be: disable coalescing, keep the code (for now), and add a
path/stream capability to indicate merging.
with a max size for range merging of 2MB, it'll only surface in production
if ORC/parquet had rowgroups of size < 2MB, so not realistic at all. But it
exists.
On Fri, 6 Jun 2025 at 09:54, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Thanks everyone for testing out this RC.
>
>
> For this RC, the current status is:
>
>
> * Issue with buffers allocated by ParquetFileReader.readVectored() are
> not being released. While this is not new, we should root cause and fix in
> case it's a hadoop issue for 3.4.2.
> * Need to figure out what to do with AWS SDK bundle, the transitive
> dependency should ideally be restored. I will discuss with Steve.
> * Documentation updates - I will fix.
> * PR's are up for the YARN issues which was failing the ARM64 builds,
> thank you Masatake, so the next RC will include ARM64 binaries.
>
>
>
> Once the parquet buffer issue is resolved, I will begin work on RC-2. If
> there are any other issues in this RC that I've missed in the above list
> please let me know so I can be sure to include the fix in the next RC.
>
>
> Thank you,
> Ahmar
>
> ________________________________
> From: Masatake Iwasaki <iw...@oss.nttdata.com>
> Sent: Thursday, June 5, 2025 4:46:06 PM
> To: Wei-Chiu Chuang; Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi Wei-Chiu Chuang,
>
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
>
> I submitted PRs for YARN-11712 and YARN-11713.
> Could you check the patches?
>
> Thanks,
> Masatake Iwasaki
>
> On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
> >
> > On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> >
> >> Hey all,
> >>
> >> The first release candidate for Hadoop 3.4.2 is now available for
> voting.
> >>
> >> There are a couple of things to note:
> >>
> >> 1/ No Arm64 artifacts. This is due to previously reported issues:
> >> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> >> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> >> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> >> mean that the build fails on arm64.
> >>
> >> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> >> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> >> which makes the size of tar > 1GB, and it can no longer be uploaded to
> SVN.
> >> For S3A, download SDK bundle v2.29.52 from:
> >>
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> >> and drop it into /share/hadoop/common/lib. Release notes will be updated
> >> with these instructions.
> >>
> >>
> >> The RC is available at:
> >>
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
> >>
> >> The git tag is release-3.4.2-RC1, commit
> >> 09870840ec35b48cd107972eb24d25e8aece04c9
> >>
> >> The maven artifacts are staged at:
> >>
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1437
> >>
> >>
> >> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
> at:
> >>
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> This release has been created off of branch-3.4. Key changes include:
> >>
> >> * S3A: Integration with S3 Analytics Accelerator input stream
> >> * S3A: Support for S3 conditional writes
> >> * ABFS: Deprecation of WASB driver
> >> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
> >>
> >>
> >> This is my first attempt at managing a release, please do test the
> release
> >> and let me know in case of any issues.
> >>
> >> Thanks,
> >> Ahmar
> >>
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
regarding parquet, the jira there is :
https://github.com/apache/parquet-java/issues/3237
and while the change in that lib is "release the buffers", it hits a
problem with coalesced reads as there the releasing doesn't work.
my fix will be: disable coalescing, keep the code (for now), and add a
path/stream capability to indicate merging.
with a max size for range merging of 2MB, it'll only surface in production
if ORC/parquet had rowgroups of size < 2MB, so not realistic at all. But it
exists.
On Fri, 6 Jun 2025 at 09:54, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Thanks everyone for testing out this RC.
>
>
> For this RC, the current status is:
>
>
> * Issue with buffers allocated by ParquetFileReader.readVectored() are
> not being released. While this is not new, we should root cause and fix in
> case it's a hadoop issue for 3.4.2.
> * Need to figure out what to do with AWS SDK bundle, the transitive
> dependency should ideally be restored. I will discuss with Steve.
> * Documentation updates - I will fix.
> * PR's are up for the YARN issues which was failing the ARM64 builds,
> thank you Masatake, so the next RC will include ARM64 binaries.
>
>
>
> Once the parquet buffer issue is resolved, I will begin work on RC-2. If
> there are any other issues in this RC that I've missed in the above list
> please let me know so I can be sure to include the fix in the next RC.
>
>
> Thank you,
> Ahmar
>
> ________________________________
> From: Masatake Iwasaki <iw...@oss.nttdata.com>
> Sent: Thursday, June 5, 2025 4:46:06 PM
> To: Wei-Chiu Chuang; Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi Wei-Chiu Chuang,
>
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
>
> I submitted PRs for YARN-11712 and YARN-11713.
> Could you check the patches?
>
> Thanks,
> Masatake Iwasaki
>
> On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
> >
> > On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> >
> >> Hey all,
> >>
> >> The first release candidate for Hadoop 3.4.2 is now available for
> voting.
> >>
> >> There are a couple of things to note:
> >>
> >> 1/ No Arm64 artifacts. This is due to previously reported issues:
> >> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> >> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> >> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> >> mean that the build fails on arm64.
> >>
> >> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> >> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> >> which makes the size of tar > 1GB, and it can no longer be uploaded to
> SVN.
> >> For S3A, download SDK bundle v2.29.52 from:
> >>
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> >> and drop it into /share/hadoop/common/lib. Release notes will be updated
> >> with these instructions.
> >>
> >>
> >> The RC is available at:
> >>
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
> >>
> >> The git tag is release-3.4.2-RC1, commit
> >> 09870840ec35b48cd107972eb24d25e8aece04c9
> >>
> >> The maven artifacts are staged at:
> >>
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1437
> >>
> >>
> >> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
> at:
> >>
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> This release has been created off of branch-3.4. Key changes include:
> >>
> >> * S3A: Integration with S3 Analytics Accelerator input stream
> >> * S3A: Support for S3 conditional writes
> >> * ABFS: Deprecation of WASB driver
> >> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
> >>
> >>
> >> This is my first attempt at managing a release, please do test the
> release
> >> and let me know in case of any issues.
> >>
> >> Thanks,
> >> Ahmar
> >>
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
regarding parquet, the jira there is :
https://github.com/apache/parquet-java/issues/3237
and while the change in that lib is "release the buffers", it hits a
problem with coalesced reads as there the releasing doesn't work.
my fix will be: disable coalescing, keep the code (for now), and add a
path/stream capability to indicate merging.
with a max size for range merging of 2MB, it'll only surface in production
if ORC/parquet had rowgroups of size < 2MB, so not realistic at all. But it
exists.
On Fri, 6 Jun 2025 at 09:54, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Thanks everyone for testing out this RC.
>
>
> For this RC, the current status is:
>
>
> * Issue with buffers allocated by ParquetFileReader.readVectored() are
> not being released. While this is not new, we should root cause and fix in
> case it's a hadoop issue for 3.4.2.
> * Need to figure out what to do with AWS SDK bundle, the transitive
> dependency should ideally be restored. I will discuss with Steve.
> * Documentation updates - I will fix.
> * PR's are up for the YARN issues which was failing the ARM64 builds,
> thank you Masatake, so the next RC will include ARM64 binaries.
>
>
>
> Once the parquet buffer issue is resolved, I will begin work on RC-2. If
> there are any other issues in this RC that I've missed in the above list
> please let me know so I can be sure to include the fix in the next RC.
>
>
> Thank you,
> Ahmar
>
> ________________________________
> From: Masatake Iwasaki <iw...@oss.nttdata.com>
> Sent: Thursday, June 5, 2025 4:46:06 PM
> To: Wei-Chiu Chuang; Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi Wei-Chiu Chuang,
>
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
>
> I submitted PRs for YARN-11712 and YARN-11713.
> Could you check the patches?
>
> Thanks,
> Masatake Iwasaki
>
> On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
> >
> > On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> >
> >> Hey all,
> >>
> >> The first release candidate for Hadoop 3.4.2 is now available for
> voting.
> >>
> >> There are a couple of things to note:
> >>
> >> 1/ No Arm64 artifacts. This is due to previously reported issues:
> >> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> >> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> >> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> >> mean that the build fails on arm64.
> >>
> >> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> >> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> >> which makes the size of tar > 1GB, and it can no longer be uploaded to
> SVN.
> >> For S3A, download SDK bundle v2.29.52 from:
> >>
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> >> and drop it into /share/hadoop/common/lib. Release notes will be updated
> >> with these instructions.
> >>
> >>
> >> The RC is available at:
> >>
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
> >>
> >> The git tag is release-3.4.2-RC1, commit
> >> 09870840ec35b48cd107972eb24d25e8aece04c9
> >>
> >> The maven artifacts are staged at:
> >>
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1437
> >>
> >>
> >> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
> at:
> >>
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> This release has been created off of branch-3.4. Key changes include:
> >>
> >> * S3A: Integration with S3 Analytics Accelerator input stream
> >> * S3A: Support for S3 conditional writes
> >> * ABFS: Deprecation of WASB driver
> >> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
> >>
> >>
> >> This is my first attempt at managing a release, please do test the
> release
> >> and let me know in case of any issues.
> >>
> >> Thanks,
> >> Ahmar
> >>
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
regarding parquet, the jira there is :
https://github.com/apache/parquet-java/issues/3237
and while the change in that lib is "release the buffers", it hits a
problem with coalesced reads as there the releasing doesn't work.
my fix will be: disable coalescing, keep the code (for now), and add a
path/stream capability to indicate merging.
with a max size for range merging of 2MB, it'll only surface in production
if ORC/parquet had rowgroups of size < 2MB, so not realistic at all. But it
exists.
On Fri, 6 Jun 2025 at 09:54, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Thanks everyone for testing out this RC.
>
>
> For this RC, the current status is:
>
>
> * Issue with buffers allocated by ParquetFileReader.readVectored() are
> not being released. While this is not new, we should root cause and fix in
> case it's a hadoop issue for 3.4.2.
> * Need to figure out what to do with AWS SDK bundle, the transitive
> dependency should ideally be restored. I will discuss with Steve.
> * Documentation updates - I will fix.
> * PR's are up for the YARN issues which was failing the ARM64 builds,
> thank you Masatake, so the next RC will include ARM64 binaries.
>
>
>
> Once the parquet buffer issue is resolved, I will begin work on RC-2. If
> there are any other issues in this RC that I've missed in the above list
> please let me know so I can be sure to include the fix in the next RC.
>
>
> Thank you,
> Ahmar
>
> ________________________________
> From: Masatake Iwasaki <iw...@oss.nttdata.com>
> Sent: Thursday, June 5, 2025 4:46:06 PM
> To: Wei-Chiu Chuang; Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi Wei-Chiu Chuang,
>
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
>
> I submitted PRs for YARN-11712 and YARN-11713.
> Could you check the patches?
>
> Thanks,
> Masatake Iwasaki
>
> On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> > I really wish we can make ARM64 binaries. Would like to find time to work
> > on the two jiras mentioned.
> >
> > On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> >
> >> Hey all,
> >>
> >> The first release candidate for Hadoop 3.4.2 is now available for
> voting.
> >>
> >> There are a couple of things to note:
> >>
> >> 1/ No Arm64 artifacts. This is due to previously reported issues:
> >> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> >> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> >> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> >> mean that the build fails on arm64.
> >>
> >> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> >> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> >> which makes the size of tar > 1GB, and it can no longer be uploaded to
> SVN.
> >> For S3A, download SDK bundle v2.29.52 from:
> >>
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> >> and drop it into /share/hadoop/common/lib. Release notes will be updated
> >> with these instructions.
> >>
> >>
> >> The RC is available at:
> >>
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
> >>
> >> The git tag is release-3.4.2-RC1, commit
> >> 09870840ec35b48cd107972eb24d25e8aece04c9
> >>
> >> The maven artifacts are staged at:
> >>
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1437
> >>
> >>
> >> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
> at:
> >>
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> This release has been created off of branch-3.4. Key changes include:
> >>
> >> * S3A: Integration with S3 Analytics Accelerator input stream
> >> * S3A: Support for S3 conditional writes
> >> * ABFS: Deprecation of WASB driver
> >> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
> >>
> >>
> >> This is my first attempt at managing a release, please do test the
> release
> >> and let me know in case of any issues.
> >>
> >> Thanks,
> >> Ahmar
> >>
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Thanks everyone for testing out this RC.
For this RC, the current status is:
* Issue with buffers allocated by ParquetFileReader.readVectored() are not being released. While this is not new, we should root cause and fix in case it's a hadoop issue for 3.4.2.
* Need to figure out what to do with AWS SDK bundle, the transitive dependency should ideally be restored. I will discuss with Steve.
* Documentation updates - I will fix.
* PR's are up for the YARN issues which was failing the ARM64 builds, thank you Masatake, so the next RC will include ARM64 binaries.
Once the parquet buffer issue is resolved, I will begin work on RC-2. If there are any other issues in this RC that I've missed in the above list please let me know so I can be sure to include the fix in the next RC.
Thank you,
Ahmar
________________________________
From: Masatake Iwasaki <iw...@oss.nttdata.com>
Sent: Thursday, June 5, 2025 4:46:06 PM
To: Wei-Chiu Chuang; Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Thanks everyone for testing out this RC.
For this RC, the current status is:
* Issue with buffers allocated by ParquetFileReader.readVectored() are not being released. While this is not new, we should root cause and fix in case it's a hadoop issue for 3.4.2.
* Need to figure out what to do with AWS SDK bundle, the transitive dependency should ideally be restored. I will discuss with Steve.
* Documentation updates - I will fix.
* PR's are up for the YARN issues which was failing the ARM64 builds, thank you Masatake, so the next RC will include ARM64 binaries.
Once the parquet buffer issue is resolved, I will begin work on RC-2. If there are any other issues in this RC that I've missed in the above list please let me know so I can be sure to include the fix in the next RC.
Thank you,
Ahmar
________________________________
From: Masatake Iwasaki <iw...@oss.nttdata.com>
Sent: Thursday, June 5, 2025 4:46:06 PM
To: Wei-Chiu Chuang; Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Thanks everyone for testing out this RC.
For this RC, the current status is:
* Issue with buffers allocated by ParquetFileReader.readVectored() are not being released. While this is not new, we should root cause and fix in case it's a hadoop issue for 3.4.2.
* Need to figure out what to do with AWS SDK bundle, the transitive dependency should ideally be restored. I will discuss with Steve.
* Documentation updates - I will fix.
* PR's are up for the YARN issues which was failing the ARM64 builds, thank you Masatake, so the next RC will include ARM64 binaries.
Once the parquet buffer issue is resolved, I will begin work on RC-2. If there are any other issues in this RC that I've missed in the above list please let me know so I can be sure to include the fix in the next RC.
Thank you,
Ahmar
________________________________
From: Masatake Iwasaki <iw...@oss.nttdata.com>
Sent: Thursday, June 5, 2025 4:46:06 PM
To: Wei-Chiu Chuang; Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Thanks everyone for testing out this RC.
For this RC, the current status is:
* Issue with buffers allocated by ParquetFileReader.readVectored() are not being released. While this is not new, we should root cause and fix in case it's a hadoop issue for 3.4.2.
* Need to figure out what to do with AWS SDK bundle, the transitive dependency should ideally be restored. I will discuss with Steve.
* Documentation updates - I will fix.
* PR's are up for the YARN issues which was failing the ARM64 builds, thank you Masatake, so the next RC will include ARM64 binaries.
Once the parquet buffer issue is resolved, I will begin work on RC-2. If there are any other issues in this RC that I've missed in the above list please let me know so I can be sure to include the fix in the next RC.
Thank you,
Ahmar
________________________________
From: Masatake Iwasaki <iw...@oss.nttdata.com>
Sent: Thursday, June 5, 2025 4:46:06 PM
To: Wei-Chiu Chuang; Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi Wei-Chiu Chuang,
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
I submitted PRs for YARN-11712 and YARN-11713.
Could you check the patches?
Thanks,
Masatake Iwasaki
On 2025/06/05 1:14, Wei-Chiu Chuang wrote:
> I really wish we can make ARM64 binaries. Would like to find time to work
> on the two jiras mentioned.
>
> On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
>
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Wei-Chiu Chuang <we...@apache.org>.
I really wish we can make ARM64 binaries. Would like to find time to work
on the two jiras mentioned.
On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
and I've done the bytebuffer flip for read vectored
https://github.com/apache/hadoop/pull/7725
but .flip() is used in so many other places, it's not enough. Given that
the overloaded flip() method appears in a lot of the jdk8 releases.
So how about "this releases uses the overloaded ByteByffer.flip() method,
which is absent from very old JDK releases. If you encounter the error "
java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;" you need to upgrade to a
more recent version of java 8 or later.
On Wed, 4 Jun 2025 at 16:45, Steve Loughran <st...@cloudera.com> wrote:
> parquet stuff is unrelated, filed
> https://github.com/apache/parquet-java/issues/3237 . still my fault
> though.
>
> overall, I'm going to give a quick -1 with the commitment to provide one
> of the patches.
>
> my comments in markdown format:
>
>
> ## relesse docs index.html
>
> the summary in index.html are from 3.4.1, so we need to update them to
> 3.4.2 with key features anre issues fixed.
>
>
> Call out avro upgrade in the release notes -this is incompatible but
> required for security reasons.
>
> > We have upgraded to Avro 1.11.4, which is incompatible with previous
> versions.
> { This is critical for security reasons -everyone needs to upgrade their
> own uses of Avro too.
>
> A side-effect of excluding the AWS SDK from the release is that the
> hadoop-aws module does not
> declare a transitive dependency on the AWS SDK. This means that if you use
> hadoop-aws, you must explicitly include the AWS SDK in your project.
>
> I'd like to see this (somehow) returned, at the very least we explicitly
> add it into the hadoop-cloud-connectors dependencies so you can get it that
> way.
>
> otherwise it's a bit trickier to get right
>
> doc changes
>
> > This release was qualified against the 2.29.52 release, which at the
> time of the qualification was the last release compatible with third-party
> stores. This regression may have been fixed in later releases.
>
> <dependency>
> <groupId>software.amazon.awssdk</groupId>
> <artifactId>bundle</artifactId>
> <version>2.29.52</version>
> <exclusions>
> <exclusion>
> <groupId>*</groupId>
> <artifactId>*</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
>
>
>
> > The minimum version of the AWS SDK bundle.jar which can be used in this
> release is 2.29.52; later versions may work, but no later versions have
> been qualified. Upgrade with care, especially when working with third-party
> stores.
>
> > see: https://issues.apache.org/jira/browse/HADOOP-19490
>
> ## java8
>
> my setup has an old java8 release
>
> java -version
> openjdk version "1.8.0_362"
> OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 1.8.0_362-b09)
> OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 25.362-b09, mixed mode)
>
> it is getting that recurrent bytebuffer flip error, from an overloaded
> version of ByteBuffer...this can happen even with recent java8 releases.
>
> We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
>
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.parquet.hadoop.TestParquetReader
> Exception in thread "Thread-8" java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at
> sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
>
> this can actually be fixed trivially, casting to java.io.Buffer first:
> ((Byffer)buff).flip())
>
> I'll do this
>
> On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> I'm sort of leaning towards a -1, though the regression is actually in
>> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>>
>>
>> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
>> -- Time elapsed: 0.033 s <<< ERROR!
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
>> 4 ByteBuffer object(s) is/are remained unreleased after closing this
>> allocator.
>> at
>> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
>> at
>> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>> Caused by:
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
>> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
>> more info
>>
>>
>> 1. This is very much the vector io stuff.
>> 2. It's not a regression since 3.4.0
>> 3. but it's indicative of a memory leak. Surprised and Annoyed I
>> hadn't spotted this before...the tests went in last year.
>>
>>
>> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>>
>>>
>>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>>> other people can revoke if you break things.
>>>
>>> will do the merge and test
>>>
>>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>>> wrote:
>>>
>>>> Steve - I created a PR on hadoop-release-support with my properties
>>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>>
>>>>
>>>> Masatake - Yes, sounds good, will update the documentation for the new
>>>> RC if it is created.
>>>>
>>>>
>>>> Mukund - I realised too late that I messed up the commit for that one.
>>>> Will figure out how to fix..
>>>>
>>>> ________________________________
>>>> From: Steve Loughran <st...@cloudera.com>
>>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>>> To: Ahmar Suhail
>>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not
>>>> click links or open attachments unless you can confirm the sender and know
>>>> the content is safe.
>>>>
>>>>
>>>>
>>>> can you put up the relevant changes to the hadoop-release-support
>>>> module; I'd like to use it as part of my validation and I'm assuming you
>>>> have that src/releases/release-info-3.4.2.properties file
>>>>
>>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>>> ahmar@apache.org>> wrote:
>>>> Hey all,
>>>>
>>>> The first release candidate for Hadoop 3.4.2 is now available for
>>>> voting.
>>>>
>>>> There are a couple of things to note:
>>>>
>>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>>> mean that the build fails on arm64.
>>>>
>>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>>> SVN.
>>>> For S3A, download SDK bundle v2.29.52 from:
>>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>>> ,
>>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>>> with these instructions.
>>>>
>>>>
>>>> The RC is available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>>
>>>> The git tag is release-3.4.2-RC1, commit
>>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>>
>>>> The maven artifacts are staged at:
>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>>
>>>>
>>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
>>>> at:
>>>>
>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>
>>>>
>>>> This release has been created off of branch-3.4. Key changes include:
>>>>
>>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>>> * S3A: Support for S3 conditional writes
>>>> * ABFS: Deprecation of WASB driver
>>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>>
>>>>
>>>> This is my first attempt at managing a release, please do test the
>>>> release
>>>> and let me know in case of any issues.
>>>>
>>>> Thanks,
>>>> Ahmar
>>>>
>>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
and I've done the bytebuffer flip for read vectored
https://github.com/apache/hadoop/pull/7725
but .flip() is used in so many other places, it's not enough. Given that
the overloaded flip() method appears in a lot of the jdk8 releases.
So how about "this releases uses the overloaded ByteByffer.flip() method,
which is absent from very old JDK releases. If you encounter the error "
java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;" you need to upgrade to a
more recent version of java 8 or later.
On Wed, 4 Jun 2025 at 16:45, Steve Loughran <st...@cloudera.com> wrote:
> parquet stuff is unrelated, filed
> https://github.com/apache/parquet-java/issues/3237 . still my fault
> though.
>
> overall, I'm going to give a quick -1 with the commitment to provide one
> of the patches.
>
> my comments in markdown format:
>
>
> ## relesse docs index.html
>
> the summary in index.html are from 3.4.1, so we need to update them to
> 3.4.2 with key features anre issues fixed.
>
>
> Call out avro upgrade in the release notes -this is incompatible but
> required for security reasons.
>
> > We have upgraded to Avro 1.11.4, which is incompatible with previous
> versions.
> { This is critical for security reasons -everyone needs to upgrade their
> own uses of Avro too.
>
> A side-effect of excluding the AWS SDK from the release is that the
> hadoop-aws module does not
> declare a transitive dependency on the AWS SDK. This means that if you use
> hadoop-aws, you must explicitly include the AWS SDK in your project.
>
> I'd like to see this (somehow) returned, at the very least we explicitly
> add it into the hadoop-cloud-connectors dependencies so you can get it that
> way.
>
> otherwise it's a bit trickier to get right
>
> doc changes
>
> > This release was qualified against the 2.29.52 release, which at the
> time of the qualification was the last release compatible with third-party
> stores. This regression may have been fixed in later releases.
>
> <dependency>
> <groupId>software.amazon.awssdk</groupId>
> <artifactId>bundle</artifactId>
> <version>2.29.52</version>
> <exclusions>
> <exclusion>
> <groupId>*</groupId>
> <artifactId>*</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
>
>
>
> > The minimum version of the AWS SDK bundle.jar which can be used in this
> release is 2.29.52; later versions may work, but no later versions have
> been qualified. Upgrade with care, especially when working with third-party
> stores.
>
> > see: https://issues.apache.org/jira/browse/HADOOP-19490
>
> ## java8
>
> my setup has an old java8 release
>
> java -version
> openjdk version "1.8.0_362"
> OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 1.8.0_362-b09)
> OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 25.362-b09, mixed mode)
>
> it is getting that recurrent bytebuffer flip error, from an overloaded
> version of ByteBuffer...this can happen even with recent java8 releases.
>
> We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
>
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.parquet.hadoop.TestParquetReader
> Exception in thread "Thread-8" java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at
> sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
>
> this can actually be fixed trivially, casting to java.io.Buffer first:
> ((Byffer)buff).flip())
>
> I'll do this
>
> On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> I'm sort of leaning towards a -1, though the regression is actually in
>> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>>
>>
>> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
>> -- Time elapsed: 0.033 s <<< ERROR!
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
>> 4 ByteBuffer object(s) is/are remained unreleased after closing this
>> allocator.
>> at
>> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
>> at
>> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>> Caused by:
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
>> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
>> more info
>>
>>
>> 1. This is very much the vector io stuff.
>> 2. It's not a regression since 3.4.0
>> 3. but it's indicative of a memory leak. Surprised and Annoyed I
>> hadn't spotted this before...the tests went in last year.
>>
>>
>> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>>
>>>
>>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>>> other people can revoke if you break things.
>>>
>>> will do the merge and test
>>>
>>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>>> wrote:
>>>
>>>> Steve - I created a PR on hadoop-release-support with my properties
>>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>>
>>>>
>>>> Masatake - Yes, sounds good, will update the documentation for the new
>>>> RC if it is created.
>>>>
>>>>
>>>> Mukund - I realised too late that I messed up the commit for that one.
>>>> Will figure out how to fix..
>>>>
>>>> ________________________________
>>>> From: Steve Loughran <st...@cloudera.com>
>>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>>> To: Ahmar Suhail
>>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not
>>>> click links or open attachments unless you can confirm the sender and know
>>>> the content is safe.
>>>>
>>>>
>>>>
>>>> can you put up the relevant changes to the hadoop-release-support
>>>> module; I'd like to use it as part of my validation and I'm assuming you
>>>> have that src/releases/release-info-3.4.2.properties file
>>>>
>>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>>> ahmar@apache.org>> wrote:
>>>> Hey all,
>>>>
>>>> The first release candidate for Hadoop 3.4.2 is now available for
>>>> voting.
>>>>
>>>> There are a couple of things to note:
>>>>
>>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>>> mean that the build fails on arm64.
>>>>
>>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>>> SVN.
>>>> For S3A, download SDK bundle v2.29.52 from:
>>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>>> ,
>>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>>> with these instructions.
>>>>
>>>>
>>>> The RC is available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>>
>>>> The git tag is release-3.4.2-RC1, commit
>>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>>
>>>> The maven artifacts are staged at:
>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>>
>>>>
>>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
>>>> at:
>>>>
>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>
>>>>
>>>> This release has been created off of branch-3.4. Key changes include:
>>>>
>>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>>> * S3A: Support for S3 conditional writes
>>>> * ABFS: Deprecation of WASB driver
>>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>>
>>>>
>>>> This is my first attempt at managing a release, please do test the
>>>> release
>>>> and let me know in case of any issues.
>>>>
>>>> Thanks,
>>>> Ahmar
>>>>
>>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
and I've done the bytebuffer flip for read vectored
https://github.com/apache/hadoop/pull/7725
but .flip() is used in so many other places, it's not enough. Given that
the overloaded flip() method appears in a lot of the jdk8 releases.
So how about "this releases uses the overloaded ByteByffer.flip() method,
which is absent from very old JDK releases. If you encounter the error "
java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;" you need to upgrade to a
more recent version of java 8 or later.
On Wed, 4 Jun 2025 at 16:45, Steve Loughran <st...@cloudera.com> wrote:
> parquet stuff is unrelated, filed
> https://github.com/apache/parquet-java/issues/3237 . still my fault
> though.
>
> overall, I'm going to give a quick -1 with the commitment to provide one
> of the patches.
>
> my comments in markdown format:
>
>
> ## relesse docs index.html
>
> the summary in index.html are from 3.4.1, so we need to update them to
> 3.4.2 with key features anre issues fixed.
>
>
> Call out avro upgrade in the release notes -this is incompatible but
> required for security reasons.
>
> > We have upgraded to Avro 1.11.4, which is incompatible with previous
> versions.
> { This is critical for security reasons -everyone needs to upgrade their
> own uses of Avro too.
>
> A side-effect of excluding the AWS SDK from the release is that the
> hadoop-aws module does not
> declare a transitive dependency on the AWS SDK. This means that if you use
> hadoop-aws, you must explicitly include the AWS SDK in your project.
>
> I'd like to see this (somehow) returned, at the very least we explicitly
> add it into the hadoop-cloud-connectors dependencies so you can get it that
> way.
>
> otherwise it's a bit trickier to get right
>
> doc changes
>
> > This release was qualified against the 2.29.52 release, which at the
> time of the qualification was the last release compatible with third-party
> stores. This regression may have been fixed in later releases.
>
> <dependency>
> <groupId>software.amazon.awssdk</groupId>
> <artifactId>bundle</artifactId>
> <version>2.29.52</version>
> <exclusions>
> <exclusion>
> <groupId>*</groupId>
> <artifactId>*</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
>
>
>
> > The minimum version of the AWS SDK bundle.jar which can be used in this
> release is 2.29.52; later versions may work, but no later versions have
> been qualified. Upgrade with care, especially when working with third-party
> stores.
>
> > see: https://issues.apache.org/jira/browse/HADOOP-19490
>
> ## java8
>
> my setup has an old java8 release
>
> java -version
> openjdk version "1.8.0_362"
> OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 1.8.0_362-b09)
> OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 25.362-b09, mixed mode)
>
> it is getting that recurrent bytebuffer flip error, from an overloaded
> version of ByteBuffer...this can happen even with recent java8 releases.
>
> We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
>
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.parquet.hadoop.TestParquetReader
> Exception in thread "Thread-8" java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at
> sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
>
> this can actually be fixed trivially, casting to java.io.Buffer first:
> ((Byffer)buff).flip())
>
> I'll do this
>
> On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> I'm sort of leaning towards a -1, though the regression is actually in
>> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>>
>>
>> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
>> -- Time elapsed: 0.033 s <<< ERROR!
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
>> 4 ByteBuffer object(s) is/are remained unreleased after closing this
>> allocator.
>> at
>> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
>> at
>> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>> Caused by:
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
>> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
>> more info
>>
>>
>> 1. This is very much the vector io stuff.
>> 2. It's not a regression since 3.4.0
>> 3. but it's indicative of a memory leak. Surprised and Annoyed I
>> hadn't spotted this before...the tests went in last year.
>>
>>
>> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>>
>>>
>>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>>> other people can revoke if you break things.
>>>
>>> will do the merge and test
>>>
>>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>>> wrote:
>>>
>>>> Steve - I created a PR on hadoop-release-support with my properties
>>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>>
>>>>
>>>> Masatake - Yes, sounds good, will update the documentation for the new
>>>> RC if it is created.
>>>>
>>>>
>>>> Mukund - I realised too late that I messed up the commit for that one.
>>>> Will figure out how to fix..
>>>>
>>>> ________________________________
>>>> From: Steve Loughran <st...@cloudera.com>
>>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>>> To: Ahmar Suhail
>>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not
>>>> click links or open attachments unless you can confirm the sender and know
>>>> the content is safe.
>>>>
>>>>
>>>>
>>>> can you put up the relevant changes to the hadoop-release-support
>>>> module; I'd like to use it as part of my validation and I'm assuming you
>>>> have that src/releases/release-info-3.4.2.properties file
>>>>
>>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>>> ahmar@apache.org>> wrote:
>>>> Hey all,
>>>>
>>>> The first release candidate for Hadoop 3.4.2 is now available for
>>>> voting.
>>>>
>>>> There are a couple of things to note:
>>>>
>>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>>> mean that the build fails on arm64.
>>>>
>>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>>> SVN.
>>>> For S3A, download SDK bundle v2.29.52 from:
>>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>>> ,
>>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>>> with these instructions.
>>>>
>>>>
>>>> The RC is available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>>
>>>> The git tag is release-3.4.2-RC1, commit
>>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>>
>>>> The maven artifacts are staged at:
>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>>
>>>>
>>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
>>>> at:
>>>>
>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>
>>>>
>>>> This release has been created off of branch-3.4. Key changes include:
>>>>
>>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>>> * S3A: Support for S3 conditional writes
>>>> * ABFS: Deprecation of WASB driver
>>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>>
>>>>
>>>> This is my first attempt at managing a release, please do test the
>>>> release
>>>> and let me know in case of any issues.
>>>>
>>>> Thanks,
>>>> Ahmar
>>>>
>>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
and I've done the bytebuffer flip for read vectored
https://github.com/apache/hadoop/pull/7725
but .flip() is used in so many other places, it's not enough. Given that
the overloaded flip() method appears in a lot of the jdk8 releases.
So how about "this releases uses the overloaded ByteByffer.flip() method,
which is absent from very old JDK releases. If you encounter the error "
java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;" you need to upgrade to a
more recent version of java 8 or later.
On Wed, 4 Jun 2025 at 16:45, Steve Loughran <st...@cloudera.com> wrote:
> parquet stuff is unrelated, filed
> https://github.com/apache/parquet-java/issues/3237 . still my fault
> though.
>
> overall, I'm going to give a quick -1 with the commitment to provide one
> of the patches.
>
> my comments in markdown format:
>
>
> ## relesse docs index.html
>
> the summary in index.html are from 3.4.1, so we need to update them to
> 3.4.2 with key features anre issues fixed.
>
>
> Call out avro upgrade in the release notes -this is incompatible but
> required for security reasons.
>
> > We have upgraded to Avro 1.11.4, which is incompatible with previous
> versions.
> { This is critical for security reasons -everyone needs to upgrade their
> own uses of Avro too.
>
> A side-effect of excluding the AWS SDK from the release is that the
> hadoop-aws module does not
> declare a transitive dependency on the AWS SDK. This means that if you use
> hadoop-aws, you must explicitly include the AWS SDK in your project.
>
> I'd like to see this (somehow) returned, at the very least we explicitly
> add it into the hadoop-cloud-connectors dependencies so you can get it that
> way.
>
> otherwise it's a bit trickier to get right
>
> doc changes
>
> > This release was qualified against the 2.29.52 release, which at the
> time of the qualification was the last release compatible with third-party
> stores. This regression may have been fixed in later releases.
>
> <dependency>
> <groupId>software.amazon.awssdk</groupId>
> <artifactId>bundle</artifactId>
> <version>2.29.52</version>
> <exclusions>
> <exclusion>
> <groupId>*</groupId>
> <artifactId>*</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
>
>
>
> > The minimum version of the AWS SDK bundle.jar which can be used in this
> release is 2.29.52; later versions may work, but no later versions have
> been qualified. Upgrade with care, especially when working with third-party
> stores.
>
> > see: https://issues.apache.org/jira/browse/HADOOP-19490
>
> ## java8
>
> my setup has an old java8 release
>
> java -version
> openjdk version "1.8.0_362"
> OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 1.8.0_362-b09)
> OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
> 25.362-b09, mixed mode)
>
> it is getting that recurrent bytebuffer flip error, from an overloaded
> version of ByteBuffer...this can happen even with recent java8 releases.
>
> We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
>
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.parquet.hadoop.TestParquetReader
> Exception in thread "Thread-8" java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at
> sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
>
> this can actually be fixed trivially, casting to java.io.Buffer first:
> ((Byffer)buff).flip())
>
> I'll do this
>
> On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> I'm sort of leaning towards a -1, though the regression is actually in
>> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>>
>>
>> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
>> -- Time elapsed: 0.033 s <<< ERROR!
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
>> 4 ByteBuffer object(s) is/are remained unreleased after closing this
>> allocator.
>> at
>> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
>> at
>> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>> Caused by:
>> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
>> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
>> more info
>>
>>
>> 1. This is very much the vector io stuff.
>> 2. It's not a regression since 3.4.0
>> 3. but it's indicative of a memory leak. Surprised and Annoyed I
>> hadn't spotted this before...the tests went in last year.
>>
>>
>> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>>
>>>
>>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>>> other people can revoke if you break things.
>>>
>>> will do the merge and test
>>>
>>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>>> wrote:
>>>
>>>> Steve - I created a PR on hadoop-release-support with my properties
>>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>>
>>>>
>>>> Masatake - Yes, sounds good, will update the documentation for the new
>>>> RC if it is created.
>>>>
>>>>
>>>> Mukund - I realised too late that I messed up the commit for that one.
>>>> Will figure out how to fix..
>>>>
>>>> ________________________________
>>>> From: Steve Loughran <st...@cloudera.com>
>>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>>> To: Ahmar Suhail
>>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not
>>>> click links or open attachments unless you can confirm the sender and know
>>>> the content is safe.
>>>>
>>>>
>>>>
>>>> can you put up the relevant changes to the hadoop-release-support
>>>> module; I'd like to use it as part of my validation and I'm assuming you
>>>> have that src/releases/release-info-3.4.2.properties file
>>>>
>>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>>> ahmar@apache.org>> wrote:
>>>> Hey all,
>>>>
>>>> The first release candidate for Hadoop 3.4.2 is now available for
>>>> voting.
>>>>
>>>> There are a couple of things to note:
>>>>
>>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>>> mean that the build fails on arm64.
>>>>
>>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>>> SVN.
>>>> For S3A, download SDK bundle v2.29.52 from:
>>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>>> ,
>>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>>> with these instructions.
>>>>
>>>>
>>>> The RC is available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>>
>>>> The git tag is release-3.4.2-RC1, commit
>>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>>
>>>> The maven artifacts are staged at:
>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>>
>>>>
>>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98)
>>>> at:
>>>>
>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>
>>>>
>>>> This release has been created off of branch-3.4. Key changes include:
>>>>
>>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>>> * S3A: Support for S3 conditional writes
>>>> * ABFS: Deprecation of WASB driver
>>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>>
>>>>
>>>> This is my first attempt at managing a release, please do test the
>>>> release
>>>> and let me know in case of any issues.
>>>>
>>>> Thanks,
>>>> Ahmar
>>>>
>>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
parquet stuff is unrelated, filed
https://github.com/apache/parquet-java/issues/3237 . still my fault though.
overall, I'm going to give a quick -1 with the commitment to provide one of
the patches.
my comments in markdown format:
## relesse docs index.html
the summary in index.html are from 3.4.1, so we need to update them to
3.4.2 with key features anre issues fixed.
Call out avro upgrade in the release notes -this is incompatible but
required for security reasons.
> We have upgraded to Avro 1.11.4, which is incompatible with previous
versions.
{ This is critical for security reasons -everyone needs to upgrade their
own uses of Avro too.
A side-effect of excluding the AWS SDK from the release is that the
hadoop-aws module does not
declare a transitive dependency on the AWS SDK. This means that if you use
hadoop-aws, you must explicitly include the AWS SDK in your project.
I'd like to see this (somehow) returned, at the very least we explicitly
add it into the hadoop-cloud-connectors dependencies so you can get it that
way.
otherwise it's a bit trickier to get right
doc changes
> This release was qualified against the 2.29.52 release, which at the time
of the qualification was the last release compatible with third-party
stores. This regression may have been fixed in later releases.
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
<version>2.29.52</version>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
> The minimum version of the AWS SDK bundle.jar which can be used in this
release is 2.29.52; later versions may work, but no later versions have
been qualified. Upgrade with care, especially when working with third-party
stores.
> see: https://issues.apache.org/jira/browse/HADOOP-19490
## java8
my setup has an old java8 release
java -version
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
1.8.0_362-b09)
OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
25.362-b09, mixed mode)
it is getting that recurrent bytebuffer flip error, from an overloaded
version of ByteBuffer...this can happen even with recent java8 releases.
We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
[INFO] -------------------------------------------------------
[INFO] Running org.apache.parquet.hadoop.TestParquetReader
Exception in thread "Thread-8" java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at
sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
this can actually be fixed trivially, casting to java.io.Buffer first:
((Byffer)buff).flip())
I'll do this
On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
> I'm sort of leaning towards a -1, though the regression is actually in
> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>
>
> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
> -- Time elapsed: 0.033 s <<< ERROR!
> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
> 4 ByteBuffer object(s) is/are remained unreleased after closing this
> allocator.
> at
> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
> at
> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Caused by:
> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
> more info
>
>
> 1. This is very much the vector io stuff.
> 2. It's not a regression since 3.4.0
> 3. but it's indicative of a memory leak. Surprised and Annoyed I
> hadn't spotted this before...the tests went in last year.
>
>
> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>> other people can revoke if you break things.
>>
>> will do the merge and test
>>
>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>> wrote:
>>
>>> Steve - I created a PR on hadoop-release-support with my properties
>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>
>>>
>>> Masatake - Yes, sounds good, will update the documentation for the new
>>> RC if it is created.
>>>
>>>
>>> Mukund - I realised too late that I messed up the commit for that one.
>>> Will figure out how to fix..
>>>
>>> ________________________________
>>> From: Steve Loughran <st...@cloudera.com>
>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>> To: Ahmar Suhail
>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not
>>> click links or open attachments unless you can confirm the sender and know
>>> the content is safe.
>>>
>>>
>>>
>>> can you put up the relevant changes to the hadoop-release-support
>>> module; I'd like to use it as part of my validation and I'm assuming you
>>> have that src/releases/release-info-3.4.2.properties file
>>>
>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>> ahmar@apache.org>> wrote:
>>> Hey all,
>>>
>>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>>
>>> There are a couple of things to note:
>>>
>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>> mean that the build fails on arm64.
>>>
>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>> SVN.
>>> For S3A, download SDK bundle v2.29.52 from:
>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>> ,
>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>> with these instructions.
>>>
>>>
>>> The RC is available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>
>>> The git tag is release-3.4.2-RC1, commit
>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>
>>> The maven artifacts are staged at:
>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>
>>>
>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>>
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>>
>>> This release has been created off of branch-3.4. Key changes include:
>>>
>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>> * S3A: Support for S3 conditional writes
>>> * ABFS: Deprecation of WASB driver
>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>
>>>
>>> This is my first attempt at managing a release, please do test the
>>> release
>>> and let me know in case of any issues.
>>>
>>> Thanks,
>>> Ahmar
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
parquet stuff is unrelated, filed
https://github.com/apache/parquet-java/issues/3237 . still my fault though.
overall, I'm going to give a quick -1 with the commitment to provide one of
the patches.
my comments in markdown format:
## relesse docs index.html
the summary in index.html are from 3.4.1, so we need to update them to
3.4.2 with key features anre issues fixed.
Call out avro upgrade in the release notes -this is incompatible but
required for security reasons.
> We have upgraded to Avro 1.11.4, which is incompatible with previous
versions.
{ This is critical for security reasons -everyone needs to upgrade their
own uses of Avro too.
A side-effect of excluding the AWS SDK from the release is that the
hadoop-aws module does not
declare a transitive dependency on the AWS SDK. This means that if you use
hadoop-aws, you must explicitly include the AWS SDK in your project.
I'd like to see this (somehow) returned, at the very least we explicitly
add it into the hadoop-cloud-connectors dependencies so you can get it that
way.
otherwise it's a bit trickier to get right
doc changes
> This release was qualified against the 2.29.52 release, which at the time
of the qualification was the last release compatible with third-party
stores. This regression may have been fixed in later releases.
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
<version>2.29.52</version>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
> The minimum version of the AWS SDK bundle.jar which can be used in this
release is 2.29.52; later versions may work, but no later versions have
been qualified. Upgrade with care, especially when working with third-party
stores.
> see: https://issues.apache.org/jira/browse/HADOOP-19490
## java8
my setup has an old java8 release
java -version
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
1.8.0_362-b09)
OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
25.362-b09, mixed mode)
it is getting that recurrent bytebuffer flip error, from an overloaded
version of ByteBuffer...this can happen even with recent java8 releases.
We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
[INFO] -------------------------------------------------------
[INFO] Running org.apache.parquet.hadoop.TestParquetReader
Exception in thread "Thread-8" java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at
sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
this can actually be fixed trivially, casting to java.io.Buffer first:
((Byffer)buff).flip())
I'll do this
On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
> I'm sort of leaning towards a -1, though the regression is actually in
> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>
>
> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
> -- Time elapsed: 0.033 s <<< ERROR!
> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
> 4 ByteBuffer object(s) is/are remained unreleased after closing this
> allocator.
> at
> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
> at
> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Caused by:
> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
> more info
>
>
> 1. This is very much the vector io stuff.
> 2. It's not a regression since 3.4.0
> 3. but it's indicative of a memory leak. Surprised and Annoyed I
> hadn't spotted this before...the tests went in last year.
>
>
> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>> other people can revoke if you break things.
>>
>> will do the merge and test
>>
>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>> wrote:
>>
>>> Steve - I created a PR on hadoop-release-support with my properties
>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>
>>>
>>> Masatake - Yes, sounds good, will update the documentation for the new
>>> RC if it is created.
>>>
>>>
>>> Mukund - I realised too late that I messed up the commit for that one.
>>> Will figure out how to fix..
>>>
>>> ________________________________
>>> From: Steve Loughran <st...@cloudera.com>
>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>> To: Ahmar Suhail
>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not
>>> click links or open attachments unless you can confirm the sender and know
>>> the content is safe.
>>>
>>>
>>>
>>> can you put up the relevant changes to the hadoop-release-support
>>> module; I'd like to use it as part of my validation and I'm assuming you
>>> have that src/releases/release-info-3.4.2.properties file
>>>
>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>> ahmar@apache.org>> wrote:
>>> Hey all,
>>>
>>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>>
>>> There are a couple of things to note:
>>>
>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>> mean that the build fails on arm64.
>>>
>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>> SVN.
>>> For S3A, download SDK bundle v2.29.52 from:
>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>> ,
>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>> with these instructions.
>>>
>>>
>>> The RC is available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>
>>> The git tag is release-3.4.2-RC1, commit
>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>
>>> The maven artifacts are staged at:
>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>
>>>
>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>>
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>>
>>> This release has been created off of branch-3.4. Key changes include:
>>>
>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>> * S3A: Support for S3 conditional writes
>>> * ABFS: Deprecation of WASB driver
>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>
>>>
>>> This is my first attempt at managing a release, please do test the
>>> release
>>> and let me know in case of any issues.
>>>
>>> Thanks,
>>> Ahmar
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
parquet stuff is unrelated, filed
https://github.com/apache/parquet-java/issues/3237 . still my fault though.
overall, I'm going to give a quick -1 with the commitment to provide one of
the patches.
my comments in markdown format:
## relesse docs index.html
the summary in index.html are from 3.4.1, so we need to update them to
3.4.2 with key features anre issues fixed.
Call out avro upgrade in the release notes -this is incompatible but
required for security reasons.
> We have upgraded to Avro 1.11.4, which is incompatible with previous
versions.
{ This is critical for security reasons -everyone needs to upgrade their
own uses of Avro too.
A side-effect of excluding the AWS SDK from the release is that the
hadoop-aws module does not
declare a transitive dependency on the AWS SDK. This means that if you use
hadoop-aws, you must explicitly include the AWS SDK in your project.
I'd like to see this (somehow) returned, at the very least we explicitly
add it into the hadoop-cloud-connectors dependencies so you can get it that
way.
otherwise it's a bit trickier to get right
doc changes
> This release was qualified against the 2.29.52 release, which at the time
of the qualification was the last release compatible with third-party
stores. This regression may have been fixed in later releases.
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
<version>2.29.52</version>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
> The minimum version of the AWS SDK bundle.jar which can be used in this
release is 2.29.52; later versions may work, but no later versions have
been qualified. Upgrade with care, especially when working with third-party
stores.
> see: https://issues.apache.org/jira/browse/HADOOP-19490
## java8
my setup has an old java8 release
java -version
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
1.8.0_362-b09)
OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
25.362-b09, mixed mode)
it is getting that recurrent bytebuffer flip error, from an overloaded
version of ByteBuffer...this can happen even with recent java8 releases.
We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
[INFO] -------------------------------------------------------
[INFO] Running org.apache.parquet.hadoop.TestParquetReader
Exception in thread "Thread-8" java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at
sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
this can actually be fixed trivially, casting to java.io.Buffer first:
((Byffer)buff).flip())
I'll do this
On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
> I'm sort of leaning towards a -1, though the regression is actually in
> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>
>
> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
> -- Time elapsed: 0.033 s <<< ERROR!
> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
> 4 ByteBuffer object(s) is/are remained unreleased after closing this
> allocator.
> at
> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
> at
> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Caused by:
> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
> more info
>
>
> 1. This is very much the vector io stuff.
> 2. It's not a regression since 3.4.0
> 3. but it's indicative of a memory leak. Surprised and Annoyed I
> hadn't spotted this before...the tests went in last year.
>
>
> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>> other people can revoke if you break things.
>>
>> will do the merge and test
>>
>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>> wrote:
>>
>>> Steve - I created a PR on hadoop-release-support with my properties
>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>
>>>
>>> Masatake - Yes, sounds good, will update the documentation for the new
>>> RC if it is created.
>>>
>>>
>>> Mukund - I realised too late that I messed up the commit for that one.
>>> Will figure out how to fix..
>>>
>>> ________________________________
>>> From: Steve Loughran <st...@cloudera.com>
>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>> To: Ahmar Suhail
>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not
>>> click links or open attachments unless you can confirm the sender and know
>>> the content is safe.
>>>
>>>
>>>
>>> can you put up the relevant changes to the hadoop-release-support
>>> module; I'd like to use it as part of my validation and I'm assuming you
>>> have that src/releases/release-info-3.4.2.properties file
>>>
>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>> ahmar@apache.org>> wrote:
>>> Hey all,
>>>
>>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>>
>>> There are a couple of things to note:
>>>
>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>> mean that the build fails on arm64.
>>>
>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>> SVN.
>>> For S3A, download SDK bundle v2.29.52 from:
>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>> ,
>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>> with these instructions.
>>>
>>>
>>> The RC is available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>
>>> The git tag is release-3.4.2-RC1, commit
>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>
>>> The maven artifacts are staged at:
>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>
>>>
>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>>
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>>
>>> This release has been created off of branch-3.4. Key changes include:
>>>
>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>> * S3A: Support for S3 conditional writes
>>> * ABFS: Deprecation of WASB driver
>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>
>>>
>>> This is my first attempt at managing a release, please do test the
>>> release
>>> and let me know in case of any issues.
>>>
>>> Thanks,
>>> Ahmar
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
parquet stuff is unrelated, filed
https://github.com/apache/parquet-java/issues/3237 . still my fault though.
overall, I'm going to give a quick -1 with the commitment to provide one of
the patches.
my comments in markdown format:
## relesse docs index.html
the summary in index.html are from 3.4.1, so we need to update them to
3.4.2 with key features anre issues fixed.
Call out avro upgrade in the release notes -this is incompatible but
required for security reasons.
> We have upgraded to Avro 1.11.4, which is incompatible with previous
versions.
{ This is critical for security reasons -everyone needs to upgrade their
own uses of Avro too.
A side-effect of excluding the AWS SDK from the release is that the
hadoop-aws module does not
declare a transitive dependency on the AWS SDK. This means that if you use
hadoop-aws, you must explicitly include the AWS SDK in your project.
I'd like to see this (somehow) returned, at the very least we explicitly
add it into the hadoop-cloud-connectors dependencies so you can get it that
way.
otherwise it's a bit trickier to get right
doc changes
> This release was qualified against the 2.29.52 release, which at the time
of the qualification was the last release compatible with third-party
stores. This regression may have been fixed in later releases.
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
<version>2.29.52</version>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
> The minimum version of the AWS SDK bundle.jar which can be used in this
release is 2.29.52; later versions may work, but no later versions have
been qualified. Upgrade with care, especially when working with third-party
stores.
> see: https://issues.apache.org/jira/browse/HADOOP-19490
## java8
my setup has an old java8 release
java -version
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build
1.8.0_362-b09)
OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build
25.362-b09, mixed mode)
it is getting that recurrent bytebuffer flip error, from an overloaded
version of ByteBuffer...this can happen even with recent java8 releases.
We cast the ByteBuffer to (java.nio.Buffer) before calling flip().
[INFO] -------------------------------------------------------
[INFO] Running org.apache.parquet.hadoop.TestParquetReader
Exception in thread "Thread-8" java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:428)
at
org.apache.hadoop.fs.RawLocalFileSystem$AsyncHandler.completed(RawLocalFileSystem.java:362)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at
sun.nio.ch.SimpleAsynchronousFileChannelImpl$2.run(SimpleAsynchronousFileChannelImpl.java:335)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
this can actually be fixed trivially, casting to java.io.Buffer first:
((Byffer)buff).flip())
I'll do this
On Tue, 3 Jun 2025 at 16:04, Steve Loughran <st...@cloudera.com> wrote:
>
> I'm sort of leaning towards a -1, though the regression is actually in
> 3.4.0, and surfacing as a failure to release buffers in localfs IO.
>
>
> [ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
> -- Time elapsed: 0.033 s <<< ERROR!
> org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
> 4 ByteBuffer object(s) is/are remained unreleased after closing this
> allocator.
> at
> org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
> at
> org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Caused by:
> org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
> Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
> more info
>
>
> 1. This is very much the vector io stuff.
> 2. It's not a regression since 3.4.0
> 3. but it's indicative of a memory leak. Surprised and Annoyed I
> hadn't spotted this before...the tests went in last year.
>
>
> On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> Ahmar, no need for prs there, it's a "commit then review" repo, as in
>> other people can revoke if you break things.
>>
>> will do the merge and test
>>
>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
>> wrote:
>>
>>> Steve - I created a PR on hadoop-release-support with my properties
>>> file: https://github.com/apache/hadoop-release-support/pull/4
>>>
>>>
>>> Masatake - Yes, sounds good, will update the documentation for the new
>>> RC if it is created.
>>>
>>>
>>> Mukund - I realised too late that I messed up the commit for that one.
>>> Will figure out how to fix..
>>>
>>> ________________________________
>>> From: Steve Loughran <st...@cloudera.com>
>>> Sent: Monday, June 2, 2025 4:25:00 PM
>>> To: Ahmar Suhail
>>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not
>>> click links or open attachments unless you can confirm the sender and know
>>> the content is safe.
>>>
>>>
>>>
>>> can you put up the relevant changes to the hadoop-release-support
>>> module; I'd like to use it as part of my validation and I'm assuming you
>>> have that src/releases/release-info-3.4.2.properties file
>>>
>>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>>> ahmar@apache.org>> wrote:
>>> Hey all,
>>>
>>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>>
>>> There are a couple of things to note:
>>>
>>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>>> mean that the build fails on arm64.
>>>
>>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>>> SVN.
>>> For S3A, download SDK bundle v2.29.52 from:
>>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52
>>> ,
>>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>>> with these instructions.
>>>
>>>
>>> The RC is available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>>
>>> The git tag is release-3.4.2-RC1, commit
>>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>>
>>> The maven artifacts are staged at:
>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>>
>>>
>>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>>
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>>
>>> This release has been created off of branch-3.4. Key changes include:
>>>
>>> * S3A: Integration with S3 Analytics Accelerator input stream
>>> * S3A: Support for S3 conditional writes
>>> * ABFS: Deprecation of WASB driver
>>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>>
>>>
>>> This is my first attempt at managing a release, please do test the
>>> release
>>> and let me know in case of any issues.
>>>
>>> Thanks,
>>> Ahmar
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
I'm sort of leaning towards a -1, though the regression is actually in
3.4.0, and surfacing as a failure to release buffers in localfs IO.
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
-- Time elapsed: 0.033 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
4 ByteBuffer object(s) is/are remained unreleased after closing this
allocator.
at
org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
at
org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by:
org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
more info
1. This is very much the vector io stuff.
2. It's not a regression since 3.4.0
3. but it's indicative of a memory leak. Surprised and Annoyed I hadn't
spotted this before...the tests went in last year.
On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
> Ahmar, no need for prs there, it's a "commit then review" repo, as in
> other people can revoke if you break things.
>
> will do the merge and test
>
> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
> wrote:
>
>> Steve - I created a PR on hadoop-release-support with my properties
>> file: https://github.com/apache/hadoop-release-support/pull/4
>>
>>
>> Masatake - Yes, sounds good, will update the documentation for the new RC
>> if it is created.
>>
>>
>> Mukund - I realised too late that I messed up the commit for that one.
>> Will figure out how to fix..
>>
>> ________________________________
>> From: Steve Loughran <st...@cloudera.com>
>> Sent: Monday, June 2, 2025 4:25:00 PM
>> To: Ahmar Suhail
>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> can you put up the relevant changes to the hadoop-release-support module;
>> I'd like to use it as part of my validation and I'm assuming you have that
>> src/releases/release-info-3.4.2.properties file
>>
>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>> ahmar@apache.org>> wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>> SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
I'm sort of leaning towards a -1, though the regression is actually in
3.4.0, and surfacing as a failure to release buffers in localfs IO.
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
-- Time elapsed: 0.033 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
4 ByteBuffer object(s) is/are remained unreleased after closing this
allocator.
at
org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
at
org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by:
org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
more info
1. This is very much the vector io stuff.
2. It's not a regression since 3.4.0
3. but it's indicative of a memory leak. Surprised and Annoyed I hadn't
spotted this before...the tests went in last year.
On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
> Ahmar, no need for prs there, it's a "commit then review" repo, as in
> other people can revoke if you break things.
>
> will do the merge and test
>
> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
> wrote:
>
>> Steve - I created a PR on hadoop-release-support with my properties
>> file: https://github.com/apache/hadoop-release-support/pull/4
>>
>>
>> Masatake - Yes, sounds good, will update the documentation for the new RC
>> if it is created.
>>
>>
>> Mukund - I realised too late that I messed up the commit for that one.
>> Will figure out how to fix..
>>
>> ________________________________
>> From: Steve Loughran <st...@cloudera.com>
>> Sent: Monday, June 2, 2025 4:25:00 PM
>> To: Ahmar Suhail
>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> can you put up the relevant changes to the hadoop-release-support module;
>> I'd like to use it as part of my validation and I'm assuming you have that
>> src/releases/release-info-3.4.2.properties file
>>
>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>> ahmar@apache.org>> wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>> SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
I'm sort of leaning towards a -1, though the regression is actually in
3.4.0, and surfacing as a failure to release buffers in localfs IO.
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
-- Time elapsed: 0.033 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
4 ByteBuffer object(s) is/are remained unreleased after closing this
allocator.
at
org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
at
org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by:
org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
more info
1. This is very much the vector io stuff.
2. It's not a regression since 3.4.0
3. but it's indicative of a memory leak. Surprised and Annoyed I hadn't
spotted this before...the tests went in last year.
On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
> Ahmar, no need for prs there, it's a "commit then review" repo, as in
> other people can revoke if you break things.
>
> will do the merge and test
>
> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
> wrote:
>
>> Steve - I created a PR on hadoop-release-support with my properties
>> file: https://github.com/apache/hadoop-release-support/pull/4
>>
>>
>> Masatake - Yes, sounds good, will update the documentation for the new RC
>> if it is created.
>>
>>
>> Mukund - I realised too late that I messed up the commit for that one.
>> Will figure out how to fix..
>>
>> ________________________________
>> From: Steve Loughran <st...@cloudera.com>
>> Sent: Monday, June 2, 2025 4:25:00 PM
>> To: Ahmar Suhail
>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> can you put up the relevant changes to the hadoop-release-support module;
>> I'd like to use it as part of my validation and I'm assuming you have that
>> src/releases/release-info-3.4.2.properties file
>>
>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>> ahmar@apache.org>> wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>> SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
I'm sort of leaning towards a -1, though the regression is actually in
3.4.0, and surfacing as a failure to release buffers in localfs IO.
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
-- Time elapsed: 0.033 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
4 ByteBuffer object(s) is/are remained unreleased after closing this
allocator.
at
org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
at
org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by:
org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
more info
1. This is very much the vector io stuff.
2. It's not a regression since 3.4.0
3. but it's indicative of a memory leak. Surprised and Annoyed I hadn't
spotted this before...the tests went in last year.
On Tue, 3 Jun 2025 at 10:55, Steve Loughran <st...@cloudera.com> wrote:
>
> Ahmar, no need for prs there, it's a "commit then review" repo, as in
> other people can revoke if you break things.
>
> will do the merge and test
>
> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
> wrote:
>
>> Steve - I created a PR on hadoop-release-support with my properties
>> file: https://github.com/apache/hadoop-release-support/pull/4
>>
>>
>> Masatake - Yes, sounds good, will update the documentation for the new RC
>> if it is created.
>>
>>
>> Mukund - I realised too late that I messed up the commit for that one.
>> Will figure out how to fix..
>>
>> ________________________________
>> From: Steve Loughran <st...@cloudera.com>
>> Sent: Monday, June 2, 2025 4:25:00 PM
>> To: Ahmar Suhail
>> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> can you put up the relevant changes to the hadoop-release-support module;
>> I'd like to use it as part of my validation and I'm assuming you have that
>> src/releases/release-info-3.4.2.properties file
>>
>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
>> ahmar@apache.org>> wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>> SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Ahmar, no need for prs there, it's a "commit then review" repo, as in other
people can revoke if you break things.
will do the merge and test
On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Steve - I created a PR on hadoop-release-support with my properties file:
> https://github.com/apache/hadoop-release-support/pull/4
>
>
> Masatake - Yes, sounds good, will update the documentation for the new RC
> if it is created.
>
>
> Mukund - I realised too late that I messed up the commit for that one.
> Will figure out how to fix..
>
> ________________________________
> From: Steve Loughran <st...@cloudera.com>
> Sent: Monday, June 2, 2025 4:25:00 PM
> To: Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> can you put up the relevant changes to the hadoop-release-support module;
> I'd like to use it as part of my validation and I'm assuming you have that
> src/releases/release-info-3.4.2.properties file
>
> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
> ahmar@apache.org>> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Ahmar, no need for prs there, it's a "commit then review" repo, as in other
people can revoke if you break things.
will do the merge and test
On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Steve - I created a PR on hadoop-release-support with my properties file:
> https://github.com/apache/hadoop-release-support/pull/4
>
>
> Masatake - Yes, sounds good, will update the documentation for the new RC
> if it is created.
>
>
> Mukund - I realised too late that I messed up the commit for that one.
> Will figure out how to fix..
>
> ________________________________
> From: Steve Loughran <st...@cloudera.com>
> Sent: Monday, June 2, 2025 4:25:00 PM
> To: Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> can you put up the relevant changes to the hadoop-release-support module;
> I'd like to use it as part of my validation and I'm assuming you have that
> src/releases/release-info-3.4.2.properties file
>
> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
> ahmar@apache.org>> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Ahmar, no need for prs there, it's a "commit then review" repo, as in other
people can revoke if you break things.
will do the merge and test
On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Steve - I created a PR on hadoop-release-support with my properties file:
> https://github.com/apache/hadoop-release-support/pull/4
>
>
> Masatake - Yes, sounds good, will update the documentation for the new RC
> if it is created.
>
>
> Mukund - I realised too late that I messed up the commit for that one.
> Will figure out how to fix..
>
> ________________________________
> From: Steve Loughran <st...@cloudera.com>
> Sent: Monday, June 2, 2025 4:25:00 PM
> To: Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> can you put up the relevant changes to the hadoop-release-support module;
> I'd like to use it as part of my validation and I'm assuming you have that
> src/releases/release-info-3.4.2.properties file
>
> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
> ahmar@apache.org>> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Ahmar, no need for prs there, it's a "commit then review" repo, as in other
people can revoke if you break things.
will do the merge and test
On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ah...@amazon.co.uk.invalid>
wrote:
> Steve - I created a PR on hadoop-release-support with my properties file:
> https://github.com/apache/hadoop-release-support/pull/4
>
>
> Masatake - Yes, sounds good, will update the documentation for the new RC
> if it is created.
>
>
> Mukund - I realised too late that I messed up the commit for that one.
> Will figure out how to fix..
>
> ________________________________
> From: Steve Loughran <st...@cloudera.com>
> Sent: Monday, June 2, 2025 4:25:00 PM
> To: Ahmar Suhail
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> can you put up the relevant changes to the hadoop-release-support module;
> I'd like to use it as part of my validation and I'm assuming you have that
> src/releases/release-info-3.4.2.properties file
>
> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ahmar@apache.org<mailto:
> ahmar@apache.org>> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Steve - I created a PR on hadoop-release-support with my properties file: https://github.com/apache/hadoop-release-support/pull/4
Masatake - Yes, sounds good, will update the documentation for the new RC if it is created.
Mukund - I realised too late that I messed up the commit for that one. Will figure out how to fix..
________________________________
From: Steve Loughran <st...@cloudera.com>
Sent: Monday, June 2, 2025 4:25:00 PM
To: Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
can you put up the relevant changes to the hadoop-release-support module; I'd like to use it as part of my validation and I'm assuming you have that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org>> wrote:
Hey all,
The first release candidate for Hadoop 3.4.2 is now available for voting.
There are a couple of things to note:
1/ No Arm64 artifacts. This is due to previously reported issues:
https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
<https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
mean that the build fails on arm64.
2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
For S3A, download SDK bundle v2.29.52 from:
https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
and drop it into /share/hadoop/common/lib. Release notes will be updated
with these instructions.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
The git tag is release-3.4.2-RC1, commit
09870840ec35b48cd107972eb24d25e8aece04c9
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1437
You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
This release has been created off of branch-3.4. Key changes include:
* S3A: Integration with S3 Analytics Accelerator input stream
* S3A: Support for S3 conditional writes
* ABFS: Deprecation of WASB driver
* ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
This is my first attempt at managing a release, please do test the release
and let me know in case of any issues.
Thanks,
Ahmar
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Steve - I created a PR on hadoop-release-support with my properties file: https://github.com/apache/hadoop-release-support/pull/4
Masatake - Yes, sounds good, will update the documentation for the new RC if it is created.
Mukund - I realised too late that I messed up the commit for that one. Will figure out how to fix..
________________________________
From: Steve Loughran <st...@cloudera.com>
Sent: Monday, June 2, 2025 4:25:00 PM
To: Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
can you put up the relevant changes to the hadoop-release-support module; I'd like to use it as part of my validation and I'm assuming you have that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org>> wrote:
Hey all,
The first release candidate for Hadoop 3.4.2 is now available for voting.
There are a couple of things to note:
1/ No Arm64 artifacts. This is due to previously reported issues:
https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
<https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
mean that the build fails on arm64.
2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
For S3A, download SDK bundle v2.29.52 from:
https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
and drop it into /share/hadoop/common/lib. Release notes will be updated
with these instructions.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
The git tag is release-3.4.2-RC1, commit
09870840ec35b48cd107972eb24d25e8aece04c9
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1437
You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
This release has been created off of branch-3.4. Key changes include:
* S3A: Integration with S3 Analytics Accelerator input stream
* S3A: Support for S3 conditional writes
* ABFS: Deprecation of WASB driver
* ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
This is my first attempt at managing a release, please do test the release
and let me know in case of any issues.
Thanks,
Ahmar
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Steve - I created a PR on hadoop-release-support with my properties file: https://github.com/apache/hadoop-release-support/pull/4
Masatake - Yes, sounds good, will update the documentation for the new RC if it is created.
Mukund - I realised too late that I messed up the commit for that one. Will figure out how to fix..
________________________________
From: Steve Loughran <st...@cloudera.com>
Sent: Monday, June 2, 2025 4:25:00 PM
To: Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
can you put up the relevant changes to the hadoop-release-support module; I'd like to use it as part of my validation and I'm assuming you have that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org>> wrote:
Hey all,
The first release candidate for Hadoop 3.4.2 is now available for voting.
There are a couple of things to note:
1/ No Arm64 artifacts. This is due to previously reported issues:
https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
<https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
mean that the build fails on arm64.
2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
For S3A, download SDK bundle v2.29.52 from:
https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
and drop it into /share/hadoop/common/lib. Release notes will be updated
with these instructions.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
The git tag is release-3.4.2-RC1, commit
09870840ec35b48cd107972eb24d25e8aece04c9
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1437
You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
This release has been created off of branch-3.4. Key changes include:
* S3A: Integration with S3 Analytics Accelerator input stream
* S3A: Support for S3 conditional writes
* ABFS: Deprecation of WASB driver
* ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
This is my first attempt at managing a release, please do test the release
and let me know in case of any issues.
Thanks,
Ahmar
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
Steve - I created a PR on hadoop-release-support with my properties file: https://github.com/apache/hadoop-release-support/pull/4
Masatake - Yes, sounds good, will update the documentation for the new RC if it is created.
Mukund - I realised too late that I messed up the commit for that one. Will figure out how to fix..
________________________________
From: Steve Loughran <st...@cloudera.com>
Sent: Monday, June 2, 2025 4:25:00 PM
To: Ahmar Suhail
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
can you put up the relevant changes to the hadoop-release-support module; I'd like to use it as part of my validation and I'm assuming you have that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org>> wrote:
Hey all,
The first release candidate for Hadoop 3.4.2 is now available for voting.
There are a couple of things to note:
1/ No Arm64 artifacts. This is due to previously reported issues:
https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
<https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
mean that the build fails on arm64.
2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
For S3A, download SDK bundle v2.29.52 from:
https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
and drop it into /share/hadoop/common/lib. Release notes will be updated
with these instructions.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
The git tag is release-3.4.2-RC1, commit
09870840ec35b48cd107972eb24d25e8aece04c9
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1437
You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
This release has been created off of branch-3.4. Key changes include:
* S3A: Integration with S3 Analytics Accelerator input stream
* S3A: Support for S3 conditional writes
* ABFS: Deprecation of WASB driver
* ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
This is my first attempt at managing a release, please do test the release
and let me know in case of any issues.
Thanks,
Ahmar
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
can you put up the relevant changes to the hadoop-release-support module;
I'd like to use it as part of my validation and I'm assuming you have
that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Wei-Chiu Chuang <we...@apache.org>.
I really wish we can make ARM64 binaries. Would like to find time to work
on the two jiras mentioned.
On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Wei-Chiu Chuang <we...@apache.org>.
I really wish we can make ARM64 binaries. Would like to find time to work
on the two jiras mentioned.
On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
Thanks for starting this! I tried accessing the S3 bucket using the RC by
dropping in the bundle jar and it works. Will test more!
On Wed, May 28, 2025 at 7:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
can you put up the relevant changes to the hadoop-release-support module;
I'd like to use it as part of my validation and I'm assuming you have
that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
Thanks for starting this! I tried accessing the S3 bucket using the RC by
dropping in the bundle jar and it works. Will test more!
On Wed, May 28, 2025 at 7:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
I updated JIRA's "Fix Versions/s" and "Resolution" along with the commit status.
Since HADOOP-19580 was commited to branch-3.4.2 but not in the RC1,
it will be contained if RC2 is created.
"Fix Versions/s" should be updated later.
https://issues.apache.org/jira/browse/HADOOP-19580
On 2025/06/02 18:47, Masatake Iwasaki wrote:
> Thanks for driving the release, Ahmar.
>
> I found some issues on documentations.
> While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
>
> * The top page of site documentation looks mentioning changes of 3.4.1.
> https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
>
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
> https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
>
>
> Jira/Git commit message diff starting: ##############################################
> WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
> WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
> Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
> Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
> Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
> Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
> Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
> Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
> Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
> Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
> Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
> WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
> Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
> Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
> Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
> Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
> Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
> Exiting successfully
> Jira/Git commit message diff completed: ##############################################
>
> Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
> Starting diff: ##############################################
> HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> Completed diff: ##############################################
>
>
> I will vote after some more local testing.
>
> Thanks,
> Masatake Iwasaki
>
>
> On 2025/05/28 21:24, Ahmar Suhail wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
I updated JIRA's "Fix Versions/s" and "Resolution" along with the commit status.
Since HADOOP-19580 was commited to branch-3.4.2 but not in the RC1,
it will be contained if RC2 is created.
"Fix Versions/s" should be updated later.
https://issues.apache.org/jira/browse/HADOOP-19580
On 2025/06/02 18:47, Masatake Iwasaki wrote:
> Thanks for driving the release, Ahmar.
>
> I found some issues on documentations.
> While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
>
> * The top page of site documentation looks mentioning changes of 3.4.1.
> https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
>
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
> https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
>
>
> Jira/Git commit message diff starting: ##############################################
> WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
> WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
> Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
> Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
> Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
> Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
> Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
> Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
> Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
> Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
> Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
> WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
> Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
> Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
> Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
> Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
> Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
> Exiting successfully
> Jira/Git commit message diff completed: ##############################################
>
> Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
> Starting diff: ##############################################
> HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> Completed diff: ##############################################
>
>
> I will vote after some more local testing.
>
> Thanks,
> Masatake Iwasaki
>
>
> On 2025/05/28 21:24, Ahmar Suhail wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
I updated JIRA's "Fix Versions/s" and "Resolution" along with the commit status.
Since HADOOP-19580 was commited to branch-3.4.2 but not in the RC1,
it will be contained if RC2 is created.
"Fix Versions/s" should be updated later.
https://issues.apache.org/jira/browse/HADOOP-19580
On 2025/06/02 18:47, Masatake Iwasaki wrote:
> Thanks for driving the release, Ahmar.
>
> I found some issues on documentations.
> While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
>
> * The top page of site documentation looks mentioning changes of 3.4.1.
> https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
>
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
> https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
>
>
> Jira/Git commit message diff starting: ##############################################
> WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
> WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
> Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
> Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
> Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
> Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
> Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
> Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
> Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
> Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
> Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
> WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
> Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
> Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
> Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
> Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
> Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
> Exiting successfully
> Jira/Git commit message diff completed: ##############################################
>
> Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
> Starting diff: ##############################################
> HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> Completed diff: ##############################################
>
>
> I will vote after some more local testing.
>
> Thanks,
> Masatake Iwasaki
>
>
> On 2025/05/28 21:24, Ahmar Suhail wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
I updated JIRA's "Fix Versions/s" and "Resolution" along with the commit status.
Since HADOOP-19580 was commited to branch-3.4.2 but not in the RC1,
it will be contained if RC2 is created.
"Fix Versions/s" should be updated later.
https://issues.apache.org/jira/browse/HADOOP-19580
On 2025/06/02 18:47, Masatake Iwasaki wrote:
> Thanks for driving the release, Ahmar.
>
> I found some issues on documentations.
> While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
>
> * The top page of site documentation looks mentioning changes of 3.4.1.
> https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
>
> * git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
> https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
>
>
> Jira/Git commit message diff starting: ##############################################
> WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
> WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
> Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
> Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
> Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
> Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
> Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
> Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
> Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
> Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
> Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
> WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
> Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
> Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
> Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
> Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
> Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
> Exiting successfully
> Jira/Git commit message diff completed: ##############################################
>
> Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
> Starting diff: ##############################################
> HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
> Completed diff: ##############################################
>
>
> I will vote after some more local testing.
>
> Thanks,
> Masatake Iwasaki
>
>
> On 2025/05/28 21:24, Ahmar Suhail wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for driving the release, Ahmar.
I found some issues on documentations.
While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
* The top page of site documentation looks mentioning changes of 3.4.1.
https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
* git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
Jira/Git commit message diff starting: ##############################################
WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
Exiting successfully
Jira/Git commit message diff completed: ##############################################
Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
Starting diff: ##############################################
HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
Completed diff: ##############################################
I will vote after some more local testing.
Thanks,
Masatake Iwasaki
On 2025/05/28 21:24, Ahmar Suhail wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for driving the release, Ahmar.
I found some issues on documentations.
While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
* The top page of site documentation looks mentioning changes of 3.4.1.
https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
* git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
Jira/Git commit message diff starting: ##############################################
WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
Exiting successfully
Jira/Git commit message diff completed: ##############################################
Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
Starting diff: ##############################################
HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
Completed diff: ##############################################
I will vote after some more local testing.
Thanks,
Masatake Iwasaki
On 2025/05/28 21:24, Ahmar Suhail wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
can you put up the relevant changes to the hadoop-release-support module;
I'd like to use it as part of my validation and I'm assuming you have
that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
Thanks for starting this! I tried accessing the S3 bucket using the RC by
dropping in the bundle jar and it works. Will test more!
On Wed, May 28, 2025 at 7:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for driving the release, Ahmar.
I found some issues on documentations.
While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
* The top page of site documentation looks mentioning changes of 3.4.1.
https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
* git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
Jira/Git commit message diff starting: ##############################################
WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
Exiting successfully
Jira/Git commit message diff completed: ##############################################
Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
Starting diff: ##############################################
HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
Completed diff: ##############################################
I will vote after some more local testing.
Thanks,
Masatake Iwasaki
On 2025/05/28 21:24, Ahmar Suhail wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Wei-Chiu Chuang <we...@apache.org>.
I really wish we can make ARM64 binaries. Would like to find time to work
on the two jiras mentioned.
On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
Thanks for starting this! I tried accessing the S3 bucket using the RC by
dropping in the bundle jar and it works. Will test more!
On Wed, May 28, 2025 at 7:25 AM Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for driving the release, Ahmar.
I found some issues on documentations.
While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2.
* The top page of site documentation looks mentioning changes of 3.4.1.
https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hadoop-project/src/site/markdown/index.md.vm
* git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA.
https://github.com/apache/hadoop/tree/release-3.4.2-RC1/dev-support/git-jira-validation
Jira/Git commit message diff starting: ##############################################
WARN: Jira not found. Commit: 09870840ec35b48cd107972eb24d25e8aece04c9 updates SDK dependency to provided (#7688)
WARN: Jira not found. Commit: 02b4849886c943be635af521714934b5bf6c32c1 Preparing for 3.4.2 release
Jira not present with version: 3.4.2. Commit: 7f170e9788f8074091f98527cdd9552acd1d05db HADOOP-19548: [ABFS] Fix Logging in FSDataInputStream to Mention Correct Buffersize (#7642) (#7660)
Jira is not resolved yet? Commit: f3ec55b5911d1e5e7da438c64f614a59a7db62a3 HADOOP-19516. S3A: SDK reads content twice during PUT to S3 Express store.
Commit seems reverted. Commit: 9d3082e130227fdd05216eb4a0ecd374bce447f2 Revert "HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh." (#7529)
Jira not present with version: 3.4.2. Commit: c7d292ecd1078f72e234ae1e84e70d7ee280ad92 HADOOP-19488 fix temporary directory creation (branch-3.4) (#7530)
Jira not present with version: 3.4.2. Commit: a5fb37e6547e152fabbc47520b9c8f7fc448a1df HADOOP-18940. [ABFS] Remove commons IOUtils.close() from AbfsOutputStream (#7506)
Commit seems reverted. Commit: 474f5343a4676832cf514ff37c03d25915472752 Revert "YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions."
Jira not present with version: 3.4.2. Commit: 7e682e0f976694bf3030309f73ac2264694f6d70 YARN-11793: Remove grizzly-http-* dependencies from Hadoop to avoid transitive inclusions.
Jira not present with version: 3.4.2. Commit: c122ff33b82a727ac1f6c83760efcb7fd5c0bac7 [YARN-11776] Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null (#7431) Contributed by Abhey Rana.
Jira not present with version: 3.4.2. Commit: b3ae9d1eb6841a6c9f9928a917cc50818a319f71 HADOOP-19152. Do not hard code security providers. (#6739) (#7439)
WARN: Jira not found. Commit: 04ca53410beb18bbd279219275f59f04e6df293a Integrate analytics accelerator into S3A. (#7471)
Jira not present with version: 3.4.2. Commit: fd445c215459f2b7604e2cd383af98c040861cc2 HADOOP-19407: Update 3.4.1 docs landing page to highlight changes shipped in the release.
Jira not present with version: 3.4.2. Commit: 7d699e5ffb7dee613ac6bc6bccb529ef6bb4a2a6 HADOOP-19401: Improve error message when OS can't identify the current user.
Jira not present with version: 3.4.2. Commit: 57d821e6aad23b604833272071c8481567363e63 HADOOP-19143. Upgrade commons-cli to 1.9.0. (#7126) (#7146) Contributed by Min Yan.
Jira not present with version: 3.4.2. Commit: 0b74fbe9479a88b154c5d5d5acc5920c4a50e65d HDFS-17637. [BackPort] Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) (#7195) Contributed by Hualong Zhang.
Found first commit hash after which git history is redundant. commit: 23f45d0b26d4a883e7a29278df0ec75bc5c1fda3
Exiting successfully
Jira/Git commit message diff completed: ##############################################
Any resolved Jira with fixVersion 3.4.2 but corresponding commit not present
Starting diff: ##############################################
HADOOP-19580 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19512 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19492 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19491 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
HADOOP-19382 is marked resolved with fixVersion 3.4.2 but no corresponding commit found
Completed diff: ##############################################
I will vote after some more local testing.
Thanks,
Masatake Iwasaki
On 2025/05/28 21:24, Ahmar Suhail wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
can you put up the relevant changes to the hadoop-release-support module;
I'd like to use it as part of my validation and I'm assuming you have
that src/releases/release-info-3.4.2.properties file
On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org> wrote:
> Hey all,
>
> The first release candidate for Hadoop 3.4.2 is now available for voting.
>
> There are a couple of things to note:
>
> 1/ No Arm64 artifacts. This is due to previously reported issues:
> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713
> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
> mean that the build fails on arm64.
>
> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
> which makes the size of tar > 1GB, and it can no longer be uploaded to SVN.
> For S3A, download SDK bundle v2.29.52 from:
> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
> and drop it into /share/hadoop/common/lib. Release notes will be updated
> with these instructions.
>
>
> The RC is available at:
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>
> The git tag is release-3.4.2-RC1, commit
> 09870840ec35b48cd107972eb24d25e8aece04c9
>
> The maven artifacts are staged at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>
>
> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> This release has been created off of branch-3.4. Key changes include:
>
> * S3A: Integration with S3 Analytics Accelerator input stream
> * S3A: Support for S3 conditional writes
> * ABFS: Deprecation of WASB driver
> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>
>
> This is my first attempt at managing a release, please do test the release
> and let me know in case of any issues.
>
> Thanks,
> Ahmar
>