You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by slfan1989 <sl...@apache.org> on 2024/03/04 22:07:05 UTC
[VOTE] Release Apache Hadoop 3.4.0 (RC3)
Hi folks,
Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
The git tag is release-3.4.0-RC3, commit bd8b77f398f
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408
You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
This is off branch-3.4.0 and is the first big release since 3.3.6.
Key changes include
* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes
Differences from Hadoop-3.4.0-RC2
* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description
Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).
The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.
[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.
Best Regards,
Shilun Fan.
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
Just gave a quick try, Hive master didn't compile. I gave up post that...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hive-common: Compilation failure
[ERROR]
/Users/ayushsaxena/code/hive/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java:[23,37]
package org.apache.hadoop.log.metrics does not exist
I think it due to HADOOP-17524
<https://issues.apache.org/jira/browse/HADOOP-17524>, it will be quite a
effort to upgrade from 3.3.x to 3.4.x, I don't think we would chase that
upgrade in hive anytime soon
I will abstain
-Ayush
On Thu, 7 Mar 2024 at 08:33, Xiaoqiao He <he...@apache.org> wrote:
> cc @Chao Sun <su...@apache.org> @zhangduo@apache.org
> <zh...@apache.org> @Ayush Saxena <ay...@gmail.com>
> Would you mind helping to check the upstream systems(Spark, HBase, Hive)
> dependency with
> the new hadoop release version if necessary? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
>
>> Shilun, Thanks for your great work. I am checking but not finished yet.
>> Will confirm once done.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>>> @Steve
>>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>>> <mt...@cloudera.com> @Takanobu
>>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>>
>>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>>
>>> > Hi folks,
>>> >
>>> > Xiaoqiao He and I have put together a release candidate (RC3) for
>>> Hadoop
>>> > 3.4.0.
>>> >
>>> > What we would like is for anyone who can to verify the tarballs,
>>> especially
>>> > anyone who can try the arm64 binaries as we want to include them too.
>>> >
>>> > The RC is available at:
>>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >
>>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >
>>> > The maven artifacts are staged at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >
>>> > You can find my public key at:
>>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >
>>> > Change log
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >
>>> > Release notes
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >
>>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>>> >
>>> > Key changes include
>>> >
>>> > * S3A: Upgrade AWS SDK to V2
>>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>>> > * YARN Federation improvements
>>> > * YARN Capacity Scheduler improvements
>>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> > * HDFS EC: Code Enhancements and Bug Fixes
>>> > * Transitive CVE fixes
>>> >
>>> > Differences from Hadoop-3.4.0-RC2
>>> >
>>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>>> > (80b4bb68159c)
>>> > * Use hadoop-release-support[1] for packaging and verification.
>>> > * Add protobuf compatibility issue description
>>> >
>>> > Note, because the arm64 binaries are built separately on a different
>>> > platform and JVM, their jar files may not match those of the x86
>>> > release -and therefore the maven artifacts. I don't think this is
>>> > an issue (the ASF actually releases source tarballs, the binaries are
>>> > there for help only, though with the maven repo that's a bit blurred).
>>> >
>>> > The only way to be consistent would actually untar the x86.tar.gz,
>>> > overwrite its binaries with the arm stuff, retar, sign and push out
>>> > for the vote. Even automating that would be risky.
>>> >
>>> > [1] hadoop-release-support:
>>> > https://github.com/apache/hadoop-release-support
>>> > Thanks to steve for providing hadoop-release-support.
>>> >
>>> > Best Regards,
>>> > Shilun Fan.
>>> >
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
Just gave a quick try, Hive master didn't compile. I gave up post that...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hive-common: Compilation failure
[ERROR]
/Users/ayushsaxena/code/hive/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java:[23,37]
package org.apache.hadoop.log.metrics does not exist
I think it due to HADOOP-17524
<https://issues.apache.org/jira/browse/HADOOP-17524>, it will be quite a
effort to upgrade from 3.3.x to 3.4.x, I don't think we would chase that
upgrade in hive anytime soon
I will abstain
-Ayush
On Thu, 7 Mar 2024 at 08:33, Xiaoqiao He <he...@apache.org> wrote:
> cc @Chao Sun <su...@apache.org> @zhangduo@apache.org
> <zh...@apache.org> @Ayush Saxena <ay...@gmail.com>
> Would you mind helping to check the upstream systems(Spark, HBase, Hive)
> dependency with
> the new hadoop release version if necessary? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
>
>> Shilun, Thanks for your great work. I am checking but not finished yet.
>> Will confirm once done.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>>> @Steve
>>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>>> <mt...@cloudera.com> @Takanobu
>>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>>
>>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>>
>>> > Hi folks,
>>> >
>>> > Xiaoqiao He and I have put together a release candidate (RC3) for
>>> Hadoop
>>> > 3.4.0.
>>> >
>>> > What we would like is for anyone who can to verify the tarballs,
>>> especially
>>> > anyone who can try the arm64 binaries as we want to include them too.
>>> >
>>> > The RC is available at:
>>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >
>>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >
>>> > The maven artifacts are staged at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >
>>> > You can find my public key at:
>>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >
>>> > Change log
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >
>>> > Release notes
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >
>>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>>> >
>>> > Key changes include
>>> >
>>> > * S3A: Upgrade AWS SDK to V2
>>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>>> > * YARN Federation improvements
>>> > * YARN Capacity Scheduler improvements
>>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> > * HDFS EC: Code Enhancements and Bug Fixes
>>> > * Transitive CVE fixes
>>> >
>>> > Differences from Hadoop-3.4.0-RC2
>>> >
>>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>>> > (80b4bb68159c)
>>> > * Use hadoop-release-support[1] for packaging and verification.
>>> > * Add protobuf compatibility issue description
>>> >
>>> > Note, because the arm64 binaries are built separately on a different
>>> > platform and JVM, their jar files may not match those of the x86
>>> > release -and therefore the maven artifacts. I don't think this is
>>> > an issue (the ASF actually releases source tarballs, the binaries are
>>> > there for help only, though with the maven repo that's a bit blurred).
>>> >
>>> > The only way to be consistent would actually untar the x86.tar.gz,
>>> > overwrite its binaries with the arm stuff, retar, sign and push out
>>> > for the vote. Even automating that would be risky.
>>> >
>>> > [1] hadoop-release-support:
>>> > https://github.com/apache/hadoop-release-support
>>> > Thanks to steve for providing hadoop-release-support.
>>> >
>>> > Best Regards,
>>> > Shilun Fan.
>>> >
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
Just gave a quick try, Hive master didn't compile. I gave up post that...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hive-common: Compilation failure
[ERROR]
/Users/ayushsaxena/code/hive/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java:[23,37]
package org.apache.hadoop.log.metrics does not exist
I think it due to HADOOP-17524
<https://issues.apache.org/jira/browse/HADOOP-17524>, it will be quite a
effort to upgrade from 3.3.x to 3.4.x, I don't think we would chase that
upgrade in hive anytime soon
I will abstain
-Ayush
On Thu, 7 Mar 2024 at 08:33, Xiaoqiao He <he...@apache.org> wrote:
> cc @Chao Sun <su...@apache.org> @zhangduo@apache.org
> <zh...@apache.org> @Ayush Saxena <ay...@gmail.com>
> Would you mind helping to check the upstream systems(Spark, HBase, Hive)
> dependency with
> the new hadoop release version if necessary? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
>
>> Shilun, Thanks for your great work. I am checking but not finished yet.
>> Will confirm once done.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>>> @Steve
>>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>>> <mt...@cloudera.com> @Takanobu
>>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>>
>>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>>
>>> > Hi folks,
>>> >
>>> > Xiaoqiao He and I have put together a release candidate (RC3) for
>>> Hadoop
>>> > 3.4.0.
>>> >
>>> > What we would like is for anyone who can to verify the tarballs,
>>> especially
>>> > anyone who can try the arm64 binaries as we want to include them too.
>>> >
>>> > The RC is available at:
>>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >
>>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >
>>> > The maven artifacts are staged at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >
>>> > You can find my public key at:
>>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >
>>> > Change log
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >
>>> > Release notes
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >
>>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>>> >
>>> > Key changes include
>>> >
>>> > * S3A: Upgrade AWS SDK to V2
>>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>>> > * YARN Federation improvements
>>> > * YARN Capacity Scheduler improvements
>>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> > * HDFS EC: Code Enhancements and Bug Fixes
>>> > * Transitive CVE fixes
>>> >
>>> > Differences from Hadoop-3.4.0-RC2
>>> >
>>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>>> > (80b4bb68159c)
>>> > * Use hadoop-release-support[1] for packaging and verification.
>>> > * Add protobuf compatibility issue description
>>> >
>>> > Note, because the arm64 binaries are built separately on a different
>>> > platform and JVM, their jar files may not match those of the x86
>>> > release -and therefore the maven artifacts. I don't think this is
>>> > an issue (the ASF actually releases source tarballs, the binaries are
>>> > there for help only, though with the maven repo that's a bit blurred).
>>> >
>>> > The only way to be consistent would actually untar the x86.tar.gz,
>>> > overwrite its binaries with the arm stuff, retar, sign and push out
>>> > for the vote. Even automating that would be risky.
>>> >
>>> > [1] hadoop-release-support:
>>> > https://github.com/apache/hadoop-release-support
>>> > Thanks to steve for providing hadoop-release-support.
>>> >
>>> > Best Regards,
>>> > Shilun Fan.
>>> >
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
Just gave a quick try, Hive master didn't compile. I gave up post that...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hive-common: Compilation failure
[ERROR]
/Users/ayushsaxena/code/hive/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java:[23,37]
package org.apache.hadoop.log.metrics does not exist
I think it due to HADOOP-17524
<https://issues.apache.org/jira/browse/HADOOP-17524>, it will be quite a
effort to upgrade from 3.3.x to 3.4.x, I don't think we would chase that
upgrade in hive anytime soon
I will abstain
-Ayush
On Thu, 7 Mar 2024 at 08:33, Xiaoqiao He <he...@apache.org> wrote:
> cc @Chao Sun <su...@apache.org> @zhangduo@apache.org
> <zh...@apache.org> @Ayush Saxena <ay...@gmail.com>
> Would you mind helping to check the upstream systems(Spark, HBase, Hive)
> dependency with
> the new hadoop release version if necessary? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
>
>> Shilun, Thanks for your great work. I am checking but not finished yet.
>> Will confirm once done.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>>
>>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>>> @Steve
>>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>>> <mt...@cloudera.com> @Takanobu
>>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>>
>>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>>
>>> > Hi folks,
>>> >
>>> > Xiaoqiao He and I have put together a release candidate (RC3) for
>>> Hadoop
>>> > 3.4.0.
>>> >
>>> > What we would like is for anyone who can to verify the tarballs,
>>> especially
>>> > anyone who can try the arm64 binaries as we want to include them too.
>>> >
>>> > The RC is available at:
>>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >
>>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >
>>> > The maven artifacts are staged at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >
>>> > You can find my public key at:
>>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >
>>> > Change log
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >
>>> > Release notes
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >
>>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>>> >
>>> > Key changes include
>>> >
>>> > * S3A: Upgrade AWS SDK to V2
>>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>>> > * YARN Federation improvements
>>> > * YARN Capacity Scheduler improvements
>>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> > * HDFS EC: Code Enhancements and Bug Fixes
>>> > * Transitive CVE fixes
>>> >
>>> > Differences from Hadoop-3.4.0-RC2
>>> >
>>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>>> > (80b4bb68159c)
>>> > * Use hadoop-release-support[1] for packaging and verification.
>>> > * Add protobuf compatibility issue description
>>> >
>>> > Note, because the arm64 binaries are built separately on a different
>>> > platform and JVM, their jar files may not match those of the x86
>>> > release -and therefore the maven artifacts. I don't think this is
>>> > an issue (the ASF actually releases source tarballs, the binaries are
>>> > there for help only, though with the maven repo that's a bit blurred).
>>> >
>>> > The only way to be consistent would actually untar the x86.tar.gz,
>>> > overwrite its binaries with the arm stuff, retar, sign and push out
>>> > for the vote. Even automating that would be risky.
>>> >
>>> > [1] hadoop-release-support:
>>> > https://github.com/apache/hadoop-release-support
>>> > Thanks to steve for providing hadoop-release-support.
>>> >
>>> > Best Regards,
>>> > Shilun Fan.
>>> >
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
cc @Chao Sun <su...@apache.org> @zhangduo@apache.org <zh...@apache.org>
@Ayush Saxena <ay...@gmail.com>
Would you mind helping to check the upstream systems(Spark, HBase, Hive)
dependency with
the new hadoop release version if necessary? Thanks.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
> Shilun, Thanks for your great work. I am checking but not finished yet.
> Will confirm once done.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>
>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>> @Steve
>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>> <mt...@cloudera.com> @Takanobu
>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>
>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hi folks,
>> >
>> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
>> > 3.4.0.
>> >
>> > What we would like is for anyone who can to verify the tarballs,
>> especially
>> > anyone who can try the arm64 binaries as we want to include them too.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >
>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >
>> > The maven artifacts are staged at
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >
>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> >
>> > Key changes include
>> >
>> > * S3A: Upgrade AWS SDK to V2
>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > * YARN Federation improvements
>> > * YARN Capacity Scheduler improvements
>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > * HDFS EC: Code Enhancements and Bug Fixes
>> > * Transitive CVE fixes
>> >
>> > Differences from Hadoop-3.4.0-RC2
>> >
>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > (80b4bb68159c)
>> > * Use hadoop-release-support[1] for packaging and verification.
>> > * Add protobuf compatibility issue description
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > [1] hadoop-release-support:
>> > https://github.com/apache/hadoop-release-support
>> > Thanks to steve for providing hadoop-release-support.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
cc @Chao Sun <su...@apache.org> @zhangduo@apache.org <zh...@apache.org>
@Ayush Saxena <ay...@gmail.com>
Would you mind helping to check the upstream systems(Spark, HBase, Hive)
dependency with
the new hadoop release version if necessary? Thanks.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
> Shilun, Thanks for your great work. I am checking but not finished yet.
> Will confirm once done.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>
>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>> @Steve
>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>> <mt...@cloudera.com> @Takanobu
>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>
>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hi folks,
>> >
>> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
>> > 3.4.0.
>> >
>> > What we would like is for anyone who can to verify the tarballs,
>> especially
>> > anyone who can try the arm64 binaries as we want to include them too.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >
>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >
>> > The maven artifacts are staged at
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >
>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> >
>> > Key changes include
>> >
>> > * S3A: Upgrade AWS SDK to V2
>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > * YARN Federation improvements
>> > * YARN Capacity Scheduler improvements
>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > * HDFS EC: Code Enhancements and Bug Fixes
>> > * Transitive CVE fixes
>> >
>> > Differences from Hadoop-3.4.0-RC2
>> >
>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > (80b4bb68159c)
>> > * Use hadoop-release-support[1] for packaging and verification.
>> > * Add protobuf compatibility issue description
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > [1] hadoop-release-support:
>> > https://github.com/apache/hadoop-release-support
>> > Thanks to steve for providing hadoop-release-support.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
cc @Chao Sun <su...@apache.org> @zhangduo@apache.org <zh...@apache.org>
@Ayush Saxena <ay...@gmail.com>
Would you mind helping to check the upstream systems(Spark, HBase, Hive)
dependency with
the new hadoop release version if necessary? Thanks.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
> Shilun, Thanks for your great work. I am checking but not finished yet.
> Will confirm once done.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>
>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>> @Steve
>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>> <mt...@cloudera.com> @Takanobu
>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>
>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hi folks,
>> >
>> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
>> > 3.4.0.
>> >
>> > What we would like is for anyone who can to verify the tarballs,
>> especially
>> > anyone who can try the arm64 binaries as we want to include them too.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >
>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >
>> > The maven artifacts are staged at
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >
>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> >
>> > Key changes include
>> >
>> > * S3A: Upgrade AWS SDK to V2
>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > * YARN Federation improvements
>> > * YARN Capacity Scheduler improvements
>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > * HDFS EC: Code Enhancements and Bug Fixes
>> > * Transitive CVE fixes
>> >
>> > Differences from Hadoop-3.4.0-RC2
>> >
>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > (80b4bb68159c)
>> > * Use hadoop-release-support[1] for packaging and verification.
>> > * Add protobuf compatibility issue description
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > [1] hadoop-release-support:
>> > https://github.com/apache/hadoop-release-support
>> > Thanks to steve for providing hadoop-release-support.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
cc @Chao Sun <su...@apache.org> @zhangduo@apache.org <zh...@apache.org>
@Ayush Saxena <ay...@gmail.com>
Would you mind helping to check the upstream systems(Spark, HBase, Hive)
dependency with
the new hadoop release version if necessary? Thanks.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He <he...@apache.org> wrote:
> Shilun, Thanks for your great work. I am checking but not finished yet.
> Will confirm once done.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
>
>> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
>> @Steve
>> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
>> <mt...@cloudera.com> @Takanobu
>> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>>
>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>>
>> > Hi folks,
>> >
>> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
>> > 3.4.0.
>> >
>> > What we would like is for anyone who can to verify the tarballs,
>> especially
>> > anyone who can try the arm64 binaries as we want to include them too.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >
>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >
>> > The maven artifacts are staged at
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >
>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> >
>> > Key changes include
>> >
>> > * S3A: Upgrade AWS SDK to V2
>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > * YARN Federation improvements
>> > * YARN Capacity Scheduler improvements
>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > * HDFS EC: Code Enhancements and Bug Fixes
>> > * Transitive CVE fixes
>> >
>> > Differences from Hadoop-3.4.0-RC2
>> >
>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > (80b4bb68159c)
>> > * Use hadoop-release-support[1] for packaging and verification.
>> > * Add protobuf compatibility issue description
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > [1] hadoop-release-support:
>> > https://github.com/apache/hadoop-release-support
>> > Thanks to steve for providing hadoop-release-support.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Shilun, Thanks for your great work. I am checking but not finished yet.
Will confirm once done.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
> @Steve
> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
> <mt...@cloudera.com> @Takanobu
> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>
> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Shilun, Thanks for your great work. I am checking but not finished yet.
Will confirm once done.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
> @Steve
> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
> <mt...@cloudera.com> @Takanobu
> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>
> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Shilun, Thanks for your great work. I am checking but not finished yet.
Will confirm once done.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
> @Steve
> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
> <mt...@cloudera.com> @Takanobu
> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>
> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Shilun, Thanks for your great work. I am checking but not finished yet.
Will confirm once done.
Best Regards,
- He Xiaoqiao
On Thu, Mar 7, 2024 at 7:23 AM slfan1989 <sl...@apache.org> wrote:
> @Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com>
> @Steve
> Loughran <st...@cloudera.com> @Mukund Madhav Thakur
> <mt...@cloudera.com> @Takanobu
> Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
>
> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
@Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com> @Steve
Loughran <st...@cloudera.com> @Mukund Madhav Thakur
<mt...@cloudera.com> @Takanobu
Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
Best Regards,
Shilun Fan.
On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
+1 (binding)
Thanks for driving the release, Shilun Fan.
- Verified checksums and signatures.
- Build the source on macOS and Java 8.
- Ran AWS and Azure tests ( some known failures, created Jira's
HADOOP-19106 it is a test side issue only mostly related to test configs,
also as Ahmar pointed out)
- Untar the binaries and ran hadoop fs commands on S3 and ABFS buckets.
- Compiled gcs-connector and ran tests using the 3.4.0 version.
On Mon, Mar 11, 2024 at 4:10 AM Xiaoqiao He <he...@apache.org> wrote:
> +1 (binding)
>
> [x] Verified signature and checksum.
> [x] LICENSE files exist and NOTICE is included.
> [x] Rat check is ok. mvn clean apache-rat:check
> [x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> [x] Setup pseudo cluster with HDFS and YARN.
> [x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> [x] Run example mr jobs and check the result - Pi & wordcount.
> [x] Spot-check and run some unit tests.
> [x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> [x] Skimmed over the contents of site documentation.
> [x] Skimmed over the contents of maven repo.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
>
> > +1 (non-binding)
> >
> >
> > Thanks for driving the release, Shilun.
> >
> >
> > * Verified sha512 checksum and signature was correct for source tarball
> > * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> > * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> > * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> > agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> > * untar the binaries and able to access objects in S3 (for both standard
> > and S3 express) via hadoop fs commands.
> >
> > We have some failures in S3A tests, but these are expected due to one of
> > the public landsat buckets we relied on no longer being available, these
> > tests were fixed in HADOOP-19057
> > <https://issues.apache.org/jira/browse/HADOOP-19057>.
> >
> > ------------------------------
> > *From:* slfan1989 <sl...@apache.org>
> > *Sent:* Monday, March 4, 2024 10:07:05 PM
> > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> > private@hadoop.apache.org>
> > *Cc:* Steve Loughran; Xiaoqiao He
> > *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
+1 (binding)
Thanks for driving the release, Shilun Fan.
- Verified checksums and signatures.
- Build the source on macOS and Java 8.
- Ran AWS and Azure tests ( some known failures, created Jira's
HADOOP-19106 it is a test side issue only mostly related to test configs,
also as Ahmar pointed out)
- Untar the binaries and ran hadoop fs commands on S3 and ABFS buckets.
- Compiled gcs-connector and ran tests using the 3.4.0 version.
On Mon, Mar 11, 2024 at 4:10 AM Xiaoqiao He <he...@apache.org> wrote:
> +1 (binding)
>
> [x] Verified signature and checksum.
> [x] LICENSE files exist and NOTICE is included.
> [x] Rat check is ok. mvn clean apache-rat:check
> [x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> [x] Setup pseudo cluster with HDFS and YARN.
> [x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> [x] Run example mr jobs and check the result - Pi & wordcount.
> [x] Spot-check and run some unit tests.
> [x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> [x] Skimmed over the contents of site documentation.
> [x] Skimmed over the contents of maven repo.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
>
> > +1 (non-binding)
> >
> >
> > Thanks for driving the release, Shilun.
> >
> >
> > * Verified sha512 checksum and signature was correct for source tarball
> > * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> > * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> > * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> > agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> > * untar the binaries and able to access objects in S3 (for both standard
> > and S3 express) via hadoop fs commands.
> >
> > We have some failures in S3A tests, but these are expected due to one of
> > the public landsat buckets we relied on no longer being available, these
> > tests were fixed in HADOOP-19057
> > <https://issues.apache.org/jira/browse/HADOOP-19057>.
> >
> > ------------------------------
> > *From:* slfan1989 <sl...@apache.org>
> > *Sent:* Monday, March 4, 2024 10:07:05 PM
> > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> > private@hadoop.apache.org>
> > *Cc:* Steve Loughran; Xiaoqiao He
> > *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
+1 (binding)
Thanks for driving the release, Shilun Fan.
- Verified checksums and signatures.
- Build the source on macOS and Java 8.
- Ran AWS and Azure tests ( some known failures, created Jira's
HADOOP-19106 it is a test side issue only mostly related to test configs,
also as Ahmar pointed out)
- Untar the binaries and ran hadoop fs commands on S3 and ABFS buckets.
- Compiled gcs-connector and ran tests using the 3.4.0 version.
On Mon, Mar 11, 2024 at 4:10 AM Xiaoqiao He <he...@apache.org> wrote:
> +1 (binding)
>
> [x] Verified signature and checksum.
> [x] LICENSE files exist and NOTICE is included.
> [x] Rat check is ok. mvn clean apache-rat:check
> [x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> [x] Setup pseudo cluster with HDFS and YARN.
> [x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> [x] Run example mr jobs and check the result - Pi & wordcount.
> [x] Spot-check and run some unit tests.
> [x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> [x] Skimmed over the contents of site documentation.
> [x] Skimmed over the contents of maven repo.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
>
> > +1 (non-binding)
> >
> >
> > Thanks for driving the release, Shilun.
> >
> >
> > * Verified sha512 checksum and signature was correct for source tarball
> > * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> > * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> > * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> > agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> > * untar the binaries and able to access objects in S3 (for both standard
> > and S3 express) via hadoop fs commands.
> >
> > We have some failures in S3A tests, but these are expected due to one of
> > the public landsat buckets we relied on no longer being available, these
> > tests were fixed in HADOOP-19057
> > <https://issues.apache.org/jira/browse/HADOOP-19057>.
> >
> > ------------------------------
> > *From:* slfan1989 <sl...@apache.org>
> > *Sent:* Monday, March 4, 2024 10:07:05 PM
> > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> > private@hadoop.apache.org>
> > *Cc:* Steve Loughran; Xiaoqiao He
> > *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
+1 (binding)
Thanks for driving the release, Shilun Fan.
- Verified checksums and signatures.
- Build the source on macOS and Java 8.
- Ran AWS and Azure tests ( some known failures, created Jira's
HADOOP-19106 it is a test side issue only mostly related to test configs,
also as Ahmar pointed out)
- Untar the binaries and ran hadoop fs commands on S3 and ABFS buckets.
- Compiled gcs-connector and ran tests using the 3.4.0 version.
On Mon, Mar 11, 2024 at 4:10 AM Xiaoqiao He <he...@apache.org> wrote:
> +1 (binding)
>
> [x] Verified signature and checksum.
> [x] LICENSE files exist and NOTICE is included.
> [x] Rat check is ok. mvn clean apache-rat:check
> [x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> [x] Setup pseudo cluster with HDFS and YARN.
> [x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> [x] Run example mr jobs and check the result - Pi & wordcount.
> [x] Spot-check and run some unit tests.
> [x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> [x] Skimmed over the contents of site documentation.
> [x] Skimmed over the contents of maven repo.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
>
> > +1 (non-binding)
> >
> >
> > Thanks for driving the release, Shilun.
> >
> >
> > * Verified sha512 checksum and signature was correct for source tarball
> > * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> > * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> > * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> > agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> > * untar the binaries and able to access objects in S3 (for both standard
> > and S3 express) via hadoop fs commands.
> >
> > We have some failures in S3A tests, but these are expected due to one of
> > the public landsat buckets we relied on no longer being available, these
> > tests were fixed in HADOOP-19057
> > <https://issues.apache.org/jira/browse/HADOOP-19057>.
> >
> > ------------------------------
> > *From:* slfan1989 <sl...@apache.org>
> > *Sent:* Monday, March 4, 2024 10:07:05 PM
> > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> > private@hadoop.apache.org>
> > *Cc:* Steve Loughran; Xiaoqiao He
> > *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
+1 (binding)
[x] Verified signature and checksum.
[x] LICENSE files exist and NOTICE is included.
[x] Rat check is ok. mvn clean apache-rat:check
[x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
[x] Setup pseudo cluster with HDFS and YARN.
[x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
result.
[x] Run example mr jobs and check the result - Pi & wordcount.
[x] Spot-check and run some unit tests.
[x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
[x] Skimmed over the contents of site documentation.
[x] Skimmed over the contents of maven repo.
Best Regards,
- He Xiaoqiao
On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
> +1 (non-binding)
>
>
> Thanks for driving the release, Shilun.
>
>
> * Verified sha512 checksum and signature was correct for source tarball
> * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> * untar the binaries and able to access objects in S3 (for both standard
> and S3 express) via hadoop fs commands.
>
> We have some failures in S3A tests, but these are expected due to one of
> the public landsat buckets we relied on no longer being available, these
> tests were fixed in HADOOP-19057
> <https://issues.apache.org/jira/browse/HADOOP-19057>.
>
> ------------------------------
> *From:* slfan1989 <sl...@apache.org>
> *Sent:* Monday, March 4, 2024 10:07:05 PM
> *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> private@hadoop.apache.org>
> *Cc:* Steve Loughran; Xiaoqiao He
> *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
+1 (binding)
[x] Verified signature and checksum.
[x] LICENSE files exist and NOTICE is included.
[x] Rat check is ok. mvn clean apache-rat:check
[x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
[x] Setup pseudo cluster with HDFS and YARN.
[x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
result.
[x] Run example mr jobs and check the result - Pi & wordcount.
[x] Spot-check and run some unit tests.
[x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
[x] Skimmed over the contents of site documentation.
[x] Skimmed over the contents of maven repo.
Best Regards,
- He Xiaoqiao
On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
> +1 (non-binding)
>
>
> Thanks for driving the release, Shilun.
>
>
> * Verified sha512 checksum and signature was correct for source tarball
> * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> * untar the binaries and able to access objects in S3 (for both standard
> and S3 express) via hadoop fs commands.
>
> We have some failures in S3A tests, but these are expected due to one of
> the public landsat buckets we relied on no longer being available, these
> tests were fixed in HADOOP-19057
> <https://issues.apache.org/jira/browse/HADOOP-19057>.
>
> ------------------------------
> *From:* slfan1989 <sl...@apache.org>
> *Sent:* Monday, March 4, 2024 10:07:05 PM
> *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> private@hadoop.apache.org>
> *Cc:* Steve Loughran; Xiaoqiao He
> *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
+1 (binding)
[x] Verified signature and checksum.
[x] LICENSE files exist and NOTICE is included.
[x] Rat check is ok. mvn clean apache-rat:check
[x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
[x] Setup pseudo cluster with HDFS and YARN.
[x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
result.
[x] Run example mr jobs and check the result - Pi & wordcount.
[x] Spot-check and run some unit tests.
[x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
[x] Skimmed over the contents of site documentation.
[x] Skimmed over the contents of maven repo.
Best Regards,
- He Xiaoqiao
On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
> +1 (non-binding)
>
>
> Thanks for driving the release, Shilun.
>
>
> * Verified sha512 checksum and signature was correct for source tarball
> * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> * untar the binaries and able to access objects in S3 (for both standard
> and S3 express) via hadoop fs commands.
>
> We have some failures in S3A tests, but these are expected due to one of
> the public landsat buckets we relied on no longer being available, these
> tests were fixed in HADOOP-19057
> <https://issues.apache.org/jira/browse/HADOOP-19057>.
>
> ------------------------------
> *From:* slfan1989 <sl...@apache.org>
> *Sent:* Monday, March 4, 2024 10:07:05 PM
> *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> private@hadoop.apache.org>
> *Cc:* Steve Loughran; Xiaoqiao He
> *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
+1 (binding)
[x] Verified signature and checksum.
[x] LICENSE files exist and NOTICE is included.
[x] Rat check is ok. mvn clean apache-rat:check
[x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
[x] Setup pseudo cluster with HDFS and YARN.
[x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
result.
[x] Run example mr jobs and check the result - Pi & wordcount.
[x] Spot-check and run some unit tests.
[x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
[x] Skimmed over the contents of site documentation.
[x] Skimmed over the contents of maven repo.
Best Regards,
- He Xiaoqiao
On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar <ah...@amazon.co.uk> wrote:
> +1 (non-binding)
>
>
> Thanks for driving the release, Shilun.
>
>
> * Verified sha512 checksum and signature was correct for source tarball
> * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> * untar the binaries and able to access objects in S3 (for both standard
> and S3 express) via hadoop fs commands.
>
> We have some failures in S3A tests, but these are expected due to one of
> the public landsat buckets we relied on no longer being available, these
> tests were fixed in HADOOP-19057
> <https://issues.apache.org/jira/browse/HADOOP-19057>.
>
> ------------------------------
> *From:* slfan1989 <sl...@apache.org>
> *Sent:* Monday, March 4, 2024 10:07:05 PM
> *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> private@hadoop.apache.org>
> *Cc:* Steve Loughran; Xiaoqiao He
> *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
+1 (non-binding)
Thanks for driving the release, Shilun.
* Verified sha512 checksum and signature was correct for source tarball
* Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
* Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
* untar the binaries and able to access objects in S3 (for both standard and S3 express) via hadoop fs commands.
We have some failures in S3A tests, but these are expected due to one of the public landsat buckets we relied on no longer being available, these tests were fixed in HADOOP-19057<https://issues.apache.org/jira/browse/HADOOP-19057>.
________________________________
From: slfan1989 <sl...@apache.org>
Sent: Monday, March 4, 2024 10:07:05 PM
To: Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <pr...@hadoop.apache.org>
Cc: Steve Loughran; Xiaoqiao He
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi folks,
Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
The git tag is release-3.4.0-RC3, commit bd8b77f398f
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408
You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
This is off branch-3.4.0 and is the first big release since 3.3.6.
Key changes include
* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes
Differences from Hadoop-3.4.0-RC2
* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description
Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).
The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.
[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.
Best Regards,
Shilun Fan.
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Dongjoon Hyun <do...@apache.org>.
+1
I tested with Apache ORC repo.
Thank you for the new release.
Dongjoon.
On 2024/03/11 20:28:22 Steve Loughran wrote:
> +1 binding
>
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
>
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
>
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
>
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
>
> The core changes I needed in are at the dependency level and I've
> verified they are good.
>
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
>
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
>
>
>
>
>
> On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs, especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Dongjoon Hyun <do...@apache.org>.
+1
I tested with Apache ORC repo.
Thank you for the new release.
Dongjoon.
(PS. My previous vote went to yarn-dev@ for some reasons, too)
On 2024/03/14 08:40:29 Takanobu Asanuma wrote:
> +1 (binding).
>
> Thanks for the great work, Shilun Fan.
>
> * Verified signatures and checksums
> * Successfully built from source with native code
> * Deployed a distributed cluster (on K8s)
> * Successfully ran some Erasure Coding operations with ISA-L codec
> * Successfully ran some HDFS RBF operations
>
> Regards,
> - Takanobu Asanuma
>
> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>
> > Thanks Ayush for highlighting this information. Absolutely true, we should
> > count RM's vote when explicit +1 here.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > > Counter should be with yourself vote, where the current summary
> > > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > >
> > > Just on the process: The release manager needs to "explicitly" vote like
> > > any other before counting their own vote, there has been a lot of
> > > discussions around that at multiple places & the official apache doc has
> > > been updated as well [1], the last paragraph reads:
> > >
> > > "Note that there is no implicit +1 from the release manager, or from
> > > anyone in any ASF vote. Only explicit votes are valid. The release
> > manager
> > > is encouraged to vote on releases, like any reviewer would do."
> > >
> > > So, do put an explicit +1, before you count yourself. Good Luck!!!
> > >
> > > -Ayush
> > >
> > > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> > >
> > > On Tue, 12 Mar 2024 at 17:27, Steve Loughran <stevel@cloudera.com.invalid
> > >
> > > wrote:
> > >
> > >> followup: overnight work happy too.
> > >>
> > >> one interesting pain point is that on a raspberry pi 64 os checknative
> > >> complains that libcrypto is missing
> > >>
> > >> > bin/hadoop checknative
> > >>
> > >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> > >> initialized native-bzip2 library system-native
> > >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> > >> initialized native-zlib library
> > >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
> > support
> > >> is not available in your platform... using builtin-java codec where
> > >> applicable
> > >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
> > built
> > >> without PMDK support.
> > >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
> > OpenSSL
> > >> Cipher.
> > >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> > >> cannot open shared object file: No such file or directory)!
> > >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> > >> at
> > >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> > >> at
> > >>
> > >>
> > org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> > >> Native library checking:
> > >> hadoop: true
> > >>
> > >>
> > /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> > >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> > >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> > >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> > >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
> > shared
> > >> object file: No such file or directory)!
> > >> ISA-L: false libhadoop was built without ISA-L support
> > >> PMDK: false The native code was built without PMDK support.
> > >>
> > >> which happens because its not in /lib/aarch64-linux-gnu but instead in
> > >> /usr/lib/aarch64-linux-gnu/l
> > >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> > >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> > >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> > >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> > >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
> > >>
> > >> Anyone got any insights on how I should set up this (debian-based) OS
> > >> here?
> > >> I know it's only a small box but with arm64 VMs becoming available in
> > >> cloud
> > >> infras, it'd be good to know if they are similar.
> > >>
> > >> Note: checknative itself is happy; but checknative -a will fail because
> > of
> > >> this -though it's an OS setup issue, nothing related to the hadoop
> > >> binaries.
> > >>
> > >> steve
> > >>
> > >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
> > wrote:
> > >>
> > >> > Hi Shilun, Counter should be with yourself vote, where the current
> > >> summary
> > >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > >> > Thanks again.
> > >> >
> > >> > Best Regards,
> > >> > - He Xiaoqiao
> > >> >
> > >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
> > wrote:
> > >> >
> > >> > > As of now, we have collected 5 affirmative votes, with 4 votes
> > binding
> > >> > and
> > >> > > 1 vote non-binding.
> > >> > >
> > >> > > Thank you very much for voting and verifying!
> > >> > >
> > >> > > This voting will continue until March 15th, this Friday.
> > >> > >
> > >> > > Best Regards,
> > >> > > Shilun Fan.
> > >> > >
> > >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> > >> > <stevel@cloudera.com.invalid
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > +1 binding
> > >> > > >
> > >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> > >> seen
> > >> > it
> > >> > > > earlier. been testing it this afternoon:
> > >> > > >
> > >> > > > pulled the latest version of
> > >> > > > https://github.com/apache/hadoop-release-support
> > >> > > > (note, this module is commit-then-review; whoever is working
> > >> > > on/validating
> > >> > > > a release can commit as they go along. This is not production
> > >> code...)
> > >> > > >
> > >> > > > * went through the "validating a release" step, validating maven
> > >> > > artifacts
> > >> > > > * building the same downstream modules which built for me last
> > time
> > >> > (avro
> > >> > > > too complex; hboss not aws v2 in apache yet)
> > >> > > >
> > >> > > > spark build is still ongoing, but I'm not going to wait. It is
> > >> > building,
> > >> > > > which is key.
> > >> > > >
> > >> > > > The core changes I needed in are at the dependency level and I've
> > >> > > > verified they are good.
> > >> > > >
> > >> > > > Oh, and I've also got my raspberry p5 doing the download of the
> > arm
> > >> > > > stuff for its checknative; not expecting problems.
> > >> > > >
> > >> > > > So: i've got some stuff still ongoing, but the core changes to
> > >> > packaging
> > >> > > > are in and the rest I'm not worried about -they shouldn't block
> > the
> > >> > > release
> > >> > > > as I already validated them on RC2
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
> > >> wrote:
> > >> > > >
> > >> > > > > Hi folks,
> > >> > > > >
> > >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
> > for
> > >> > > Hadoop
> > >> > > > > 3.4.0.
> > >> > > > >
> > >> > > > > What we would like is for anyone who can to verify the tarballs,
> > >> > > > especially
> > >> > > > > anyone who can try the arm64 binaries as we want to include them
> > >> too.
> > >> > > > >
> > >> > > > > The RC is available at:
> > >> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > >> > > > >
> > >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > >> > > > >
> > >> > > > > The maven artifacts are staged at
> > >> > > > >
> > >> > >
> > >> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > >> > > > >
> > >> > > > > You can find my public key at:
> > >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >> > > > >
> > >> > > > > Change log
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > >> > > > >
> > >> > > > > Release notes
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > >> > > > >
> > >> > > > > This is off branch-3.4.0 and is the first big release since
> > 3.3.6.
> > >> > > > >
> > >> > > > > Key changes include
> > >> > > > >
> > >> > > > > * S3A: Upgrade AWS SDK to V2
> > >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
> > locks
> > >> > > > > * YARN Federation improvements
> > >> > > > > * YARN Capacity Scheduler improvements
> > >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> > >> > > > > * Transitive CVE fixes
> > >> > > > >
> > >> > > > > Differences from Hadoop-3.4.0-RC2
> > >> > > > >
> > >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> > >> (ad8b6541117b)
> > >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
> > dependencies.
> > >> > > > > (80b4bb68159c)
> > >> > > > > * Use hadoop-release-support[1] for packaging and verification.
> > >> > > > > * Add protobuf compatibility issue description
> > >> > > > >
> > >> > > > > Note, because the arm64 binaries are built separately on a
> > >> different
> > >> > > > > platform and JVM, their jar files may not match those of the x86
> > >> > > > > release -and therefore the maven artifacts. I don't think this
> > is
> > >> > > > > an issue (the ASF actually releases source tarballs, the
> > binaries
> > >> are
> > >> > > > > there for help only, though with the maven repo that's a bit
> > >> > blurred).
> > >> > > > >
> > >> > > > > The only way to be consistent would actually untar the
> > x86.tar.gz,
> > >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
> > >> out
> > >> > > > > for the vote. Even automating that would be risky.
> > >> > > > >
> > >> > > > > [1] hadoop-release-support:
> > >> > > > > https://github.com/apache/hadoop-release-support
> > >> > > > > Thanks to steve for providing hadoop-release-support.
> > >> > > > >
> > >> > > > > Best Regards,
> > >> > > > > Shilun Fan.
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
Hi folks,
The voting for hadoop-3.4.0-RC3 has ended, and we have received a total of
8 affirmative votes, with 6 binding and 2 non-binding.
In the order of the voting timeline:
binding (+6)
Masatake Iwasaki
Xiaoqiao He
Mukund Madhav Thakur
Steve Loughran
Takanobu Asanuma
Shilun Fan
non-binding(+2)
Suhail, Ahmar
Dongjoon Hyun
Thank you very much for helping with the review and validation!
We have received 6 binding PMC affirmative votes. hadoop-3.4.0 will be
officially released in the next 1-2 days.
Best Regards,
Shilun Fan.
On Fri, Mar 15, 2024 at 8:32 AM slfan1989 <sl...@apache.org> wrote:
> +1 (binding).
>
> * Verified the signature and checksum of all tarballs.
> * Built from the source tarball on macOS 10.14 and OpenJDK 8.
> * Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
> * Check the Yarn federation page and basic functionalities.
> * Read the basic help documentation.
>
> Best Regards,
> Shilun Fan.
>
> On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1 (binding).
>>
>> Thanks for the great work, Shilun Fan.
>>
>> * Verified signatures and checksums
>> * Successfully built from source with native code
>> * Deployed a distributed cluster (on K8s)
>> * Successfully ran some Erasure Coding operations with ISA-L codec
>> * Successfully ran some HDFS RBF operations
>>
>> Regards,
>> - Takanobu Asanuma
>>
>> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>>
>>> Thanks Ayush for highlighting this information. Absolutely true, we
>>> should
>>> count RM's vote when explicit +1 here.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>>
>>> > > Counter should be with yourself vote, where the current summary
>>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >
>>> > Just on the process: The release manager needs to "explicitly" vote
>>> like
>>> > any other before counting their own vote, there has been a lot of
>>> > discussions around that at multiple places & the official apache doc
>>> has
>>> > been updated as well [1], the last paragraph reads:
>>> >
>>> > "Note that there is no implicit +1 from the release manager, or from
>>> > anyone in any ASF vote. Only explicit votes are valid. The release
>>> manager
>>> > is encouraged to vote on releases, like any reviewer would do."
>>> >
>>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>>> >
>>> > -Ayush
>>> >
>>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>>> >
>>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>>> <st...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> followup: overnight work happy too.
>>> >>
>>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>>> >> complains that libcrypto is missing
>>> >>
>>> >> > bin/hadoop checknative
>>> >>
>>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>>> >> initialized native-bzip2 library system-native
>>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>>> >> initialized native-zlib library
>>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>>> support
>>> >> is not available in your platform... using builtin-java codec where
>>> >> applicable
>>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>>> built
>>> >> without PMDK support.
>>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>>> OpenSSL
>>> >> Cipher.
>>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so
>>> (libcrypto.so:
>>> >> cannot open shared object file: No such file or directory)!
>>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>>> Method)
>>> >> at
>>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>>> >> at
>>> >>
>>> >>
>>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>>> >> Native library checking:
>>> >> hadoop: true
>>> >>
>>> >>
>>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>>> shared
>>> >> object file: No such file or directory)!
>>> >> ISA-L: false libhadoop was built without ISA-L support
>>> >> PMDK: false The native code was built without PMDK support.
>>> >>
>>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>>> >> /usr/lib/aarch64-linux-gnu/l
>>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>> >>
>>> >> Anyone got any insights on how I should set up this (debian-based) OS
>>> >> here?
>>> >> I know it's only a small box but with arm64 VMs becoming available in
>>> >> cloud
>>> >> infras, it'd be good to know if they are similar.
>>> >>
>>> >> Note: checknative itself is happy; but checknative -a will fail
>>> because of
>>> >> this -though it's an OS setup issue, nothing related to the hadoop
>>> >> binaries.
>>> >>
>>> >> steve
>>> >>
>>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>>> wrote:
>>> >>
>>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>>> >> summary
>>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >> > Thanks again.
>>> >> >
>>> >> > Best Regards,
>>> >> > - He Xiaoqiao
>>> >> >
>>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>>> wrote:
>>> >> >
>>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>>> binding
>>> >> > and
>>> >> > > 1 vote non-binding.
>>> >> > >
>>> >> > > Thank you very much for voting and verifying!
>>> >> > >
>>> >> > > This voting will continue until March 15th, this Friday.
>>> >> > >
>>> >> > > Best Regards,
>>> >> > > Shilun Fan.
>>> >> > >
>>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>>> >> > <stevel@cloudera.com.invalid
>>> >> > > >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > +1 binding
>>> >> > > >
>>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd
>>> have
>>> >> seen
>>> >> > it
>>> >> > > > earlier. been testing it this afternoon:
>>> >> > > >
>>> >> > > > pulled the latest version of
>>> >> > > > https://github.com/apache/hadoop-release-support
>>> >> > > > (note, this module is commit-then-review; whoever is working
>>> >> > > on/validating
>>> >> > > > a release can commit as they go along. This is not production
>>> >> code...)
>>> >> > > >
>>> >> > > > * went through the "validating a release" step, validating maven
>>> >> > > artifacts
>>> >> > > > * building the same downstream modules which built for me last
>>> time
>>> >> > (avro
>>> >> > > > too complex; hboss not aws v2 in apache yet)
>>> >> > > >
>>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>>> >> > building,
>>> >> > > > which is key.
>>> >> > > >
>>> >> > > > The core changes I needed in are at the dependency level and
>>> I've
>>> >> > > > verified they are good.
>>> >> > > >
>>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>>> arm
>>> >> > > > stuff for its checknative; not expecting problems.
>>> >> > > >
>>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>>> >> > packaging
>>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>>> the
>>> >> > > release
>>> >> > > > as I already validated them on RC2
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>>> >> wrote:
>>> >> > > >
>>> >> > > > > Hi folks,
>>> >> > > > >
>>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>>> for
>>> >> > > Hadoop
>>> >> > > > > 3.4.0.
>>> >> > > > >
>>> >> > > > > What we would like is for anyone who can to verify the
>>> tarballs,
>>> >> > > > especially
>>> >> > > > > anyone who can try the arm64 binaries as we want to include
>>> them
>>> >> too.
>>> >> > > > >
>>> >> > > > > The RC is available at:
>>> >> > > > >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >> > > > >
>>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >> > > > >
>>> >> > > > > The maven artifacts are staged at
>>> >> > > > >
>>> >> > >
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >> > > > >
>>> >> > > > > You can find my public key at:
>>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >> > > > >
>>> >> > > > > Change log
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >> > > > >
>>> >> > > > > Release notes
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >> > > > >
>>> >> > > > > This is off branch-3.4.0 and is the first big release since
>>> 3.3.6.
>>> >> > > > >
>>> >> > > > > Key changes include
>>> >> > > > >
>>> >> > > > > * S3A: Upgrade AWS SDK to V2
>>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>>> locks
>>> >> > > > > * YARN Federation improvements
>>> >> > > > > * YARN Capacity Scheduler improvements
>>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>>> >> > > > > * Transitive CVE fixes
>>> >> > > > >
>>> >> > > > > Differences from Hadoop-3.4.0-RC2
>>> >> > > > >
>>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>>> >> (ad8b6541117b)
>>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>>> dependencies.
>>> >> > > > > (80b4bb68159c)
>>> >> > > > > * Use hadoop-release-support[1] for packaging and
>>> verification.
>>> >> > > > > * Add protobuf compatibility issue description
>>> >> > > > >
>>> >> > > > > Note, because the arm64 binaries are built separately on a
>>> >> different
>>> >> > > > > platform and JVM, their jar files may not match those of the
>>> x86
>>> >> > > > > release -and therefore the maven artifacts. I don't think
>>> this is
>>> >> > > > > an issue (the ASF actually releases source tarballs, the
>>> binaries
>>> >> are
>>> >> > > > > there for help only, though with the maven repo that's a bit
>>> >> > blurred).
>>> >> > > > >
>>> >> > > > > The only way to be consistent would actually untar the
>>> x86.tar.gz,
>>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and
>>> push
>>> >> out
>>> >> > > > > for the vote. Even automating that would be risky.
>>> >> > > > >
>>> >> > > > > [1] hadoop-release-support:
>>> >> > > > > https://github.com/apache/hadoop-release-support
>>> >> > > > > Thanks to steve for providing hadoop-release-support.
>>> >> > > > >
>>> >> > > > > Best Regards,
>>> >> > > > > Shilun Fan.
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
Hi folks,
The voting for hadoop-3.4.0-RC3 has ended, and we have received a total of
8 affirmative votes, with 6 binding and 2 non-binding.
In the order of the voting timeline:
binding (+6)
Masatake Iwasaki
Xiaoqiao He
Mukund Madhav Thakur
Steve Loughran
Takanobu Asanuma
Shilun Fan
non-binding(+2)
Suhail, Ahmar
Dongjoon Hyun
Thank you very much for helping with the review and validation!
We have received 6 binding PMC affirmative votes. hadoop-3.4.0 will be
officially released in the next 1-2 days.
Best Regards,
Shilun Fan.
On Fri, Mar 15, 2024 at 8:32 AM slfan1989 <sl...@apache.org> wrote:
> +1 (binding).
>
> * Verified the signature and checksum of all tarballs.
> * Built from the source tarball on macOS 10.14 and OpenJDK 8.
> * Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
> * Check the Yarn federation page and basic functionalities.
> * Read the basic help documentation.
>
> Best Regards,
> Shilun Fan.
>
> On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1 (binding).
>>
>> Thanks for the great work, Shilun Fan.
>>
>> * Verified signatures and checksums
>> * Successfully built from source with native code
>> * Deployed a distributed cluster (on K8s)
>> * Successfully ran some Erasure Coding operations with ISA-L codec
>> * Successfully ran some HDFS RBF operations
>>
>> Regards,
>> - Takanobu Asanuma
>>
>> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>>
>>> Thanks Ayush for highlighting this information. Absolutely true, we
>>> should
>>> count RM's vote when explicit +1 here.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>>
>>> > > Counter should be with yourself vote, where the current summary
>>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >
>>> > Just on the process: The release manager needs to "explicitly" vote
>>> like
>>> > any other before counting their own vote, there has been a lot of
>>> > discussions around that at multiple places & the official apache doc
>>> has
>>> > been updated as well [1], the last paragraph reads:
>>> >
>>> > "Note that there is no implicit +1 from the release manager, or from
>>> > anyone in any ASF vote. Only explicit votes are valid. The release
>>> manager
>>> > is encouraged to vote on releases, like any reviewer would do."
>>> >
>>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>>> >
>>> > -Ayush
>>> >
>>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>>> >
>>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>>> <st...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> followup: overnight work happy too.
>>> >>
>>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>>> >> complains that libcrypto is missing
>>> >>
>>> >> > bin/hadoop checknative
>>> >>
>>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>>> >> initialized native-bzip2 library system-native
>>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>>> >> initialized native-zlib library
>>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>>> support
>>> >> is not available in your platform... using builtin-java codec where
>>> >> applicable
>>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>>> built
>>> >> without PMDK support.
>>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>>> OpenSSL
>>> >> Cipher.
>>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so
>>> (libcrypto.so:
>>> >> cannot open shared object file: No such file or directory)!
>>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>>> Method)
>>> >> at
>>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>>> >> at
>>> >>
>>> >>
>>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>>> >> Native library checking:
>>> >> hadoop: true
>>> >>
>>> >>
>>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>>> shared
>>> >> object file: No such file or directory)!
>>> >> ISA-L: false libhadoop was built without ISA-L support
>>> >> PMDK: false The native code was built without PMDK support.
>>> >>
>>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>>> >> /usr/lib/aarch64-linux-gnu/l
>>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>> >>
>>> >> Anyone got any insights on how I should set up this (debian-based) OS
>>> >> here?
>>> >> I know it's only a small box but with arm64 VMs becoming available in
>>> >> cloud
>>> >> infras, it'd be good to know if they are similar.
>>> >>
>>> >> Note: checknative itself is happy; but checknative -a will fail
>>> because of
>>> >> this -though it's an OS setup issue, nothing related to the hadoop
>>> >> binaries.
>>> >>
>>> >> steve
>>> >>
>>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>>> wrote:
>>> >>
>>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>>> >> summary
>>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >> > Thanks again.
>>> >> >
>>> >> > Best Regards,
>>> >> > - He Xiaoqiao
>>> >> >
>>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>>> wrote:
>>> >> >
>>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>>> binding
>>> >> > and
>>> >> > > 1 vote non-binding.
>>> >> > >
>>> >> > > Thank you very much for voting and verifying!
>>> >> > >
>>> >> > > This voting will continue until March 15th, this Friday.
>>> >> > >
>>> >> > > Best Regards,
>>> >> > > Shilun Fan.
>>> >> > >
>>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>>> >> > <stevel@cloudera.com.invalid
>>> >> > > >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > +1 binding
>>> >> > > >
>>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd
>>> have
>>> >> seen
>>> >> > it
>>> >> > > > earlier. been testing it this afternoon:
>>> >> > > >
>>> >> > > > pulled the latest version of
>>> >> > > > https://github.com/apache/hadoop-release-support
>>> >> > > > (note, this module is commit-then-review; whoever is working
>>> >> > > on/validating
>>> >> > > > a release can commit as they go along. This is not production
>>> >> code...)
>>> >> > > >
>>> >> > > > * went through the "validating a release" step, validating maven
>>> >> > > artifacts
>>> >> > > > * building the same downstream modules which built for me last
>>> time
>>> >> > (avro
>>> >> > > > too complex; hboss not aws v2 in apache yet)
>>> >> > > >
>>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>>> >> > building,
>>> >> > > > which is key.
>>> >> > > >
>>> >> > > > The core changes I needed in are at the dependency level and
>>> I've
>>> >> > > > verified they are good.
>>> >> > > >
>>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>>> arm
>>> >> > > > stuff for its checknative; not expecting problems.
>>> >> > > >
>>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>>> >> > packaging
>>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>>> the
>>> >> > > release
>>> >> > > > as I already validated them on RC2
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>>> >> wrote:
>>> >> > > >
>>> >> > > > > Hi folks,
>>> >> > > > >
>>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>>> for
>>> >> > > Hadoop
>>> >> > > > > 3.4.0.
>>> >> > > > >
>>> >> > > > > What we would like is for anyone who can to verify the
>>> tarballs,
>>> >> > > > especially
>>> >> > > > > anyone who can try the arm64 binaries as we want to include
>>> them
>>> >> too.
>>> >> > > > >
>>> >> > > > > The RC is available at:
>>> >> > > > >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >> > > > >
>>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >> > > > >
>>> >> > > > > The maven artifacts are staged at
>>> >> > > > >
>>> >> > >
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >> > > > >
>>> >> > > > > You can find my public key at:
>>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >> > > > >
>>> >> > > > > Change log
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >> > > > >
>>> >> > > > > Release notes
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >> > > > >
>>> >> > > > > This is off branch-3.4.0 and is the first big release since
>>> 3.3.6.
>>> >> > > > >
>>> >> > > > > Key changes include
>>> >> > > > >
>>> >> > > > > * S3A: Upgrade AWS SDK to V2
>>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>>> locks
>>> >> > > > > * YARN Federation improvements
>>> >> > > > > * YARN Capacity Scheduler improvements
>>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>>> >> > > > > * Transitive CVE fixes
>>> >> > > > >
>>> >> > > > > Differences from Hadoop-3.4.0-RC2
>>> >> > > > >
>>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>>> >> (ad8b6541117b)
>>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>>> dependencies.
>>> >> > > > > (80b4bb68159c)
>>> >> > > > > * Use hadoop-release-support[1] for packaging and
>>> verification.
>>> >> > > > > * Add protobuf compatibility issue description
>>> >> > > > >
>>> >> > > > > Note, because the arm64 binaries are built separately on a
>>> >> different
>>> >> > > > > platform and JVM, their jar files may not match those of the
>>> x86
>>> >> > > > > release -and therefore the maven artifacts. I don't think
>>> this is
>>> >> > > > > an issue (the ASF actually releases source tarballs, the
>>> binaries
>>> >> are
>>> >> > > > > there for help only, though with the maven repo that's a bit
>>> >> > blurred).
>>> >> > > > >
>>> >> > > > > The only way to be consistent would actually untar the
>>> x86.tar.gz,
>>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and
>>> push
>>> >> out
>>> >> > > > > for the vote. Even automating that would be risky.
>>> >> > > > >
>>> >> > > > > [1] hadoop-release-support:
>>> >> > > > > https://github.com/apache/hadoop-release-support
>>> >> > > > > Thanks to steve for providing hadoop-release-support.
>>> >> > > > >
>>> >> > > > > Best Regards,
>>> >> > > > > Shilun Fan.
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
Hi folks,
The voting for hadoop-3.4.0-RC3 has ended, and we have received a total of
8 affirmative votes, with 6 binding and 2 non-binding.
In the order of the voting timeline:
binding (+6)
Masatake Iwasaki
Xiaoqiao He
Mukund Madhav Thakur
Steve Loughran
Takanobu Asanuma
Shilun Fan
non-binding(+2)
Suhail, Ahmar
Dongjoon Hyun
Thank you very much for helping with the review and validation!
We have received 6 binding PMC affirmative votes. hadoop-3.4.0 will be
officially released in the next 1-2 days.
Best Regards,
Shilun Fan.
On Fri, Mar 15, 2024 at 8:32 AM slfan1989 <sl...@apache.org> wrote:
> +1 (binding).
>
> * Verified the signature and checksum of all tarballs.
> * Built from the source tarball on macOS 10.14 and OpenJDK 8.
> * Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
> * Check the Yarn federation page and basic functionalities.
> * Read the basic help documentation.
>
> Best Regards,
> Shilun Fan.
>
> On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1 (binding).
>>
>> Thanks for the great work, Shilun Fan.
>>
>> * Verified signatures and checksums
>> * Successfully built from source with native code
>> * Deployed a distributed cluster (on K8s)
>> * Successfully ran some Erasure Coding operations with ISA-L codec
>> * Successfully ran some HDFS RBF operations
>>
>> Regards,
>> - Takanobu Asanuma
>>
>> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>>
>>> Thanks Ayush for highlighting this information. Absolutely true, we
>>> should
>>> count RM's vote when explicit +1 here.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>>
>>> > > Counter should be with yourself vote, where the current summary
>>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >
>>> > Just on the process: The release manager needs to "explicitly" vote
>>> like
>>> > any other before counting their own vote, there has been a lot of
>>> > discussions around that at multiple places & the official apache doc
>>> has
>>> > been updated as well [1], the last paragraph reads:
>>> >
>>> > "Note that there is no implicit +1 from the release manager, or from
>>> > anyone in any ASF vote. Only explicit votes are valid. The release
>>> manager
>>> > is encouraged to vote on releases, like any reviewer would do."
>>> >
>>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>>> >
>>> > -Ayush
>>> >
>>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>>> >
>>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>>> <st...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> followup: overnight work happy too.
>>> >>
>>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>>> >> complains that libcrypto is missing
>>> >>
>>> >> > bin/hadoop checknative
>>> >>
>>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>>> >> initialized native-bzip2 library system-native
>>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>>> >> initialized native-zlib library
>>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>>> support
>>> >> is not available in your platform... using builtin-java codec where
>>> >> applicable
>>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>>> built
>>> >> without PMDK support.
>>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>>> OpenSSL
>>> >> Cipher.
>>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so
>>> (libcrypto.so:
>>> >> cannot open shared object file: No such file or directory)!
>>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>>> Method)
>>> >> at
>>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>>> >> at
>>> >>
>>> >>
>>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>>> >> Native library checking:
>>> >> hadoop: true
>>> >>
>>> >>
>>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>>> shared
>>> >> object file: No such file or directory)!
>>> >> ISA-L: false libhadoop was built without ISA-L support
>>> >> PMDK: false The native code was built without PMDK support.
>>> >>
>>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>>> >> /usr/lib/aarch64-linux-gnu/l
>>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>> >>
>>> >> Anyone got any insights on how I should set up this (debian-based) OS
>>> >> here?
>>> >> I know it's only a small box but with arm64 VMs becoming available in
>>> >> cloud
>>> >> infras, it'd be good to know if they are similar.
>>> >>
>>> >> Note: checknative itself is happy; but checknative -a will fail
>>> because of
>>> >> this -though it's an OS setup issue, nothing related to the hadoop
>>> >> binaries.
>>> >>
>>> >> steve
>>> >>
>>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>>> wrote:
>>> >>
>>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>>> >> summary
>>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >> > Thanks again.
>>> >> >
>>> >> > Best Regards,
>>> >> > - He Xiaoqiao
>>> >> >
>>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>>> wrote:
>>> >> >
>>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>>> binding
>>> >> > and
>>> >> > > 1 vote non-binding.
>>> >> > >
>>> >> > > Thank you very much for voting and verifying!
>>> >> > >
>>> >> > > This voting will continue until March 15th, this Friday.
>>> >> > >
>>> >> > > Best Regards,
>>> >> > > Shilun Fan.
>>> >> > >
>>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>>> >> > <stevel@cloudera.com.invalid
>>> >> > > >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > +1 binding
>>> >> > > >
>>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd
>>> have
>>> >> seen
>>> >> > it
>>> >> > > > earlier. been testing it this afternoon:
>>> >> > > >
>>> >> > > > pulled the latest version of
>>> >> > > > https://github.com/apache/hadoop-release-support
>>> >> > > > (note, this module is commit-then-review; whoever is working
>>> >> > > on/validating
>>> >> > > > a release can commit as they go along. This is not production
>>> >> code...)
>>> >> > > >
>>> >> > > > * went through the "validating a release" step, validating maven
>>> >> > > artifacts
>>> >> > > > * building the same downstream modules which built for me last
>>> time
>>> >> > (avro
>>> >> > > > too complex; hboss not aws v2 in apache yet)
>>> >> > > >
>>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>>> >> > building,
>>> >> > > > which is key.
>>> >> > > >
>>> >> > > > The core changes I needed in are at the dependency level and
>>> I've
>>> >> > > > verified they are good.
>>> >> > > >
>>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>>> arm
>>> >> > > > stuff for its checknative; not expecting problems.
>>> >> > > >
>>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>>> >> > packaging
>>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>>> the
>>> >> > > release
>>> >> > > > as I already validated them on RC2
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>>> >> wrote:
>>> >> > > >
>>> >> > > > > Hi folks,
>>> >> > > > >
>>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>>> for
>>> >> > > Hadoop
>>> >> > > > > 3.4.0.
>>> >> > > > >
>>> >> > > > > What we would like is for anyone who can to verify the
>>> tarballs,
>>> >> > > > especially
>>> >> > > > > anyone who can try the arm64 binaries as we want to include
>>> them
>>> >> too.
>>> >> > > > >
>>> >> > > > > The RC is available at:
>>> >> > > > >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >> > > > >
>>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >> > > > >
>>> >> > > > > The maven artifacts are staged at
>>> >> > > > >
>>> >> > >
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >> > > > >
>>> >> > > > > You can find my public key at:
>>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >> > > > >
>>> >> > > > > Change log
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >> > > > >
>>> >> > > > > Release notes
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >> > > > >
>>> >> > > > > This is off branch-3.4.0 and is the first big release since
>>> 3.3.6.
>>> >> > > > >
>>> >> > > > > Key changes include
>>> >> > > > >
>>> >> > > > > * S3A: Upgrade AWS SDK to V2
>>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>>> locks
>>> >> > > > > * YARN Federation improvements
>>> >> > > > > * YARN Capacity Scheduler improvements
>>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>>> >> > > > > * Transitive CVE fixes
>>> >> > > > >
>>> >> > > > > Differences from Hadoop-3.4.0-RC2
>>> >> > > > >
>>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>>> >> (ad8b6541117b)
>>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>>> dependencies.
>>> >> > > > > (80b4bb68159c)
>>> >> > > > > * Use hadoop-release-support[1] for packaging and
>>> verification.
>>> >> > > > > * Add protobuf compatibility issue description
>>> >> > > > >
>>> >> > > > > Note, because the arm64 binaries are built separately on a
>>> >> different
>>> >> > > > > platform and JVM, their jar files may not match those of the
>>> x86
>>> >> > > > > release -and therefore the maven artifacts. I don't think
>>> this is
>>> >> > > > > an issue (the ASF actually releases source tarballs, the
>>> binaries
>>> >> are
>>> >> > > > > there for help only, though with the maven repo that's a bit
>>> >> > blurred).
>>> >> > > > >
>>> >> > > > > The only way to be consistent would actually untar the
>>> x86.tar.gz,
>>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and
>>> push
>>> >> out
>>> >> > > > > for the vote. Even automating that would be risky.
>>> >> > > > >
>>> >> > > > > [1] hadoop-release-support:
>>> >> > > > > https://github.com/apache/hadoop-release-support
>>> >> > > > > Thanks to steve for providing hadoop-release-support.
>>> >> > > > >
>>> >> > > > > Best Regards,
>>> >> > > > > Shilun Fan.
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
Hi folks,
The voting for hadoop-3.4.0-RC3 has ended, and we have received a total of
8 affirmative votes, with 6 binding and 2 non-binding.
In the order of the voting timeline:
binding (+6)
Masatake Iwasaki
Xiaoqiao He
Mukund Madhav Thakur
Steve Loughran
Takanobu Asanuma
Shilun Fan
non-binding(+2)
Suhail, Ahmar
Dongjoon Hyun
Thank you very much for helping with the review and validation!
We have received 6 binding PMC affirmative votes. hadoop-3.4.0 will be
officially released in the next 1-2 days.
Best Regards,
Shilun Fan.
On Fri, Mar 15, 2024 at 8:32 AM slfan1989 <sl...@apache.org> wrote:
> +1 (binding).
>
> * Verified the signature and checksum of all tarballs.
> * Built from the source tarball on macOS 10.14 and OpenJDK 8.
> * Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
> * Check the Yarn federation page and basic functionalities.
> * Read the basic help documentation.
>
> Best Regards,
> Shilun Fan.
>
> On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1 (binding).
>>
>> Thanks for the great work, Shilun Fan.
>>
>> * Verified signatures and checksums
>> * Successfully built from source with native code
>> * Deployed a distributed cluster (on K8s)
>> * Successfully ran some Erasure Coding operations with ISA-L codec
>> * Successfully ran some HDFS RBF operations
>>
>> Regards,
>> - Takanobu Asanuma
>>
>> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>>
>>> Thanks Ayush for highlighting this information. Absolutely true, we
>>> should
>>> count RM's vote when explicit +1 here.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>>
>>> > > Counter should be with yourself vote, where the current summary
>>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >
>>> > Just on the process: The release manager needs to "explicitly" vote
>>> like
>>> > any other before counting their own vote, there has been a lot of
>>> > discussions around that at multiple places & the official apache doc
>>> has
>>> > been updated as well [1], the last paragraph reads:
>>> >
>>> > "Note that there is no implicit +1 from the release manager, or from
>>> > anyone in any ASF vote. Only explicit votes are valid. The release
>>> manager
>>> > is encouraged to vote on releases, like any reviewer would do."
>>> >
>>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>>> >
>>> > -Ayush
>>> >
>>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>>> >
>>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>>> <st...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> followup: overnight work happy too.
>>> >>
>>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>>> >> complains that libcrypto is missing
>>> >>
>>> >> > bin/hadoop checknative
>>> >>
>>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>>> >> initialized native-bzip2 library system-native
>>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>>> >> initialized native-zlib library
>>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>>> support
>>> >> is not available in your platform... using builtin-java codec where
>>> >> applicable
>>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>>> built
>>> >> without PMDK support.
>>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>>> OpenSSL
>>> >> Cipher.
>>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so
>>> (libcrypto.so:
>>> >> cannot open shared object file: No such file or directory)!
>>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>>> Method)
>>> >> at
>>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>>> >> at
>>> >>
>>> >>
>>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>>> >> Native library checking:
>>> >> hadoop: true
>>> >>
>>> >>
>>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>>> shared
>>> >> object file: No such file or directory)!
>>> >> ISA-L: false libhadoop was built without ISA-L support
>>> >> PMDK: false The native code was built without PMDK support.
>>> >>
>>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>>> >> /usr/lib/aarch64-linux-gnu/l
>>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>> >>
>>> >> Anyone got any insights on how I should set up this (debian-based) OS
>>> >> here?
>>> >> I know it's only a small box but with arm64 VMs becoming available in
>>> >> cloud
>>> >> infras, it'd be good to know if they are similar.
>>> >>
>>> >> Note: checknative itself is happy; but checknative -a will fail
>>> because of
>>> >> this -though it's an OS setup issue, nothing related to the hadoop
>>> >> binaries.
>>> >>
>>> >> steve
>>> >>
>>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>>> wrote:
>>> >>
>>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>>> >> summary
>>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >> > Thanks again.
>>> >> >
>>> >> > Best Regards,
>>> >> > - He Xiaoqiao
>>> >> >
>>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>>> wrote:
>>> >> >
>>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>>> binding
>>> >> > and
>>> >> > > 1 vote non-binding.
>>> >> > >
>>> >> > > Thank you very much for voting and verifying!
>>> >> > >
>>> >> > > This voting will continue until March 15th, this Friday.
>>> >> > >
>>> >> > > Best Regards,
>>> >> > > Shilun Fan.
>>> >> > >
>>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>>> >> > <stevel@cloudera.com.invalid
>>> >> > > >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > +1 binding
>>> >> > > >
>>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd
>>> have
>>> >> seen
>>> >> > it
>>> >> > > > earlier. been testing it this afternoon:
>>> >> > > >
>>> >> > > > pulled the latest version of
>>> >> > > > https://github.com/apache/hadoop-release-support
>>> >> > > > (note, this module is commit-then-review; whoever is working
>>> >> > > on/validating
>>> >> > > > a release can commit as they go along. This is not production
>>> >> code...)
>>> >> > > >
>>> >> > > > * went through the "validating a release" step, validating maven
>>> >> > > artifacts
>>> >> > > > * building the same downstream modules which built for me last
>>> time
>>> >> > (avro
>>> >> > > > too complex; hboss not aws v2 in apache yet)
>>> >> > > >
>>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>>> >> > building,
>>> >> > > > which is key.
>>> >> > > >
>>> >> > > > The core changes I needed in are at the dependency level and
>>> I've
>>> >> > > > verified they are good.
>>> >> > > >
>>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>>> arm
>>> >> > > > stuff for its checknative; not expecting problems.
>>> >> > > >
>>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>>> >> > packaging
>>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>>> the
>>> >> > > release
>>> >> > > > as I already validated them on RC2
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>>> >> wrote:
>>> >> > > >
>>> >> > > > > Hi folks,
>>> >> > > > >
>>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>>> for
>>> >> > > Hadoop
>>> >> > > > > 3.4.0.
>>> >> > > > >
>>> >> > > > > What we would like is for anyone who can to verify the
>>> tarballs,
>>> >> > > > especially
>>> >> > > > > anyone who can try the arm64 binaries as we want to include
>>> them
>>> >> too.
>>> >> > > > >
>>> >> > > > > The RC is available at:
>>> >> > > > >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >> > > > >
>>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >> > > > >
>>> >> > > > > The maven artifacts are staged at
>>> >> > > > >
>>> >> > >
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >> > > > >
>>> >> > > > > You can find my public key at:
>>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >> > > > >
>>> >> > > > > Change log
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >> > > > >
>>> >> > > > > Release notes
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >> > > > >
>>> >> > > > > This is off branch-3.4.0 and is the first big release since
>>> 3.3.6.
>>> >> > > > >
>>> >> > > > > Key changes include
>>> >> > > > >
>>> >> > > > > * S3A: Upgrade AWS SDK to V2
>>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>>> locks
>>> >> > > > > * YARN Federation improvements
>>> >> > > > > * YARN Capacity Scheduler improvements
>>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>>> >> > > > > * Transitive CVE fixes
>>> >> > > > >
>>> >> > > > > Differences from Hadoop-3.4.0-RC2
>>> >> > > > >
>>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>>> >> (ad8b6541117b)
>>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>>> dependencies.
>>> >> > > > > (80b4bb68159c)
>>> >> > > > > * Use hadoop-release-support[1] for packaging and
>>> verification.
>>> >> > > > > * Add protobuf compatibility issue description
>>> >> > > > >
>>> >> > > > > Note, because the arm64 binaries are built separately on a
>>> >> different
>>> >> > > > > platform and JVM, their jar files may not match those of the
>>> x86
>>> >> > > > > release -and therefore the maven artifacts. I don't think
>>> this is
>>> >> > > > > an issue (the ASF actually releases source tarballs, the
>>> binaries
>>> >> are
>>> >> > > > > there for help only, though with the maven repo that's a bit
>>> >> > blurred).
>>> >> > > > >
>>> >> > > > > The only way to be consistent would actually untar the
>>> x86.tar.gz,
>>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and
>>> push
>>> >> out
>>> >> > > > > for the vote. Even automating that would be risky.
>>> >> > > > >
>>> >> > > > > [1] hadoop-release-support:
>>> >> > > > > https://github.com/apache/hadoop-release-support
>>> >> > > > > Thanks to steve for providing hadoop-release-support.
>>> >> > > > >
>>> >> > > > > Best Regards,
>>> >> > > > > Shilun Fan.
>>> >> > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> >
>>>
>>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
+1 (binding).
* Verified the signature and checksum of all tarballs.
* Built from the source tarball on macOS 10.14 and OpenJDK 8.
* Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
* Check the Yarn federation page and basic functionalities.
* Read the basic help documentation.
Best Regards,
Shilun Fan.
On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
wrote:
> +1 (binding).
>
> Thanks for the great work, Shilun Fan.
>
> * Verified signatures and checksums
> * Successfully built from source with native code
> * Deployed a distributed cluster (on K8s)
> * Successfully ran some Erasure Coding operations with ISA-L codec
> * Successfully ran some HDFS RBF operations
>
> Regards,
> - Takanobu Asanuma
>
> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>
>> Thanks Ayush for highlighting this information. Absolutely true, we should
>> count RM's vote when explicit +1 here.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > > Counter should be with yourself vote, where the current summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >
>> > Just on the process: The release manager needs to "explicitly" vote like
>> > any other before counting their own vote, there has been a lot of
>> > discussions around that at multiple places & the official apache doc has
>> > been updated as well [1], the last paragraph reads:
>> >
>> > "Note that there is no implicit +1 from the release manager, or from
>> > anyone in any ASF vote. Only explicit votes are valid. The release
>> manager
>> > is encouraged to vote on releases, like any reviewer would do."
>> >
>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>> >
>> > -Ayush
>> >
>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>> >
>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>> <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> followup: overnight work happy too.
>> >>
>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>> >> complains that libcrypto is missing
>> >>
>> >> > bin/hadoop checknative
>> >>
>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> >> initialized native-bzip2 library system-native
>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> >> initialized native-zlib library
>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>> support
>> >> is not available in your platform... using builtin-java codec where
>> >> applicable
>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>> built
>> >> without PMDK support.
>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>> OpenSSL
>> >> Cipher.
>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> >> cannot open shared object file: No such file or directory)!
>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>> Method)
>> >> at
>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> >> Native library checking:
>> >> hadoop: true
>> >>
>> >>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>> shared
>> >> object file: No such file or directory)!
>> >> ISA-L: false libhadoop was built without ISA-L support
>> >> PMDK: false The native code was built without PMDK support.
>> >>
>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> >> /usr/lib/aarch64-linux-gnu/l
>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>> >>
>> >> Anyone got any insights on how I should set up this (debian-based) OS
>> >> here?
>> >> I know it's only a small box but with arm64 VMs becoming available in
>> >> cloud
>> >> infras, it'd be good to know if they are similar.
>> >>
>> >> Note: checknative itself is happy; but checknative -a will fail
>> because of
>> >> this -though it's an OS setup issue, nothing related to the hadoop
>> >> binaries.
>> >>
>> >> steve
>> >>
>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>> wrote:
>> >>
>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>> >> summary
>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >> > Thanks again.
>> >> >
>> >> > Best Regards,
>> >> > - He Xiaoqiao
>> >> >
>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>> wrote:
>> >> >
>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>> binding
>> >> > and
>> >> > > 1 vote non-binding.
>> >> > >
>> >> > > Thank you very much for voting and verifying!
>> >> > >
>> >> > > This voting will continue until March 15th, this Friday.
>> >> > >
>> >> > > Best Regards,
>> >> > > Shilun Fan.
>> >> > >
>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> >> > <stevel@cloudera.com.invalid
>> >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > +1 binding
>> >> > > >
>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> >> seen
>> >> > it
>> >> > > > earlier. been testing it this afternoon:
>> >> > > >
>> >> > > > pulled the latest version of
>> >> > > > https://github.com/apache/hadoop-release-support
>> >> > > > (note, this module is commit-then-review; whoever is working
>> >> > > on/validating
>> >> > > > a release can commit as they go along. This is not production
>> >> code...)
>> >> > > >
>> >> > > > * went through the "validating a release" step, validating maven
>> >> > > artifacts
>> >> > > > * building the same downstream modules which built for me last
>> time
>> >> > (avro
>> >> > > > too complex; hboss not aws v2 in apache yet)
>> >> > > >
>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>> >> > building,
>> >> > > > which is key.
>> >> > > >
>> >> > > > The core changes I needed in are at the dependency level and I've
>> >> > > > verified they are good.
>> >> > > >
>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>> arm
>> >> > > > stuff for its checknative; not expecting problems.
>> >> > > >
>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>> >> > packaging
>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>> the
>> >> > > release
>> >> > > > as I already validated them on RC2
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> >> wrote:
>> >> > > >
>> >> > > > > Hi folks,
>> >> > > > >
>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>> for
>> >> > > Hadoop
>> >> > > > > 3.4.0.
>> >> > > > >
>> >> > > > > What we would like is for anyone who can to verify the
>> tarballs,
>> >> > > > especially
>> >> > > > > anyone who can try the arm64 binaries as we want to include
>> them
>> >> too.
>> >> > > > >
>> >> > > > > The RC is available at:
>> >> > > > >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >> > > > >
>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >> > > > >
>> >> > > > > The maven artifacts are staged at
>> >> > > > >
>> >> > >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >> > > > >
>> >> > > > > You can find my public key at:
>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> > > > >
>> >> > > > > Change log
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >> > > > >
>> >> > > > > Release notes
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >> > > > >
>> >> > > > > This is off branch-3.4.0 and is the first big release since
>> 3.3.6.
>> >> > > > >
>> >> > > > > Key changes include
>> >> > > > >
>> >> > > > > * S3A: Upgrade AWS SDK to V2
>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>> locks
>> >> > > > > * YARN Federation improvements
>> >> > > > > * YARN Capacity Scheduler improvements
>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> >> > > > > * Transitive CVE fixes
>> >> > > > >
>> >> > > > > Differences from Hadoop-3.4.0-RC2
>> >> > > > >
>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> >> (ad8b6541117b)
>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>> dependencies.
>> >> > > > > (80b4bb68159c)
>> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> >> > > > > * Add protobuf compatibility issue description
>> >> > > > >
>> >> > > > > Note, because the arm64 binaries are built separately on a
>> >> different
>> >> > > > > platform and JVM, their jar files may not match those of the
>> x86
>> >> > > > > release -and therefore the maven artifacts. I don't think this
>> is
>> >> > > > > an issue (the ASF actually releases source tarballs, the
>> binaries
>> >> are
>> >> > > > > there for help only, though with the maven repo that's a bit
>> >> > blurred).
>> >> > > > >
>> >> > > > > The only way to be consistent would actually untar the
>> x86.tar.gz,
>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> >> out
>> >> > > > > for the vote. Even automating that would be risky.
>> >> > > > >
>> >> > > > > [1] hadoop-release-support:
>> >> > > > > https://github.com/apache/hadoop-release-support
>> >> > > > > Thanks to steve for providing hadoop-release-support.
>> >> > > > >
>> >> > > > > Best Regards,
>> >> > > > > Shilun Fan.
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
+1 (binding).
* Verified the signature and checksum of all tarballs.
* Built from the source tarball on macOS 10.14 and OpenJDK 8.
* Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
* Check the Yarn federation page and basic functionalities.
* Read the basic help documentation.
Best Regards,
Shilun Fan.
On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
wrote:
> +1 (binding).
>
> Thanks for the great work, Shilun Fan.
>
> * Verified signatures and checksums
> * Successfully built from source with native code
> * Deployed a distributed cluster (on K8s)
> * Successfully ran some Erasure Coding operations with ISA-L codec
> * Successfully ran some HDFS RBF operations
>
> Regards,
> - Takanobu Asanuma
>
> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>
>> Thanks Ayush for highlighting this information. Absolutely true, we should
>> count RM's vote when explicit +1 here.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > > Counter should be with yourself vote, where the current summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >
>> > Just on the process: The release manager needs to "explicitly" vote like
>> > any other before counting their own vote, there has been a lot of
>> > discussions around that at multiple places & the official apache doc has
>> > been updated as well [1], the last paragraph reads:
>> >
>> > "Note that there is no implicit +1 from the release manager, or from
>> > anyone in any ASF vote. Only explicit votes are valid. The release
>> manager
>> > is encouraged to vote on releases, like any reviewer would do."
>> >
>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>> >
>> > -Ayush
>> >
>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>> >
>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>> <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> followup: overnight work happy too.
>> >>
>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>> >> complains that libcrypto is missing
>> >>
>> >> > bin/hadoop checknative
>> >>
>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> >> initialized native-bzip2 library system-native
>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> >> initialized native-zlib library
>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>> support
>> >> is not available in your platform... using builtin-java codec where
>> >> applicable
>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>> built
>> >> without PMDK support.
>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>> OpenSSL
>> >> Cipher.
>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> >> cannot open shared object file: No such file or directory)!
>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>> Method)
>> >> at
>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> >> Native library checking:
>> >> hadoop: true
>> >>
>> >>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>> shared
>> >> object file: No such file or directory)!
>> >> ISA-L: false libhadoop was built without ISA-L support
>> >> PMDK: false The native code was built without PMDK support.
>> >>
>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> >> /usr/lib/aarch64-linux-gnu/l
>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>> >>
>> >> Anyone got any insights on how I should set up this (debian-based) OS
>> >> here?
>> >> I know it's only a small box but with arm64 VMs becoming available in
>> >> cloud
>> >> infras, it'd be good to know if they are similar.
>> >>
>> >> Note: checknative itself is happy; but checknative -a will fail
>> because of
>> >> this -though it's an OS setup issue, nothing related to the hadoop
>> >> binaries.
>> >>
>> >> steve
>> >>
>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>> wrote:
>> >>
>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>> >> summary
>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >> > Thanks again.
>> >> >
>> >> > Best Regards,
>> >> > - He Xiaoqiao
>> >> >
>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>> wrote:
>> >> >
>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>> binding
>> >> > and
>> >> > > 1 vote non-binding.
>> >> > >
>> >> > > Thank you very much for voting and verifying!
>> >> > >
>> >> > > This voting will continue until March 15th, this Friday.
>> >> > >
>> >> > > Best Regards,
>> >> > > Shilun Fan.
>> >> > >
>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> >> > <stevel@cloudera.com.invalid
>> >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > +1 binding
>> >> > > >
>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> >> seen
>> >> > it
>> >> > > > earlier. been testing it this afternoon:
>> >> > > >
>> >> > > > pulled the latest version of
>> >> > > > https://github.com/apache/hadoop-release-support
>> >> > > > (note, this module is commit-then-review; whoever is working
>> >> > > on/validating
>> >> > > > a release can commit as they go along. This is not production
>> >> code...)
>> >> > > >
>> >> > > > * went through the "validating a release" step, validating maven
>> >> > > artifacts
>> >> > > > * building the same downstream modules which built for me last
>> time
>> >> > (avro
>> >> > > > too complex; hboss not aws v2 in apache yet)
>> >> > > >
>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>> >> > building,
>> >> > > > which is key.
>> >> > > >
>> >> > > > The core changes I needed in are at the dependency level and I've
>> >> > > > verified they are good.
>> >> > > >
>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>> arm
>> >> > > > stuff for its checknative; not expecting problems.
>> >> > > >
>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>> >> > packaging
>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>> the
>> >> > > release
>> >> > > > as I already validated them on RC2
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> >> wrote:
>> >> > > >
>> >> > > > > Hi folks,
>> >> > > > >
>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>> for
>> >> > > Hadoop
>> >> > > > > 3.4.0.
>> >> > > > >
>> >> > > > > What we would like is for anyone who can to verify the
>> tarballs,
>> >> > > > especially
>> >> > > > > anyone who can try the arm64 binaries as we want to include
>> them
>> >> too.
>> >> > > > >
>> >> > > > > The RC is available at:
>> >> > > > >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >> > > > >
>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >> > > > >
>> >> > > > > The maven artifacts are staged at
>> >> > > > >
>> >> > >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >> > > > >
>> >> > > > > You can find my public key at:
>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> > > > >
>> >> > > > > Change log
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >> > > > >
>> >> > > > > Release notes
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >> > > > >
>> >> > > > > This is off branch-3.4.0 and is the first big release since
>> 3.3.6.
>> >> > > > >
>> >> > > > > Key changes include
>> >> > > > >
>> >> > > > > * S3A: Upgrade AWS SDK to V2
>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>> locks
>> >> > > > > * YARN Federation improvements
>> >> > > > > * YARN Capacity Scheduler improvements
>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> >> > > > > * Transitive CVE fixes
>> >> > > > >
>> >> > > > > Differences from Hadoop-3.4.0-RC2
>> >> > > > >
>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> >> (ad8b6541117b)
>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>> dependencies.
>> >> > > > > (80b4bb68159c)
>> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> >> > > > > * Add protobuf compatibility issue description
>> >> > > > >
>> >> > > > > Note, because the arm64 binaries are built separately on a
>> >> different
>> >> > > > > platform and JVM, their jar files may not match those of the
>> x86
>> >> > > > > release -and therefore the maven artifacts. I don't think this
>> is
>> >> > > > > an issue (the ASF actually releases source tarballs, the
>> binaries
>> >> are
>> >> > > > > there for help only, though with the maven repo that's a bit
>> >> > blurred).
>> >> > > > >
>> >> > > > > The only way to be consistent would actually untar the
>> x86.tar.gz,
>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> >> out
>> >> > > > > for the vote. Even automating that would be risky.
>> >> > > > >
>> >> > > > > [1] hadoop-release-support:
>> >> > > > > https://github.com/apache/hadoop-release-support
>> >> > > > > Thanks to steve for providing hadoop-release-support.
>> >> > > > >
>> >> > > > > Best Regards,
>> >> > > > > Shilun Fan.
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
+1 (binding).
* Verified the signature and checksum of all tarballs.
* Built from the source tarball on macOS 10.14 and OpenJDK 8.
* Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
* Check the Yarn federation page and basic functionalities.
* Read the basic help documentation.
Best Regards,
Shilun Fan.
On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
wrote:
> +1 (binding).
>
> Thanks for the great work, Shilun Fan.
>
> * Verified signatures and checksums
> * Successfully built from source with native code
> * Deployed a distributed cluster (on K8s)
> * Successfully ran some Erasure Coding operations with ISA-L codec
> * Successfully ran some HDFS RBF operations
>
> Regards,
> - Takanobu Asanuma
>
> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>
>> Thanks Ayush for highlighting this information. Absolutely true, we should
>> count RM's vote when explicit +1 here.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > > Counter should be with yourself vote, where the current summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >
>> > Just on the process: The release manager needs to "explicitly" vote like
>> > any other before counting their own vote, there has been a lot of
>> > discussions around that at multiple places & the official apache doc has
>> > been updated as well [1], the last paragraph reads:
>> >
>> > "Note that there is no implicit +1 from the release manager, or from
>> > anyone in any ASF vote. Only explicit votes are valid. The release
>> manager
>> > is encouraged to vote on releases, like any reviewer would do."
>> >
>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>> >
>> > -Ayush
>> >
>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>> >
>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>> <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> followup: overnight work happy too.
>> >>
>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>> >> complains that libcrypto is missing
>> >>
>> >> > bin/hadoop checknative
>> >>
>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> >> initialized native-bzip2 library system-native
>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> >> initialized native-zlib library
>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>> support
>> >> is not available in your platform... using builtin-java codec where
>> >> applicable
>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>> built
>> >> without PMDK support.
>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>> OpenSSL
>> >> Cipher.
>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> >> cannot open shared object file: No such file or directory)!
>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>> Method)
>> >> at
>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> >> Native library checking:
>> >> hadoop: true
>> >>
>> >>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>> shared
>> >> object file: No such file or directory)!
>> >> ISA-L: false libhadoop was built without ISA-L support
>> >> PMDK: false The native code was built without PMDK support.
>> >>
>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> >> /usr/lib/aarch64-linux-gnu/l
>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>> >>
>> >> Anyone got any insights on how I should set up this (debian-based) OS
>> >> here?
>> >> I know it's only a small box but with arm64 VMs becoming available in
>> >> cloud
>> >> infras, it'd be good to know if they are similar.
>> >>
>> >> Note: checknative itself is happy; but checknative -a will fail
>> because of
>> >> this -though it's an OS setup issue, nothing related to the hadoop
>> >> binaries.
>> >>
>> >> steve
>> >>
>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>> wrote:
>> >>
>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>> >> summary
>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >> > Thanks again.
>> >> >
>> >> > Best Regards,
>> >> > - He Xiaoqiao
>> >> >
>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>> wrote:
>> >> >
>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>> binding
>> >> > and
>> >> > > 1 vote non-binding.
>> >> > >
>> >> > > Thank you very much for voting and verifying!
>> >> > >
>> >> > > This voting will continue until March 15th, this Friday.
>> >> > >
>> >> > > Best Regards,
>> >> > > Shilun Fan.
>> >> > >
>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> >> > <stevel@cloudera.com.invalid
>> >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > +1 binding
>> >> > > >
>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> >> seen
>> >> > it
>> >> > > > earlier. been testing it this afternoon:
>> >> > > >
>> >> > > > pulled the latest version of
>> >> > > > https://github.com/apache/hadoop-release-support
>> >> > > > (note, this module is commit-then-review; whoever is working
>> >> > > on/validating
>> >> > > > a release can commit as they go along. This is not production
>> >> code...)
>> >> > > >
>> >> > > > * went through the "validating a release" step, validating maven
>> >> > > artifacts
>> >> > > > * building the same downstream modules which built for me last
>> time
>> >> > (avro
>> >> > > > too complex; hboss not aws v2 in apache yet)
>> >> > > >
>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>> >> > building,
>> >> > > > which is key.
>> >> > > >
>> >> > > > The core changes I needed in are at the dependency level and I've
>> >> > > > verified they are good.
>> >> > > >
>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>> arm
>> >> > > > stuff for its checknative; not expecting problems.
>> >> > > >
>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>> >> > packaging
>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>> the
>> >> > > release
>> >> > > > as I already validated them on RC2
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> >> wrote:
>> >> > > >
>> >> > > > > Hi folks,
>> >> > > > >
>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>> for
>> >> > > Hadoop
>> >> > > > > 3.4.0.
>> >> > > > >
>> >> > > > > What we would like is for anyone who can to verify the
>> tarballs,
>> >> > > > especially
>> >> > > > > anyone who can try the arm64 binaries as we want to include
>> them
>> >> too.
>> >> > > > >
>> >> > > > > The RC is available at:
>> >> > > > >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >> > > > >
>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >> > > > >
>> >> > > > > The maven artifacts are staged at
>> >> > > > >
>> >> > >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >> > > > >
>> >> > > > > You can find my public key at:
>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> > > > >
>> >> > > > > Change log
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >> > > > >
>> >> > > > > Release notes
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >> > > > >
>> >> > > > > This is off branch-3.4.0 and is the first big release since
>> 3.3.6.
>> >> > > > >
>> >> > > > > Key changes include
>> >> > > > >
>> >> > > > > * S3A: Upgrade AWS SDK to V2
>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>> locks
>> >> > > > > * YARN Federation improvements
>> >> > > > > * YARN Capacity Scheduler improvements
>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> >> > > > > * Transitive CVE fixes
>> >> > > > >
>> >> > > > > Differences from Hadoop-3.4.0-RC2
>> >> > > > >
>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> >> (ad8b6541117b)
>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>> dependencies.
>> >> > > > > (80b4bb68159c)
>> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> >> > > > > * Add protobuf compatibility issue description
>> >> > > > >
>> >> > > > > Note, because the arm64 binaries are built separately on a
>> >> different
>> >> > > > > platform and JVM, their jar files may not match those of the
>> x86
>> >> > > > > release -and therefore the maven artifacts. I don't think this
>> is
>> >> > > > > an issue (the ASF actually releases source tarballs, the
>> binaries
>> >> are
>> >> > > > > there for help only, though with the maven repo that's a bit
>> >> > blurred).
>> >> > > > >
>> >> > > > > The only way to be consistent would actually untar the
>> x86.tar.gz,
>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> >> out
>> >> > > > > for the vote. Even automating that would be risky.
>> >> > > > >
>> >> > > > > [1] hadoop-release-support:
>> >> > > > > https://github.com/apache/hadoop-release-support
>> >> > > > > Thanks to steve for providing hadoop-release-support.
>> >> > > > >
>> >> > > > > Best Regards,
>> >> > > > > Shilun Fan.
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
+1 (binding).
* Verified the signature and checksum of all tarballs.
* Built from the source tarball on macOS 10.14 and OpenJDK 8.
* Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
* Check the Yarn federation page and basic functionalities.
* Read the basic help documentation.
Best Regards,
Shilun Fan.
On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma <ta...@apache.org>
wrote:
> +1 (binding).
>
> Thanks for the great work, Shilun Fan.
>
> * Verified signatures and checksums
> * Successfully built from source with native code
> * Deployed a distributed cluster (on K8s)
> * Successfully ran some Erasure Coding operations with ISA-L codec
> * Successfully ran some HDFS RBF operations
>
> Regards,
> - Takanobu Asanuma
>
> 2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
>
>> Thanks Ayush for highlighting this information. Absolutely true, we should
>> count RM's vote when explicit +1 here.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > > Counter should be with yourself vote, where the current summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >
>> > Just on the process: The release manager needs to "explicitly" vote like
>> > any other before counting their own vote, there has been a lot of
>> > discussions around that at multiple places & the official apache doc has
>> > been updated as well [1], the last paragraph reads:
>> >
>> > "Note that there is no implicit +1 from the release manager, or from
>> > anyone in any ASF vote. Only explicit votes are valid. The release
>> manager
>> > is encouraged to vote on releases, like any reviewer would do."
>> >
>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>> >
>> > -Ayush
>> >
>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>> >
>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>> <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> followup: overnight work happy too.
>> >>
>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>> >> complains that libcrypto is missing
>> >>
>> >> > bin/hadoop checknative
>> >>
>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> >> initialized native-bzip2 library system-native
>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> >> initialized native-zlib library
>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>> support
>> >> is not available in your platform... using builtin-java codec where
>> >> applicable
>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>> built
>> >> without PMDK support.
>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>> OpenSSL
>> >> Cipher.
>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> >> cannot open shared object file: No such file or directory)!
>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>> Method)
>> >> at
>> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> >> Native library checking:
>> >> hadoop: true
>> >>
>> >>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>> shared
>> >> object file: No such file or directory)!
>> >> ISA-L: false libhadoop was built without ISA-L support
>> >> PMDK: false The native code was built without PMDK support.
>> >>
>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> >> /usr/lib/aarch64-linux-gnu/l
>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>> >>
>> >> Anyone got any insights on how I should set up this (debian-based) OS
>> >> here?
>> >> I know it's only a small box but with arm64 VMs becoming available in
>> >> cloud
>> >> infras, it'd be good to know if they are similar.
>> >>
>> >> Note: checknative itself is happy; but checknative -a will fail
>> because of
>> >> this -though it's an OS setup issue, nothing related to the hadoop
>> >> binaries.
>> >>
>> >> steve
>> >>
>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
>> wrote:
>> >>
>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>> >> summary
>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >> > Thanks again.
>> >> >
>> >> > Best Regards,
>> >> > - He Xiaoqiao
>> >> >
>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
>> wrote:
>> >> >
>> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
>> binding
>> >> > and
>> >> > > 1 vote non-binding.
>> >> > >
>> >> > > Thank you very much for voting and verifying!
>> >> > >
>> >> > > This voting will continue until March 15th, this Friday.
>> >> > >
>> >> > > Best Regards,
>> >> > > Shilun Fan.
>> >> > >
>> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> >> > <stevel@cloudera.com.invalid
>> >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > +1 binding
>> >> > > >
>> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> >> seen
>> >> > it
>> >> > > > earlier. been testing it this afternoon:
>> >> > > >
>> >> > > > pulled the latest version of
>> >> > > > https://github.com/apache/hadoop-release-support
>> >> > > > (note, this module is commit-then-review; whoever is working
>> >> > > on/validating
>> >> > > > a release can commit as they go along. This is not production
>> >> code...)
>> >> > > >
>> >> > > > * went through the "validating a release" step, validating maven
>> >> > > artifacts
>> >> > > > * building the same downstream modules which built for me last
>> time
>> >> > (avro
>> >> > > > too complex; hboss not aws v2 in apache yet)
>> >> > > >
>> >> > > > spark build is still ongoing, but I'm not going to wait. It is
>> >> > building,
>> >> > > > which is key.
>> >> > > >
>> >> > > > The core changes I needed in are at the dependency level and I've
>> >> > > > verified they are good.
>> >> > > >
>> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
>> arm
>> >> > > > stuff for its checknative; not expecting problems.
>> >> > > >
>> >> > > > So: i've got some stuff still ongoing, but the core changes to
>> >> > packaging
>> >> > > > are in and the rest I'm not worried about -they shouldn't block
>> the
>> >> > > release
>> >> > > > as I already validated them on RC2
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> >> wrote:
>> >> > > >
>> >> > > > > Hi folks,
>> >> > > > >
>> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
>> for
>> >> > > Hadoop
>> >> > > > > 3.4.0.
>> >> > > > >
>> >> > > > > What we would like is for anyone who can to verify the
>> tarballs,
>> >> > > > especially
>> >> > > > > anyone who can try the arm64 binaries as we want to include
>> them
>> >> too.
>> >> > > > >
>> >> > > > > The RC is available at:
>> >> > > > >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >> > > > >
>> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >> > > > >
>> >> > > > > The maven artifacts are staged at
>> >> > > > >
>> >> > >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >> > > > >
>> >> > > > > You can find my public key at:
>> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> > > > >
>> >> > > > > Change log
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >> > > > >
>> >> > > > > Release notes
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >> > > > >
>> >> > > > > This is off branch-3.4.0 and is the first big release since
>> 3.3.6.
>> >> > > > >
>> >> > > > > Key changes include
>> >> > > > >
>> >> > > > > * S3A: Upgrade AWS SDK to V2
>> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
>> locks
>> >> > > > > * YARN Federation improvements
>> >> > > > > * YARN Capacity Scheduler improvements
>> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> >> > > > > * Transitive CVE fixes
>> >> > > > >
>> >> > > > > Differences from Hadoop-3.4.0-RC2
>> >> > > > >
>> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> >> (ad8b6541117b)
>> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
>> dependencies.
>> >> > > > > (80b4bb68159c)
>> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> >> > > > > * Add protobuf compatibility issue description
>> >> > > > >
>> >> > > > > Note, because the arm64 binaries are built separately on a
>> >> different
>> >> > > > > platform and JVM, their jar files may not match those of the
>> x86
>> >> > > > > release -and therefore the maven artifacts. I don't think this
>> is
>> >> > > > > an issue (the ASF actually releases source tarballs, the
>> binaries
>> >> are
>> >> > > > > there for help only, though with the maven repo that's a bit
>> >> > blurred).
>> >> > > > >
>> >> > > > > The only way to be consistent would actually untar the
>> x86.tar.gz,
>> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> >> out
>> >> > > > > for the vote. Even automating that would be risky.
>> >> > > > >
>> >> > > > > [1] hadoop-release-support:
>> >> > > > > https://github.com/apache/hadoop-release-support
>> >> > > > > Thanks to steve for providing hadoop-release-support.
>> >> > > > >
>> >> > > > > Best Regards,
>> >> > > > > Shilun Fan.
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Takanobu Asanuma <ta...@apache.org>.
+1 (binding).
Thanks for the great work, Shilun Fan.
* Verified signatures and checksums
* Successfully built from source with native code
* Deployed a distributed cluster (on K8s)
* Successfully ran some Erasure Coding operations with ISA-L codec
* Successfully ran some HDFS RBF operations
Regards,
- Takanobu Asanuma
2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
> Thanks Ayush for highlighting this information. Absolutely true, we should
> count RM's vote when explicit +1 here.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > > Counter should be with yourself vote, where the current summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >
> > Just on the process: The release manager needs to "explicitly" vote like
> > any other before counting their own vote, there has been a lot of
> > discussions around that at multiple places & the official apache doc has
> > been updated as well [1], the last paragraph reads:
> >
> > "Note that there is no implicit +1 from the release manager, or from
> > anyone in any ASF vote. Only explicit votes are valid. The release
> manager
> > is encouraged to vote on releases, like any reviewer would do."
> >
> > So, do put an explicit +1, before you count yourself. Good Luck!!!
> >
> > -Ayush
> >
> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> >
> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran <stevel@cloudera.com.invalid
> >
> > wrote:
> >
> >> followup: overnight work happy too.
> >>
> >> one interesting pain point is that on a raspberry pi 64 os checknative
> >> complains that libcrypto is missing
> >>
> >> > bin/hadoop checknative
> >>
> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> >> initialized native-bzip2 library system-native
> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> >> initialized native-zlib library
> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
> support
> >> is not available in your platform... using builtin-java codec where
> >> applicable
> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
> built
> >> without PMDK support.
> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
> OpenSSL
> >> Cipher.
> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> >> cannot open shared object file: No such file or directory)!
> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> >> at
> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> >> Native library checking:
> >> hadoop: true
> >>
> >>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
> shared
> >> object file: No such file or directory)!
> >> ISA-L: false libhadoop was built without ISA-L support
> >> PMDK: false The native code was built without PMDK support.
> >>
> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
> >> /usr/lib/aarch64-linux-gnu/l
> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
> >>
> >> Anyone got any insights on how I should set up this (debian-based) OS
> >> here?
> >> I know it's only a small box but with arm64 VMs becoming available in
> >> cloud
> >> infras, it'd be good to know if they are similar.
> >>
> >> Note: checknative itself is happy; but checknative -a will fail because
> of
> >> this -though it's an OS setup issue, nothing related to the hadoop
> >> binaries.
> >>
> >> steve
> >>
> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
> wrote:
> >>
> >> > Hi Shilun, Counter should be with yourself vote, where the current
> >> summary
> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >> > Thanks again.
> >> >
> >> > Best Regards,
> >> > - He Xiaoqiao
> >> >
> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
> wrote:
> >> >
> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
> binding
> >> > and
> >> > > 1 vote non-binding.
> >> > >
> >> > > Thank you very much for voting and verifying!
> >> > >
> >> > > This voting will continue until March 15th, this Friday.
> >> > >
> >> > > Best Regards,
> >> > > Shilun Fan.
> >> > >
> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >> > <stevel@cloudera.com.invalid
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > +1 binding
> >> > > >
> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> >> seen
> >> > it
> >> > > > earlier. been testing it this afternoon:
> >> > > >
> >> > > > pulled the latest version of
> >> > > > https://github.com/apache/hadoop-release-support
> >> > > > (note, this module is commit-then-review; whoever is working
> >> > > on/validating
> >> > > > a release can commit as they go along. This is not production
> >> code...)
> >> > > >
> >> > > > * went through the "validating a release" step, validating maven
> >> > > artifacts
> >> > > > * building the same downstream modules which built for me last
> time
> >> > (avro
> >> > > > too complex; hboss not aws v2 in apache yet)
> >> > > >
> >> > > > spark build is still ongoing, but I'm not going to wait. It is
> >> > building,
> >> > > > which is key.
> >> > > >
> >> > > > The core changes I needed in are at the dependency level and I've
> >> > > > verified they are good.
> >> > > >
> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
> arm
> >> > > > stuff for its checknative; not expecting problems.
> >> > > >
> >> > > > So: i've got some stuff still ongoing, but the core changes to
> >> > packaging
> >> > > > are in and the rest I'm not worried about -they shouldn't block
> the
> >> > > release
> >> > > > as I already validated them on RC2
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
> >> wrote:
> >> > > >
> >> > > > > Hi folks,
> >> > > > >
> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
> for
> >> > > Hadoop
> >> > > > > 3.4.0.
> >> > > > >
> >> > > > > What we would like is for anyone who can to verify the tarballs,
> >> > > > especially
> >> > > > > anyone who can try the arm64 binaries as we want to include them
> >> too.
> >> > > > >
> >> > > > > The RC is available at:
> >> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >> > > > >
> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >> > > > >
> >> > > > > The maven artifacts are staged at
> >> > > > >
> >> > >
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >> > > > >
> >> > > > > You can find my public key at:
> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > > >
> >> > > > > Change log
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >> > > > >
> >> > > > > Release notes
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >> > > > >
> >> > > > > This is off branch-3.4.0 and is the first big release since
> 3.3.6.
> >> > > > >
> >> > > > > Key changes include
> >> > > > >
> >> > > > > * S3A: Upgrade AWS SDK to V2
> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
> locks
> >> > > > > * YARN Federation improvements
> >> > > > > * YARN Capacity Scheduler improvements
> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> >> > > > > * Transitive CVE fixes
> >> > > > >
> >> > > > > Differences from Hadoop-3.4.0-RC2
> >> > > > >
> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> >> (ad8b6541117b)
> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
> dependencies.
> >> > > > > (80b4bb68159c)
> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
> >> > > > > * Add protobuf compatibility issue description
> >> > > > >
> >> > > > > Note, because the arm64 binaries are built separately on a
> >> different
> >> > > > > platform and JVM, their jar files may not match those of the x86
> >> > > > > release -and therefore the maven artifacts. I don't think this
> is
> >> > > > > an issue (the ASF actually releases source tarballs, the
> binaries
> >> are
> >> > > > > there for help only, though with the maven repo that's a bit
> >> > blurred).
> >> > > > >
> >> > > > > The only way to be consistent would actually untar the
> x86.tar.gz,
> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
> >> out
> >> > > > > for the vote. Even automating that would be risky.
> >> > > > >
> >> > > > > [1] hadoop-release-support:
> >> > > > > https://github.com/apache/hadoop-release-support
> >> > > > > Thanks to steve for providing hadoop-release-support.
> >> > > > >
> >> > > > > Best Regards,
> >> > > > > Shilun Fan.
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Takanobu Asanuma <ta...@apache.org>.
+1 (binding).
Thanks for the great work, Shilun Fan.
* Verified signatures and checksums
* Successfully built from source with native code
* Deployed a distributed cluster (on K8s)
* Successfully ran some Erasure Coding operations with ISA-L codec
* Successfully ran some HDFS RBF operations
Regards,
- Takanobu Asanuma
2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
> Thanks Ayush for highlighting this information. Absolutely true, we should
> count RM's vote when explicit +1 here.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > > Counter should be with yourself vote, where the current summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >
> > Just on the process: The release manager needs to "explicitly" vote like
> > any other before counting their own vote, there has been a lot of
> > discussions around that at multiple places & the official apache doc has
> > been updated as well [1], the last paragraph reads:
> >
> > "Note that there is no implicit +1 from the release manager, or from
> > anyone in any ASF vote. Only explicit votes are valid. The release
> manager
> > is encouraged to vote on releases, like any reviewer would do."
> >
> > So, do put an explicit +1, before you count yourself. Good Luck!!!
> >
> > -Ayush
> >
> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> >
> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran <stevel@cloudera.com.invalid
> >
> > wrote:
> >
> >> followup: overnight work happy too.
> >>
> >> one interesting pain point is that on a raspberry pi 64 os checknative
> >> complains that libcrypto is missing
> >>
> >> > bin/hadoop checknative
> >>
> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> >> initialized native-bzip2 library system-native
> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> >> initialized native-zlib library
> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
> support
> >> is not available in your platform... using builtin-java codec where
> >> applicable
> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
> built
> >> without PMDK support.
> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
> OpenSSL
> >> Cipher.
> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> >> cannot open shared object file: No such file or directory)!
> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> >> at
> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> >> Native library checking:
> >> hadoop: true
> >>
> >>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
> shared
> >> object file: No such file or directory)!
> >> ISA-L: false libhadoop was built without ISA-L support
> >> PMDK: false The native code was built without PMDK support.
> >>
> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
> >> /usr/lib/aarch64-linux-gnu/l
> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
> >>
> >> Anyone got any insights on how I should set up this (debian-based) OS
> >> here?
> >> I know it's only a small box but with arm64 VMs becoming available in
> >> cloud
> >> infras, it'd be good to know if they are similar.
> >>
> >> Note: checknative itself is happy; but checknative -a will fail because
> of
> >> this -though it's an OS setup issue, nothing related to the hadoop
> >> binaries.
> >>
> >> steve
> >>
> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
> wrote:
> >>
> >> > Hi Shilun, Counter should be with yourself vote, where the current
> >> summary
> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >> > Thanks again.
> >> >
> >> > Best Regards,
> >> > - He Xiaoqiao
> >> >
> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
> wrote:
> >> >
> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
> binding
> >> > and
> >> > > 1 vote non-binding.
> >> > >
> >> > > Thank you very much for voting and verifying!
> >> > >
> >> > > This voting will continue until March 15th, this Friday.
> >> > >
> >> > > Best Regards,
> >> > > Shilun Fan.
> >> > >
> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >> > <stevel@cloudera.com.invalid
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > +1 binding
> >> > > >
> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> >> seen
> >> > it
> >> > > > earlier. been testing it this afternoon:
> >> > > >
> >> > > > pulled the latest version of
> >> > > > https://github.com/apache/hadoop-release-support
> >> > > > (note, this module is commit-then-review; whoever is working
> >> > > on/validating
> >> > > > a release can commit as they go along. This is not production
> >> code...)
> >> > > >
> >> > > > * went through the "validating a release" step, validating maven
> >> > > artifacts
> >> > > > * building the same downstream modules which built for me last
> time
> >> > (avro
> >> > > > too complex; hboss not aws v2 in apache yet)
> >> > > >
> >> > > > spark build is still ongoing, but I'm not going to wait. It is
> >> > building,
> >> > > > which is key.
> >> > > >
> >> > > > The core changes I needed in are at the dependency level and I've
> >> > > > verified they are good.
> >> > > >
> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
> arm
> >> > > > stuff for its checknative; not expecting problems.
> >> > > >
> >> > > > So: i've got some stuff still ongoing, but the core changes to
> >> > packaging
> >> > > > are in and the rest I'm not worried about -they shouldn't block
> the
> >> > > release
> >> > > > as I already validated them on RC2
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
> >> wrote:
> >> > > >
> >> > > > > Hi folks,
> >> > > > >
> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
> for
> >> > > Hadoop
> >> > > > > 3.4.0.
> >> > > > >
> >> > > > > What we would like is for anyone who can to verify the tarballs,
> >> > > > especially
> >> > > > > anyone who can try the arm64 binaries as we want to include them
> >> too.
> >> > > > >
> >> > > > > The RC is available at:
> >> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >> > > > >
> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >> > > > >
> >> > > > > The maven artifacts are staged at
> >> > > > >
> >> > >
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >> > > > >
> >> > > > > You can find my public key at:
> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > > >
> >> > > > > Change log
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >> > > > >
> >> > > > > Release notes
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >> > > > >
> >> > > > > This is off branch-3.4.0 and is the first big release since
> 3.3.6.
> >> > > > >
> >> > > > > Key changes include
> >> > > > >
> >> > > > > * S3A: Upgrade AWS SDK to V2
> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
> locks
> >> > > > > * YARN Federation improvements
> >> > > > > * YARN Capacity Scheduler improvements
> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> >> > > > > * Transitive CVE fixes
> >> > > > >
> >> > > > > Differences from Hadoop-3.4.0-RC2
> >> > > > >
> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> >> (ad8b6541117b)
> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
> dependencies.
> >> > > > > (80b4bb68159c)
> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
> >> > > > > * Add protobuf compatibility issue description
> >> > > > >
> >> > > > > Note, because the arm64 binaries are built separately on a
> >> different
> >> > > > > platform and JVM, their jar files may not match those of the x86
> >> > > > > release -and therefore the maven artifacts. I don't think this
> is
> >> > > > > an issue (the ASF actually releases source tarballs, the
> binaries
> >> are
> >> > > > > there for help only, though with the maven repo that's a bit
> >> > blurred).
> >> > > > >
> >> > > > > The only way to be consistent would actually untar the
> x86.tar.gz,
> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
> >> out
> >> > > > > for the vote. Even automating that would be risky.
> >> > > > >
> >> > > > > [1] hadoop-release-support:
> >> > > > > https://github.com/apache/hadoop-release-support
> >> > > > > Thanks to steve for providing hadoop-release-support.
> >> > > > >
> >> > > > > Best Regards,
> >> > > > > Shilun Fan.
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Takanobu Asanuma <ta...@apache.org>.
+1 (binding).
Thanks for the great work, Shilun Fan.
* Verified signatures and checksums
* Successfully built from source with native code
* Deployed a distributed cluster (on K8s)
* Successfully ran some Erasure Coding operations with ISA-L codec
* Successfully ran some HDFS RBF operations
Regards,
- Takanobu Asanuma
2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
> Thanks Ayush for highlighting this information. Absolutely true, we should
> count RM's vote when explicit +1 here.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > > Counter should be with yourself vote, where the current summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >
> > Just on the process: The release manager needs to "explicitly" vote like
> > any other before counting their own vote, there has been a lot of
> > discussions around that at multiple places & the official apache doc has
> > been updated as well [1], the last paragraph reads:
> >
> > "Note that there is no implicit +1 from the release manager, or from
> > anyone in any ASF vote. Only explicit votes are valid. The release
> manager
> > is encouraged to vote on releases, like any reviewer would do."
> >
> > So, do put an explicit +1, before you count yourself. Good Luck!!!
> >
> > -Ayush
> >
> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> >
> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran <stevel@cloudera.com.invalid
> >
> > wrote:
> >
> >> followup: overnight work happy too.
> >>
> >> one interesting pain point is that on a raspberry pi 64 os checknative
> >> complains that libcrypto is missing
> >>
> >> > bin/hadoop checknative
> >>
> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> >> initialized native-bzip2 library system-native
> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> >> initialized native-zlib library
> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
> support
> >> is not available in your platform... using builtin-java codec where
> >> applicable
> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
> built
> >> without PMDK support.
> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
> OpenSSL
> >> Cipher.
> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> >> cannot open shared object file: No such file or directory)!
> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> >> at
> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> >> Native library checking:
> >> hadoop: true
> >>
> >>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
> shared
> >> object file: No such file or directory)!
> >> ISA-L: false libhadoop was built without ISA-L support
> >> PMDK: false The native code was built without PMDK support.
> >>
> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
> >> /usr/lib/aarch64-linux-gnu/l
> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
> >>
> >> Anyone got any insights on how I should set up this (debian-based) OS
> >> here?
> >> I know it's only a small box but with arm64 VMs becoming available in
> >> cloud
> >> infras, it'd be good to know if they are similar.
> >>
> >> Note: checknative itself is happy; but checknative -a will fail because
> of
> >> this -though it's an OS setup issue, nothing related to the hadoop
> >> binaries.
> >>
> >> steve
> >>
> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
> wrote:
> >>
> >> > Hi Shilun, Counter should be with yourself vote, where the current
> >> summary
> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >> > Thanks again.
> >> >
> >> > Best Regards,
> >> > - He Xiaoqiao
> >> >
> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
> wrote:
> >> >
> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
> binding
> >> > and
> >> > > 1 vote non-binding.
> >> > >
> >> > > Thank you very much for voting and verifying!
> >> > >
> >> > > This voting will continue until March 15th, this Friday.
> >> > >
> >> > > Best Regards,
> >> > > Shilun Fan.
> >> > >
> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >> > <stevel@cloudera.com.invalid
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > +1 binding
> >> > > >
> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> >> seen
> >> > it
> >> > > > earlier. been testing it this afternoon:
> >> > > >
> >> > > > pulled the latest version of
> >> > > > https://github.com/apache/hadoop-release-support
> >> > > > (note, this module is commit-then-review; whoever is working
> >> > > on/validating
> >> > > > a release can commit as they go along. This is not production
> >> code...)
> >> > > >
> >> > > > * went through the "validating a release" step, validating maven
> >> > > artifacts
> >> > > > * building the same downstream modules which built for me last
> time
> >> > (avro
> >> > > > too complex; hboss not aws v2 in apache yet)
> >> > > >
> >> > > > spark build is still ongoing, but I'm not going to wait. It is
> >> > building,
> >> > > > which is key.
> >> > > >
> >> > > > The core changes I needed in are at the dependency level and I've
> >> > > > verified they are good.
> >> > > >
> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
> arm
> >> > > > stuff for its checknative; not expecting problems.
> >> > > >
> >> > > > So: i've got some stuff still ongoing, but the core changes to
> >> > packaging
> >> > > > are in and the rest I'm not worried about -they shouldn't block
> the
> >> > > release
> >> > > > as I already validated them on RC2
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
> >> wrote:
> >> > > >
> >> > > > > Hi folks,
> >> > > > >
> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
> for
> >> > > Hadoop
> >> > > > > 3.4.0.
> >> > > > >
> >> > > > > What we would like is for anyone who can to verify the tarballs,
> >> > > > especially
> >> > > > > anyone who can try the arm64 binaries as we want to include them
> >> too.
> >> > > > >
> >> > > > > The RC is available at:
> >> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >> > > > >
> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >> > > > >
> >> > > > > The maven artifacts are staged at
> >> > > > >
> >> > >
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >> > > > >
> >> > > > > You can find my public key at:
> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > > >
> >> > > > > Change log
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >> > > > >
> >> > > > > Release notes
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >> > > > >
> >> > > > > This is off branch-3.4.0 and is the first big release since
> 3.3.6.
> >> > > > >
> >> > > > > Key changes include
> >> > > > >
> >> > > > > * S3A: Upgrade AWS SDK to V2
> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
> locks
> >> > > > > * YARN Federation improvements
> >> > > > > * YARN Capacity Scheduler improvements
> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> >> > > > > * Transitive CVE fixes
> >> > > > >
> >> > > > > Differences from Hadoop-3.4.0-RC2
> >> > > > >
> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> >> (ad8b6541117b)
> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
> dependencies.
> >> > > > > (80b4bb68159c)
> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
> >> > > > > * Add protobuf compatibility issue description
> >> > > > >
> >> > > > > Note, because the arm64 binaries are built separately on a
> >> different
> >> > > > > platform and JVM, their jar files may not match those of the x86
> >> > > > > release -and therefore the maven artifacts. I don't think this
> is
> >> > > > > an issue (the ASF actually releases source tarballs, the
> binaries
> >> are
> >> > > > > there for help only, though with the maven repo that's a bit
> >> > blurred).
> >> > > > >
> >> > > > > The only way to be consistent would actually untar the
> x86.tar.gz,
> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
> >> out
> >> > > > > for the vote. Even automating that would be risky.
> >> > > > >
> >> > > > > [1] hadoop-release-support:
> >> > > > > https://github.com/apache/hadoop-release-support
> >> > > > > Thanks to steve for providing hadoop-release-support.
> >> > > > >
> >> > > > > Best Regards,
> >> > > > > Shilun Fan.
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Takanobu Asanuma <ta...@apache.org>.
+1 (binding).
Thanks for the great work, Shilun Fan.
* Verified signatures and checksums
* Successfully built from source with native code
* Deployed a distributed cluster (on K8s)
* Successfully ran some Erasure Coding operations with ISA-L codec
* Successfully ran some HDFS RBF operations
Regards,
- Takanobu Asanuma
2024年3月14日(木) 15:19 Xiaoqiao He <he...@apache.org>:
> Thanks Ayush for highlighting this information. Absolutely true, we should
> count RM's vote when explicit +1 here.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > > Counter should be with yourself vote, where the current summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >
> > Just on the process: The release manager needs to "explicitly" vote like
> > any other before counting their own vote, there has been a lot of
> > discussions around that at multiple places & the official apache doc has
> > been updated as well [1], the last paragraph reads:
> >
> > "Note that there is no implicit +1 from the release manager, or from
> > anyone in any ASF vote. Only explicit votes are valid. The release
> manager
> > is encouraged to vote on releases, like any reviewer would do."
> >
> > So, do put an explicit +1, before you count yourself. Good Luck!!!
> >
> > -Ayush
> >
> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> >
> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran <stevel@cloudera.com.invalid
> >
> > wrote:
> >
> >> followup: overnight work happy too.
> >>
> >> one interesting pain point is that on a raspberry pi 64 os checknative
> >> complains that libcrypto is missing
> >>
> >> > bin/hadoop checknative
> >>
> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> >> initialized native-bzip2 library system-native
> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> >> initialized native-zlib library
> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
> support
> >> is not available in your platform... using builtin-java codec where
> >> applicable
> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
> built
> >> without PMDK support.
> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
> OpenSSL
> >> Cipher.
> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> >> cannot open shared object file: No such file or directory)!
> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> >> at
> >> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> >> Native library checking:
> >> hadoop: true
> >>
> >>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> >> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
> shared
> >> object file: No such file or directory)!
> >> ISA-L: false libhadoop was built without ISA-L support
> >> PMDK: false The native code was built without PMDK support.
> >>
> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
> >> /usr/lib/aarch64-linux-gnu/l
> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
> >>
> >> Anyone got any insights on how I should set up this (debian-based) OS
> >> here?
> >> I know it's only a small box but with arm64 VMs becoming available in
> >> cloud
> >> infras, it'd be good to know if they are similar.
> >>
> >> Note: checknative itself is happy; but checknative -a will fail because
> of
> >> this -though it's an OS setup issue, nothing related to the hadoop
> >> binaries.
> >>
> >> steve
> >>
> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org>
> wrote:
> >>
> >> > Hi Shilun, Counter should be with yourself vote, where the current
> >> summary
> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >> > Thanks again.
> >> >
> >> > Best Regards,
> >> > - He Xiaoqiao
> >> >
> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org>
> wrote:
> >> >
> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
> binding
> >> > and
> >> > > 1 vote non-binding.
> >> > >
> >> > > Thank you very much for voting and verifying!
> >> > >
> >> > > This voting will continue until March 15th, this Friday.
> >> > >
> >> > > Best Regards,
> >> > > Shilun Fan.
> >> > >
> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >> > <stevel@cloudera.com.invalid
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > +1 binding
> >> > > >
> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> >> seen
> >> > it
> >> > > > earlier. been testing it this afternoon:
> >> > > >
> >> > > > pulled the latest version of
> >> > > > https://github.com/apache/hadoop-release-support
> >> > > > (note, this module is commit-then-review; whoever is working
> >> > > on/validating
> >> > > > a release can commit as they go along. This is not production
> >> code...)
> >> > > >
> >> > > > * went through the "validating a release" step, validating maven
> >> > > artifacts
> >> > > > * building the same downstream modules which built for me last
> time
> >> > (avro
> >> > > > too complex; hboss not aws v2 in apache yet)
> >> > > >
> >> > > > spark build is still ongoing, but I'm not going to wait. It is
> >> > building,
> >> > > > which is key.
> >> > > >
> >> > > > The core changes I needed in are at the dependency level and I've
> >> > > > verified they are good.
> >> > > >
> >> > > > Oh, and I've also got my raspberry p5 doing the download of the
> arm
> >> > > > stuff for its checknative; not expecting problems.
> >> > > >
> >> > > > So: i've got some stuff still ongoing, but the core changes to
> >> > packaging
> >> > > > are in and the rest I'm not worried about -they shouldn't block
> the
> >> > > release
> >> > > > as I already validated them on RC2
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
> >> wrote:
> >> > > >
> >> > > > > Hi folks,
> >> > > > >
> >> > > > > Xiaoqiao He and I have put together a release candidate (RC3)
> for
> >> > > Hadoop
> >> > > > > 3.4.0.
> >> > > > >
> >> > > > > What we would like is for anyone who can to verify the tarballs,
> >> > > > especially
> >> > > > > anyone who can try the arm64 binaries as we want to include them
> >> too.
> >> > > > >
> >> > > > > The RC is available at:
> >> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >> > > > >
> >> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >> > > > >
> >> > > > > The maven artifacts are staged at
> >> > > > >
> >> > >
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >> > > > >
> >> > > > > You can find my public key at:
> >> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > > >
> >> > > > > Change log
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >> > > > >
> >> > > > > Release notes
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >> > > > >
> >> > > > > This is off branch-3.4.0 and is the first big release since
> 3.3.6.
> >> > > > >
> >> > > > > Key changes include
> >> > > > >
> >> > > > > * S3A: Upgrade AWS SDK to V2
> >> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain
> locks
> >> > > > > * YARN Federation improvements
> >> > > > > * YARN Capacity Scheduler improvements
> >> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> >> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> >> > > > > * Transitive CVE fixes
> >> > > > >
> >> > > > > Differences from Hadoop-3.4.0-RC2
> >> > > > >
> >> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> >> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> >> (ad8b6541117b)
> >> > > > > * HADOOP-19084: Pruning hadoop-common transitive
> dependencies.
> >> > > > > (80b4bb68159c)
> >> > > > > * Use hadoop-release-support[1] for packaging and verification.
> >> > > > > * Add protobuf compatibility issue description
> >> > > > >
> >> > > > > Note, because the arm64 binaries are built separately on a
> >> different
> >> > > > > platform and JVM, their jar files may not match those of the x86
> >> > > > > release -and therefore the maven artifacts. I don't think this
> is
> >> > > > > an issue (the ASF actually releases source tarballs, the
> binaries
> >> are
> >> > > > > there for help only, though with the maven repo that's a bit
> >> > blurred).
> >> > > > >
> >> > > > > The only way to be consistent would actually untar the
> x86.tar.gz,
> >> > > > > overwrite its binaries with the arm stuff, retar, sign and push
> >> out
> >> > > > > for the vote. Even automating that would be risky.
> >> > > > >
> >> > > > > [1] hadoop-release-support:
> >> > > > > https://github.com/apache/hadoop-release-support
> >> > > > > Thanks to steve for providing hadoop-release-support.
> >> > > > >
> >> > > > > Best Regards,
> >> > > > > Shilun Fan.
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Thanks Ayush for highlighting this information. Absolutely true, we should
count RM's vote when explicit +1 here.
Best Regards,
- He Xiaoqiao
On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
> > Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>
> Just on the process: The release manager needs to "explicitly" vote like
> any other before counting their own vote, there has been a lot of
> discussions around that at multiple places & the official apache doc has
> been updated as well [1], the last paragraph reads:
>
> "Note that there is no implicit +1 from the release manager, or from
> anyone in any ASF vote. Only explicit votes are valid. The release manager
> is encouraged to vote on releases, like any reviewer would do."
>
> So, do put an explicit +1, before you count yourself. Good Luck!!!
>
> -Ayush
>
> [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>
> On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> followup: overnight work happy too.
>>
>> one interesting pain point is that on a raspberry pi 64 os checknative
>> complains that libcrypto is missing
>>
>> > bin/hadoop checknative
>>
>> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> initialized native-bzip2 library system-native
>> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized native-zlib library
>> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
>> is not available in your platform... using builtin-java codec where
>> applicable
>> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
>> without PMDK support.
>> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
>> Cipher.
>> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> cannot open shared object file: No such file or directory)!
>> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
>> at
>> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> at
>>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> Native library checking:
>> hadoop: true
>>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
>> object file: No such file or directory)!
>> ISA-L: false libhadoop was built without ISA-L support
>> PMDK: false The native code was built without PMDK support.
>>
>> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> /usr/lib/aarch64-linux-gnu/l
>> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>
>> Anyone got any insights on how I should set up this (debian-based) OS
>> here?
>> I know it's only a small box but with arm64 VMs becoming available in
>> cloud
>> infras, it'd be good to know if they are similar.
>>
>> Note: checknative itself is happy; but checknative -a will fail because of
>> this -though it's an OS setup issue, nothing related to the hadoop
>> binaries.
>>
>> steve
>>
>> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>>
>> > Hi Shilun, Counter should be with yourself vote, where the current
>> summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> > Thanks again.
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>> >
>> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
>> > and
>> > > 1 vote non-binding.
>> > >
>> > > Thank you very much for voting and verifying!
>> > >
>> > > This voting will continue until March 15th, this Friday.
>> > >
>> > > Best Regards,
>> > > Shilun Fan.
>> > >
>> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> > <stevel@cloudera.com.invalid
>> > > >
>> > > wrote:
>> > >
>> > > > +1 binding
>> > > >
>> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> seen
>> > it
>> > > > earlier. been testing it this afternoon:
>> > > >
>> > > > pulled the latest version of
>> > > > https://github.com/apache/hadoop-release-support
>> > > > (note, this module is commit-then-review; whoever is working
>> > > on/validating
>> > > > a release can commit as they go along. This is not production
>> code...)
>> > > >
>> > > > * went through the "validating a release" step, validating maven
>> > > artifacts
>> > > > * building the same downstream modules which built for me last time
>> > (avro
>> > > > too complex; hboss not aws v2 in apache yet)
>> > > >
>> > > > spark build is still ongoing, but I'm not going to wait. It is
>> > building,
>> > > > which is key.
>> > > >
>> > > > The core changes I needed in are at the dependency level and I've
>> > > > verified they are good.
>> > > >
>> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
>> > > > stuff for its checknative; not expecting problems.
>> > > >
>> > > > So: i've got some stuff still ongoing, but the core changes to
>> > packaging
>> > > > are in and the rest I'm not worried about -they shouldn't block the
>> > > release
>> > > > as I already validated them on RC2
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
>> > > Hadoop
>> > > > > 3.4.0.
>> > > > >
>> > > > > What we would like is for anyone who can to verify the tarballs,
>> > > > especially
>> > > > > anyone who can try the arm64 binaries as we want to include them
>> too.
>> > > > >
>> > > > > The RC is available at:
>> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> > > > >
>> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> > > > >
>> > > > > The maven artifacts are staged at
>> > > > >
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> > > > >
>> > > > > You can find my public key at:
>> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > > >
>> > > > > Change log
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> > > > >
>> > > > > Release notes
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> > > > >
>> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> > > > >
>> > > > > Key changes include
>> > > > >
>> > > > > * S3A: Upgrade AWS SDK to V2
>> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > > > > * YARN Federation improvements
>> > > > > * YARN Capacity Scheduler improvements
>> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> > > > > * Transitive CVE fixes
>> > > > >
>> > > > > Differences from Hadoop-3.4.0-RC2
>> > > > >
>> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> (ad8b6541117b)
>> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > > > > (80b4bb68159c)
>> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> > > > > * Add protobuf compatibility issue description
>> > > > >
>> > > > > Note, because the arm64 binaries are built separately on a
>> different
>> > > > > platform and JVM, their jar files may not match those of the x86
>> > > > > release -and therefore the maven artifacts. I don't think this is
>> > > > > an issue (the ASF actually releases source tarballs, the binaries
>> are
>> > > > > there for help only, though with the maven repo that's a bit
>> > blurred).
>> > > > >
>> > > > > The only way to be consistent would actually untar the x86.tar.gz,
>> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> out
>> > > > > for the vote. Even automating that would be risky.
>> > > > >
>> > > > > [1] hadoop-release-support:
>> > > > > https://github.com/apache/hadoop-release-support
>> > > > > Thanks to steve for providing hadoop-release-support.
>> > > > >
>> > > > > Best Regards,
>> > > > > Shilun Fan.
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Thanks Ayush for highlighting this information. Absolutely true, we should
count RM's vote when explicit +1 here.
Best Regards,
- He Xiaoqiao
On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
> > Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>
> Just on the process: The release manager needs to "explicitly" vote like
> any other before counting their own vote, there has been a lot of
> discussions around that at multiple places & the official apache doc has
> been updated as well [1], the last paragraph reads:
>
> "Note that there is no implicit +1 from the release manager, or from
> anyone in any ASF vote. Only explicit votes are valid. The release manager
> is encouraged to vote on releases, like any reviewer would do."
>
> So, do put an explicit +1, before you count yourself. Good Luck!!!
>
> -Ayush
>
> [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>
> On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> followup: overnight work happy too.
>>
>> one interesting pain point is that on a raspberry pi 64 os checknative
>> complains that libcrypto is missing
>>
>> > bin/hadoop checknative
>>
>> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> initialized native-bzip2 library system-native
>> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized native-zlib library
>> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
>> is not available in your platform... using builtin-java codec where
>> applicable
>> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
>> without PMDK support.
>> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
>> Cipher.
>> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> cannot open shared object file: No such file or directory)!
>> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
>> at
>> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> at
>>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> Native library checking:
>> hadoop: true
>>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
>> object file: No such file or directory)!
>> ISA-L: false libhadoop was built without ISA-L support
>> PMDK: false The native code was built without PMDK support.
>>
>> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> /usr/lib/aarch64-linux-gnu/l
>> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>
>> Anyone got any insights on how I should set up this (debian-based) OS
>> here?
>> I know it's only a small box but with arm64 VMs becoming available in
>> cloud
>> infras, it'd be good to know if they are similar.
>>
>> Note: checknative itself is happy; but checknative -a will fail because of
>> this -though it's an OS setup issue, nothing related to the hadoop
>> binaries.
>>
>> steve
>>
>> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>>
>> > Hi Shilun, Counter should be with yourself vote, where the current
>> summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> > Thanks again.
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>> >
>> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
>> > and
>> > > 1 vote non-binding.
>> > >
>> > > Thank you very much for voting and verifying!
>> > >
>> > > This voting will continue until March 15th, this Friday.
>> > >
>> > > Best Regards,
>> > > Shilun Fan.
>> > >
>> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> > <stevel@cloudera.com.invalid
>> > > >
>> > > wrote:
>> > >
>> > > > +1 binding
>> > > >
>> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> seen
>> > it
>> > > > earlier. been testing it this afternoon:
>> > > >
>> > > > pulled the latest version of
>> > > > https://github.com/apache/hadoop-release-support
>> > > > (note, this module is commit-then-review; whoever is working
>> > > on/validating
>> > > > a release can commit as they go along. This is not production
>> code...)
>> > > >
>> > > > * went through the "validating a release" step, validating maven
>> > > artifacts
>> > > > * building the same downstream modules which built for me last time
>> > (avro
>> > > > too complex; hboss not aws v2 in apache yet)
>> > > >
>> > > > spark build is still ongoing, but I'm not going to wait. It is
>> > building,
>> > > > which is key.
>> > > >
>> > > > The core changes I needed in are at the dependency level and I've
>> > > > verified they are good.
>> > > >
>> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
>> > > > stuff for its checknative; not expecting problems.
>> > > >
>> > > > So: i've got some stuff still ongoing, but the core changes to
>> > packaging
>> > > > are in and the rest I'm not worried about -they shouldn't block the
>> > > release
>> > > > as I already validated them on RC2
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
>> > > Hadoop
>> > > > > 3.4.0.
>> > > > >
>> > > > > What we would like is for anyone who can to verify the tarballs,
>> > > > especially
>> > > > > anyone who can try the arm64 binaries as we want to include them
>> too.
>> > > > >
>> > > > > The RC is available at:
>> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> > > > >
>> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> > > > >
>> > > > > The maven artifacts are staged at
>> > > > >
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> > > > >
>> > > > > You can find my public key at:
>> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > > >
>> > > > > Change log
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> > > > >
>> > > > > Release notes
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> > > > >
>> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> > > > >
>> > > > > Key changes include
>> > > > >
>> > > > > * S3A: Upgrade AWS SDK to V2
>> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > > > > * YARN Federation improvements
>> > > > > * YARN Capacity Scheduler improvements
>> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> > > > > * Transitive CVE fixes
>> > > > >
>> > > > > Differences from Hadoop-3.4.0-RC2
>> > > > >
>> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> (ad8b6541117b)
>> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > > > > (80b4bb68159c)
>> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> > > > > * Add protobuf compatibility issue description
>> > > > >
>> > > > > Note, because the arm64 binaries are built separately on a
>> different
>> > > > > platform and JVM, their jar files may not match those of the x86
>> > > > > release -and therefore the maven artifacts. I don't think this is
>> > > > > an issue (the ASF actually releases source tarballs, the binaries
>> are
>> > > > > there for help only, though with the maven repo that's a bit
>> > blurred).
>> > > > >
>> > > > > The only way to be consistent would actually untar the x86.tar.gz,
>> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> out
>> > > > > for the vote. Even automating that would be risky.
>> > > > >
>> > > > > [1] hadoop-release-support:
>> > > > > https://github.com/apache/hadoop-release-support
>> > > > > Thanks to steve for providing hadoop-release-support.
>> > > > >
>> > > > > Best Regards,
>> > > > > Shilun Fan.
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Thanks Ayush for highlighting this information. Absolutely true, we should
count RM's vote when explicit +1 here.
Best Regards,
- He Xiaoqiao
On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
> > Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>
> Just on the process: The release manager needs to "explicitly" vote like
> any other before counting their own vote, there has been a lot of
> discussions around that at multiple places & the official apache doc has
> been updated as well [1], the last paragraph reads:
>
> "Note that there is no implicit +1 from the release manager, or from
> anyone in any ASF vote. Only explicit votes are valid. The release manager
> is encouraged to vote on releases, like any reviewer would do."
>
> So, do put an explicit +1, before you count yourself. Good Luck!!!
>
> -Ayush
>
> [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>
> On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> followup: overnight work happy too.
>>
>> one interesting pain point is that on a raspberry pi 64 os checknative
>> complains that libcrypto is missing
>>
>> > bin/hadoop checknative
>>
>> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> initialized native-bzip2 library system-native
>> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized native-zlib library
>> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
>> is not available in your platform... using builtin-java codec where
>> applicable
>> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
>> without PMDK support.
>> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
>> Cipher.
>> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> cannot open shared object file: No such file or directory)!
>> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
>> at
>> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> at
>>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> Native library checking:
>> hadoop: true
>>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
>> object file: No such file or directory)!
>> ISA-L: false libhadoop was built without ISA-L support
>> PMDK: false The native code was built without PMDK support.
>>
>> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> /usr/lib/aarch64-linux-gnu/l
>> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>
>> Anyone got any insights on how I should set up this (debian-based) OS
>> here?
>> I know it's only a small box but with arm64 VMs becoming available in
>> cloud
>> infras, it'd be good to know if they are similar.
>>
>> Note: checknative itself is happy; but checknative -a will fail because of
>> this -though it's an OS setup issue, nothing related to the hadoop
>> binaries.
>>
>> steve
>>
>> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>>
>> > Hi Shilun, Counter should be with yourself vote, where the current
>> summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> > Thanks again.
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>> >
>> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
>> > and
>> > > 1 vote non-binding.
>> > >
>> > > Thank you very much for voting and verifying!
>> > >
>> > > This voting will continue until March 15th, this Friday.
>> > >
>> > > Best Regards,
>> > > Shilun Fan.
>> > >
>> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> > <stevel@cloudera.com.invalid
>> > > >
>> > > wrote:
>> > >
>> > > > +1 binding
>> > > >
>> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> seen
>> > it
>> > > > earlier. been testing it this afternoon:
>> > > >
>> > > > pulled the latest version of
>> > > > https://github.com/apache/hadoop-release-support
>> > > > (note, this module is commit-then-review; whoever is working
>> > > on/validating
>> > > > a release can commit as they go along. This is not production
>> code...)
>> > > >
>> > > > * went through the "validating a release" step, validating maven
>> > > artifacts
>> > > > * building the same downstream modules which built for me last time
>> > (avro
>> > > > too complex; hboss not aws v2 in apache yet)
>> > > >
>> > > > spark build is still ongoing, but I'm not going to wait. It is
>> > building,
>> > > > which is key.
>> > > >
>> > > > The core changes I needed in are at the dependency level and I've
>> > > > verified they are good.
>> > > >
>> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
>> > > > stuff for its checknative; not expecting problems.
>> > > >
>> > > > So: i've got some stuff still ongoing, but the core changes to
>> > packaging
>> > > > are in and the rest I'm not worried about -they shouldn't block the
>> > > release
>> > > > as I already validated them on RC2
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
>> > > Hadoop
>> > > > > 3.4.0.
>> > > > >
>> > > > > What we would like is for anyone who can to verify the tarballs,
>> > > > especially
>> > > > > anyone who can try the arm64 binaries as we want to include them
>> too.
>> > > > >
>> > > > > The RC is available at:
>> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> > > > >
>> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> > > > >
>> > > > > The maven artifacts are staged at
>> > > > >
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> > > > >
>> > > > > You can find my public key at:
>> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > > >
>> > > > > Change log
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> > > > >
>> > > > > Release notes
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> > > > >
>> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> > > > >
>> > > > > Key changes include
>> > > > >
>> > > > > * S3A: Upgrade AWS SDK to V2
>> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > > > > * YARN Federation improvements
>> > > > > * YARN Capacity Scheduler improvements
>> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> > > > > * Transitive CVE fixes
>> > > > >
>> > > > > Differences from Hadoop-3.4.0-RC2
>> > > > >
>> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> (ad8b6541117b)
>> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > > > > (80b4bb68159c)
>> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> > > > > * Add protobuf compatibility issue description
>> > > > >
>> > > > > Note, because the arm64 binaries are built separately on a
>> different
>> > > > > platform and JVM, their jar files may not match those of the x86
>> > > > > release -and therefore the maven artifacts. I don't think this is
>> > > > > an issue (the ASF actually releases source tarballs, the binaries
>> are
>> > > > > there for help only, though with the maven repo that's a bit
>> > blurred).
>> > > > >
>> > > > > The only way to be consistent would actually untar the x86.tar.gz,
>> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> out
>> > > > > for the vote. Even automating that would be risky.
>> > > > >
>> > > > > [1] hadoop-release-support:
>> > > > > https://github.com/apache/hadoop-release-support
>> > > > > Thanks to steve for providing hadoop-release-support.
>> > > > >
>> > > > > Best Regards,
>> > > > > Shilun Fan.
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Thanks Ayush for highlighting this information. Absolutely true, we should
count RM's vote when explicit +1 here.
Best Regards,
- He Xiaoqiao
On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena <ay...@gmail.com> wrote:
> > Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>
> Just on the process: The release manager needs to "explicitly" vote like
> any other before counting their own vote, there has been a lot of
> discussions around that at multiple places & the official apache doc has
> been updated as well [1], the last paragraph reads:
>
> "Note that there is no implicit +1 from the release manager, or from
> anyone in any ASF vote. Only explicit votes are valid. The release manager
> is encouraged to vote on releases, like any reviewer would do."
>
> So, do put an explicit +1, before you count yourself. Good Luck!!!
>
> -Ayush
>
> [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>
> On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> followup: overnight work happy too.
>>
>> one interesting pain point is that on a raspberry pi 64 os checknative
>> complains that libcrypto is missing
>>
>> > bin/hadoop checknative
>>
>> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> initialized native-bzip2 library system-native
>> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized native-zlib library
>> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
>> is not available in your platform... using builtin-java codec where
>> applicable
>> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
>> without PMDK support.
>> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
>> Cipher.
>> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> cannot open shared object file: No such file or directory)!
>> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
>> at
>> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
>> at
>>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> Native library checking:
>> hadoop: true
>>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> zlib: true /lib/aarch64-linux-gnu/libz.so.1
>> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
>> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
>> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
>> object file: No such file or directory)!
>> ISA-L: false libhadoop was built without ISA-L support
>> PMDK: false The native code was built without PMDK support.
>>
>> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> /usr/lib/aarch64-linux-gnu/l
>> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>
>> Anyone got any insights on how I should set up this (debian-based) OS
>> here?
>> I know it's only a small box but with arm64 VMs becoming available in
>> cloud
>> infras, it'd be good to know if they are similar.
>>
>> Note: checknative itself is happy; but checknative -a will fail because of
>> this -though it's an OS setup issue, nothing related to the hadoop
>> binaries.
>>
>> steve
>>
>> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>>
>> > Hi Shilun, Counter should be with yourself vote, where the current
>> summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> > Thanks again.
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>> >
>> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
>> > and
>> > > 1 vote non-binding.
>> > >
>> > > Thank you very much for voting and verifying!
>> > >
>> > > This voting will continue until March 15th, this Friday.
>> > >
>> > > Best Regards,
>> > > Shilun Fan.
>> > >
>> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> > <stevel@cloudera.com.invalid
>> > > >
>> > > wrote:
>> > >
>> > > > +1 binding
>> > > >
>> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> seen
>> > it
>> > > > earlier. been testing it this afternoon:
>> > > >
>> > > > pulled the latest version of
>> > > > https://github.com/apache/hadoop-release-support
>> > > > (note, this module is commit-then-review; whoever is working
>> > > on/validating
>> > > > a release can commit as they go along. This is not production
>> code...)
>> > > >
>> > > > * went through the "validating a release" step, validating maven
>> > > artifacts
>> > > > * building the same downstream modules which built for me last time
>> > (avro
>> > > > too complex; hboss not aws v2 in apache yet)
>> > > >
>> > > > spark build is still ongoing, but I'm not going to wait. It is
>> > building,
>> > > > which is key.
>> > > >
>> > > > The core changes I needed in are at the dependency level and I've
>> > > > verified they are good.
>> > > >
>> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
>> > > > stuff for its checknative; not expecting problems.
>> > > >
>> > > > So: i've got some stuff still ongoing, but the core changes to
>> > packaging
>> > > > are in and the rest I'm not worried about -they shouldn't block the
>> > > release
>> > > > as I already validated them on RC2
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
>> > > Hadoop
>> > > > > 3.4.0.
>> > > > >
>> > > > > What we would like is for anyone who can to verify the tarballs,
>> > > > especially
>> > > > > anyone who can try the arm64 binaries as we want to include them
>> too.
>> > > > >
>> > > > > The RC is available at:
>> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> > > > >
>> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> > > > >
>> > > > > The maven artifacts are staged at
>> > > > >
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> > > > >
>> > > > > You can find my public key at:
>> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > > >
>> > > > > Change log
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> > > > >
>> > > > > Release notes
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> > > > >
>> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> > > > >
>> > > > > Key changes include
>> > > > >
>> > > > > * S3A: Upgrade AWS SDK to V2
>> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > > > > * YARN Federation improvements
>> > > > > * YARN Capacity Scheduler improvements
>> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > > > > * HDFS EC: Code Enhancements and Bug Fixes
>> > > > > * Transitive CVE fixes
>> > > > >
>> > > > > Differences from Hadoop-3.4.0-RC2
>> > > > >
>> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
>> (ad8b6541117b)
>> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > > > > (80b4bb68159c)
>> > > > > * Use hadoop-release-support[1] for packaging and verification.
>> > > > > * Add protobuf compatibility issue description
>> > > > >
>> > > > > Note, because the arm64 binaries are built separately on a
>> different
>> > > > > platform and JVM, their jar files may not match those of the x86
>> > > > > release -and therefore the maven artifacts. I don't think this is
>> > > > > an issue (the ASF actually releases source tarballs, the binaries
>> are
>> > > > > there for help only, though with the maven repo that's a bit
>> > blurred).
>> > > > >
>> > > > > The only way to be consistent would actually untar the x86.tar.gz,
>> > > > > overwrite its binaries with the arm stuff, retar, sign and push
>> out
>> > > > > for the vote. Even automating that would be risky.
>> > > > >
>> > > > > [1] hadoop-release-support:
>> > > > > https://github.com/apache/hadoop-release-support
>> > > > > Thanks to steve for providing hadoop-release-support.
>> > > > >
>> > > > > Best Regards,
>> > > > > Shilun Fan.
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
> Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Just on the process: The release manager needs to "explicitly" vote like
any other before counting their own vote, there has been a lot of
discussions around that at multiple places & the official apache doc has
been updated as well [1], the last paragraph reads:
"Note that there is no implicit +1 from the release manager, or from anyone
in any ASF vote. Only explicit votes are valid. The release manager is
encouraged to vote on releases, like any reviewer would do."
So, do put an explicit +1, before you count yourself. Good Luck!!!
-Ayush
[1] https://www.apache.org/foundation/voting.html#ReleaseVotes
On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
wrote:
> followup: overnight work happy too.
>
> one interesting pain point is that on a raspberry pi 64 os checknative
> complains that libcrypto is missing
>
> > bin/hadoop checknative
>
> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
> is not available in your platform... using builtin-java codec where
> applicable
> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
> Cipher.
> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> cannot open shared object file: No such file or directory)!
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> at
>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> Native library checking:
> hadoop: true
>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
> object file: No such file or directory)!
> ISA-L: false libhadoop was built without ISA-L support
> PMDK: false The native code was built without PMDK support.
>
> which happens because its not in /lib/aarch64-linux-gnu but instead in
> /usr/lib/aarch64-linux-gnu/l
> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>
> Anyone got any insights on how I should set up this (debian-based) OS here?
> I know it's only a small box but with arm64 VMs becoming available in cloud
> infras, it'd be good to know if they are similar.
>
> Note: checknative itself is happy; but checknative -a will fail because of
> this -though it's an OS setup issue, nothing related to the hadoop
> binaries.
>
> steve
>
> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>
> > Hi Shilun, Counter should be with yourself vote, where the current
> summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > Thanks again.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> >
> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
> > and
> > > 1 vote non-binding.
> > >
> > > Thank you very much for voting and verifying!
> > >
> > > This voting will continue until March 15th, this Friday.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> seen
> > it
> > > > earlier. been testing it this afternoon:
> > > >
> > > > pulled the latest version of
> > > > https://github.com/apache/hadoop-release-support
> > > > (note, this module is commit-then-review; whoever is working
> > > on/validating
> > > > a release can commit as they go along. This is not production
> code...)
> > > >
> > > > * went through the "validating a release" step, validating maven
> > > artifacts
> > > > * building the same downstream modules which built for me last time
> > (avro
> > > > too complex; hboss not aws v2 in apache yet)
> > > >
> > > > spark build is still ongoing, but I'm not going to wait. It is
> > building,
> > > > which is key.
> > > >
> > > > The core changes I needed in are at the dependency level and I've
> > > > verified they are good.
> > > >
> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > > stuff for its checknative; not expecting problems.
> > > >
> > > > So: i've got some stuff still ongoing, but the core changes to
> > packaging
> > > > are in and the rest I'm not worried about -they shouldn't block the
> > > release
> > > > as I already validated them on RC2
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > > Hadoop
> > > > > 3.4.0.
> > > > >
> > > > > What we would like is for anyone who can to verify the tarballs,
> > > > especially
> > > > > anyone who can try the arm64 binaries as we want to include them
> too.
> > > > >
> > > > > The RC is available at:
> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > > >
> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > > >
> > > > > The maven artifacts are staged at
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > > >
> > > > > You can find my public key at:
> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >
> > > > > Change log
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > > >
> > > > > Release notes
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > > >
> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > > >
> > > > > Key changes include
> > > > >
> > > > > * S3A: Upgrade AWS SDK to V2
> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > > * YARN Federation improvements
> > > > > * YARN Capacity Scheduler improvements
> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > > * Transitive CVE fixes
> > > > >
> > > > > Differences from Hadoop-3.4.0-RC2
> > > > >
> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> (ad8b6541117b)
> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > > (80b4bb68159c)
> > > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > > * Add protobuf compatibility issue description
> > > > >
> > > > > Note, because the arm64 binaries are built separately on a
> different
> > > > > platform and JVM, their jar files may not match those of the x86
> > > > > release -and therefore the maven artifacts. I don't think this is
> > > > > an issue (the ASF actually releases source tarballs, the binaries
> are
> > > > > there for help only, though with the maven repo that's a bit
> > blurred).
> > > > >
> > > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > > for the vote. Even automating that would be risky.
> > > > >
> > > > > [1] hadoop-release-support:
> > > > > https://github.com/apache/hadoop-release-support
> > > > > Thanks to steve for providing hadoop-release-support.
> > > > >
> > > > > Best Regards,
> > > > > Shilun Fan.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
> Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Just on the process: The release manager needs to "explicitly" vote like
any other before counting their own vote, there has been a lot of
discussions around that at multiple places & the official apache doc has
been updated as well [1], the last paragraph reads:
"Note that there is no implicit +1 from the release manager, or from anyone
in any ASF vote. Only explicit votes are valid. The release manager is
encouraged to vote on releases, like any reviewer would do."
So, do put an explicit +1, before you count yourself. Good Luck!!!
-Ayush
[1] https://www.apache.org/foundation/voting.html#ReleaseVotes
On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
wrote:
> followup: overnight work happy too.
>
> one interesting pain point is that on a raspberry pi 64 os checknative
> complains that libcrypto is missing
>
> > bin/hadoop checknative
>
> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
> is not available in your platform... using builtin-java codec where
> applicable
> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
> Cipher.
> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> cannot open shared object file: No such file or directory)!
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> at
>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> Native library checking:
> hadoop: true
>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
> object file: No such file or directory)!
> ISA-L: false libhadoop was built without ISA-L support
> PMDK: false The native code was built without PMDK support.
>
> which happens because its not in /lib/aarch64-linux-gnu but instead in
> /usr/lib/aarch64-linux-gnu/l
> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>
> Anyone got any insights on how I should set up this (debian-based) OS here?
> I know it's only a small box but with arm64 VMs becoming available in cloud
> infras, it'd be good to know if they are similar.
>
> Note: checknative itself is happy; but checknative -a will fail because of
> this -though it's an OS setup issue, nothing related to the hadoop
> binaries.
>
> steve
>
> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>
> > Hi Shilun, Counter should be with yourself vote, where the current
> summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > Thanks again.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> >
> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
> > and
> > > 1 vote non-binding.
> > >
> > > Thank you very much for voting and verifying!
> > >
> > > This voting will continue until March 15th, this Friday.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> seen
> > it
> > > > earlier. been testing it this afternoon:
> > > >
> > > > pulled the latest version of
> > > > https://github.com/apache/hadoop-release-support
> > > > (note, this module is commit-then-review; whoever is working
> > > on/validating
> > > > a release can commit as they go along. This is not production
> code...)
> > > >
> > > > * went through the "validating a release" step, validating maven
> > > artifacts
> > > > * building the same downstream modules which built for me last time
> > (avro
> > > > too complex; hboss not aws v2 in apache yet)
> > > >
> > > > spark build is still ongoing, but I'm not going to wait. It is
> > building,
> > > > which is key.
> > > >
> > > > The core changes I needed in are at the dependency level and I've
> > > > verified they are good.
> > > >
> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > > stuff for its checknative; not expecting problems.
> > > >
> > > > So: i've got some stuff still ongoing, but the core changes to
> > packaging
> > > > are in and the rest I'm not worried about -they shouldn't block the
> > > release
> > > > as I already validated them on RC2
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > > Hadoop
> > > > > 3.4.0.
> > > > >
> > > > > What we would like is for anyone who can to verify the tarballs,
> > > > especially
> > > > > anyone who can try the arm64 binaries as we want to include them
> too.
> > > > >
> > > > > The RC is available at:
> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > > >
> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > > >
> > > > > The maven artifacts are staged at
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > > >
> > > > > You can find my public key at:
> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >
> > > > > Change log
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > > >
> > > > > Release notes
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > > >
> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > > >
> > > > > Key changes include
> > > > >
> > > > > * S3A: Upgrade AWS SDK to V2
> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > > * YARN Federation improvements
> > > > > * YARN Capacity Scheduler improvements
> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > > * Transitive CVE fixes
> > > > >
> > > > > Differences from Hadoop-3.4.0-RC2
> > > > >
> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> (ad8b6541117b)
> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > > (80b4bb68159c)
> > > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > > * Add protobuf compatibility issue description
> > > > >
> > > > > Note, because the arm64 binaries are built separately on a
> different
> > > > > platform and JVM, their jar files may not match those of the x86
> > > > > release -and therefore the maven artifacts. I don't think this is
> > > > > an issue (the ASF actually releases source tarballs, the binaries
> are
> > > > > there for help only, though with the maven repo that's a bit
> > blurred).
> > > > >
> > > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > > for the vote. Even automating that would be risky.
> > > > >
> > > > > [1] hadoop-release-support:
> > > > > https://github.com/apache/hadoop-release-support
> > > > > Thanks to steve for providing hadoop-release-support.
> > > > >
> > > > > Best Regards,
> > > > > Shilun Fan.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
> Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Just on the process: The release manager needs to "explicitly" vote like
any other before counting their own vote, there has been a lot of
discussions around that at multiple places & the official apache doc has
been updated as well [1], the last paragraph reads:
"Note that there is no implicit +1 from the release manager, or from anyone
in any ASF vote. Only explicit votes are valid. The release manager is
encouraged to vote on releases, like any reviewer would do."
So, do put an explicit +1, before you count yourself. Good Luck!!!
-Ayush
[1] https://www.apache.org/foundation/voting.html#ReleaseVotes
On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
wrote:
> followup: overnight work happy too.
>
> one interesting pain point is that on a raspberry pi 64 os checknative
> complains that libcrypto is missing
>
> > bin/hadoop checknative
>
> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
> is not available in your platform... using builtin-java codec where
> applicable
> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
> Cipher.
> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> cannot open shared object file: No such file or directory)!
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> at
>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> Native library checking:
> hadoop: true
>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
> object file: No such file or directory)!
> ISA-L: false libhadoop was built without ISA-L support
> PMDK: false The native code was built without PMDK support.
>
> which happens because its not in /lib/aarch64-linux-gnu but instead in
> /usr/lib/aarch64-linux-gnu/l
> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>
> Anyone got any insights on how I should set up this (debian-based) OS here?
> I know it's only a small box but with arm64 VMs becoming available in cloud
> infras, it'd be good to know if they are similar.
>
> Note: checknative itself is happy; but checknative -a will fail because of
> this -though it's an OS setup issue, nothing related to the hadoop
> binaries.
>
> steve
>
> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>
> > Hi Shilun, Counter should be with yourself vote, where the current
> summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > Thanks again.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> >
> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
> > and
> > > 1 vote non-binding.
> > >
> > > Thank you very much for voting and verifying!
> > >
> > > This voting will continue until March 15th, this Friday.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> seen
> > it
> > > > earlier. been testing it this afternoon:
> > > >
> > > > pulled the latest version of
> > > > https://github.com/apache/hadoop-release-support
> > > > (note, this module is commit-then-review; whoever is working
> > > on/validating
> > > > a release can commit as they go along. This is not production
> code...)
> > > >
> > > > * went through the "validating a release" step, validating maven
> > > artifacts
> > > > * building the same downstream modules which built for me last time
> > (avro
> > > > too complex; hboss not aws v2 in apache yet)
> > > >
> > > > spark build is still ongoing, but I'm not going to wait. It is
> > building,
> > > > which is key.
> > > >
> > > > The core changes I needed in are at the dependency level and I've
> > > > verified they are good.
> > > >
> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > > stuff for its checknative; not expecting problems.
> > > >
> > > > So: i've got some stuff still ongoing, but the core changes to
> > packaging
> > > > are in and the rest I'm not worried about -they shouldn't block the
> > > release
> > > > as I already validated them on RC2
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > > Hadoop
> > > > > 3.4.0.
> > > > >
> > > > > What we would like is for anyone who can to verify the tarballs,
> > > > especially
> > > > > anyone who can try the arm64 binaries as we want to include them
> too.
> > > > >
> > > > > The RC is available at:
> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > > >
> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > > >
> > > > > The maven artifacts are staged at
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > > >
> > > > > You can find my public key at:
> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >
> > > > > Change log
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > > >
> > > > > Release notes
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > > >
> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > > >
> > > > > Key changes include
> > > > >
> > > > > * S3A: Upgrade AWS SDK to V2
> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > > * YARN Federation improvements
> > > > > * YARN Capacity Scheduler improvements
> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > > * Transitive CVE fixes
> > > > >
> > > > > Differences from Hadoop-3.4.0-RC2
> > > > >
> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> (ad8b6541117b)
> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > > (80b4bb68159c)
> > > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > > * Add protobuf compatibility issue description
> > > > >
> > > > > Note, because the arm64 binaries are built separately on a
> different
> > > > > platform and JVM, their jar files may not match those of the x86
> > > > > release -and therefore the maven artifacts. I don't think this is
> > > > > an issue (the ASF actually releases source tarballs, the binaries
> are
> > > > > there for help only, though with the maven repo that's a bit
> > blurred).
> > > > >
> > > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > > for the vote. Even automating that would be risky.
> > > > >
> > > > > [1] hadoop-release-support:
> > > > > https://github.com/apache/hadoop-release-support
> > > > > Thanks to steve for providing hadoop-release-support.
> > > > >
> > > > > Best Regards,
> > > > > Shilun Fan.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Ayush Saxena <ay...@gmail.com>.
> Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Just on the process: The release manager needs to "explicitly" vote like
any other before counting their own vote, there has been a lot of
discussions around that at multiple places & the official apache doc has
been updated as well [1], the last paragraph reads:
"Note that there is no implicit +1 from the release manager, or from anyone
in any ASF vote. Only explicit votes are valid. The release manager is
encouraged to vote on releases, like any reviewer would do."
So, do put an explicit +1, before you count yourself. Good Luck!!!
-Ayush
[1] https://www.apache.org/foundation/voting.html#ReleaseVotes
On Tue, 12 Mar 2024 at 17:27, Steve Loughran <st...@cloudera.com.invalid>
wrote:
> followup: overnight work happy too.
>
> one interesting pain point is that on a raspberry pi 64 os checknative
> complains that libcrypto is missing
>
> > bin/hadoop checknative
>
> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
> is not available in your platform... using builtin-java codec where
> applicable
> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
> Cipher.
> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> cannot open shared object file: No such file or directory)!
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
> at
>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> Native library checking:
> hadoop: true
>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> zlib: true /lib/aarch64-linux-gnu/libz.so.1
> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
> object file: No such file or directory)!
> ISA-L: false libhadoop was built without ISA-L support
> PMDK: false The native code was built without PMDK support.
>
> which happens because its not in /lib/aarch64-linux-gnu but instead in
> /usr/lib/aarch64-linux-gnu/l
> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>
> Anyone got any insights on how I should set up this (debian-based) OS here?
> I know it's only a small box but with arm64 VMs becoming available in cloud
> infras, it'd be good to know if they are similar.
>
> Note: checknative itself is happy; but checknative -a will fail because of
> this -though it's an OS setup issue, nothing related to the hadoop
> binaries.
>
> steve
>
> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
>
> > Hi Shilun, Counter should be with yourself vote, where the current
> summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > Thanks again.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> >
> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
> > and
> > > 1 vote non-binding.
> > >
> > > Thank you very much for voting and verifying!
> > >
> > > This voting will continue until March 15th, this Friday.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> seen
> > it
> > > > earlier. been testing it this afternoon:
> > > >
> > > > pulled the latest version of
> > > > https://github.com/apache/hadoop-release-support
> > > > (note, this module is commit-then-review; whoever is working
> > > on/validating
> > > > a release can commit as they go along. This is not production
> code...)
> > > >
> > > > * went through the "validating a release" step, validating maven
> > > artifacts
> > > > * building the same downstream modules which built for me last time
> > (avro
> > > > too complex; hboss not aws v2 in apache yet)
> > > >
> > > > spark build is still ongoing, but I'm not going to wait. It is
> > building,
> > > > which is key.
> > > >
> > > > The core changes I needed in are at the dependency level and I've
> > > > verified they are good.
> > > >
> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > > stuff for its checknative; not expecting problems.
> > > >
> > > > So: i've got some stuff still ongoing, but the core changes to
> > packaging
> > > > are in and the rest I'm not worried about -they shouldn't block the
> > > release
> > > > as I already validated them on RC2
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > > Hadoop
> > > > > 3.4.0.
> > > > >
> > > > > What we would like is for anyone who can to verify the tarballs,
> > > > especially
> > > > > anyone who can try the arm64 binaries as we want to include them
> too.
> > > > >
> > > > > The RC is available at:
> > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > > >
> > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > > >
> > > > > The maven artifacts are staged at
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > > >
> > > > > You can find my public key at:
> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >
> > > > > Change log
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > > >
> > > > > Release notes
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > > >
> > > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > > >
> > > > > Key changes include
> > > > >
> > > > > * S3A: Upgrade AWS SDK to V2
> > > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > > * YARN Federation improvements
> > > > > * YARN Capacity Scheduler improvements
> > > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > > * Transitive CVE fixes
> > > > >
> > > > > Differences from Hadoop-3.4.0-RC2
> > > > >
> > > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > > * HADOOP-18088: Replacing log4j 1.x with reload4j.
> (ad8b6541117b)
> > > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > > (80b4bb68159c)
> > > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > > * Add protobuf compatibility issue description
> > > > >
> > > > > Note, because the arm64 binaries are built separately on a
> different
> > > > > platform and JVM, their jar files may not match those of the x86
> > > > > release -and therefore the maven artifacts. I don't think this is
> > > > > an issue (the ASF actually releases source tarballs, the binaries
> are
> > > > > there for help only, though with the maven repo that's a bit
> > blurred).
> > > > >
> > > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > > for the vote. Even automating that would be risky.
> > > > >
> > > > > [1] hadoop-release-support:
> > > > > https://github.com/apache/hadoop-release-support
> > > > > Thanks to steve for providing hadoop-release-support.
> > > > >
> > > > > Best Regards,
> > > > > Shilun Fan.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
followup: overnight work happy too.
one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing
> bin/hadoop checknative
2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop: true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib: true /lib/aarch64-linux-gnu/libz.so.1
zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L: false libhadoop was built without ISA-L support
PMDK: false The native code was built without PMDK support.
which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3
Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.
Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.
steve
On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > > especially
> > > > anyone who can try the arm64 binaries as we want to include them too.
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > >
> > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > >
> > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > >
> > > > Key changes include
> > > >
> > > > * S3A: Upgrade AWS SDK to V2
> > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > * YARN Federation improvements
> > > > * YARN Capacity Scheduler improvements
> > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > * Transitive CVE fixes
> > > >
> > > > Differences from Hadoop-3.4.0-RC2
> > > >
> > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > (80b4bb68159c)
> > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > * Add protobuf compatibility issue description
> > > >
> > > > Note, because the arm64 binaries are built separately on a different
> > > > platform and JVM, their jar files may not match those of the x86
> > > > release -and therefore the maven artifacts. I don't think this is
> > > > an issue (the ASF actually releases source tarballs, the binaries are
> > > > there for help only, though with the maven repo that's a bit
> blurred).
> > > >
> > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > for the vote. Even automating that would be risky.
> > > >
> > > > [1] hadoop-release-support:
> > > > https://github.com/apache/hadoop-release-support
> > > > Thanks to steve for providing hadoop-release-support.
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
followup: overnight work happy too.
one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing
> bin/hadoop checknative
2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop: true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib: true /lib/aarch64-linux-gnu/libz.so.1
zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L: false libhadoop was built without ISA-L support
PMDK: false The native code was built without PMDK support.
which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3
Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.
Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.
steve
On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > > especially
> > > > anyone who can try the arm64 binaries as we want to include them too.
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > >
> > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > >
> > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > >
> > > > Key changes include
> > > >
> > > > * S3A: Upgrade AWS SDK to V2
> > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > * YARN Federation improvements
> > > > * YARN Capacity Scheduler improvements
> > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > * Transitive CVE fixes
> > > >
> > > > Differences from Hadoop-3.4.0-RC2
> > > >
> > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > (80b4bb68159c)
> > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > * Add protobuf compatibility issue description
> > > >
> > > > Note, because the arm64 binaries are built separately on a different
> > > > platform and JVM, their jar files may not match those of the x86
> > > > release -and therefore the maven artifacts. I don't think this is
> > > > an issue (the ASF actually releases source tarballs, the binaries are
> > > > there for help only, though with the maven repo that's a bit
> blurred).
> > > >
> > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > for the vote. Even automating that would be risky.
> > > >
> > > > [1] hadoop-release-support:
> > > > https://github.com/apache/hadoop-release-support
> > > > Thanks to steve for providing hadoop-release-support.
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
followup: overnight work happy too.
one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing
> bin/hadoop checknative
2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop: true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib: true /lib/aarch64-linux-gnu/libz.so.1
zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L: false libhadoop was built without ISA-L support
PMDK: false The native code was built without PMDK support.
which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3
Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.
Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.
steve
On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > > especially
> > > > anyone who can try the arm64 binaries as we want to include them too.
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > >
> > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > >
> > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > >
> > > > Key changes include
> > > >
> > > > * S3A: Upgrade AWS SDK to V2
> > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > * YARN Federation improvements
> > > > * YARN Capacity Scheduler improvements
> > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > * Transitive CVE fixes
> > > >
> > > > Differences from Hadoop-3.4.0-RC2
> > > >
> > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > (80b4bb68159c)
> > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > * Add protobuf compatibility issue description
> > > >
> > > > Note, because the arm64 binaries are built separately on a different
> > > > platform and JVM, their jar files may not match those of the x86
> > > > release -and therefore the maven artifacts. I don't think this is
> > > > an issue (the ASF actually releases source tarballs, the binaries are
> > > > there for help only, though with the maven repo that's a bit
> blurred).
> > > >
> > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > for the vote. Even automating that would be risky.
> > > >
> > > > [1] hadoop-release-support:
> > > > https://github.com/apache/hadoop-release-support
> > > > Thanks to steve for providing hadoop-release-support.
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
followup: overnight work happy too.
one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing
> bin/hadoop checknative
2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop: true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib: true /lib/aarch64-linux-gnu/libz.so.1
zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L: false libhadoop was built without ISA-L support
PMDK: false The native code was built without PMDK support.
which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3
Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.
Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.
steve
On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <he...@apache.org> wrote:
> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > > especially
> > > > anyone who can try the arm64 binaries as we want to include them too.
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > >
> > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > >
> > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > >
> > > > Key changes include
> > > >
> > > > * S3A: Upgrade AWS SDK to V2
> > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > * YARN Federation improvements
> > > > * YARN Capacity Scheduler improvements
> > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > * Transitive CVE fixes
> > > >
> > > > Differences from Hadoop-3.4.0-RC2
> > > >
> > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > (80b4bb68159c)
> > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > * Add protobuf compatibility issue description
> > > >
> > > > Note, because the arm64 binaries are built separately on a different
> > > > platform and JVM, their jar files may not match those of the x86
> > > > release -and therefore the maven artifacts. I don't think this is
> > > > an issue (the ASF actually releases source tarballs, the binaries are
> > > > there for help only, though with the maven repo that's a bit
> blurred).
> > > >
> > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > for the vote. Even automating that would be risky.
> > > >
> > > > [1] hadoop-release-support:
> > > > https://github.com/apache/hadoop-release-support
> > > > Thanks to steve for providing hadoop-release-support.
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Hi Shilun, Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Thanks again.
Best Regards,
- He Xiaoqiao
On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> As of now, we have collected 5 affirmative votes, with 4 votes binding and
> 1 vote non-binding.
>
> Thank you very much for voting and verifying!
>
> This voting will continue until March 15th, this Friday.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > +1 binding
> >
> > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> > earlier. been testing it this afternoon:
> >
> > pulled the latest version of
> > https://github.com/apache/hadoop-release-support
> > (note, this module is commit-then-review; whoever is working
> on/validating
> > a release can commit as they go along. This is not production code...)
> >
> > * went through the "validating a release" step, validating maven
> artifacts
> > * building the same downstream modules which built for me last time (avro
> > too complex; hboss not aws v2 in apache yet)
> >
> > spark build is still ongoing, but I'm not going to wait. It is building,
> > which is key.
> >
> > The core changes I needed in are at the dependency level and I've
> > verified they are good.
> >
> > Oh, and I've also got my raspberry p5 doing the download of the arm
> > stuff for its checknative; not expecting problems.
> >
> > So: i've got some stuff still ongoing, but the core changes to packaging
> > are in and the rest I'm not worried about -they shouldn't block the
> release
> > as I already validated them on RC2
> >
> >
> >
> >
> >
> > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC3) for
> Hadoop
> > > 3.4.0.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > >
> > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > >
> > > Key changes include
> > >
> > > * S3A: Upgrade AWS SDK to V2
> > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > * YARN Federation improvements
> > > * YARN Capacity Scheduler improvements
> > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > * HDFS EC: Code Enhancements and Bug Fixes
> > > * Transitive CVE fixes
> > >
> > > Differences from Hadoop-3.4.0-RC2
> > >
> > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > (80b4bb68159c)
> > > * Use hadoop-release-support[1] for packaging and verification.
> > > * Add protobuf compatibility issue description
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > [1] hadoop-release-support:
> > > https://github.com/apache/hadoop-release-support
> > > Thanks to steve for providing hadoop-release-support.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Hi Shilun, Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Thanks again.
Best Regards,
- He Xiaoqiao
On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> As of now, we have collected 5 affirmative votes, with 4 votes binding and
> 1 vote non-binding.
>
> Thank you very much for voting and verifying!
>
> This voting will continue until March 15th, this Friday.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > +1 binding
> >
> > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> > earlier. been testing it this afternoon:
> >
> > pulled the latest version of
> > https://github.com/apache/hadoop-release-support
> > (note, this module is commit-then-review; whoever is working
> on/validating
> > a release can commit as they go along. This is not production code...)
> >
> > * went through the "validating a release" step, validating maven
> artifacts
> > * building the same downstream modules which built for me last time (avro
> > too complex; hboss not aws v2 in apache yet)
> >
> > spark build is still ongoing, but I'm not going to wait. It is building,
> > which is key.
> >
> > The core changes I needed in are at the dependency level and I've
> > verified they are good.
> >
> > Oh, and I've also got my raspberry p5 doing the download of the arm
> > stuff for its checknative; not expecting problems.
> >
> > So: i've got some stuff still ongoing, but the core changes to packaging
> > are in and the rest I'm not worried about -they shouldn't block the
> release
> > as I already validated them on RC2
> >
> >
> >
> >
> >
> > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC3) for
> Hadoop
> > > 3.4.0.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > >
> > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > >
> > > Key changes include
> > >
> > > * S3A: Upgrade AWS SDK to V2
> > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > * YARN Federation improvements
> > > * YARN Capacity Scheduler improvements
> > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > * HDFS EC: Code Enhancements and Bug Fixes
> > > * Transitive CVE fixes
> > >
> > > Differences from Hadoop-3.4.0-RC2
> > >
> > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > (80b4bb68159c)
> > > * Use hadoop-release-support[1] for packaging and verification.
> > > * Add protobuf compatibility issue description
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > [1] hadoop-release-support:
> > > https://github.com/apache/hadoop-release-support
> > > Thanks to steve for providing hadoop-release-support.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Hi Shilun, Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Thanks again.
Best Regards,
- He Xiaoqiao
On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> As of now, we have collected 5 affirmative votes, with 4 votes binding and
> 1 vote non-binding.
>
> Thank you very much for voting and verifying!
>
> This voting will continue until March 15th, this Friday.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > +1 binding
> >
> > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> > earlier. been testing it this afternoon:
> >
> > pulled the latest version of
> > https://github.com/apache/hadoop-release-support
> > (note, this module is commit-then-review; whoever is working
> on/validating
> > a release can commit as they go along. This is not production code...)
> >
> > * went through the "validating a release" step, validating maven
> artifacts
> > * building the same downstream modules which built for me last time (avro
> > too complex; hboss not aws v2 in apache yet)
> >
> > spark build is still ongoing, but I'm not going to wait. It is building,
> > which is key.
> >
> > The core changes I needed in are at the dependency level and I've
> > verified they are good.
> >
> > Oh, and I've also got my raspberry p5 doing the download of the arm
> > stuff for its checknative; not expecting problems.
> >
> > So: i've got some stuff still ongoing, but the core changes to packaging
> > are in and the rest I'm not worried about -they shouldn't block the
> release
> > as I already validated them on RC2
> >
> >
> >
> >
> >
> > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC3) for
> Hadoop
> > > 3.4.0.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > >
> > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > >
> > > Key changes include
> > >
> > > * S3A: Upgrade AWS SDK to V2
> > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > * YARN Federation improvements
> > > * YARN Capacity Scheduler improvements
> > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > * HDFS EC: Code Enhancements and Bug Fixes
> > > * Transitive CVE fixes
> > >
> > > Differences from Hadoop-3.4.0-RC2
> > >
> > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > (80b4bb68159c)
> > > * Use hadoop-release-support[1] for packaging and verification.
> > > * Add protobuf compatibility issue description
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > [1] hadoop-release-support:
> > > https://github.com/apache/hadoop-release-support
> > > Thanks to steve for providing hadoop-release-support.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Xiaoqiao He <he...@apache.org>.
Hi Shilun, Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Thanks again.
Best Regards,
- He Xiaoqiao
On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <sl...@apache.org> wrote:
> As of now, we have collected 5 affirmative votes, with 4 votes binding and
> 1 vote non-binding.
>
> Thank you very much for voting and verifying!
>
> This voting will continue until March 15th, this Friday.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > +1 binding
> >
> > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> > earlier. been testing it this afternoon:
> >
> > pulled the latest version of
> > https://github.com/apache/hadoop-release-support
> > (note, this module is commit-then-review; whoever is working
> on/validating
> > a release can commit as they go along. This is not production code...)
> >
> > * went through the "validating a release" step, validating maven
> artifacts
> > * building the same downstream modules which built for me last time (avro
> > too complex; hboss not aws v2 in apache yet)
> >
> > spark build is still ongoing, but I'm not going to wait. It is building,
> > which is key.
> >
> > The core changes I needed in are at the dependency level and I've
> > verified they are good.
> >
> > Oh, and I've also got my raspberry p5 doing the download of the arm
> > stuff for its checknative; not expecting problems.
> >
> > So: i've got some stuff still ongoing, but the core changes to packaging
> > are in and the rest I'm not worried about -they shouldn't block the
> release
> > as I already validated them on RC2
> >
> >
> >
> >
> >
> > On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC3) for
> Hadoop
> > > 3.4.0.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > >
> > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > >
> > > Key changes include
> > >
> > > * S3A: Upgrade AWS SDK to V2
> > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > * YARN Federation improvements
> > > * YARN Capacity Scheduler improvements
> > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > * HDFS EC: Code Enhancements and Bug Fixes
> > > * Transitive CVE fixes
> > >
> > > Differences from Hadoop-3.4.0-RC2
> > >
> > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > (80b4bb68159c)
> > > * Use hadoop-release-support[1] for packaging and verification.
> > > * Add protobuf compatibility issue description
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > [1] hadoop-release-support:
> > > https://github.com/apache/hadoop-release-support
> > > Thanks to steve for providing hadoop-release-support.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
As of now, we have collected 5 affirmative votes, with 4 votes binding and
1 vote non-binding.
Thank you very much for voting and verifying!
This voting will continue until March 15th, this Friday.
Best Regards,
Shilun Fan.
On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> +1 binding
>
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
>
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
>
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
>
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
>
> The core changes I needed in are at the dependency level and I've
> verified they are good.
>
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
>
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
>
>
>
>
>
> On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
As of now, we have collected 5 affirmative votes, with 4 votes binding and
1 vote non-binding.
Thank you very much for voting and verifying!
This voting will continue until March 15th, this Friday.
Best Regards,
Shilun Fan.
On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> +1 binding
>
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
>
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
>
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
>
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
>
> The core changes I needed in are at the dependency level and I've
> verified they are good.
>
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
>
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
>
>
>
>
>
> On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
As of now, we have collected 5 affirmative votes, with 4 votes binding and
1 vote non-binding.
Thank you very much for voting and verifying!
This voting will continue until March 15th, this Friday.
Best Regards,
Shilun Fan.
On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> +1 binding
>
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
>
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
>
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
>
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
>
> The core changes I needed in are at the dependency level and I've
> verified they are good.
>
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
>
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
>
>
>
>
>
> On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
As of now, we have collected 5 affirmative votes, with 4 votes binding and
1 vote non-binding.
Thank you very much for voting and verifying!
This voting will continue until March 15th, this Friday.
Best Regards,
Shilun Fan.
On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> +1 binding
>
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
>
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
>
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
>
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
>
> The core changes I needed in are at the dependency level and I've
> verified they are good.
>
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
>
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
>
>
>
>
>
> On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1 binding
(sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
earlier. been testing it this afternoon:
pulled the latest version of
https://github.com/apache/hadoop-release-support
(note, this module is commit-then-review; whoever is working on/validating
a release can commit as they go along. This is not production code...)
* went through the "validating a release" step, validating maven artifacts
* building the same downstream modules which built for me last time (avro
too complex; hboss not aws v2 in apache yet)
spark build is still ongoing, but I'm not going to wait. It is building,
which is key.
The core changes I needed in are at the dependency level and I've
verified they are good.
Oh, and I've also got my raspberry p5 doing the download of the arm
stuff for its checknative; not expecting problems.
So: i've got some stuff still ongoing, but the core changes to packaging
are in and the rest I'm not worried about -they shouldn't block the release
as I already validated them on RC2
On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
@Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com> @Steve
Loughran <st...@cloudera.com> @Mukund Madhav Thakur
<mt...@cloudera.com> @Takanobu
Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
Best Regards,
Shilun Fan.
On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 (binding)
Thanks for the great work, Shilun Fan.
+ verified checksums and signatures of src and site tarballs.
+ built from source tarball with native profile enabled on Rocky Linux 8
and Ubuntu 22.04 (x86_64).
+ deployed pseudo cluster with and without kerberos enabled.
+ ran some example jobs on encryption zone.
+ ran some simple query against httpfs.
+ built site documentation and skimmed the contents.
+ skimmed the contents of pre-built site documentation.
- downstream like HBase can not be built against 3.4.0 without addressing
changes of transitive dependencies.
- OpenSSL feature of libhadoop.so does not work on the platform without
SM4 support (e.g. Red Hat) while it falls back to Java implementation.
I want HADOOP-17609 be in the next release (or OpenSSL 3 support considering this)::
$ bin/hdfs dfs -put README.txt /zone1/
2024-03-08 15:57:42,685 WARN crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot find AES-CTR/SM4-CTR support, is your version of Openssl new enough?
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
Masatake Iwasaki
On 2024/03/05 7:07, slfan1989 wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support: https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
+1 (non-binding)
Thanks for driving the release, Shilun.
* Verified sha512 checksum and signature was correct for source tarball
* Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
* Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
* untar the binaries and able to access objects in S3 (for both standard and S3 express) via hadoop fs commands.
We have some failures in S3A tests, but these are expected due to one of the public landsat buckets we relied on no longer being available, these tests were fixed in HADOOP-19057<https://issues.apache.org/jira/browse/HADOOP-19057>.
________________________________
From: slfan1989 <sl...@apache.org>
Sent: Monday, March 4, 2024 10:07:05 PM
To: Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <pr...@hadoop.apache.org>
Cc: Steve Loughran; Xiaoqiao He
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi folks,
Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
The git tag is release-3.4.0-RC3, commit bd8b77f398f
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408
You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
This is off branch-3.4.0 and is the first big release since 3.3.6.
Key changes include
* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes
Differences from Hadoop-3.4.0-RC2
* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description
Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).
The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.
[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.
Best Regards,
Shilun Fan.
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1 binding
(sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
earlier. been testing it this afternoon:
pulled the latest version of
https://github.com/apache/hadoop-release-support
(note, this module is commit-then-review; whoever is working on/validating
a release can commit as they go along. This is not production code...)
* went through the "validating a release" step, validating maven artifacts
* building the same downstream modules which built for me last time (avro
too complex; hboss not aws v2 in apache yet)
spark build is still ongoing, but I'm not going to wait. It is building,
which is key.
The core changes I needed in are at the dependency level and I've
verified they are good.
Oh, and I've also got my raspberry p5 doing the download of the arm
stuff for its checknative; not expecting problems.
So: i've got some stuff still ongoing, but the core changes to packaging
are in and the rest I'm not worried about -they shouldn't block the release
as I already validated them on RC2
On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 (binding)
Thanks for the great work, Shilun Fan.
+ verified checksums and signatures of src and site tarballs.
+ built from source tarball with native profile enabled on Rocky Linux 8
and Ubuntu 22.04 (x86_64).
+ deployed pseudo cluster with and without kerberos enabled.
+ ran some example jobs on encryption zone.
+ ran some simple query against httpfs.
+ built site documentation and skimmed the contents.
+ skimmed the contents of pre-built site documentation.
- downstream like HBase can not be built against 3.4.0 without addressing
changes of transitive dependencies.
- OpenSSL feature of libhadoop.so does not work on the platform without
SM4 support (e.g. Red Hat) while it falls back to Java implementation.
I want HADOOP-17609 be in the next release (or OpenSSL 3 support considering this)::
$ bin/hdfs dfs -put README.txt /zone1/
2024-03-08 15:57:42,685 WARN crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot find AES-CTR/SM4-CTR support, is your version of Openssl new enough?
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
Masatake Iwasaki
On 2024/03/05 7:07, slfan1989 wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support: https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
+1 (non-binding)
Thanks for driving the release, Shilun.
* Verified sha512 checksum and signature was correct for source tarball
* Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
* Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
* untar the binaries and able to access objects in S3 (for both standard and S3 express) via hadoop fs commands.
We have some failures in S3A tests, but these are expected due to one of the public landsat buckets we relied on no longer being available, these tests were fixed in HADOOP-19057<https://issues.apache.org/jira/browse/HADOOP-19057>.
________________________________
From: slfan1989 <sl...@apache.org>
Sent: Monday, March 4, 2024 10:07:05 PM
To: Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <pr...@hadoop.apache.org>
Cc: Steve Loughran; Xiaoqiao He
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi folks,
Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
The git tag is release-3.4.0-RC3, commit bd8b77f398f
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408
You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
This is off branch-3.4.0 and is the first big release since 3.3.6.
Key changes include
* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes
Differences from Hadoop-3.4.0-RC2
* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description
Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).
The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.
[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.
Best Regards,
Shilun Fan.
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by "Suhail, Ahmar" <ah...@amazon.co.uk.INVALID>.
+1 (non-binding)
Thanks for driving the release, Shilun.
* Verified sha512 checksum and signature was correct for source tarball
* Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
* Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
* untar the binaries and able to access objects in S3 (for both standard and S3 express) via hadoop fs commands.
We have some failures in S3A tests, but these are expected due to one of the public landsat buckets we relied on no longer being available, these tests were fixed in HADOOP-19057<https://issues.apache.org/jira/browse/HADOOP-19057>.
________________________________
From: slfan1989 <sl...@apache.org>
Sent: Monday, March 4, 2024 10:07:05 PM
To: Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <pr...@hadoop.apache.org>
Cc: Steve Loughran; Xiaoqiao He
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi folks,
Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.
What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
The git tag is release-3.4.0-RC3, commit bd8b77f398f
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408
You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
This is off branch-3.4.0 and is the first big release since 3.3.6.
Key changes include
* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes
Differences from Hadoop-3.4.0-RC2
* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description
Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).
The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.
[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.
Best Regards,
Shilun Fan.
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1 binding
(sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
earlier. been testing it this afternoon:
pulled the latest version of
https://github.com/apache/hadoop-release-support
(note, this module is commit-then-review; whoever is working on/validating
a release can commit as they go along. This is not production code...)
* went through the "validating a release" step, validating maven artifacts
* building the same downstream modules which built for me last time (avro
too complex; hboss not aws v2 in apache yet)
spark build is still ongoing, but I'm not going to wait. It is building,
which is key.
The core changes I needed in are at the dependency level and I've
verified they are good.
Oh, and I've also got my raspberry p5 doing the download of the arm
stuff for its checknative; not expecting problems.
So: i've got some stuff still ongoing, but the core changes to packaging
are in and the rest I'm not worried about -they shouldn't block the release
as I already validated them on RC2
On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
@Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com> @Steve
Loughran <st...@cloudera.com> @Mukund Madhav Thakur
<mt...@cloudera.com> @Takanobu
Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
Best Regards,
Shilun Fan.
On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by slfan1989 <sl...@apache.org>.
@Xiaoqiao He <he...@apache.org> @Ayush Saxena <ay...@gmail.com> @Steve
Loughran <st...@cloudera.com> @Mukund Madhav Thakur
<mt...@cloudera.com> @Takanobu
Asanuma <ta...@apache.org> @Masatake Iwasaki <iw...@apache.org>
Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
Best Regards,
Shilun Fan.
On Tue, Mar 5, 2024 at 6:07 AM slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 (binding)
Thanks for the great work, Shilun Fan.
+ verified checksums and signatures of src and site tarballs.
+ built from source tarball with native profile enabled on Rocky Linux 8
and Ubuntu 22.04 (x86_64).
+ deployed pseudo cluster with and without kerberos enabled.
+ ran some example jobs on encryption zone.
+ ran some simple query against httpfs.
+ built site documentation and skimmed the contents.
+ skimmed the contents of pre-built site documentation.
- downstream like HBase can not be built against 3.4.0 without addressing
changes of transitive dependencies.
- OpenSSL feature of libhadoop.so does not work on the platform without
SM4 support (e.g. Red Hat) while it falls back to Java implementation.
I want HADOOP-17609 be in the next release (or OpenSSL 3 support considering this)::
$ bin/hdfs dfs -put README.txt /zone1/
2024-03-08 15:57:42,685 WARN crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot find AES-CTR/SM4-CTR support, is your version of Openssl new enough?
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
Masatake Iwasaki
On 2024/03/05 7:07, slfan1989 wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support: https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 (binding)
Thanks for the great work, Shilun Fan.
+ verified checksums and signatures of src and site tarballs.
+ built from source tarball with native profile enabled on Rocky Linux 8
and Ubuntu 22.04 (x86_64).
+ deployed pseudo cluster with and without kerberos enabled.
+ ran some example jobs on encryption zone.
+ ran some simple query against httpfs.
+ built site documentation and skimmed the contents.
+ skimmed the contents of pre-built site documentation.
- downstream like HBase can not be built against 3.4.0 without addressing
changes of transitive dependencies.
- OpenSSL feature of libhadoop.so does not work on the platform without
SM4 support (e.g. Red Hat) while it falls back to Java implementation.
I want HADOOP-17609 be in the next release (or OpenSSL 3 support considering this)::
$ bin/hdfs dfs -put README.txt /zone1/
2024-03-08 15:57:42,685 WARN crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot find AES-CTR/SM4-CTR support, is your version of Openssl new enough?
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
Masatake Iwasaki
On 2024/03/05 7:07, slfan1989 wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support: https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1 binding
(sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
earlier. been testing it this afternoon:
pulled the latest version of
https://github.com/apache/hadoop-release-support
(note, this module is commit-then-review; whoever is working on/validating
a release can commit as they go along. This is not production code...)
* went through the "validating a release" step, validating maven artifacts
* building the same downstream modules which built for me last time (avro
too complex; hboss not aws v2 in apache yet)
spark build is still ongoing, but I'm not going to wait. It is building,
which is key.
The core changes I needed in are at the dependency level and I've
verified they are good.
Oh, and I've also got my raspberry p5 doing the download of the arm
stuff for its checknative; not expecting problems.
So: i've got some stuff still ongoing, but the core changes to packaging
are in and the rest I'm not worried about -they shouldn't block the release
as I already validated them on RC2
On Mon, 4 Mar 2024 at 22:08, slfan1989 <sl...@apache.org> wrote:
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>