You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Wei-Chiu Chuang <we...@apache.org> on 2021/06/01 10:29:21 UTC

[VOTE] Release Apache Hadoop 3.3.1 RC3

Hi community,

This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
issues have been resolved [1] again.

There are 2 additional issues resolved for RC3:
* Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
HADOOP-16878
* Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
and destination are the same

There are 4 issues resolved for RC2:
* HADOOP-17666. Update LICENSE for 3.3.1
* MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
* Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
* HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)

The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
fixes compared to hadoop-thirdparty 1.1.0:
* HADOOP-17707. Remove jaeger document from site index.
* HADOOP-17730. Add back error_prone

*RC tag is release-3.3.1-RC3
https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3

*The RC3 artifacts are at*:
https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/

*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1320/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS


Things I've verified:
* all blocker issues targeting 3.3.1 have been resolved.
* stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
* LICENSE and NOTICE files checked
* RELEASENOTES and CHANGELOG
* rat check passed.
* Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
* Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
* Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
dependency divergence. Issues are being identified but so far nothing
blocker for Hadoop itself.

Please try the release and vote. The vote will run for 5 days.

My +1 to start,

[1] https://issues.apache.org/jira/issues/?filter=12350491
[2]
https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 (non-binding)

- verified signature and checksum
- launched a single docker based cluster and ran some simple HDFS commands
- build Spark master with 3.3.1 RC and : 1) run full Spark test suites and
all success; 2) tested simple Spark commands against a S3 endpoint; 3)
tested Spark on YARN with a simple example job.

Thanks Wei-Chiu for the great work!

On Thu, Jun 10, 2021 at 5:55 AM Masatake Iwasaki <
iwasakims@oss.nttdata.co.jp> wrote:

> +1
>
> Thanks for the great work, Wei-Chiu Chuang.
>
> * verified signature and checksum.
> * built site documentation by `mvn site` and skimmed the contents.
>    # found that top-level index.html is not updated.
> * built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>    * launched pseudo cluster with security enabled and ran sample MR jobs.
>    * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
> * built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>
> * built Hive with the patch of HIVE-24484 against hadoop-3.3.1
>    and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).
>
> Masatake Iwasaki
>
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 (non-binding)

- verified signature and checksum
- launched a single docker based cluster and ran some simple HDFS commands
- build Spark master with 3.3.1 RC and : 1) run full Spark test suites and
all success; 2) tested simple Spark commands against a S3 endpoint; 3)
tested Spark on YARN with a simple example job.

Thanks Wei-Chiu for the great work!

On Thu, Jun 10, 2021 at 5:55 AM Masatake Iwasaki <
iwasakims@oss.nttdata.co.jp> wrote:

> +1
>
> Thanks for the great work, Wei-Chiu Chuang.
>
> * verified signature and checksum.
> * built site documentation by `mvn site` and skimmed the contents.
>    # found that top-level index.html is not updated.
> * built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>    * launched pseudo cluster with security enabled and ran sample MR jobs.
>    * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
> * built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>
> * built Hive with the patch of HIVE-24484 against hadoop-3.3.1
>    and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).
>
> Masatake Iwasaki
>
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 (non-binding)

- verified signature and checksum
- launched a single docker based cluster and ran some simple HDFS commands
- build Spark master with 3.3.1 RC and : 1) run full Spark test suites and
all success; 2) tested simple Spark commands against a S3 endpoint; 3)
tested Spark on YARN with a simple example job.

Thanks Wei-Chiu for the great work!

On Thu, Jun 10, 2021 at 5:55 AM Masatake Iwasaki <
iwasakims@oss.nttdata.co.jp> wrote:

> +1
>
> Thanks for the great work, Wei-Chiu Chuang.
>
> * verified signature and checksum.
> * built site documentation by `mvn site` and skimmed the contents.
>    # found that top-level index.html is not updated.
> * built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>    * launched pseudo cluster with security enabled and ran sample MR jobs.
>    * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
> * built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>
> * built Hive with the patch of HIVE-24484 against hadoop-3.3.1
>    and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).
>
> Masatake Iwasaki
>
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 (non-binding)

- verified signature and checksum
- launched a single docker based cluster and ran some simple HDFS commands
- build Spark master with 3.3.1 RC and : 1) run full Spark test suites and
all success; 2) tested simple Spark commands against a S3 endpoint; 3)
tested Spark on YARN with a simple example job.

Thanks Wei-Chiu for the great work!

On Thu, Jun 10, 2021 at 5:55 AM Masatake Iwasaki <
iwasakims@oss.nttdata.co.jp> wrote:

> +1
>
> Thanks for the great work, Wei-Chiu Chuang.
>
> * verified signature and checksum.
> * built site documentation by `mvn site` and skimmed the contents.
>    # found that top-level index.html is not updated.
> * built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>    * launched pseudo cluster with security enabled and ran sample MR jobs.
>    * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
> * built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>
> * built Hive with the patch of HIVE-24484 against hadoop-3.3.1
>    and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).
>
> Masatake Iwasaki
>
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 (non-binding)

- verified signature and checksum
- launched a single docker based cluster and ran some simple HDFS commands
- build Spark master with 3.3.1 RC and : 1) run full Spark test suites and
all success; 2) tested simple Spark commands against a S3 endpoint; 3)
tested Spark on YARN with a simple example job.

Thanks Wei-Chiu for the great work!

On Thu, Jun 10, 2021 at 5:55 AM Masatake Iwasaki <
iwasakims@oss.nttdata.co.jp> wrote:

> +1
>
> Thanks for the great work, Wei-Chiu Chuang.
>
> * verified signature and checksum.
> * built site documentation by `mvn site` and skimmed the contents.
>    # found that top-level index.html is not updated.
> * built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>    * launched pseudo cluster with security enabled and ran sample MR jobs.
>    * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
> * built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests
> -Pnative -Pdist`.
>
> * built Hive with the patch of HIVE-24484 against hadoop-3.3.1
>    and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).
>
> Masatake Iwasaki
>
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
+1

Thanks for the great work, Wei-Chiu Chuang.

* verified signature and checksum.
* built site documentation by `mvn site` and skimmed the contents.
   # found that top-level index.html is not updated.
* built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.
   * launched pseudo cluster with security enabled and ran sample MR jobs.
   * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
* built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.

* built Hive with the patch of HIVE-24484 against hadoop-3.3.1
   and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).

Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Viraj Jasani <vj...@apache.org>.
+1 (non-binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_171): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_171): ok
 - mvn clean install  -DskipTests
* HDFS basic testing in pseudo-distributed mode: ok
* Built HBase 2.4.4 with Hadoop 3.3.1 RC and tested some basic scenarios,
looks good

On Wed, Jun 9, 2021 at 10:55 PM Stack <st...@duboce.net> wrote:

> +1
>
>
>
>         * Signature: ok
>
>         * Checksum : ok
>
>         * Rat check (1.8.0_191): ok
>
>          - mvn clean apache-rat:check
>
>         * Built from source (1.8.0_191): ok
>
>          - mvn clean install -DskipTests
>
>
> Ran a ten node cluster w/ hbase on top running its verification loadings w/
> (gentle) chaos. Had trouble getting the rig running but mostly pilot error
> and none that I could particularly attribute to hdfs after poking in logs.
>
> Messed in UI and shell some. Nothing untoward.
>
> Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
> there (a classic flakie seems more-so on 3.3.1... will dig in more on why).
>
>
> Thanks,
>
> S
>
>
> On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Viraj Jasani <vj...@apache.org>.
+1 (non-binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_171): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_171): ok
 - mvn clean install  -DskipTests
* HDFS basic testing in pseudo-distributed mode: ok
* Built HBase 2.4.4 with Hadoop 3.3.1 RC and tested some basic scenarios,
looks good

On Wed, Jun 9, 2021 at 10:55 PM Stack <st...@duboce.net> wrote:

> +1
>
>
>
>         * Signature: ok
>
>         * Checksum : ok
>
>         * Rat check (1.8.0_191): ok
>
>          - mvn clean apache-rat:check
>
>         * Built from source (1.8.0_191): ok
>
>          - mvn clean install -DskipTests
>
>
> Ran a ten node cluster w/ hbase on top running its verification loadings w/
> (gentle) chaos. Had trouble getting the rig running but mostly pilot error
> and none that I could particularly attribute to hdfs after poking in logs.
>
> Messed in UI and shell some. Nothing untoward.
>
> Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
> there (a classic flakie seems more-so on 3.3.1... will dig in more on why).
>
>
> Thanks,
>
> S
>
>
> On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Viraj Jasani <vj...@apache.org>.
+1 (non-binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_171): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_171): ok
 - mvn clean install  -DskipTests
* HDFS basic testing in pseudo-distributed mode: ok
* Built HBase 2.4.4 with Hadoop 3.3.1 RC and tested some basic scenarios,
looks good

On Wed, Jun 9, 2021 at 10:55 PM Stack <st...@duboce.net> wrote:

> +1
>
>
>
>         * Signature: ok
>
>         * Checksum : ok
>
>         * Rat check (1.8.0_191): ok
>
>          - mvn clean apache-rat:check
>
>         * Built from source (1.8.0_191): ok
>
>          - mvn clean install -DskipTests
>
>
> Ran a ten node cluster w/ hbase on top running its verification loadings w/
> (gentle) chaos. Had trouble getting the rig running but mostly pilot error
> and none that I could particularly attribute to hdfs after poking in logs.
>
> Messed in UI and shell some. Nothing untoward.
>
> Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
> there (a classic flakie seems more-so on 3.3.1... will dig in more on why).
>
>
> Thanks,
>
> S
>
>
> On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Viraj Jasani <vj...@apache.org>.
+1 (non-binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_171): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_171): ok
 - mvn clean install  -DskipTests
* HDFS basic testing in pseudo-distributed mode: ok
* Built HBase 2.4.4 with Hadoop 3.3.1 RC and tested some basic scenarios,
looks good

On Wed, Jun 9, 2021 at 10:55 PM Stack <st...@duboce.net> wrote:

> +1
>
>
>
>         * Signature: ok
>
>         * Checksum : ok
>
>         * Rat check (1.8.0_191): ok
>
>          - mvn clean apache-rat:check
>
>         * Built from source (1.8.0_191): ok
>
>          - mvn clean install -DskipTests
>
>
> Ran a ten node cluster w/ hbase on top running its verification loadings w/
> (gentle) chaos. Had trouble getting the rig running but mostly pilot error
> and none that I could particularly attribute to hdfs after poking in logs.
>
> Messed in UI and shell some. Nothing untoward.
>
> Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
> there (a classic flakie seems more-so on 3.3.1... will dig in more on why).
>
>
> Thanks,
>
> S
>
>
> On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Viraj Jasani <vj...@apache.org>.
+1 (non-binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_171): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_171): ok
 - mvn clean install  -DskipTests
* HDFS basic testing in pseudo-distributed mode: ok
* Built HBase 2.4.4 with Hadoop 3.3.1 RC and tested some basic scenarios,
looks good

On Wed, Jun 9, 2021 at 10:55 PM Stack <st...@duboce.net> wrote:

> +1
>
>
>
>         * Signature: ok
>
>         * Checksum : ok
>
>         * Rat check (1.8.0_191): ok
>
>          - mvn clean apache-rat:check
>
>         * Built from source (1.8.0_191): ok
>
>          - mvn clean install -DskipTests
>
>
> Ran a ten node cluster w/ hbase on top running its verification loadings w/
> (gentle) chaos. Had trouble getting the rig running but mostly pilot error
> and none that I could particularly attribute to hdfs after poking in logs.
>
> Messed in UI and shell some. Nothing untoward.
>
> Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
> there (a classic flakie seems more-so on 3.3.1... will dig in more on why).
>
>
> Thanks,
>
> S
>
>
> On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Stack <st...@duboce.net>.
+1



        * Signature: ok

        * Checksum : ok

        * Rat check (1.8.0_191): ok

         - mvn clean apache-rat:check

        * Built from source (1.8.0_191): ok

         - mvn clean install -DskipTests


Ran a ten node cluster w/ hbase on top running its verification loadings w/
(gentle) chaos. Had trouble getting the rig running but mostly pilot error
and none that I could particularly attribute to hdfs after poking in logs.

Messed in UI and shell some. Nothing untoward.

Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
there (a classic flakie seems more-so on 3.3.1... will dig in more on why).


Thanks,

S


On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
For someone having similar issue.

When I tested the RC3, ResourceManager failed to start due to following exception.::

   2021-06-07 10:50:54,715 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
   java.lang.IllegalAccessError: tried to access class org.bouncycastle.asn1.DEROutputStream from class org.bouncycastle.cert.CertUtils
           at org.bouncycastle.cert.CertUtils.generateSig(Unknown Source)
           at org.bouncycastle.cert.CertUtils.generateFullCert(Unknown Source)
           at org.bouncycastle.cert.X509v3CertificateBuilder.build(Unknown Source)
           at org.apache.hadoop.yarn.server.webproxy.ProxyCA.createCert(ProxyCA.java:180)
           ...

The cause turned out to be my local environment.

When I was working on HADOOP-17609, in order to try SM4 feature,
I installed bcprov-ext-jdk15on-168.jar to $JAVA_HOME/jre/lib/ext
and added the line
"security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider"
to $JAVA_HOME/jre/lib/security/java.security
based on the comment on HDFS-15098 [1] (and fotgot doing so).

Removing the bcprov-ext-jdk15on-168.jar from $JAVA_HOME/jre/lib/ext fixed the issue.

[1] https://issues.apache.org/jira/browse/HDFS-15098?focusedCommentId=17112893&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17112893

Masatake Iwasaki

On 2021/06/07 18:04, Masatake Iwasaki wrote:
> I found that the top page of site documentation (index.html on the top directory)
> has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).
> 
> While I think it is not worth for sinking the RC3,
> you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
> if you need to cut RC4.
> 
> Thanks,
> Masatake Iwasaki
> 
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
>> Hi community,
>>
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>>
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>>
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>>
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>>
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>>
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>>
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>>
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> My +1 to start,
>>
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
For someone having similar issue.

When I tested the RC3, ResourceManager failed to start due to following exception.::

   2021-06-07 10:50:54,715 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
   java.lang.IllegalAccessError: tried to access class org.bouncycastle.asn1.DEROutputStream from class org.bouncycastle.cert.CertUtils
           at org.bouncycastle.cert.CertUtils.generateSig(Unknown Source)
           at org.bouncycastle.cert.CertUtils.generateFullCert(Unknown Source)
           at org.bouncycastle.cert.X509v3CertificateBuilder.build(Unknown Source)
           at org.apache.hadoop.yarn.server.webproxy.ProxyCA.createCert(ProxyCA.java:180)
           ...

The cause turned out to be my local environment.

When I was working on HADOOP-17609, in order to try SM4 feature,
I installed bcprov-ext-jdk15on-168.jar to $JAVA_HOME/jre/lib/ext
and added the line
"security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider"
to $JAVA_HOME/jre/lib/security/java.security
based on the comment on HDFS-15098 [1] (and fotgot doing so).

Removing the bcprov-ext-jdk15on-168.jar from $JAVA_HOME/jre/lib/ext fixed the issue.

[1] https://issues.apache.org/jira/browse/HDFS-15098?focusedCommentId=17112893&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17112893

Masatake Iwasaki

On 2021/06/07 18:04, Masatake Iwasaki wrote:
> I found that the top page of site documentation (index.html on the top directory)
> has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).
> 
> While I think it is not worth for sinking the RC3,
> you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
> if you need to cut RC4.
> 
> Thanks,
> Masatake Iwasaki
> 
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
>> Hi community,
>>
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>>
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>>
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>>
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>>
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>>
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>>
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>>
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> My +1 to start,
>>
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
For someone having similar issue.

When I tested the RC3, ResourceManager failed to start due to following exception.::

   2021-06-07 10:50:54,715 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
   java.lang.IllegalAccessError: tried to access class org.bouncycastle.asn1.DEROutputStream from class org.bouncycastle.cert.CertUtils
           at org.bouncycastle.cert.CertUtils.generateSig(Unknown Source)
           at org.bouncycastle.cert.CertUtils.generateFullCert(Unknown Source)
           at org.bouncycastle.cert.X509v3CertificateBuilder.build(Unknown Source)
           at org.apache.hadoop.yarn.server.webproxy.ProxyCA.createCert(ProxyCA.java:180)
           ...

The cause turned out to be my local environment.

When I was working on HADOOP-17609, in order to try SM4 feature,
I installed bcprov-ext-jdk15on-168.jar to $JAVA_HOME/jre/lib/ext
and added the line
"security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider"
to $JAVA_HOME/jre/lib/security/java.security
based on the comment on HDFS-15098 [1] (and fotgot doing so).

Removing the bcprov-ext-jdk15on-168.jar from $JAVA_HOME/jre/lib/ext fixed the issue.

[1] https://issues.apache.org/jira/browse/HDFS-15098?focusedCommentId=17112893&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17112893

Masatake Iwasaki

On 2021/06/07 18:04, Masatake Iwasaki wrote:
> I found that the top page of site documentation (index.html on the top directory)
> has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).
> 
> While I think it is not worth for sinking the RC3,
> you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
> if you need to cut RC4.
> 
> Thanks,
> Masatake Iwasaki
> 
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
>> Hi community,
>>
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>>
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>>
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>>
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>>
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>>
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>>
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>>
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> My +1 to start,
>>
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
For someone having similar issue.

When I tested the RC3, ResourceManager failed to start due to following exception.::

   2021-06-07 10:50:54,715 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
   java.lang.IllegalAccessError: tried to access class org.bouncycastle.asn1.DEROutputStream from class org.bouncycastle.cert.CertUtils
           at org.bouncycastle.cert.CertUtils.generateSig(Unknown Source)
           at org.bouncycastle.cert.CertUtils.generateFullCert(Unknown Source)
           at org.bouncycastle.cert.X509v3CertificateBuilder.build(Unknown Source)
           at org.apache.hadoop.yarn.server.webproxy.ProxyCA.createCert(ProxyCA.java:180)
           ...

The cause turned out to be my local environment.

When I was working on HADOOP-17609, in order to try SM4 feature,
I installed bcprov-ext-jdk15on-168.jar to $JAVA_HOME/jre/lib/ext
and added the line
"security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider"
to $JAVA_HOME/jre/lib/security/java.security
based on the comment on HDFS-15098 [1] (and fotgot doing so).

Removing the bcprov-ext-jdk15on-168.jar from $JAVA_HOME/jre/lib/ext fixed the issue.

[1] https://issues.apache.org/jira/browse/HDFS-15098?focusedCommentId=17112893&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17112893

Masatake Iwasaki

On 2021/06/07 18:04, Masatake Iwasaki wrote:
> I found that the top page of site documentation (index.html on the top directory)
> has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).
> 
> While I think it is not worth for sinking the RC3,
> you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
> if you need to cut RC4.
> 
> Thanks,
> Masatake Iwasaki
> 
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
>> Hi community,
>>
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>>
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>>
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>>
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>>
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>>
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>>
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>>
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> My +1 to start,
>>
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
For someone having similar issue.

When I tested the RC3, ResourceManager failed to start due to following exception.::

   2021-06-07 10:50:54,715 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
   java.lang.IllegalAccessError: tried to access class org.bouncycastle.asn1.DEROutputStream from class org.bouncycastle.cert.CertUtils
           at org.bouncycastle.cert.CertUtils.generateSig(Unknown Source)
           at org.bouncycastle.cert.CertUtils.generateFullCert(Unknown Source)
           at org.bouncycastle.cert.X509v3CertificateBuilder.build(Unknown Source)
           at org.apache.hadoop.yarn.server.webproxy.ProxyCA.createCert(ProxyCA.java:180)
           ...

The cause turned out to be my local environment.

When I was working on HADOOP-17609, in order to try SM4 feature,
I installed bcprov-ext-jdk15on-168.jar to $JAVA_HOME/jre/lib/ext
and added the line
"security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider"
to $JAVA_HOME/jre/lib/security/java.security
based on the comment on HDFS-15098 [1] (and fotgot doing so).

Removing the bcprov-ext-jdk15on-168.jar from $JAVA_HOME/jre/lib/ext fixed the issue.

[1] https://issues.apache.org/jira/browse/HDFS-15098?focusedCommentId=17112893&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17112893

Masatake Iwasaki

On 2021/06/07 18:04, Masatake Iwasaki wrote:
> I found that the top page of site documentation (index.html on the top directory)
> has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).
> 
> While I think it is not worth for sinking the RC3,
> you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
> if you need to cut RC4.
> 
> Thanks,
> Masatake Iwasaki
> 
> On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
>> Hi community,
>>
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>>
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>>
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>>
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>>
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>>
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>>
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>>
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> My +1 to start,
>>
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
I found that the top page of site documentation (index.html on the top directory)
has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).

While I think it is not worth for sinking the RC3,
you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
if you need to cut RC4.

Thanks,
Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Stack <st...@duboce.net>.
+1



        * Signature: ok

        * Checksum : ok

        * Rat check (1.8.0_191): ok

         - mvn clean apache-rat:check

        * Built from source (1.8.0_191): ok

         - mvn clean install -DskipTests


Ran a ten node cluster w/ hbase on top running its verification loadings w/
(gentle) chaos. Had trouble getting the rig running but mostly pilot error
and none that I could particularly attribute to hdfs after poking in logs.

Messed in UI and shell some. Nothing untoward.

Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
there (a classic flakie seems more-so on 3.3.1... will dig in more on why).


Thanks,

S


On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 on extending the vote. It's great to see that the S3 support in 3.3 is
so much better! So far everything looks good from Spark side (
https://github.com/apache/spark/pull/30135) but I plan to test it more in
the next few days.

Best,
Chao

On Thu, Jun 3, 2021 at 9:56 AM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:

> So I was thinking 5+7 days from Tuesday = next Sunday so that gives
> everyone a whole week to validate.
>
> On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:
>
> > Sounds good to me. That would be until Thursday June 10th, right?
> >
> > As a side note it’s concerning that a double-dot maintenance release is a
> > big release, but I get that it’s the current state of the project.
> >
> > > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > >
> > > Hello,
> > > do we want to extend the release vote? I understand a big release like
> > this
> > > takes time to validate.
> > >
> > > I am aware a number of people are testing it: Attila tested Ozone on
> > Hadoop
> > > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> > 20%
> > > over Hadoop 3.2 library.
> > >
> > > Looks like we may need some more time to test. How about extending it
> by
> > a
> > > week?
> > >
> > > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> > >
> > >> Hi community,
> > >>
> > >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > >> issues have been resolved [1] again.
> > >>
> > >> There are 2 additional issues resolved for RC3:
> > >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > >> HADOOP-16878
> > >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > >> and destination are the same
> > >>
> > >> There are 4 issues resolved for RC2:
> > >> * HADOOP-17666. Update LICENSE for 3.3.1
> > >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >>
> > >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > >> fixes compared to hadoop-thirdparty 1.1.0:
> > >> * HADOOP-17707. Remove jaeger document from site index.
> > >> * HADOOP-17730. Add back error_prone
> > >>
> > >> *RC tag is release-3.3.1-RC3
> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >>
> > >> *The RC3 artifacts are at*:
> > >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >>
> > >> *The maven artifacts are hosted here:*
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >>
> > >> *My public key is available here:*
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >>
> > >> Things I've verified:
> > >> * all blocker issues targeting 3.3.1 have been resolved.
> > >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > >> * LICENSE and NOTICE files checked
> > >> * RELEASENOTES and CHANGELOG
> > >> * rat check passed.
> > >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > >> dependency divergence. Issues are being identified but so far nothing
> > >> blocker for Hadoop itself.
> > >>
> > >> Please try the release and vote. The vote will run for 5 days.
> > >>
> > >> My +1 to start,
> > >>
> > >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> > >> [2]
> > >>
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >>
> > >>
> > >>
> >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 on extending the vote. It's great to see that the S3 support in 3.3 is
so much better! So far everything looks good from Spark side (
https://github.com/apache/spark/pull/30135) but I plan to test it more in
the next few days.

Best,
Chao

On Thu, Jun 3, 2021 at 9:56 AM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:

> So I was thinking 5+7 days from Tuesday = next Sunday so that gives
> everyone a whole week to validate.
>
> On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:
>
> > Sounds good to me. That would be until Thursday June 10th, right?
> >
> > As a side note it’s concerning that a double-dot maintenance release is a
> > big release, but I get that it’s the current state of the project.
> >
> > > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > >
> > > Hello,
> > > do we want to extend the release vote? I understand a big release like
> > this
> > > takes time to validate.
> > >
> > > I am aware a number of people are testing it: Attila tested Ozone on
> > Hadoop
> > > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> > 20%
> > > over Hadoop 3.2 library.
> > >
> > > Looks like we may need some more time to test. How about extending it
> by
> > a
> > > week?
> > >
> > > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> > >
> > >> Hi community,
> > >>
> > >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > >> issues have been resolved [1] again.
> > >>
> > >> There are 2 additional issues resolved for RC3:
> > >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > >> HADOOP-16878
> > >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > >> and destination are the same
> > >>
> > >> There are 4 issues resolved for RC2:
> > >> * HADOOP-17666. Update LICENSE for 3.3.1
> > >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >>
> > >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > >> fixes compared to hadoop-thirdparty 1.1.0:
> > >> * HADOOP-17707. Remove jaeger document from site index.
> > >> * HADOOP-17730. Add back error_prone
> > >>
> > >> *RC tag is release-3.3.1-RC3
> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >>
> > >> *The RC3 artifacts are at*:
> > >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >>
> > >> *The maven artifacts are hosted here:*
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >>
> > >> *My public key is available here:*
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >>
> > >> Things I've verified:
> > >> * all blocker issues targeting 3.3.1 have been resolved.
> > >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > >> * LICENSE and NOTICE files checked
> > >> * RELEASENOTES and CHANGELOG
> > >> * rat check passed.
> > >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > >> dependency divergence. Issues are being identified but so far nothing
> > >> blocker for Hadoop itself.
> > >>
> > >> Please try the release and vote. The vote will run for 5 days.
> > >>
> > >> My +1 to start,
> > >>
> > >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> > >> [2]
> > >>
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >>
> > >>
> > >>
> >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 on extending the vote. It's great to see that the S3 support in 3.3 is
so much better! So far everything looks good from Spark side (
https://github.com/apache/spark/pull/30135) but I plan to test it more in
the next few days.

Best,
Chao

On Thu, Jun 3, 2021 at 9:56 AM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:

> So I was thinking 5+7 days from Tuesday = next Sunday so that gives
> everyone a whole week to validate.
>
> On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:
>
> > Sounds good to me. That would be until Thursday June 10th, right?
> >
> > As a side note it’s concerning that a double-dot maintenance release is a
> > big release, but I get that it’s the current state of the project.
> >
> > > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > >
> > > Hello,
> > > do we want to extend the release vote? I understand a big release like
> > this
> > > takes time to validate.
> > >
> > > I am aware a number of people are testing it: Attila tested Ozone on
> > Hadoop
> > > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> > 20%
> > > over Hadoop 3.2 library.
> > >
> > > Looks like we may need some more time to test. How about extending it
> by
> > a
> > > week?
> > >
> > > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> > >
> > >> Hi community,
> > >>
> > >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > >> issues have been resolved [1] again.
> > >>
> > >> There are 2 additional issues resolved for RC3:
> > >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > >> HADOOP-16878
> > >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > >> and destination are the same
> > >>
> > >> There are 4 issues resolved for RC2:
> > >> * HADOOP-17666. Update LICENSE for 3.3.1
> > >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >>
> > >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > >> fixes compared to hadoop-thirdparty 1.1.0:
> > >> * HADOOP-17707. Remove jaeger document from site index.
> > >> * HADOOP-17730. Add back error_prone
> > >>
> > >> *RC tag is release-3.3.1-RC3
> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >>
> > >> *The RC3 artifacts are at*:
> > >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >>
> > >> *The maven artifacts are hosted here:*
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >>
> > >> *My public key is available here:*
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >>
> > >> Things I've verified:
> > >> * all blocker issues targeting 3.3.1 have been resolved.
> > >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > >> * LICENSE and NOTICE files checked
> > >> * RELEASENOTES and CHANGELOG
> > >> * rat check passed.
> > >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > >> dependency divergence. Issues are being identified but so far nothing
> > >> blocker for Hadoop itself.
> > >>
> > >> Please try the release and vote. The vote will run for 5 days.
> > >>
> > >> My +1 to start,
> > >>
> > >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> > >> [2]
> > >>
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >>
> > >>
> > >>
> >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 on extending the vote. It's great to see that the S3 support in 3.3 is
so much better! So far everything looks good from Spark side (
https://github.com/apache/spark/pull/30135) but I plan to test it more in
the next few days.

Best,
Chao

On Thu, Jun 3, 2021 at 9:56 AM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:

> So I was thinking 5+7 days from Tuesday = next Sunday so that gives
> everyone a whole week to validate.
>
> On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:
>
> > Sounds good to me. That would be until Thursday June 10th, right?
> >
> > As a side note it’s concerning that a double-dot maintenance release is a
> > big release, but I get that it’s the current state of the project.
> >
> > > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > >
> > > Hello,
> > > do we want to extend the release vote? I understand a big release like
> > this
> > > takes time to validate.
> > >
> > > I am aware a number of people are testing it: Attila tested Ozone on
> > Hadoop
> > > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> > 20%
> > > over Hadoop 3.2 library.
> > >
> > > Looks like we may need some more time to test. How about extending it
> by
> > a
> > > week?
> > >
> > > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> > >
> > >> Hi community,
> > >>
> > >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > >> issues have been resolved [1] again.
> > >>
> > >> There are 2 additional issues resolved for RC3:
> > >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > >> HADOOP-16878
> > >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > >> and destination are the same
> > >>
> > >> There are 4 issues resolved for RC2:
> > >> * HADOOP-17666. Update LICENSE for 3.3.1
> > >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >>
> > >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > >> fixes compared to hadoop-thirdparty 1.1.0:
> > >> * HADOOP-17707. Remove jaeger document from site index.
> > >> * HADOOP-17730. Add back error_prone
> > >>
> > >> *RC tag is release-3.3.1-RC3
> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >>
> > >> *The RC3 artifacts are at*:
> > >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >>
> > >> *The maven artifacts are hosted here:*
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >>
> > >> *My public key is available here:*
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >>
> > >> Things I've verified:
> > >> * all blocker issues targeting 3.3.1 have been resolved.
> > >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > >> * LICENSE and NOTICE files checked
> > >> * RELEASENOTES and CHANGELOG
> > >> * rat check passed.
> > >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > >> dependency divergence. Issues are being identified but so far nothing
> > >> blocker for Hadoop itself.
> > >>
> > >> Please try the release and vote. The vote will run for 5 days.
> > >>
> > >> My +1 to start,
> > >>
> > >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> > >> [2]
> > >>
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >>
> > >>
> > >>
> >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
+1 on extending the vote. It's great to see that the S3 support in 3.3 is
so much better! So far everything looks good from Spark side (
https://github.com/apache/spark/pull/30135) but I plan to test it more in
the next few days.

Best,
Chao

On Thu, Jun 3, 2021 at 9:56 AM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:

> So I was thinking 5+7 days from Tuesday = next Sunday so that gives
> everyone a whole week to validate.
>
> On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:
>
> > Sounds good to me. That would be until Thursday June 10th, right?
> >
> > As a side note it’s concerning that a double-dot maintenance release is a
> > big release, but I get that it’s the current state of the project.
> >
> > > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > >
> > > Hello,
> > > do we want to extend the release vote? I understand a big release like
> > this
> > > takes time to validate.
> > >
> > > I am aware a number of people are testing it: Attila tested Ozone on
> > Hadoop
> > > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> > 20%
> > > over Hadoop 3.2 library.
> > >
> > > Looks like we may need some more time to test. How about extending it
> by
> > a
> > > week?
> > >
> > > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> > >
> > >> Hi community,
> > >>
> > >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > >> issues have been resolved [1] again.
> > >>
> > >> There are 2 additional issues resolved for RC3:
> > >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > >> HADOOP-16878
> > >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > >> and destination are the same
> > >>
> > >> There are 4 issues resolved for RC2:
> > >> * HADOOP-17666. Update LICENSE for 3.3.1
> > >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >>
> > >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > >> fixes compared to hadoop-thirdparty 1.1.0:
> > >> * HADOOP-17707. Remove jaeger document from site index.
> > >> * HADOOP-17730. Add back error_prone
> > >>
> > >> *RC tag is release-3.3.1-RC3
> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >>
> > >> *The RC3 artifacts are at*:
> > >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >>
> > >> *The maven artifacts are hosted here:*
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >>
> > >> *My public key is available here:*
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >>
> > >> Things I've verified:
> > >> * all blocker issues targeting 3.3.1 have been resolved.
> > >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > >> * LICENSE and NOTICE files checked
> > >> * RELEASENOTES and CHANGELOG
> > >> * rat check passed.
> > >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > >> dependency divergence. Issues are being identified but so far nothing
> > >> blocker for Hadoop itself.
> > >>
> > >> Please try the release and vote. The vote will run for 5 days.
> > >>
> > >> My +1 to start,
> > >>
> > >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> > >> [2]
> > >>
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >>
> > >>
> > >>
> >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
So I was thinking 5+7 days from Tuesday = next Sunday so that gives
everyone a whole week to validate.

On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:

> Sounds good to me. That would be until Thursday June 10th, right?
>
> As a side note it’s concerning that a double-dot maintenance release is a
> big release, but I get that it’s the current state of the project.
>
> > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> >
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
> > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> >
> >> Hi community,
> >>
> >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> >> issues have been resolved [1] again.
> >>
> >> There are 2 additional issues resolved for RC3:
> >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> >> HADOOP-16878
> >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> >> and destination are the same
> >>
> >> There are 4 issues resolved for RC2:
> >> * HADOOP-17666. Update LICENSE for 3.3.1
> >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >>
> >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> >> fixes compared to hadoop-thirdparty 1.1.0:
> >> * HADOOP-17707. Remove jaeger document from site index.
> >> * HADOOP-17730. Add back error_prone
> >>
> >> *RC tag is release-3.3.1-RC3
> >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >>
> >> *The RC3 artifacts are at*:
> >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >>
> >> *The maven artifacts are hosted here:*
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >>
> >> *My public key is available here:*
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> Things I've verified:
> >> * all blocker issues targeting 3.3.1 have been resolved.
> >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> >> * LICENSE and NOTICE files checked
> >> * RELEASENOTES and CHANGELOG
> >> * rat check passed.
> >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> >> dependency divergence. Issues are being identified but so far nothing
> >> blocker for Hadoop itself.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> My +1 to start,
> >>
> >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> >> [2]
> >>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >>
> >>
> >>
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
So I was thinking 5+7 days from Tuesday = next Sunday so that gives
everyone a whole week to validate.

On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:

> Sounds good to me. That would be until Thursday June 10th, right?
>
> As a side note it’s concerning that a double-dot maintenance release is a
> big release, but I get that it’s the current state of the project.
>
> > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> >
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
> > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> >
> >> Hi community,
> >>
> >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> >> issues have been resolved [1] again.
> >>
> >> There are 2 additional issues resolved for RC3:
> >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> >> HADOOP-16878
> >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> >> and destination are the same
> >>
> >> There are 4 issues resolved for RC2:
> >> * HADOOP-17666. Update LICENSE for 3.3.1
> >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >>
> >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> >> fixes compared to hadoop-thirdparty 1.1.0:
> >> * HADOOP-17707. Remove jaeger document from site index.
> >> * HADOOP-17730. Add back error_prone
> >>
> >> *RC tag is release-3.3.1-RC3
> >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >>
> >> *The RC3 artifacts are at*:
> >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >>
> >> *The maven artifacts are hosted here:*
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >>
> >> *My public key is available here:*
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> Things I've verified:
> >> * all blocker issues targeting 3.3.1 have been resolved.
> >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> >> * LICENSE and NOTICE files checked
> >> * RELEASENOTES and CHANGELOG
> >> * rat check passed.
> >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> >> dependency divergence. Issues are being identified but so far nothing
> >> blocker for Hadoop itself.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> My +1 to start,
> >>
> >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> >> [2]
> >>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >>
> >>
> >>
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
So I was thinking 5+7 days from Tuesday = next Sunday so that gives
everyone a whole week to validate.

On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:

> Sounds good to me. That would be until Thursday June 10th, right?
>
> As a side note it’s concerning that a double-dot maintenance release is a
> big release, but I get that it’s the current state of the project.
>
> > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> >
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
> > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> >
> >> Hi community,
> >>
> >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> >> issues have been resolved [1] again.
> >>
> >> There are 2 additional issues resolved for RC3:
> >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> >> HADOOP-16878
> >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> >> and destination are the same
> >>
> >> There are 4 issues resolved for RC2:
> >> * HADOOP-17666. Update LICENSE for 3.3.1
> >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >>
> >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> >> fixes compared to hadoop-thirdparty 1.1.0:
> >> * HADOOP-17707. Remove jaeger document from site index.
> >> * HADOOP-17730. Add back error_prone
> >>
> >> *RC tag is release-3.3.1-RC3
> >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >>
> >> *The RC3 artifacts are at*:
> >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >>
> >> *The maven artifacts are hosted here:*
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >>
> >> *My public key is available here:*
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> Things I've verified:
> >> * all blocker issues targeting 3.3.1 have been resolved.
> >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> >> * LICENSE and NOTICE files checked
> >> * RELEASENOTES and CHANGELOG
> >> * rat check passed.
> >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> >> dependency divergence. Issues are being identified but so far nothing
> >> blocker for Hadoop itself.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> My +1 to start,
> >>
> >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> >> [2]
> >>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >>
> >>
> >>
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
So I was thinking 5+7 days from Tuesday = next Sunday so that gives
everyone a whole week to validate.

On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:

> Sounds good to me. That would be until Thursday June 10th, right?
>
> As a side note it’s concerning that a double-dot maintenance release is a
> big release, but I get that it’s the current state of the project.
>
> > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> >
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
> > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> >
> >> Hi community,
> >>
> >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> >> issues have been resolved [1] again.
> >>
> >> There are 2 additional issues resolved for RC3:
> >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> >> HADOOP-16878
> >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> >> and destination are the same
> >>
> >> There are 4 issues resolved for RC2:
> >> * HADOOP-17666. Update LICENSE for 3.3.1
> >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >>
> >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> >> fixes compared to hadoop-thirdparty 1.1.0:
> >> * HADOOP-17707. Remove jaeger document from site index.
> >> * HADOOP-17730. Add back error_prone
> >>
> >> *RC tag is release-3.3.1-RC3
> >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >>
> >> *The RC3 artifacts are at*:
> >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >>
> >> *The maven artifacts are hosted here:*
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >>
> >> *My public key is available here:*
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> Things I've verified:
> >> * all blocker issues targeting 3.3.1 have been resolved.
> >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> >> * LICENSE and NOTICE files checked
> >> * RELEASENOTES and CHANGELOG
> >> * rat check passed.
> >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> >> dependency divergence. Issues are being identified but so far nothing
> >> blocker for Hadoop itself.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> My +1 to start,
> >>
> >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> >> [2]
> >>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >>
> >>
> >>
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
So I was thinking 5+7 days from Tuesday = next Sunday so that gives
everyone a whole week to validate.

On Fri, Jun 4, 2021 at 12:38 AM Sean Busbey <sb...@apple.com> wrote:

> Sounds good to me. That would be until Thursday June 10th, right?
>
> As a side note it’s concerning that a double-dot maintenance release is a
> big release, but I get that it’s the current state of the project.
>
> > On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> >
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
> > On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> >
> >> Hi community,
> >>
> >> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> >> issues have been resolved [1] again.
> >>
> >> There are 2 additional issues resolved for RC3:
> >> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> >> HADOOP-16878
> >> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> >> and destination are the same
> >>
> >> There are 4 issues resolved for RC2:
> >> * HADOOP-17666. Update LICENSE for 3.3.1
> >> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> >> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> >> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >>
> >> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> >> fixes compared to hadoop-thirdparty 1.1.0:
> >> * HADOOP-17707. Remove jaeger document from site index.
> >> * HADOOP-17730. Add back error_prone
> >>
> >> *RC tag is release-3.3.1-RC3
> >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >>
> >> *The RC3 artifacts are at*:
> >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >>
> >> *The maven artifacts are hosted here:*
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >>
> >> *My public key is available here:*
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >>
> >> Things I've verified:
> >> * all blocker issues targeting 3.3.1 have been resolved.
> >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> >> * LICENSE and NOTICE files checked
> >> * RELEASENOTES and CHANGELOG
> >> * rat check passed.
> >> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> >> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> >> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> >> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> >> dependency divergence. Issues are being identified but so far nothing
> >> blocker for Hadoop itself.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> My +1 to start,
> >>
> >> [1] https://issues.apache.org/jira/issues/?filter=12350491
> >> [2]
> >>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >>
> >>
> >>
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Sean Busbey <sb...@apple.com.INVALID>.
Sounds good to me. That would be until Thursday June 10th, right?

As a side note it’s concerning that a double-dot maintenance release is a big release, but I get that it’s the current state of the project.

> On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> 
> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
> 
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
> 
> Looks like we may need some more time to test. How about extending it by a
> week?
> 
> On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:
> 
>> Hi community,
>> 
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>> 
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>> 
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> 
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>> 
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> 
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> 
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> 
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> 
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> My +1 to start,
>> 
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> 
>> 
>> 



---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Dropped them. Please check again.

On Fri, Jun 4, 2021 at 1:19 PM Chao Sun <su...@apache.org> wrote:

> Hi Wei-Chiu,
>
> It seems the Maven staging repository is still pointing at RC0:
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
> and it's probably because you need to drop the old RCs in Apache Nexus
> server. Could you do that? Thanks.
>
> Chao
>
> On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> Extend for a bit from the last RC, as it takes time to qualify.
>>
>> I'm busy testing, doing
>>
>> - packaging &c
>> - CLI working with abfs and s3, both fs and cloudstore library calls
>> - building downstream projects (so validating maven artifacts). cloudstore
>> and spark there
>> - building downstream of downstream projects, i.e. my spark cloud
>> IO/committer test module. Moving to spark 3 cost me the afternoon, not
>> through any incompatible changes there but because the upgraded scalatest
>> "moved" their foundational FunTest class to a different package and name.
>> Not happy with Team Scalatest there.
>> - reviewing the docs in the -aws and azure modules to see they link
>> together OK.
>>
>> So far so good.
>>
>> One troublespot (which isn't any reason to hold up the release), is that
>> the table in the directory_markers markdown file doesn't render right.
>>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>>
>> This is *not a blocker*
>>
>> I can prepare a fix and we can have it in so that if any other changes
>> come
>> in the page will look OK.
>>
>>
>>
>>
>> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>>
>> > Hello,
>> > do we want to extend the release vote? I understand a big release like
>> this
>> > takes time to validate.
>> >
>> > I am aware a number of people are testing it: Attila tested Ozone on
>> Hadoop
>> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
>> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
>> 20%
>> > over Hadoop 3.2 library.
>> >
>>
>> ooh. That'll be from Mukund's listing improvements translating into query
>> planning speedups,
>>
>> Nice
>>
>> If someone benchmarking this stuff were to enable directory marker
>> retention
>> fs.s3a.directory.marker.retention=keep , I'd be interested to know how
>> much
>> speedup that
>> delivers on versioned and unversioned buckets.
>>
>> Unversioned: reduces risk of IO throttling on writes
>> Versioned: that and should stop subsequent LIST operations from getting
>> slowed down from all the tombstones
>>
>>
>> >
>> > Looks like we may need some more time to test. How about extending it
>> by a
>> > week?
>> >
>>
>>
>> That would be good. This week included some holidays for people in the
>> US/UK which is why I'm a bit behind on my testing.
>>
>>
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Dropped them. Please check again.

On Fri, Jun 4, 2021 at 1:19 PM Chao Sun <su...@apache.org> wrote:

> Hi Wei-Chiu,
>
> It seems the Maven staging repository is still pointing at RC0:
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
> and it's probably because you need to drop the old RCs in Apache Nexus
> server. Could you do that? Thanks.
>
> Chao
>
> On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> Extend for a bit from the last RC, as it takes time to qualify.
>>
>> I'm busy testing, doing
>>
>> - packaging &c
>> - CLI working with abfs and s3, both fs and cloudstore library calls
>> - building downstream projects (so validating maven artifacts). cloudstore
>> and spark there
>> - building downstream of downstream projects, i.e. my spark cloud
>> IO/committer test module. Moving to spark 3 cost me the afternoon, not
>> through any incompatible changes there but because the upgraded scalatest
>> "moved" their foundational FunTest class to a different package and name.
>> Not happy with Team Scalatest there.
>> - reviewing the docs in the -aws and azure modules to see they link
>> together OK.
>>
>> So far so good.
>>
>> One troublespot (which isn't any reason to hold up the release), is that
>> the table in the directory_markers markdown file doesn't render right.
>>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>>
>> This is *not a blocker*
>>
>> I can prepare a fix and we can have it in so that if any other changes
>> come
>> in the page will look OK.
>>
>>
>>
>>
>> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>>
>> > Hello,
>> > do we want to extend the release vote? I understand a big release like
>> this
>> > takes time to validate.
>> >
>> > I am aware a number of people are testing it: Attila tested Ozone on
>> Hadoop
>> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
>> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
>> 20%
>> > over Hadoop 3.2 library.
>> >
>>
>> ooh. That'll be from Mukund's listing improvements translating into query
>> planning speedups,
>>
>> Nice
>>
>> If someone benchmarking this stuff were to enable directory marker
>> retention
>> fs.s3a.directory.marker.retention=keep , I'd be interested to know how
>> much
>> speedup that
>> delivers on versioned and unversioned buckets.
>>
>> Unversioned: reduces risk of IO throttling on writes
>> Versioned: that and should stop subsequent LIST operations from getting
>> slowed down from all the tombstones
>>
>>
>> >
>> > Looks like we may need some more time to test. How about extending it
>> by a
>> > week?
>> >
>>
>>
>> That would be good. This week included some holidays for people in the
>> US/UK which is why I'm a bit behind on my testing.
>>
>>
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Dropped them. Please check again.

On Fri, Jun 4, 2021 at 1:19 PM Chao Sun <su...@apache.org> wrote:

> Hi Wei-Chiu,
>
> It seems the Maven staging repository is still pointing at RC0:
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
> and it's probably because you need to drop the old RCs in Apache Nexus
> server. Could you do that? Thanks.
>
> Chao
>
> On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> Extend for a bit from the last RC, as it takes time to qualify.
>>
>> I'm busy testing, doing
>>
>> - packaging &c
>> - CLI working with abfs and s3, both fs and cloudstore library calls
>> - building downstream projects (so validating maven artifacts). cloudstore
>> and spark there
>> - building downstream of downstream projects, i.e. my spark cloud
>> IO/committer test module. Moving to spark 3 cost me the afternoon, not
>> through any incompatible changes there but because the upgraded scalatest
>> "moved" their foundational FunTest class to a different package and name.
>> Not happy with Team Scalatest there.
>> - reviewing the docs in the -aws and azure modules to see they link
>> together OK.
>>
>> So far so good.
>>
>> One troublespot (which isn't any reason to hold up the release), is that
>> the table in the directory_markers markdown file doesn't render right.
>>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>>
>> This is *not a blocker*
>>
>> I can prepare a fix and we can have it in so that if any other changes
>> come
>> in the page will look OK.
>>
>>
>>
>>
>> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>>
>> > Hello,
>> > do we want to extend the release vote? I understand a big release like
>> this
>> > takes time to validate.
>> >
>> > I am aware a number of people are testing it: Attila tested Ozone on
>> Hadoop
>> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
>> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
>> 20%
>> > over Hadoop 3.2 library.
>> >
>>
>> ooh. That'll be from Mukund's listing improvements translating into query
>> planning speedups,
>>
>> Nice
>>
>> If someone benchmarking this stuff were to enable directory marker
>> retention
>> fs.s3a.directory.marker.retention=keep , I'd be interested to know how
>> much
>> speedup that
>> delivers on versioned and unversioned buckets.
>>
>> Unversioned: reduces risk of IO throttling on writes
>> Versioned: that and should stop subsequent LIST operations from getting
>> slowed down from all the tombstones
>>
>>
>> >
>> > Looks like we may need some more time to test. How about extending it
>> by a
>> > week?
>> >
>>
>>
>> That would be good. This week included some holidays for people in the
>> US/UK which is why I'm a bit behind on my testing.
>>
>>
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Dropped them. Please check again.

On Fri, Jun 4, 2021 at 1:19 PM Chao Sun <su...@apache.org> wrote:

> Hi Wei-Chiu,
>
> It seems the Maven staging repository is still pointing at RC0:
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
> and it's probably because you need to drop the old RCs in Apache Nexus
> server. Could you do that? Thanks.
>
> Chao
>
> On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> Extend for a bit from the last RC, as it takes time to qualify.
>>
>> I'm busy testing, doing
>>
>> - packaging &c
>> - CLI working with abfs and s3, both fs and cloudstore library calls
>> - building downstream projects (so validating maven artifacts). cloudstore
>> and spark there
>> - building downstream of downstream projects, i.e. my spark cloud
>> IO/committer test module. Moving to spark 3 cost me the afternoon, not
>> through any incompatible changes there but because the upgraded scalatest
>> "moved" their foundational FunTest class to a different package and name.
>> Not happy with Team Scalatest there.
>> - reviewing the docs in the -aws and azure modules to see they link
>> together OK.
>>
>> So far so good.
>>
>> One troublespot (which isn't any reason to hold up the release), is that
>> the table in the directory_markers markdown file doesn't render right.
>>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>>
>> This is *not a blocker*
>>
>> I can prepare a fix and we can have it in so that if any other changes
>> come
>> in the page will look OK.
>>
>>
>>
>>
>> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>>
>> > Hello,
>> > do we want to extend the release vote? I understand a big release like
>> this
>> > takes time to validate.
>> >
>> > I am aware a number of people are testing it: Attila tested Ozone on
>> Hadoop
>> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
>> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
>> 20%
>> > over Hadoop 3.2 library.
>> >
>>
>> ooh. That'll be from Mukund's listing improvements translating into query
>> planning speedups,
>>
>> Nice
>>
>> If someone benchmarking this stuff were to enable directory marker
>> retention
>> fs.s3a.directory.marker.retention=keep , I'd be interested to know how
>> much
>> speedup that
>> delivers on versioned and unversioned buckets.
>>
>> Unversioned: reduces risk of IO throttling on writes
>> Versioned: that and should stop subsequent LIST operations from getting
>> slowed down from all the tombstones
>>
>>
>> >
>> > Looks like we may need some more time to test. How about extending it
>> by a
>> > week?
>> >
>>
>>
>> That would be good. This week included some holidays for people in the
>> US/UK which is why I'm a bit behind on my testing.
>>
>>
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Dropped them. Please check again.

On Fri, Jun 4, 2021 at 1:19 PM Chao Sun <su...@apache.org> wrote:

> Hi Wei-Chiu,
>
> It seems the Maven staging repository is still pointing at RC0:
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
> and it's probably because you need to drop the old RCs in Apache Nexus
> server. Could you do that? Thanks.
>
> Chao
>
> On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> Extend for a bit from the last RC, as it takes time to qualify.
>>
>> I'm busy testing, doing
>>
>> - packaging &c
>> - CLI working with abfs and s3, both fs and cloudstore library calls
>> - building downstream projects (so validating maven artifacts). cloudstore
>> and spark there
>> - building downstream of downstream projects, i.e. my spark cloud
>> IO/committer test module. Moving to spark 3 cost me the afternoon, not
>> through any incompatible changes there but because the upgraded scalatest
>> "moved" their foundational FunTest class to a different package and name.
>> Not happy with Team Scalatest there.
>> - reviewing the docs in the -aws and azure modules to see they link
>> together OK.
>>
>> So far so good.
>>
>> One troublespot (which isn't any reason to hold up the release), is that
>> the table in the directory_markers markdown file doesn't render right.
>>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>>
>> This is *not a blocker*
>>
>> I can prepare a fix and we can have it in so that if any other changes
>> come
>> in the page will look OK.
>>
>>
>>
>>
>> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>>
>> > Hello,
>> > do we want to extend the release vote? I understand a big release like
>> this
>> > takes time to validate.
>> >
>> > I am aware a number of people are testing it: Attila tested Ozone on
>> Hadoop
>> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
>> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
>> 20%
>> > over Hadoop 3.2 library.
>> >
>>
>> ooh. That'll be from Mukund's listing improvements translating into query
>> planning speedups,
>>
>> Nice
>>
>> If someone benchmarking this stuff were to enable directory marker
>> retention
>> fs.s3a.directory.marker.retention=keep , I'd be interested to know how
>> much
>> speedup that
>> delivers on versioned and unversioned buckets.
>>
>> Unversioned: reduces risk of IO throttling on writes
>> Versioned: that and should stop subsequent LIST operations from getting
>> slowed down from all the tombstones
>>
>>
>> >
>> > Looks like we may need some more time to test. How about extending it
>> by a
>> > week?
>> >
>>
>>
>> That would be good. This week included some holidays for people in the
>> US/UK which is why I'm a bit behind on my testing.
>>
>>
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
Hi Wei-Chiu,

It seems the Maven staging repository is still pointing at RC0:
https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
and it's probably because you need to drop the old RCs in Apache Nexus
server. Could you do that? Thanks.

Chao

On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> Extend for a bit from the last RC, as it takes time to qualify.
>
> I'm busy testing, doing
>
> - packaging &c
> - CLI working with abfs and s3, both fs and cloudstore library calls
> - building downstream projects (so validating maven artifacts). cloudstore
> and spark there
> - building downstream of downstream projects, i.e. my spark cloud
> IO/committer test module. Moving to spark 3 cost me the afternoon, not
> through any incompatible changes there but because the upgraded scalatest
> "moved" their foundational FunTest class to a different package and name.
> Not happy with Team Scalatest there.
> - reviewing the docs in the -aws and azure modules to see they link
> together OK.
>
> So far so good.
>
> One troublespot (which isn't any reason to hold up the release), is that
> the table in the directory_markers markdown file doesn't render right.
>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>
> This is *not a blocker*
>
> I can prepare a fix and we can have it in so that if any other changes come
> in the page will look OK.
>
>
>
>
> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
>
> ooh. That'll be from Mukund's listing improvements translating into query
> planning speedups,
>
> Nice
>
> If someone benchmarking this stuff were to enable directory marker
> retention
> fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
> speedup that
> delivers on versioned and unversioned buckets.
>
> Unversioned: reduces risk of IO throttling on writes
> Versioned: that and should stop subsequent LIST operations from getting
> slowed down from all the tombstones
>
>
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
>
>
> That would be good. This week included some holidays for people in the
> US/UK which is why I'm a bit behind on my testing.
>
>
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
Hi Wei-Chiu,

It seems the Maven staging repository is still pointing at RC0:
https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
and it's probably because you need to drop the old RCs in Apache Nexus
server. Could you do that? Thanks.

Chao

On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> Extend for a bit from the last RC, as it takes time to qualify.
>
> I'm busy testing, doing
>
> - packaging &c
> - CLI working with abfs and s3, both fs and cloudstore library calls
> - building downstream projects (so validating maven artifacts). cloudstore
> and spark there
> - building downstream of downstream projects, i.e. my spark cloud
> IO/committer test module. Moving to spark 3 cost me the afternoon, not
> through any incompatible changes there but because the upgraded scalatest
> "moved" their foundational FunTest class to a different package and name.
> Not happy with Team Scalatest there.
> - reviewing the docs in the -aws and azure modules to see they link
> together OK.
>
> So far so good.
>
> One troublespot (which isn't any reason to hold up the release), is that
> the table in the directory_markers markdown file doesn't render right.
>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>
> This is *not a blocker*
>
> I can prepare a fix and we can have it in so that if any other changes come
> in the page will look OK.
>
>
>
>
> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
>
> ooh. That'll be from Mukund's listing improvements translating into query
> planning speedups,
>
> Nice
>
> If someone benchmarking this stuff were to enable directory marker
> retention
> fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
> speedup that
> delivers on versioned and unversioned buckets.
>
> Unversioned: reduces risk of IO throttling on writes
> Versioned: that and should stop subsequent LIST operations from getting
> slowed down from all the tombstones
>
>
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
>
>
> That would be good. This week included some holidays for people in the
> US/UK which is why I'm a bit behind on my testing.
>
>
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
Hi Wei-Chiu,

It seems the Maven staging repository is still pointing at RC0:
https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
and it's probably because you need to drop the old RCs in Apache Nexus
server. Could you do that? Thanks.

Chao

On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> Extend for a bit from the last RC, as it takes time to qualify.
>
> I'm busy testing, doing
>
> - packaging &c
> - CLI working with abfs and s3, both fs and cloudstore library calls
> - building downstream projects (so validating maven artifacts). cloudstore
> and spark there
> - building downstream of downstream projects, i.e. my spark cloud
> IO/committer test module. Moving to spark 3 cost me the afternoon, not
> through any incompatible changes there but because the upgraded scalatest
> "moved" their foundational FunTest class to a different package and name.
> Not happy with Team Scalatest there.
> - reviewing the docs in the -aws and azure modules to see they link
> together OK.
>
> So far so good.
>
> One troublespot (which isn't any reason to hold up the release), is that
> the table in the directory_markers markdown file doesn't render right.
>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>
> This is *not a blocker*
>
> I can prepare a fix and we can have it in so that if any other changes come
> in the page will look OK.
>
>
>
>
> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
>
> ooh. That'll be from Mukund's listing improvements translating into query
> planning speedups,
>
> Nice
>
> If someone benchmarking this stuff were to enable directory marker
> retention
> fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
> speedup that
> delivers on versioned and unversioned buckets.
>
> Unversioned: reduces risk of IO throttling on writes
> Versioned: that and should stop subsequent LIST operations from getting
> slowed down from all the tombstones
>
>
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
>
>
> That would be good. This week included some holidays for people in the
> US/UK which is why I'm a bit behind on my testing.
>
>
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
Hi Wei-Chiu,

It seems the Maven staging repository is still pointing at RC0:
https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
and it's probably because you need to drop the old RCs in Apache Nexus
server. Could you do that? Thanks.

Chao

On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> Extend for a bit from the last RC, as it takes time to qualify.
>
> I'm busy testing, doing
>
> - packaging &c
> - CLI working with abfs and s3, both fs and cloudstore library calls
> - building downstream projects (so validating maven artifacts). cloudstore
> and spark there
> - building downstream of downstream projects, i.e. my spark cloud
> IO/committer test module. Moving to spark 3 cost me the afternoon, not
> through any incompatible changes there but because the upgraded scalatest
> "moved" their foundational FunTest class to a different package and name.
> Not happy with Team Scalatest there.
> - reviewing the docs in the -aws and azure modules to see they link
> together OK.
>
> So far so good.
>
> One troublespot (which isn't any reason to hold up the release), is that
> the table in the directory_markers markdown file doesn't render right.
>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>
> This is *not a blocker*
>
> I can prepare a fix and we can have it in so that if any other changes come
> in the page will look OK.
>
>
>
>
> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
>
> ooh. That'll be from Mukund's listing improvements translating into query
> planning speedups,
>
> Nice
>
> If someone benchmarking this stuff were to enable directory marker
> retention
> fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
> speedup that
> delivers on versioned and unversioned buckets.
>
> Unversioned: reduces risk of IO throttling on writes
> Versioned: that and should stop subsequent LIST operations from getting
> slowed down from all the tombstones
>
>
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
>
>
> That would be good. This week included some holidays for people in the
> US/UK which is why I'm a bit behind on my testing.
>
>
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Chao Sun <su...@apache.org>.
Hi Wei-Chiu,

It seems the Maven staging repository is still pointing at RC0:
https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/,
and it's probably because you need to drop the old RCs in Apache Nexus
server. Could you do that? Thanks.

Chao

On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> Extend for a bit from the last RC, as it takes time to qualify.
>
> I'm busy testing, doing
>
> - packaging &c
> - CLI working with abfs and s3, both fs and cloudstore library calls
> - building downstream projects (so validating maven artifacts). cloudstore
> and spark there
> - building downstream of downstream projects, i.e. my spark cloud
> IO/committer test module. Moving to spark 3 cost me the afternoon, not
> through any incompatible changes there but because the upgraded scalatest
> "moved" their foundational FunTest class to a different package and name.
> Not happy with Team Scalatest there.
> - reviewing the docs in the -aws and azure modules to see they link
> together OK.
>
> So far so good.
>
> One troublespot (which isn't any reason to hold up the release), is that
> the table in the directory_markers markdown file doesn't render right.
>  Created https://issues.apache.org/jira/browse/HADOOP-17746.
>
> This is *not a blocker*
>
> I can prepare a fix and we can have it in so that if any other changes come
> in the page will look OK.
>
>
>
>
> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:
>
> > Hello,
> > do we want to extend the release vote? I understand a big release like
> this
> > takes time to validate.
> >
> > I am aware a number of people are testing it: Attila tested Ozone on
> Hadoop
> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by
> 20%
> > over Hadoop 3.2 library.
> >
>
> ooh. That'll be from Mukund's listing improvements translating into query
> planning speedups,
>
> Nice
>
> If someone benchmarking this stuff were to enable directory marker
> retention
> fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
> speedup that
> delivers on versioned and unversioned buckets.
>
> Unversioned: reduces risk of IO throttling on writes
> Versioned: that and should stop subsequent LIST operations from getting
> slowed down from all the tombstones
>
>
> >
> > Looks like we may need some more time to test. How about extending it by
> a
> > week?
> >
>
>
> That would be good. This week included some holidays for people in the
> US/UK which is why I'm a bit behind on my testing.
>
>
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Extend for a bit from the last RC, as it takes time to qualify.

I'm busy testing, doing

- packaging &c
- CLI working with abfs and s3, both fs and cloudstore library calls
- building downstream projects (so validating maven artifacts). cloudstore
and spark there
- building downstream of downstream projects, i.e. my spark cloud
IO/committer test module. Moving to spark 3 cost me the afternoon, not
through any incompatible changes there but because the upgraded scalatest
"moved" their foundational FunTest class to a different package and name.
Not happy with Team Scalatest there.
- reviewing the docs in the -aws and azure modules to see they link
together OK.

So far so good.

One troublespot (which isn't any reason to hold up the release), is that
the table in the directory_markers markdown file doesn't render right.
 Created https://issues.apache.org/jira/browse/HADOOP-17746.

This is *not a blocker*

I can prepare a fix and we can have it in so that if any other changes come
in the page will look OK.




On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:

> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
>
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
>

ooh. That'll be from Mukund's listing improvements translating into query
planning speedups,

Nice

If someone benchmarking this stuff were to enable directory marker retention
fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
speedup that
delivers on versioned and unversioned buckets.

Unversioned: reduces risk of IO throttling on writes
Versioned: that and should stop subsequent LIST operations from getting
slowed down from all the tombstones


>
> Looks like we may need some more time to test. How about extending it by a
> week?
>


That would be good. This week included some holidays for people in the
US/UK which is why I'm a bit behind on my testing.


>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Extend for a bit from the last RC, as it takes time to qualify.

I'm busy testing, doing

- packaging &c
- CLI working with abfs and s3, both fs and cloudstore library calls
- building downstream projects (so validating maven artifacts). cloudstore
and spark there
- building downstream of downstream projects, i.e. my spark cloud
IO/committer test module. Moving to spark 3 cost me the afternoon, not
through any incompatible changes there but because the upgraded scalatest
"moved" their foundational FunTest class to a different package and name.
Not happy with Team Scalatest there.
- reviewing the docs in the -aws and azure modules to see they link
together OK.

So far so good.

One troublespot (which isn't any reason to hold up the release), is that
the table in the directory_markers markdown file doesn't render right.
 Created https://issues.apache.org/jira/browse/HADOOP-17746.

This is *not a blocker*

I can prepare a fix and we can have it in so that if any other changes come
in the page will look OK.




On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:

> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
>
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
>

ooh. That'll be from Mukund's listing improvements translating into query
planning speedups,

Nice

If someone benchmarking this stuff were to enable directory marker retention
fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
speedup that
delivers on versioned and unversioned buckets.

Unversioned: reduces risk of IO throttling on writes
Versioned: that and should stop subsequent LIST operations from getting
slowed down from all the tombstones


>
> Looks like we may need some more time to test. How about extending it by a
> week?
>


That would be good. This week included some holidays for people in the
US/UK which is why I'm a bit behind on my testing.


>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Sean Busbey <sb...@apple.com.INVALID>.
Sounds good to me. That would be until Thursday June 10th, right?

As a side note it’s concerning that a double-dot maintenance release is a big release, but I get that it’s the current state of the project.

> On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> 
> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
> 
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
> 
> Looks like we may need some more time to test. How about extending it by a
> week?
> 
> On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:
> 
>> Hi community,
>> 
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>> 
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>> 
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> 
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>> 
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> 
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> 
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> 
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> 
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> My +1 to start,
>> 
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> 
>> 
>> 



---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Extend for a bit from the last RC, as it takes time to qualify.

I'm busy testing, doing

- packaging &c
- CLI working with abfs and s3, both fs and cloudstore library calls
- building downstream projects (so validating maven artifacts). cloudstore
and spark there
- building downstream of downstream projects, i.e. my spark cloud
IO/committer test module. Moving to spark 3 cost me the afternoon, not
through any incompatible changes there but because the upgraded scalatest
"moved" their foundational FunTest class to a different package and name.
Not happy with Team Scalatest there.
- reviewing the docs in the -aws and azure modules to see they link
together OK.

So far so good.

One troublespot (which isn't any reason to hold up the release), is that
the table in the directory_markers markdown file doesn't render right.
 Created https://issues.apache.org/jira/browse/HADOOP-17746.

This is *not a blocker*

I can prepare a fix and we can have it in so that if any other changes come
in the page will look OK.




On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:

> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
>
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
>

ooh. That'll be from Mukund's listing improvements translating into query
planning speedups,

Nice

If someone benchmarking this stuff were to enable directory marker retention
fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
speedup that
delivers on versioned and unversioned buckets.

Unversioned: reduces risk of IO throttling on writes
Versioned: that and should stop subsequent LIST operations from getting
slowed down from all the tombstones


>
> Looks like we may need some more time to test. How about extending it by a
> week?
>


That would be good. This week included some holidays for people in the
US/UK which is why I'm a bit behind on my testing.


>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Sean Busbey <sb...@apple.com.INVALID>.
Sounds good to me. That would be until Thursday June 10th, right?

As a side note it’s concerning that a double-dot maintenance release is a big release, but I get that it’s the current state of the project.

> On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> 
> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
> 
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
> 
> Looks like we may need some more time to test. How about extending it by a
> week?
> 
> On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:
> 
>> Hi community,
>> 
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>> 
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>> 
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> 
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>> 
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> 
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> 
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> 
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> 
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> My +1 to start,
>> 
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> 
>> 
>> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Extend for a bit from the last RC, as it takes time to qualify.

I'm busy testing, doing

- packaging &c
- CLI working with abfs and s3, both fs and cloudstore library calls
- building downstream projects (so validating maven artifacts). cloudstore
and spark there
- building downstream of downstream projects, i.e. my spark cloud
IO/committer test module. Moving to spark 3 cost me the afternoon, not
through any incompatible changes there but because the upgraded scalatest
"moved" their foundational FunTest class to a different package and name.
Not happy with Team Scalatest there.
- reviewing the docs in the -aws and azure modules to see they link
together OK.

So far so good.

One troublespot (which isn't any reason to hold up the release), is that
the table in the directory_markers markdown file doesn't render right.
 Created https://issues.apache.org/jira/browse/HADOOP-17746.

This is *not a blocker*

I can prepare a fix and we can have it in so that if any other changes come
in the page will look OK.




On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:

> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
>
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
>

ooh. That'll be from Mukund's listing improvements translating into query
planning speedups,

Nice

If someone benchmarking this stuff were to enable directory marker retention
fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
speedup that
delivers on versioned and unversioned buckets.

Unversioned: reduces risk of IO throttling on writes
Versioned: that and should stop subsequent LIST operations from getting
slowed down from all the tombstones


>
> Looks like we may need some more time to test. How about extending it by a
> week?
>


That would be good. This week included some holidays for people in the
US/UK which is why I'm a bit behind on my testing.


>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Extend for a bit from the last RC, as it takes time to qualify.

I'm busy testing, doing

- packaging &c
- CLI working with abfs and s3, both fs and cloudstore library calls
- building downstream projects (so validating maven artifacts). cloudstore
and spark there
- building downstream of downstream projects, i.e. my spark cloud
IO/committer test module. Moving to spark 3 cost me the afternoon, not
through any incompatible changes there but because the upgraded scalatest
"moved" their foundational FunTest class to a different package and name.
Not happy with Team Scalatest there.
- reviewing the docs in the -aws and azure modules to see they link
together OK.

So far so good.

One troublespot (which isn't any reason to hold up the release), is that
the table in the directory_markers markdown file doesn't render right.
 Created https://issues.apache.org/jira/browse/HADOOP-17746.

This is *not a blocker*

I can prepare a fix and we can have it in so that if any other changes come
in the page will look OK.




On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang <we...@apache.org> wrote:

> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
>
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
>

ooh. That'll be from Mukund's listing improvements translating into query
planning speedups,

Nice

If someone benchmarking this stuff were to enable directory marker retention
fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
speedup that
delivers on versioned and unversioned buckets.

Unversioned: reduces risk of IO throttling on writes
Versioned: that and should stop subsequent LIST operations from getting
slowed down from all the tombstones


>
> Looks like we may need some more time to test. How about extending it by a
> week?
>


That would be good. This week included some holidays for people in the
US/UK which is why I'm a bit behind on my testing.


>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Sean Busbey <sb...@apple.com.INVALID>.
Sounds good to me. That would be until Thursday June 10th, right?

As a side note it’s concerning that a double-dot maintenance release is a big release, but I get that it’s the current state of the project.

> On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> 
> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
> 
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
> 
> Looks like we may need some more time to test. How about extending it by a
> week?
> 
> On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:
> 
>> Hi community,
>> 
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>> 
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>> 
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> 
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>> 
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> 
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> 
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> 
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> 
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> My +1 to start,
>> 
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> 
>> 
>> 



---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Sean Busbey <sb...@apple.com.INVALID>.
Sounds good to me. That would be until Thursday June 10th, right?

As a side note it’s concerning that a double-dot maintenance release is a big release, but I get that it’s the current state of the project.

> On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
> 
> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
> 
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
> 
> Looks like we may need some more time to test. How about extending it by a
> week?
> 
> On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:
> 
>> Hi community,
>> 
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>> 
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>> 
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> 
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>> 
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> 
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> 
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> 
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> 
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> My +1 to start,
>> 
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> 
>> 
>> 



---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@apache.org>.
Hello,
do we want to extend the release vote? I understand a big release like this
takes time to validate.

I am aware a number of people are testing it: Attila tested Ozone on Hadoop
3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
over Hadoop 3.2 library.

Looks like we may need some more time to test. How about extending it by a
week?

On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
+1

Thanks for the great work, Wei-Chiu Chuang.

* verified signature and checksum.
* built site documentation by `mvn site` and skimmed the contents.
   # found that top-level index.html is not updated.
* built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.
   * launched pseudo cluster with security enabled and ran sample MR jobs.
   * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
* built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.

* built Hive with the patch of HIVE-24484 against hadoop-3.3.1
   and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).

Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
I found that the top page of site documentation (index.html on the top directory)
has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).

While I think it is not worth for sinking the RC3,
you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
if you need to cut RC4.

Thanks,
Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Dongjoon Hyun <do...@apache.org>.
+1 (non-binding)

Bests,
Dongjoon.

On 2021/06/08 12:23:43, Steve Loughran <st...@cloudera.com.INVALID> wrote: 
> +1, binding.
> 
> Awesome piece of work!
> 
> I've done three forms of qualification, all related to s3 and azure storage
> 
>    1. tarball validate, CLI use
>    2. build/test of downstream modules off maven artifacts; mine and some
>    other ASF ones. I  (and it its very much me) have broken some downstream
>    modules tests, as I will discuss below. PRs submitted to the relevant
>    projects
>    3. local rerun of the hadoop-aws and hadoop-azure test suites
> 
> 
> *Regarding issues which surfaced*
> 
> Wei-Chiu: can you register your private GPG key with the public keystores?
> The gpg client apps let you do this? Then we can coordinate signing each
> other's keys
> 
> Filed PRs for the test regressions:
> https://github.com/apache/hbase-filesystem/pull/23
> https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569
> 
> *Artifact validation*
> 
> SHA checksum good:
> 
> 
> shasum -a 512 hadoop-3.3.1-RC3.tar.gz
> b80e0a8785b0f3d75d9db54340123872e39bad72cc60de5d263ae22024720e6e824e022090f01e248bf105e03b0f06163729adbe15b5b0978bae0447571e22eb
>  hadoop-3.3.1-RC3.tar.gz
> 
> 
> GPG: trickier, because Wei-Chiu wasn't trusted
> 
> > gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
> 
> gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
> gpg: Signature made Tue Jun  1 11:00:41 2021 BST
> gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
> gpg: requesting key 0xB362E1C021854B9D from hkps server
> hkps.pool.sks-keyservers.net
> gpg: Can't check signature: No public key
> 
> 
> *Wei-Chiu: can you add your public keys to the GPG key servers*
> 
> To validate the keys I went to the directory where I have our site under
> svn (https://dist.apache.org/repos/dist/release/hadoop/common) , and, after
> reinstalling svn (where did it go? when did it go?) did an svn update to
> get the keys
> 
> Did a gpg import of the KEYS file, added
> 
> gpg: key 0x386D80EF81E7469A: public key "Brahma Reddy Battula (CODE SIGNING
> KEY) <br...@apache.org>" imported
> gpg: key 0xFC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <
> sammichen@apache.org>" imported
> gpg: key 0x36243EECE206BB0D: public key "Masatake Iwasaki (CODE SIGNING
> KEY) <iw...@apache.org>" imported
> *gpg: key 0xB362E1C021854B9D: public key "Wei-Chiu Chuang
> <weichiu@apache.org <we...@apache.org>>" imported*
> 
> This time an import did work, but Wei-Chiu isn't trusted by anyone yet
> 
> gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
> gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
> gpg: Signature made Tue Jun  1 11:00:41 2021 BST
> gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
> gpg: Good signature from "Wei-Chiu Chuang <we...@apache.org>" [unknown]
> gpg: WARNING: This key is not certified with a trusted signature!
> gpg:          There is no indication that the signature belongs to the
> owner.
> Primary key fingerprint: CD32 D773 FF41 C3F9 E74B  DB7F B362 E1C0 2185 4B9D
> 
> (Wei-Chiu, let's coordinate signing each other's public keys via a slack
> channel; you need to be in the apache web of trust)
> 
> 
> > time gunzip hadoop-3.3.1-RC3.tar.gz
> 
> (5 seconds)
> 
> cd into the hadoop dir;
> cp my confs in: cp ~/(somewhere)/hadoop-conf/*  etc/hadoop/
> cp the hadoop-azure dependencies from share/hadoop/tools/lib/ to
> share/hadoop/common/lib (products built targeting Azure put things there)
> 
> run: all the s3a "qualifying an AWS SDK update" commands
> https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update
> 
> run: basic abfs:// FS operations; again no problems.
> FWIW I think we should consider having the hadoop-aws module and
> dependencies, and the aws ones in hadoop-common/lib. I can get them there
> through env vars and the s3guard shell sets things up, but azure is fiddly.
> 
> *Build and test cloudstore JAR; invoke from CLI*
> 
> This is my cloud-storage extension library
> https://github.com/steveloughran/cloudstore
> 
> I've always intended to put it into hadoop, but as it is where a lot of
> diagnostics and quick way to put together fixes "here's a faster du ("dux"")
> 
> https://github.com/steveloughran/cloudstore.git
> 
> modify the hadoop-3.3 profile to use 3.3.1 artifacts, then build with
> snapshots enabled. Because I'd not (yet) built any 3.3.1 artifacts locally,
> this fetched them from maven staging
> 
> mvn package -Phadoop-3.3 -Pextra -Psnapshots-and-staging
> 
> 
> Set up env var $CLOUDSTORE to point to JAR; $BUCKET to s3a bucket, run
> various commands (storediag, cloudup, ...). As an example, here's the "dux"
> command, which is "hadoop fs -du" with parallel scan underneath the dir for
> better scaling
> 
> 
> bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose
> s3a://stevel-london/
> 
> output is in
> https://gist.github.com/steveloughran/664d30cef20f605f3164ad01f92a458a
> 
> *Build and (unit test) google GCS: *
> 
> 
> Two test failures, one of which was classpath related and the other just a
> new rename contract test needing a new setting in gs.xml to declare what
> rename of file over file does.
> 
> Everything is covered in:
> https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569
> 
> Classpath: assertJ not coming through hadoop-common-test JAR dependencies.
> 
> [ERROR]
> com.google.cloud.hadoop.fs.gcs.contract.TestInMemoryGoogleContractRootDirectory.testSimpleRootListing
>  Time elapsed: 0.093 s  <<< ERROR!
> java.lang.NoClassDefFoundError: org/assertj/core/api/Assertions
> Caused by: java.lang.ClassNotFoundException: org.assertj.core.api.Assertions
> 
> 
> Happens because I added some tests to the AbstractContractRenameTest which
> use assertJ assertions.
> Assertj is declared in test scope for hadoop-common test JAR, it's somehow
> not propagating. HBoss has the same issue.
> 
> 
>     <dependency>
>       <groupId>org.assertj</groupId>
>       <artifactId>assertj-core</artifactId>
>       <scope>test</scope>
>     </dependency>
> 
> I really don't understand what is up with our declared exports; just
> reviewed them. Nothing we can do about it that I can see.
> 
> Rename test failure is from a new test, with the expected behaviour needing
> definition.
> 
> [ERROR] Failures:
> [ERROR]
> TestInMemoryGoogleContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:89
> expected
> rename(gs://fake-in-memory-test-bucket/contract-test/source-256.txt,
> gs://fake-in-memory-test-bucket/contract-test/dest-512.txt) to be rejected
> with exception, but got false
> 
> Fix, add "fs.contract.rename-returns-false-if-dest-exists" = true to the
> XML contract.
> 
> *Build and test HBoss*
> 
> This is  the HBase extension to use ZK to lock file accesses on S3
> 
> I've broken their build through to changes to the internal S3 client
> factory as some new client options were passed down (HADOOP-13551).
> That change moved to a new build parameter object, so we can add future
> changes without breaking the signature again (mehakmeet already has in
> HADOOP-17705)
> 
> https://issues.apache.org/jira/browse/HBASE-25900
> 
> 
> Got an initial PR up, though will need to do more so that it will also
> compile/test against older builds
> https://github.com/apache/hbase-filesystem/pull/23
> 
> *Build spark, then test S3A Committers through it*
> 
> Build spark-3 against 3.3.1, then ran integration tests against S3 london
> 
> Test are in: https://github.com/hortonworks-spark/cloud-integration.git
> 
> Most of an afternoon was frittered away dealing with the fact that the
> spark version move (2.4 to 3.2) meant scalatest upgrade from 3.0 to 3.20 **and
> every single test failed to compile because the scalatest project moved the
> foundational test suite into a new package**. I had to do that same upgrade
> to test my WiP manifest committer (MAPREDUCE-7341) against ABFS, so it's
> not completely wasted. It does mean that module and tests is scala 3+ only.
> 
> hadoop-aws and hadoop-azure test suites
> 
> For these I checked out branch-3.3.1, rebuilt it and ran the test suites in
> the hadoop-azure and hadoop-aws modules. This triggered a rebuild of those
> two modules.
> 
> I did this after doing all the other checks, so everything else was
> qualified against the genuine RC3 artifacts.
> 
> 
> hadoop-aws
> 
> run 1: -Dparallel-tests -DtestsThreadCount=5 -Dmarkers=keep
> run 2: -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Dscale -Ddynamo
> 
> azure
>  -Dparallel-tests=abfs -DtestsThreadCount=5 -Dscale
> 
>  [ERROR] Errors:
>  [ERROR]
> ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:642
> » TestTimedOut
>  [INFO]
> 
> This is https://issues.apache.org/jira/browse/HADOOP-17628
> 
> 
> Overall then:
> 
>    1. All production code good.
>    2. some expansion of filesystem tests require some changes downstream,
>    and the change in the S3Client from HADOOP-13551 the HBoss tests using an
>    internal interface from compiling. The move to a parameter object (and
>    documenting this use) is intended to prevent this reoccurring.
> 
> 
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1, binding.

Awesome piece of work!

I've done three forms of qualification, all related to s3 and azure storage

   1. tarball validate, CLI use
   2. build/test of downstream modules off maven artifacts; mine and some
   other ASF ones. I  (and it its very much me) have broken some downstream
   modules tests, as I will discuss below. PRs submitted to the relevant
   projects
   3. local rerun of the hadoop-aws and hadoop-azure test suites


*Regarding issues which surfaced*

Wei-Chiu: can you register your private GPG key with the public keystores?
The gpg client apps let you do this? Then we can coordinate signing each
other's keys

Filed PRs for the test regressions:
https://github.com/apache/hbase-filesystem/pull/23
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

*Artifact validation*

SHA checksum good:


shasum -a 512 hadoop-3.3.1-RC3.tar.gz
b80e0a8785b0f3d75d9db54340123872e39bad72cc60de5d263ae22024720e6e824e022090f01e248bf105e03b0f06163729adbe15b5b0978bae0447571e22eb
 hadoop-3.3.1-RC3.tar.gz


GPG: trickier, because Wei-Chiu wasn't trusted

> gpg --verify hadoop-3.3.1-RC3.tar.gz.asc

gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: requesting key 0xB362E1C021854B9D from hkps server
hkps.pool.sks-keyservers.net
gpg: Can't check signature: No public key


*Wei-Chiu: can you add your public keys to the GPG key servers*

To validate the keys I went to the directory where I have our site under
svn (https://dist.apache.org/repos/dist/release/hadoop/common) , and, after
reinstalling svn (where did it go? when did it go?) did an svn update to
get the keys

Did a gpg import of the KEYS file, added

gpg: key 0x386D80EF81E7469A: public key "Brahma Reddy Battula (CODE SIGNING
KEY) <br...@apache.org>" imported
gpg: key 0xFC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <
sammichen@apache.org>" imported
gpg: key 0x36243EECE206BB0D: public key "Masatake Iwasaki (CODE SIGNING
KEY) <iw...@apache.org>" imported
*gpg: key 0xB362E1C021854B9D: public key "Wei-Chiu Chuang
<weichiu@apache.org <we...@apache.org>>" imported*

This time an import did work, but Wei-Chiu isn't trusted by anyone yet

gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: Good signature from "Wei-Chiu Chuang <we...@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the
owner.
Primary key fingerprint: CD32 D773 FF41 C3F9 E74B  DB7F B362 E1C0 2185 4B9D

(Wei-Chiu, let's coordinate signing each other's public keys via a slack
channel; you need to be in the apache web of trust)


> time gunzip hadoop-3.3.1-RC3.tar.gz

(5 seconds)

cd into the hadoop dir;
cp my confs in: cp ~/(somewhere)/hadoop-conf/*  etc/hadoop/
cp the hadoop-azure dependencies from share/hadoop/tools/lib/ to
share/hadoop/common/lib (products built targeting Azure put things there)

run: all the s3a "qualifying an AWS SDK update" commands
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update

run: basic abfs:// FS operations; again no problems.
FWIW I think we should consider having the hadoop-aws module and
dependencies, and the aws ones in hadoop-common/lib. I can get them there
through env vars and the s3guard shell sets things up, but azure is fiddly.

*Build and test cloudstore JAR; invoke from CLI*

This is my cloud-storage extension library
https://github.com/steveloughran/cloudstore

I've always intended to put it into hadoop, but as it is where a lot of
diagnostics and quick way to put together fixes "here's a faster du ("dux"")

https://github.com/steveloughran/cloudstore.git

modify the hadoop-3.3 profile to use 3.3.1 artifacts, then build with
snapshots enabled. Because I'd not (yet) built any 3.3.1 artifacts locally,
this fetched them from maven staging

mvn package -Phadoop-3.3 -Pextra -Psnapshots-and-staging


Set up env var $CLOUDSTORE to point to JAR; $BUCKET to s3a bucket, run
various commands (storediag, cloudup, ...). As an example, here's the "dux"
command, which is "hadoop fs -du" with parallel scan underneath the dir for
better scaling


bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose
s3a://stevel-london/

output is in
https://gist.github.com/steveloughran/664d30cef20f605f3164ad01f92a458a

*Build and (unit test) google GCS: *


Two test failures, one of which was classpath related and the other just a
new rename contract test needing a new setting in gs.xml to declare what
rename of file over file does.

Everything is covered in:
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

Classpath: assertJ not coming through hadoop-common-test JAR dependencies.

[ERROR]
com.google.cloud.hadoop.fs.gcs.contract.TestInMemoryGoogleContractRootDirectory.testSimpleRootListing
 Time elapsed: 0.093 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/assertj/core/api/Assertions
Caused by: java.lang.ClassNotFoundException: org.assertj.core.api.Assertions


Happens because I added some tests to the AbstractContractRenameTest which
use assertJ assertions.
Assertj is declared in test scope for hadoop-common test JAR, it's somehow
not propagating. HBoss has the same issue.


    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <scope>test</scope>
    </dependency>

I really don't understand what is up with our declared exports; just
reviewed them. Nothing we can do about it that I can see.

Rename test failure is from a new test, with the expected behaviour needing
definition.

[ERROR] Failures:
[ERROR]
TestInMemoryGoogleContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:89
expected
rename(gs://fake-in-memory-test-bucket/contract-test/source-256.txt,
gs://fake-in-memory-test-bucket/contract-test/dest-512.txt) to be rejected
with exception, but got false

Fix, add "fs.contract.rename-returns-false-if-dest-exists" = true to the
XML contract.

*Build and test HBoss*

This is  the HBase extension to use ZK to lock file accesses on S3

I've broken their build through to changes to the internal S3 client
factory as some new client options were passed down (HADOOP-13551).
That change moved to a new build parameter object, so we can add future
changes without breaking the signature again (mehakmeet already has in
HADOOP-17705)

https://issues.apache.org/jira/browse/HBASE-25900


Got an initial PR up, though will need to do more so that it will also
compile/test against older builds
https://github.com/apache/hbase-filesystem/pull/23

*Build spark, then test S3A Committers through it*

Build spark-3 against 3.3.1, then ran integration tests against S3 london

Test are in: https://github.com/hortonworks-spark/cloud-integration.git

Most of an afternoon was frittered away dealing with the fact that the
spark version move (2.4 to 3.2) meant scalatest upgrade from 3.0 to 3.20 **and
every single test failed to compile because the scalatest project moved the
foundational test suite into a new package**. I had to do that same upgrade
to test my WiP manifest committer (MAPREDUCE-7341) against ABFS, so it's
not completely wasted. It does mean that module and tests is scala 3+ only.

hadoop-aws and hadoop-azure test suites

For these I checked out branch-3.3.1, rebuilt it and ran the test suites in
the hadoop-azure and hadoop-aws modules. This triggered a rebuild of those
two modules.

I did this after doing all the other checks, so everything else was
qualified against the genuine RC3 artifacts.


hadoop-aws

run 1: -Dparallel-tests -DtestsThreadCount=5 -Dmarkers=keep
run 2: -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Dscale -Ddynamo

azure
 -Dparallel-tests=abfs -DtestsThreadCount=5 -Dscale

 [ERROR] Errors:
 [ERROR]
ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:642
» TestTimedOut
 [INFO]

This is https://issues.apache.org/jira/browse/HADOOP-17628


Overall then:

   1. All production code good.
   2. some expansion of filesystem tests require some changes downstream,
   and the change in the S3Client from HADOOP-13551 the HBoss tests using an
   internal interface from compiling. The move to a parameter object (and
   documenting this use) is intended to prevent this reoccurring.


>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
I found that the top page of site documentation (index.html on the top directory)
has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).

While I think it is not worth for sinking the RC3,
you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
if you need to cut RC4.

Thanks,
Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1, binding.

Awesome piece of work!

I've done three forms of qualification, all related to s3 and azure storage

   1. tarball validate, CLI use
   2. build/test of downstream modules off maven artifacts; mine and some
   other ASF ones. I  (and it its very much me) have broken some downstream
   modules tests, as I will discuss below. PRs submitted to the relevant
   projects
   3. local rerun of the hadoop-aws and hadoop-azure test suites


*Regarding issues which surfaced*

Wei-Chiu: can you register your private GPG key with the public keystores?
The gpg client apps let you do this? Then we can coordinate signing each
other's keys

Filed PRs for the test regressions:
https://github.com/apache/hbase-filesystem/pull/23
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

*Artifact validation*

SHA checksum good:


shasum -a 512 hadoop-3.3.1-RC3.tar.gz
b80e0a8785b0f3d75d9db54340123872e39bad72cc60de5d263ae22024720e6e824e022090f01e248bf105e03b0f06163729adbe15b5b0978bae0447571e22eb
 hadoop-3.3.1-RC3.tar.gz


GPG: trickier, because Wei-Chiu wasn't trusted

> gpg --verify hadoop-3.3.1-RC3.tar.gz.asc

gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: requesting key 0xB362E1C021854B9D from hkps server
hkps.pool.sks-keyservers.net
gpg: Can't check signature: No public key


*Wei-Chiu: can you add your public keys to the GPG key servers*

To validate the keys I went to the directory where I have our site under
svn (https://dist.apache.org/repos/dist/release/hadoop/common) , and, after
reinstalling svn (where did it go? when did it go?) did an svn update to
get the keys

Did a gpg import of the KEYS file, added

gpg: key 0x386D80EF81E7469A: public key "Brahma Reddy Battula (CODE SIGNING
KEY) <br...@apache.org>" imported
gpg: key 0xFC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <
sammichen@apache.org>" imported
gpg: key 0x36243EECE206BB0D: public key "Masatake Iwasaki (CODE SIGNING
KEY) <iw...@apache.org>" imported
*gpg: key 0xB362E1C021854B9D: public key "Wei-Chiu Chuang
<weichiu@apache.org <we...@apache.org>>" imported*

This time an import did work, but Wei-Chiu isn't trusted by anyone yet

gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: Good signature from "Wei-Chiu Chuang <we...@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the
owner.
Primary key fingerprint: CD32 D773 FF41 C3F9 E74B  DB7F B362 E1C0 2185 4B9D

(Wei-Chiu, let's coordinate signing each other's public keys via a slack
channel; you need to be in the apache web of trust)


> time gunzip hadoop-3.3.1-RC3.tar.gz

(5 seconds)

cd into the hadoop dir;
cp my confs in: cp ~/(somewhere)/hadoop-conf/*  etc/hadoop/
cp the hadoop-azure dependencies from share/hadoop/tools/lib/ to
share/hadoop/common/lib (products built targeting Azure put things there)

run: all the s3a "qualifying an AWS SDK update" commands
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update

run: basic abfs:// FS operations; again no problems.
FWIW I think we should consider having the hadoop-aws module and
dependencies, and the aws ones in hadoop-common/lib. I can get them there
through env vars and the s3guard shell sets things up, but azure is fiddly.

*Build and test cloudstore JAR; invoke from CLI*

This is my cloud-storage extension library
https://github.com/steveloughran/cloudstore

I've always intended to put it into hadoop, but as it is where a lot of
diagnostics and quick way to put together fixes "here's a faster du ("dux"")

https://github.com/steveloughran/cloudstore.git

modify the hadoop-3.3 profile to use 3.3.1 artifacts, then build with
snapshots enabled. Because I'd not (yet) built any 3.3.1 artifacts locally,
this fetched them from maven staging

mvn package -Phadoop-3.3 -Pextra -Psnapshots-and-staging


Set up env var $CLOUDSTORE to point to JAR; $BUCKET to s3a bucket, run
various commands (storediag, cloudup, ...). As an example, here's the "dux"
command, which is "hadoop fs -du" with parallel scan underneath the dir for
better scaling


bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose
s3a://stevel-london/

output is in
https://gist.github.com/steveloughran/664d30cef20f605f3164ad01f92a458a

*Build and (unit test) google GCS: *


Two test failures, one of which was classpath related and the other just a
new rename contract test needing a new setting in gs.xml to declare what
rename of file over file does.

Everything is covered in:
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

Classpath: assertJ not coming through hadoop-common-test JAR dependencies.

[ERROR]
com.google.cloud.hadoop.fs.gcs.contract.TestInMemoryGoogleContractRootDirectory.testSimpleRootListing
 Time elapsed: 0.093 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/assertj/core/api/Assertions
Caused by: java.lang.ClassNotFoundException: org.assertj.core.api.Assertions


Happens because I added some tests to the AbstractContractRenameTest which
use assertJ assertions.
Assertj is declared in test scope for hadoop-common test JAR, it's somehow
not propagating. HBoss has the same issue.


    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <scope>test</scope>
    </dependency>

I really don't understand what is up with our declared exports; just
reviewed them. Nothing we can do about it that I can see.

Rename test failure is from a new test, with the expected behaviour needing
definition.

[ERROR] Failures:
[ERROR]
TestInMemoryGoogleContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:89
expected
rename(gs://fake-in-memory-test-bucket/contract-test/source-256.txt,
gs://fake-in-memory-test-bucket/contract-test/dest-512.txt) to be rejected
with exception, but got false

Fix, add "fs.contract.rename-returns-false-if-dest-exists" = true to the
XML contract.

*Build and test HBoss*

This is  the HBase extension to use ZK to lock file accesses on S3

I've broken their build through to changes to the internal S3 client
factory as some new client options were passed down (HADOOP-13551).
That change moved to a new build parameter object, so we can add future
changes without breaking the signature again (mehakmeet already has in
HADOOP-17705)

https://issues.apache.org/jira/browse/HBASE-25900


Got an initial PR up, though will need to do more so that it will also
compile/test against older builds
https://github.com/apache/hbase-filesystem/pull/23

*Build spark, then test S3A Committers through it*

Build spark-3 against 3.3.1, then ran integration tests against S3 london

Test are in: https://github.com/hortonworks-spark/cloud-integration.git

Most of an afternoon was frittered away dealing with the fact that the
spark version move (2.4 to 3.2) meant scalatest upgrade from 3.0 to 3.20 **and
every single test failed to compile because the scalatest project moved the
foundational test suite into a new package**. I had to do that same upgrade
to test my WiP manifest committer (MAPREDUCE-7341) against ABFS, so it's
not completely wasted. It does mean that module and tests is scala 3+ only.

hadoop-aws and hadoop-azure test suites

For these I checked out branch-3.3.1, rebuilt it and ran the test suites in
the hadoop-azure and hadoop-aws modules. This triggered a rebuild of those
two modules.

I did this after doing all the other checks, so everything else was
qualified against the genuine RC3 artifacts.


hadoop-aws

run 1: -Dparallel-tests -DtestsThreadCount=5 -Dmarkers=keep
run 2: -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Dscale -Ddynamo

azure
 -Dparallel-tests=abfs -DtestsThreadCount=5 -Dscale

 [ERROR] Errors:
 [ERROR]
ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:642
» TestTimedOut
 [INFO]

This is https://issues.apache.org/jira/browse/HADOOP-17628


Overall then:

   1. All production code good.
   2. some expansion of filesystem tests require some changes downstream,
   and the change in the S3Client from HADOOP-13551 the HBoss tests using an
   internal interface from compiling. The move to a parameter object (and
   documenting this use) is intended to prevent this reoccurring.


>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1, binding.

Awesome piece of work!

I've done three forms of qualification, all related to s3 and azure storage

   1. tarball validate, CLI use
   2. build/test of downstream modules off maven artifacts; mine and some
   other ASF ones. I  (and it its very much me) have broken some downstream
   modules tests, as I will discuss below. PRs submitted to the relevant
   projects
   3. local rerun of the hadoop-aws and hadoop-azure test suites


*Regarding issues which surfaced*

Wei-Chiu: can you register your private GPG key with the public keystores?
The gpg client apps let you do this? Then we can coordinate signing each
other's keys

Filed PRs for the test regressions:
https://github.com/apache/hbase-filesystem/pull/23
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

*Artifact validation*

SHA checksum good:


shasum -a 512 hadoop-3.3.1-RC3.tar.gz
b80e0a8785b0f3d75d9db54340123872e39bad72cc60de5d263ae22024720e6e824e022090f01e248bf105e03b0f06163729adbe15b5b0978bae0447571e22eb
 hadoop-3.3.1-RC3.tar.gz


GPG: trickier, because Wei-Chiu wasn't trusted

> gpg --verify hadoop-3.3.1-RC3.tar.gz.asc

gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: requesting key 0xB362E1C021854B9D from hkps server
hkps.pool.sks-keyservers.net
gpg: Can't check signature: No public key


*Wei-Chiu: can you add your public keys to the GPG key servers*

To validate the keys I went to the directory where I have our site under
svn (https://dist.apache.org/repos/dist/release/hadoop/common) , and, after
reinstalling svn (where did it go? when did it go?) did an svn update to
get the keys

Did a gpg import of the KEYS file, added

gpg: key 0x386D80EF81E7469A: public key "Brahma Reddy Battula (CODE SIGNING
KEY) <br...@apache.org>" imported
gpg: key 0xFC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <
sammichen@apache.org>" imported
gpg: key 0x36243EECE206BB0D: public key "Masatake Iwasaki (CODE SIGNING
KEY) <iw...@apache.org>" imported
*gpg: key 0xB362E1C021854B9D: public key "Wei-Chiu Chuang
<weichiu@apache.org <we...@apache.org>>" imported*

This time an import did work, but Wei-Chiu isn't trusted by anyone yet

gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: Good signature from "Wei-Chiu Chuang <we...@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the
owner.
Primary key fingerprint: CD32 D773 FF41 C3F9 E74B  DB7F B362 E1C0 2185 4B9D

(Wei-Chiu, let's coordinate signing each other's public keys via a slack
channel; you need to be in the apache web of trust)


> time gunzip hadoop-3.3.1-RC3.tar.gz

(5 seconds)

cd into the hadoop dir;
cp my confs in: cp ~/(somewhere)/hadoop-conf/*  etc/hadoop/
cp the hadoop-azure dependencies from share/hadoop/tools/lib/ to
share/hadoop/common/lib (products built targeting Azure put things there)

run: all the s3a "qualifying an AWS SDK update" commands
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update

run: basic abfs:// FS operations; again no problems.
FWIW I think we should consider having the hadoop-aws module and
dependencies, and the aws ones in hadoop-common/lib. I can get them there
through env vars and the s3guard shell sets things up, but azure is fiddly.

*Build and test cloudstore JAR; invoke from CLI*

This is my cloud-storage extension library
https://github.com/steveloughran/cloudstore

I've always intended to put it into hadoop, but as it is where a lot of
diagnostics and quick way to put together fixes "here's a faster du ("dux"")

https://github.com/steveloughran/cloudstore.git

modify the hadoop-3.3 profile to use 3.3.1 artifacts, then build with
snapshots enabled. Because I'd not (yet) built any 3.3.1 artifacts locally,
this fetched them from maven staging

mvn package -Phadoop-3.3 -Pextra -Psnapshots-and-staging


Set up env var $CLOUDSTORE to point to JAR; $BUCKET to s3a bucket, run
various commands (storediag, cloudup, ...). As an example, here's the "dux"
command, which is "hadoop fs -du" with parallel scan underneath the dir for
better scaling


bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose
s3a://stevel-london/

output is in
https://gist.github.com/steveloughran/664d30cef20f605f3164ad01f92a458a

*Build and (unit test) google GCS: *


Two test failures, one of which was classpath related and the other just a
new rename contract test needing a new setting in gs.xml to declare what
rename of file over file does.

Everything is covered in:
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

Classpath: assertJ not coming through hadoop-common-test JAR dependencies.

[ERROR]
com.google.cloud.hadoop.fs.gcs.contract.TestInMemoryGoogleContractRootDirectory.testSimpleRootListing
 Time elapsed: 0.093 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/assertj/core/api/Assertions
Caused by: java.lang.ClassNotFoundException: org.assertj.core.api.Assertions


Happens because I added some tests to the AbstractContractRenameTest which
use assertJ assertions.
Assertj is declared in test scope for hadoop-common test JAR, it's somehow
not propagating. HBoss has the same issue.


    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <scope>test</scope>
    </dependency>

I really don't understand what is up with our declared exports; just
reviewed them. Nothing we can do about it that I can see.

Rename test failure is from a new test, with the expected behaviour needing
definition.

[ERROR] Failures:
[ERROR]
TestInMemoryGoogleContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:89
expected
rename(gs://fake-in-memory-test-bucket/contract-test/source-256.txt,
gs://fake-in-memory-test-bucket/contract-test/dest-512.txt) to be rejected
with exception, but got false

Fix, add "fs.contract.rename-returns-false-if-dest-exists" = true to the
XML contract.

*Build and test HBoss*

This is  the HBase extension to use ZK to lock file accesses on S3

I've broken their build through to changes to the internal S3 client
factory as some new client options were passed down (HADOOP-13551).
That change moved to a new build parameter object, so we can add future
changes without breaking the signature again (mehakmeet already has in
HADOOP-17705)

https://issues.apache.org/jira/browse/HBASE-25900


Got an initial PR up, though will need to do more so that it will also
compile/test against older builds
https://github.com/apache/hbase-filesystem/pull/23

*Build spark, then test S3A Committers through it*

Build spark-3 against 3.3.1, then ran integration tests against S3 london

Test are in: https://github.com/hortonworks-spark/cloud-integration.git

Most of an afternoon was frittered away dealing with the fact that the
spark version move (2.4 to 3.2) meant scalatest upgrade from 3.0 to 3.20 **and
every single test failed to compile because the scalatest project moved the
foundational test suite into a new package**. I had to do that same upgrade
to test my WiP manifest committer (MAPREDUCE-7341) against ABFS, so it's
not completely wasted. It does mean that module and tests is scala 3+ only.

hadoop-aws and hadoop-azure test suites

For these I checked out branch-3.3.1, rebuilt it and ran the test suites in
the hadoop-azure and hadoop-aws modules. This triggered a rebuild of those
two modules.

I did this after doing all the other checks, so everything else was
qualified against the genuine RC3 artifacts.


hadoop-aws

run 1: -Dparallel-tests -DtestsThreadCount=5 -Dmarkers=keep
run 2: -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Dscale -Ddynamo

azure
 -Dparallel-tests=abfs -DtestsThreadCount=5 -Dscale

 [ERROR] Errors:
 [ERROR]
ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:642
» TestTimedOut
 [INFO]

This is https://issues.apache.org/jira/browse/HADOOP-17628


Overall then:

   1. All production code good.
   2. some expansion of filesystem tests require some changes downstream,
   and the change in the S3Client from HADOOP-13551 the HBoss tests using an
   internal interface from compiling. The move to a parameter object (and
   documenting this use) is intended to prevent this reoccurring.


>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
+1

Thanks for the great work, Wei-Chiu Chuang.

* verified signature and checksum.
* built site documentation by `mvn site` and skimmed the contents.
   # found that top-level index.html is not updated.
* built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.
   * launched pseudo cluster with security enabled and ran sample MR jobs.
   * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
* built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.

* built Hive with the patch of HIVE-24484 against hadoop-3.3.1
   and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).

Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@apache.org>.
Hello,
do we want to extend the release vote? I understand a big release like this
takes time to validate.

I am aware a number of people are testing it: Attila tested Ozone on Hadoop
3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
over Hadoop 3.2 library.

Looks like we may need some more time to test. How about extending it by a
week?

On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Stack <st...@duboce.net>.
+1



        * Signature: ok

        * Checksum : ok

        * Rat check (1.8.0_191): ok

         - mvn clean apache-rat:check

        * Built from source (1.8.0_191): ok

         - mvn clean install -DskipTests


Ran a ten node cluster w/ hbase on top running its verification loadings w/
(gentle) chaos. Had trouble getting the rig running but mostly pilot error
and none that I could particularly attribute to hdfs after poking in logs.

Messed in UI and shell some. Nothing untoward.

Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
there (a classic flakie seems more-so on 3.3.1... will dig in more on why).


Thanks,

S


On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Stack <st...@duboce.net>.
+1



        * Signature: ok

        * Checksum : ok

        * Rat check (1.8.0_191): ok

         - mvn clean apache-rat:check

        * Built from source (1.8.0_191): ok

         - mvn clean install -DskipTests


Ran a ten node cluster w/ hbase on top running its verification loadings w/
(gentle) chaos. Had trouble getting the rig running but mostly pilot error
and none that I could particularly attribute to hdfs after poking in logs.

Messed in UI and shell some. Nothing untoward.

Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
there (a classic flakie seems more-so on 3.3.1... will dig in more on why).


Thanks,

S


On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
I found that the top page of site documentation (index.html on the top directory)
has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).

While I think it is not worth for sinking the RC3,
you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
if you need to cut RC4.

Thanks,
Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks all for the input!

I released the maven artifacts earlier today, so soon you should be able to
use the 3.3.1 dependencies in downstream applications.
The official website is refreshed to include 3.3.1:
https://hadoop.apache.org/
The tarball was uploaded and verified the links are good to go.

Will send out the announcement to the user mailing list later.

On Tue, Jun 15, 2021 at 9:56 PM Xiaoqiao He <he...@apache.org> wrote:

> +0.
>
> - Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
> - Verified signature and checksum
> - Deployed pseudo-distributed cluster with 3 nodes
> - Ran basic HDFS shell commands and sample MR Jobs, it worked well.
> - Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*
>
> NOTE: attach configuration for YARN. Please correct me if I missed
> something.
>
> Thanks Wei-Chiu for your great work!
>
> - He Xiaoqiao
>
>
> On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1.
>>  - Verified hashes
>>  - Confirmed native build on CentOS7
>>  - Started kerberized cluster (using docker)
>>  - Checked NN/RBF Web UI
>>  - Ran basic Erasure Coding shell commands
>>
>> Thanks for the great work, Wei-Chiu.
>>
>> - Takanobu
>>
>> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>>
>> > +1 (Binding)
>> >
>> > 1. Built from Tag.
>> > 2. Successful Native Build on Ubuntu 20.04
>> > 3. Verified Checksums
>> > 4. Deployed the docker cluster with 3 nodes
>> > 5. Ran sample MR Jobs
>> >
>> > -Vinay
>> >
>> >
>> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >
>> > > +1,
>> > > Built from Source.
>> > > Successful Native Build on Ubuntu 20.04
>> > > Verified Checksums
>> > > Ran basic hdfs shell commands.
>> > > Ran simple MR jobs.
>> > > Browsed NN,DN,RM and NM UI.
>> > >
>> > > Thanx Wei-Chiu for driving the release.
>> > >
>> > > -Ayush
>> > >
>> > >
>> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
>> > > >
>> > > > +1 (binding)
>> > > > Eric
>> > > >
>> > > >
>> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
>> > > weichiu@apache.org> wrote:
>> > > >
>> > > > Hi community,
>> > > >
>> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
>> > > blocker
>> > > > issues have been resolved [1] again.
>> > > >
>> > > > There are 2 additional issues resolved for RC3:
>> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> > > > HADOOP-16878
>> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
>> > > source
>> > > > and destination are the same
>> > > >
>> > > > There are 4 issues resolved for RC2:
>> > > > * HADOOP-17666. Update LICENSE for 3.3.1
>> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
>> (#3055)
>> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> > > >
>> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
>> > extra
>> > > > fixes compared to hadoop-thirdparty 1.1.0:
>> > > > * HADOOP-17707. Remove jaeger document from site index.
>> > > > * HADOOP-17730. Add back error_prone
>> > > >
>> > > > *RC tag is release-3.3.1-RC3
>> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> > > >
>> > > > *The RC3 artifacts are at*:
>> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> > > > ARM artifacts:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> > > >
>> > > > *The maven artifacts are hosted here:*
>> > > >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> > > >
>> > > > *My public key is available here:*
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > >
>> > > > Things I've verified:
>> > > > * all blocker issues targeting 3.3.1 have been resolved.
>> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are
>> compatible.
>> > > > * LICENSE and NOTICE files checked
>> > > > * RELEASENOTES and CHANGELOG
>> > > > * rat check passed.
>> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
>> tests.
>> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
>> > RC2.
>> > > > Had to patch some of them due to commons-lang migration (Hadoop
>> 3.2.0)
>> > > and
>> > > > dependency divergence. Issues are being identified but so far
>> nothing
>> > > > blocker for Hadoop itself.
>> > > >
>> > > > Please try the release and vote. The vote will run for 5 days.
>> > > >
>> > > > My +1 to start,
>> > > >
>> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
>> > > > [2]
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks all for the input!

I released the maven artifacts earlier today, so soon you should be able to
use the 3.3.1 dependencies in downstream applications.
The official website is refreshed to include 3.3.1:
https://hadoop.apache.org/
The tarball was uploaded and verified the links are good to go.

Will send out the announcement to the user mailing list later.

On Tue, Jun 15, 2021 at 9:56 PM Xiaoqiao He <he...@apache.org> wrote:

> +0.
>
> - Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
> - Verified signature and checksum
> - Deployed pseudo-distributed cluster with 3 nodes
> - Ran basic HDFS shell commands and sample MR Jobs, it worked well.
> - Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*
>
> NOTE: attach configuration for YARN. Please correct me if I missed
> something.
>
> Thanks Wei-Chiu for your great work!
>
> - He Xiaoqiao
>
>
> On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1.
>>  - Verified hashes
>>  - Confirmed native build on CentOS7
>>  - Started kerberized cluster (using docker)
>>  - Checked NN/RBF Web UI
>>  - Ran basic Erasure Coding shell commands
>>
>> Thanks for the great work, Wei-Chiu.
>>
>> - Takanobu
>>
>> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>>
>> > +1 (Binding)
>> >
>> > 1. Built from Tag.
>> > 2. Successful Native Build on Ubuntu 20.04
>> > 3. Verified Checksums
>> > 4. Deployed the docker cluster with 3 nodes
>> > 5. Ran sample MR Jobs
>> >
>> > -Vinay
>> >
>> >
>> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >
>> > > +1,
>> > > Built from Source.
>> > > Successful Native Build on Ubuntu 20.04
>> > > Verified Checksums
>> > > Ran basic hdfs shell commands.
>> > > Ran simple MR jobs.
>> > > Browsed NN,DN,RM and NM UI.
>> > >
>> > > Thanx Wei-Chiu for driving the release.
>> > >
>> > > -Ayush
>> > >
>> > >
>> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
>> > > >
>> > > > +1 (binding)
>> > > > Eric
>> > > >
>> > > >
>> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
>> > > weichiu@apache.org> wrote:
>> > > >
>> > > > Hi community,
>> > > >
>> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
>> > > blocker
>> > > > issues have been resolved [1] again.
>> > > >
>> > > > There are 2 additional issues resolved for RC3:
>> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> > > > HADOOP-16878
>> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
>> > > source
>> > > > and destination are the same
>> > > >
>> > > > There are 4 issues resolved for RC2:
>> > > > * HADOOP-17666. Update LICENSE for 3.3.1
>> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
>> (#3055)
>> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> > > >
>> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
>> > extra
>> > > > fixes compared to hadoop-thirdparty 1.1.0:
>> > > > * HADOOP-17707. Remove jaeger document from site index.
>> > > > * HADOOP-17730. Add back error_prone
>> > > >
>> > > > *RC tag is release-3.3.1-RC3
>> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> > > >
>> > > > *The RC3 artifacts are at*:
>> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> > > > ARM artifacts:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> > > >
>> > > > *The maven artifacts are hosted here:*
>> > > >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> > > >
>> > > > *My public key is available here:*
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > >
>> > > > Things I've verified:
>> > > > * all blocker issues targeting 3.3.1 have been resolved.
>> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are
>> compatible.
>> > > > * LICENSE and NOTICE files checked
>> > > > * RELEASENOTES and CHANGELOG
>> > > > * rat check passed.
>> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
>> tests.
>> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
>> > RC2.
>> > > > Had to patch some of them due to commons-lang migration (Hadoop
>> 3.2.0)
>> > > and
>> > > > dependency divergence. Issues are being identified but so far
>> nothing
>> > > > blocker for Hadoop itself.
>> > > >
>> > > > Please try the release and vote. The vote will run for 5 days.
>> > > >
>> > > > My +1 to start,
>> > > >
>> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
>> > > > [2]
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks all for the input!

I released the maven artifacts earlier today, so soon you should be able to
use the 3.3.1 dependencies in downstream applications.
The official website is refreshed to include 3.3.1:
https://hadoop.apache.org/
The tarball was uploaded and verified the links are good to go.

Will send out the announcement to the user mailing list later.

On Tue, Jun 15, 2021 at 9:56 PM Xiaoqiao He <he...@apache.org> wrote:

> +0.
>
> - Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
> - Verified signature and checksum
> - Deployed pseudo-distributed cluster with 3 nodes
> - Ran basic HDFS shell commands and sample MR Jobs, it worked well.
> - Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*
>
> NOTE: attach configuration for YARN. Please correct me if I missed
> something.
>
> Thanks Wei-Chiu for your great work!
>
> - He Xiaoqiao
>
>
> On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1.
>>  - Verified hashes
>>  - Confirmed native build on CentOS7
>>  - Started kerberized cluster (using docker)
>>  - Checked NN/RBF Web UI
>>  - Ran basic Erasure Coding shell commands
>>
>> Thanks for the great work, Wei-Chiu.
>>
>> - Takanobu
>>
>> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>>
>> > +1 (Binding)
>> >
>> > 1. Built from Tag.
>> > 2. Successful Native Build on Ubuntu 20.04
>> > 3. Verified Checksums
>> > 4. Deployed the docker cluster with 3 nodes
>> > 5. Ran sample MR Jobs
>> >
>> > -Vinay
>> >
>> >
>> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >
>> > > +1,
>> > > Built from Source.
>> > > Successful Native Build on Ubuntu 20.04
>> > > Verified Checksums
>> > > Ran basic hdfs shell commands.
>> > > Ran simple MR jobs.
>> > > Browsed NN,DN,RM and NM UI.
>> > >
>> > > Thanx Wei-Chiu for driving the release.
>> > >
>> > > -Ayush
>> > >
>> > >
>> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
>> > > >
>> > > > +1 (binding)
>> > > > Eric
>> > > >
>> > > >
>> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
>> > > weichiu@apache.org> wrote:
>> > > >
>> > > > Hi community,
>> > > >
>> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
>> > > blocker
>> > > > issues have been resolved [1] again.
>> > > >
>> > > > There are 2 additional issues resolved for RC3:
>> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> > > > HADOOP-16878
>> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
>> > > source
>> > > > and destination are the same
>> > > >
>> > > > There are 4 issues resolved for RC2:
>> > > > * HADOOP-17666. Update LICENSE for 3.3.1
>> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
>> (#3055)
>> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> > > >
>> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
>> > extra
>> > > > fixes compared to hadoop-thirdparty 1.1.0:
>> > > > * HADOOP-17707. Remove jaeger document from site index.
>> > > > * HADOOP-17730. Add back error_prone
>> > > >
>> > > > *RC tag is release-3.3.1-RC3
>> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> > > >
>> > > > *The RC3 artifacts are at*:
>> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> > > > ARM artifacts:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> > > >
>> > > > *The maven artifacts are hosted here:*
>> > > >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> > > >
>> > > > *My public key is available here:*
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > >
>> > > > Things I've verified:
>> > > > * all blocker issues targeting 3.3.1 have been resolved.
>> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are
>> compatible.
>> > > > * LICENSE and NOTICE files checked
>> > > > * RELEASENOTES and CHANGELOG
>> > > > * rat check passed.
>> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
>> tests.
>> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
>> > RC2.
>> > > > Had to patch some of them due to commons-lang migration (Hadoop
>> 3.2.0)
>> > > and
>> > > > dependency divergence. Issues are being identified but so far
>> nothing
>> > > > blocker for Hadoop itself.
>> > > >
>> > > > Please try the release and vote. The vote will run for 5 days.
>> > > >
>> > > > My +1 to start,
>> > > >
>> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
>> > > > [2]
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks all for the input!

I released the maven artifacts earlier today, so soon you should be able to
use the 3.3.1 dependencies in downstream applications.
The official website is refreshed to include 3.3.1:
https://hadoop.apache.org/
The tarball was uploaded and verified the links are good to go.

Will send out the announcement to the user mailing list later.

On Tue, Jun 15, 2021 at 9:56 PM Xiaoqiao He <he...@apache.org> wrote:

> +0.
>
> - Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
> - Verified signature and checksum
> - Deployed pseudo-distributed cluster with 3 nodes
> - Ran basic HDFS shell commands and sample MR Jobs, it worked well.
> - Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*
>
> NOTE: attach configuration for YARN. Please correct me if I missed
> something.
>
> Thanks Wei-Chiu for your great work!
>
> - He Xiaoqiao
>
>
> On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1.
>>  - Verified hashes
>>  - Confirmed native build on CentOS7
>>  - Started kerberized cluster (using docker)
>>  - Checked NN/RBF Web UI
>>  - Ran basic Erasure Coding shell commands
>>
>> Thanks for the great work, Wei-Chiu.
>>
>> - Takanobu
>>
>> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>>
>> > +1 (Binding)
>> >
>> > 1. Built from Tag.
>> > 2. Successful Native Build on Ubuntu 20.04
>> > 3. Verified Checksums
>> > 4. Deployed the docker cluster with 3 nodes
>> > 5. Ran sample MR Jobs
>> >
>> > -Vinay
>> >
>> >
>> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >
>> > > +1,
>> > > Built from Source.
>> > > Successful Native Build on Ubuntu 20.04
>> > > Verified Checksums
>> > > Ran basic hdfs shell commands.
>> > > Ran simple MR jobs.
>> > > Browsed NN,DN,RM and NM UI.
>> > >
>> > > Thanx Wei-Chiu for driving the release.
>> > >
>> > > -Ayush
>> > >
>> > >
>> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
>> > > >
>> > > > +1 (binding)
>> > > > Eric
>> > > >
>> > > >
>> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
>> > > weichiu@apache.org> wrote:
>> > > >
>> > > > Hi community,
>> > > >
>> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
>> > > blocker
>> > > > issues have been resolved [1] again.
>> > > >
>> > > > There are 2 additional issues resolved for RC3:
>> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> > > > HADOOP-16878
>> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
>> > > source
>> > > > and destination are the same
>> > > >
>> > > > There are 4 issues resolved for RC2:
>> > > > * HADOOP-17666. Update LICENSE for 3.3.1
>> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
>> (#3055)
>> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> > > >
>> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
>> > extra
>> > > > fixes compared to hadoop-thirdparty 1.1.0:
>> > > > * HADOOP-17707. Remove jaeger document from site index.
>> > > > * HADOOP-17730. Add back error_prone
>> > > >
>> > > > *RC tag is release-3.3.1-RC3
>> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> > > >
>> > > > *The RC3 artifacts are at*:
>> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> > > > ARM artifacts:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> > > >
>> > > > *The maven artifacts are hosted here:*
>> > > >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> > > >
>> > > > *My public key is available here:*
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > >
>> > > > Things I've verified:
>> > > > * all blocker issues targeting 3.3.1 have been resolved.
>> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are
>> compatible.
>> > > > * LICENSE and NOTICE files checked
>> > > > * RELEASENOTES and CHANGELOG
>> > > > * rat check passed.
>> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
>> tests.
>> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
>> > RC2.
>> > > > Had to patch some of them due to commons-lang migration (Hadoop
>> 3.2.0)
>> > > and
>> > > > dependency divergence. Issues are being identified but so far
>> nothing
>> > > > blocker for Hadoop itself.
>> > > >
>> > > > Please try the release and vote. The vote will run for 5 days.
>> > > >
>> > > > My +1 to start,
>> > > >
>> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
>> > > > [2]
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks all for the input!

I released the maven artifacts earlier today, so soon you should be able to
use the 3.3.1 dependencies in downstream applications.
The official website is refreshed to include 3.3.1:
https://hadoop.apache.org/
The tarball was uploaded and verified the links are good to go.

Will send out the announcement to the user mailing list later.

On Tue, Jun 15, 2021 at 9:56 PM Xiaoqiao He <he...@apache.org> wrote:

> +0.
>
> - Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
> - Verified signature and checksum
> - Deployed pseudo-distributed cluster with 3 nodes
> - Ran basic HDFS shell commands and sample MR Jobs, it worked well.
> - Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*
>
> NOTE: attach configuration for YARN. Please correct me if I missed
> something.
>
> Thanks Wei-Chiu for your great work!
>
> - He Xiaoqiao
>
>
> On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> +1.
>>  - Verified hashes
>>  - Confirmed native build on CentOS7
>>  - Started kerberized cluster (using docker)
>>  - Checked NN/RBF Web UI
>>  - Ran basic Erasure Coding shell commands
>>
>> Thanks for the great work, Wei-Chiu.
>>
>> - Takanobu
>>
>> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>>
>> > +1 (Binding)
>> >
>> > 1. Built from Tag.
>> > 2. Successful Native Build on Ubuntu 20.04
>> > 3. Verified Checksums
>> > 4. Deployed the docker cluster with 3 nodes
>> > 5. Ran sample MR Jobs
>> >
>> > -Vinay
>> >
>> >
>> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >
>> > > +1,
>> > > Built from Source.
>> > > Successful Native Build on Ubuntu 20.04
>> > > Verified Checksums
>> > > Ran basic hdfs shell commands.
>> > > Ran simple MR jobs.
>> > > Browsed NN,DN,RM and NM UI.
>> > >
>> > > Thanx Wei-Chiu for driving the release.
>> > >
>> > > -Ayush
>> > >
>> > >
>> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
>> > > >
>> > > > +1 (binding)
>> > > > Eric
>> > > >
>> > > >
>> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
>> > > weichiu@apache.org> wrote:
>> > > >
>> > > > Hi community,
>> > > >
>> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
>> > > blocker
>> > > > issues have been resolved [1] again.
>> > > >
>> > > > There are 2 additional issues resolved for RC3:
>> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> > > > HADOOP-16878
>> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
>> > > source
>> > > > and destination are the same
>> > > >
>> > > > There are 4 issues resolved for RC2:
>> > > > * HADOOP-17666. Update LICENSE for 3.3.1
>> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
>> (#3055)
>> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> > > >
>> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
>> > extra
>> > > > fixes compared to hadoop-thirdparty 1.1.0:
>> > > > * HADOOP-17707. Remove jaeger document from site index.
>> > > > * HADOOP-17730. Add back error_prone
>> > > >
>> > > > *RC tag is release-3.3.1-RC3
>> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> > > >
>> > > > *The RC3 artifacts are at*:
>> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> > > > ARM artifacts:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> > > >
>> > > > *The maven artifacts are hosted here:*
>> > > >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> > > >
>> > > > *My public key is available here:*
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > >
>> > > > Things I've verified:
>> > > > * all blocker issues targeting 3.3.1 have been resolved.
>> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are
>> compatible.
>> > > > * LICENSE and NOTICE files checked
>> > > > * RELEASENOTES and CHANGELOG
>> > > > * rat check passed.
>> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
>> tests.
>> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
>> > RC2.
>> > > > Had to patch some of them due to commons-lang migration (Hadoop
>> 3.2.0)
>> > > and
>> > > > dependency divergence. Issues are being identified but so far
>> nothing
>> > > > blocker for Hadoop itself.
>> > > >
>> > > > Please try the release and vote. The vote will run for 5 days.
>> > > >
>> > > > My +1 to start,
>> > > >
>> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
>> > > > [2]
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Xiaoqiao He <he...@apache.org>.
+0.

- Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
- Verified signature and checksum
- Deployed pseudo-distributed cluster with 3 nodes
- Ran basic HDFS shell commands and sample MR Jobs, it worked well.
- Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*

NOTE: attach configuration for YARN. Please correct me if I missed
something.

Thanks Wei-Chiu for your great work!

- He Xiaoqiao


On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
wrote:

> +1.
>  - Verified hashes
>  - Confirmed native build on CentOS7
>  - Started kerberized cluster (using docker)
>  - Checked NN/RBF Web UI
>  - Ran basic Erasure Coding shell commands
>
> Thanks for the great work, Wei-Chiu.
>
> - Takanobu
>
> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>
> > +1 (Binding)
> >
> > 1. Built from Tag.
> > 2. Successful Native Build on Ubuntu 20.04
> > 3. Verified Checksums
> > 4. Deployed the docker cluster with 3 nodes
> > 5. Ran sample MR Jobs
> >
> > -Vinay
> >
> >
> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > +1,
> > > Built from Source.
> > > Successful Native Build on Ubuntu 20.04
> > > Verified Checksums
> > > Ran basic hdfs shell commands.
> > > Ran simple MR jobs.
> > > Browsed NN,DN,RM and NM UI.
> > >
> > > Thanx Wei-Chiu for driving the release.
> > >
> > > -Ayush
> > >
> > >
> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > > >
> > > > +1 (binding)
> > > > Eric
> > > >
> > > >
> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > > weichiu@apache.org> wrote:
> > > >
> > > > Hi community,
> > > >
> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > > blocker
> > > > issues have been resolved [1] again.
> > > >
> > > > There are 2 additional issues resolved for RC3:
> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > > HADOOP-16878
> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > > source
> > > > and destination are the same
> > > >
> > > > There are 4 issues resolved for RC2:
> > > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
> (#3055)
> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > > >
> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> > extra
> > > > fixes compared to hadoop-thirdparty 1.1.0:
> > > > * HADOOP-17707. Remove jaeger document from site index.
> > > > * HADOOP-17730. Add back error_prone
> > > >
> > > > *RC tag is release-3.3.1-RC3
> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > > >
> > > > *The RC3 artifacts are at*:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > > ARM artifacts:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > >
> > > > Things I've verified:
> > > > * all blocker issues targeting 3.3.1 have been resolved.
> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > > * LICENSE and NOTICE files checked
> > > > * RELEASENOTES and CHANGELOG
> > > > * rat check passed.
> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> > RC2.
> > > > Had to patch some of them due to commons-lang migration (Hadoop
> 3.2.0)
> > > and
> > > > dependency divergence. Issues are being identified but so far nothing
> > > > blocker for Hadoop itself.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > My +1 to start,
> > > >
> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Xiaoqiao He <he...@apache.org>.
+0.

- Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
- Verified signature and checksum
- Deployed pseudo-distributed cluster with 3 nodes
- Ran basic HDFS shell commands and sample MR Jobs, it worked well.
- Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*

NOTE: attach configuration for YARN. Please correct me if I missed
something.

Thanks Wei-Chiu for your great work!

- He Xiaoqiao


On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
wrote:

> +1.
>  - Verified hashes
>  - Confirmed native build on CentOS7
>  - Started kerberized cluster (using docker)
>  - Checked NN/RBF Web UI
>  - Ran basic Erasure Coding shell commands
>
> Thanks for the great work, Wei-Chiu.
>
> - Takanobu
>
> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>
> > +1 (Binding)
> >
> > 1. Built from Tag.
> > 2. Successful Native Build on Ubuntu 20.04
> > 3. Verified Checksums
> > 4. Deployed the docker cluster with 3 nodes
> > 5. Ran sample MR Jobs
> >
> > -Vinay
> >
> >
> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > +1,
> > > Built from Source.
> > > Successful Native Build on Ubuntu 20.04
> > > Verified Checksums
> > > Ran basic hdfs shell commands.
> > > Ran simple MR jobs.
> > > Browsed NN,DN,RM and NM UI.
> > >
> > > Thanx Wei-Chiu for driving the release.
> > >
> > > -Ayush
> > >
> > >
> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > > >
> > > > +1 (binding)
> > > > Eric
> > > >
> > > >
> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > > weichiu@apache.org> wrote:
> > > >
> > > > Hi community,
> > > >
> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > > blocker
> > > > issues have been resolved [1] again.
> > > >
> > > > There are 2 additional issues resolved for RC3:
> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > > HADOOP-16878
> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > > source
> > > > and destination are the same
> > > >
> > > > There are 4 issues resolved for RC2:
> > > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
> (#3055)
> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > > >
> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> > extra
> > > > fixes compared to hadoop-thirdparty 1.1.0:
> > > > * HADOOP-17707. Remove jaeger document from site index.
> > > > * HADOOP-17730. Add back error_prone
> > > >
> > > > *RC tag is release-3.3.1-RC3
> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > > >
> > > > *The RC3 artifacts are at*:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > > ARM artifacts:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > >
> > > > Things I've verified:
> > > > * all blocker issues targeting 3.3.1 have been resolved.
> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > > * LICENSE and NOTICE files checked
> > > > * RELEASENOTES and CHANGELOG
> > > > * rat check passed.
> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> > RC2.
> > > > Had to patch some of them due to commons-lang migration (Hadoop
> 3.2.0)
> > > and
> > > > dependency divergence. Issues are being identified but so far nothing
> > > > blocker for Hadoop itself.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > My +1 to start,
> > > >
> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Xiaoqiao He <he...@apache.org>.
+0.

- Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
- Verified signature and checksum
- Deployed pseudo-distributed cluster with 3 nodes
- Ran basic HDFS shell commands and sample MR Jobs, it worked well.
- Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*

NOTE: attach configuration for YARN. Please correct me if I missed
something.

Thanks Wei-Chiu for your great work!

- He Xiaoqiao


On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
wrote:

> +1.
>  - Verified hashes
>  - Confirmed native build on CentOS7
>  - Started kerberized cluster (using docker)
>  - Checked NN/RBF Web UI
>  - Ran basic Erasure Coding shell commands
>
> Thanks for the great work, Wei-Chiu.
>
> - Takanobu
>
> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>
> > +1 (Binding)
> >
> > 1. Built from Tag.
> > 2. Successful Native Build on Ubuntu 20.04
> > 3. Verified Checksums
> > 4. Deployed the docker cluster with 3 nodes
> > 5. Ran sample MR Jobs
> >
> > -Vinay
> >
> >
> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > +1,
> > > Built from Source.
> > > Successful Native Build on Ubuntu 20.04
> > > Verified Checksums
> > > Ran basic hdfs shell commands.
> > > Ran simple MR jobs.
> > > Browsed NN,DN,RM and NM UI.
> > >
> > > Thanx Wei-Chiu for driving the release.
> > >
> > > -Ayush
> > >
> > >
> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > > >
> > > > +1 (binding)
> > > > Eric
> > > >
> > > >
> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > > weichiu@apache.org> wrote:
> > > >
> > > > Hi community,
> > > >
> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > > blocker
> > > > issues have been resolved [1] again.
> > > >
> > > > There are 2 additional issues resolved for RC3:
> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > > HADOOP-16878
> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > > source
> > > > and destination are the same
> > > >
> > > > There are 4 issues resolved for RC2:
> > > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
> (#3055)
> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > > >
> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> > extra
> > > > fixes compared to hadoop-thirdparty 1.1.0:
> > > > * HADOOP-17707. Remove jaeger document from site index.
> > > > * HADOOP-17730. Add back error_prone
> > > >
> > > > *RC tag is release-3.3.1-RC3
> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > > >
> > > > *The RC3 artifacts are at*:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > > ARM artifacts:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > >
> > > > Things I've verified:
> > > > * all blocker issues targeting 3.3.1 have been resolved.
> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > > * LICENSE and NOTICE files checked
> > > > * RELEASENOTES and CHANGELOG
> > > > * rat check passed.
> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> > RC2.
> > > > Had to patch some of them due to commons-lang migration (Hadoop
> 3.2.0)
> > > and
> > > > dependency divergence. Issues are being identified but so far nothing
> > > > blocker for Hadoop itself.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > My +1 to start,
> > > >
> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Xiaoqiao He <he...@apache.org>.
+0.

- Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
- Verified signature and checksum
- Deployed pseudo-distributed cluster with 3 nodes
- Ran basic HDFS shell commands and sample MR Jobs, it worked well.
- Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*

NOTE: attach configuration for YARN. Please correct me if I missed
something.

Thanks Wei-Chiu for your great work!

- He Xiaoqiao


On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
wrote:

> +1.
>  - Verified hashes
>  - Confirmed native build on CentOS7
>  - Started kerberized cluster (using docker)
>  - Checked NN/RBF Web UI
>  - Ran basic Erasure Coding shell commands
>
> Thanks for the great work, Wei-Chiu.
>
> - Takanobu
>
> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>
> > +1 (Binding)
> >
> > 1. Built from Tag.
> > 2. Successful Native Build on Ubuntu 20.04
> > 3. Verified Checksums
> > 4. Deployed the docker cluster with 3 nodes
> > 5. Ran sample MR Jobs
> >
> > -Vinay
> >
> >
> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > +1,
> > > Built from Source.
> > > Successful Native Build on Ubuntu 20.04
> > > Verified Checksums
> > > Ran basic hdfs shell commands.
> > > Ran simple MR jobs.
> > > Browsed NN,DN,RM and NM UI.
> > >
> > > Thanx Wei-Chiu for driving the release.
> > >
> > > -Ayush
> > >
> > >
> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > > >
> > > > +1 (binding)
> > > > Eric
> > > >
> > > >
> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > > weichiu@apache.org> wrote:
> > > >
> > > > Hi community,
> > > >
> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > > blocker
> > > > issues have been resolved [1] again.
> > > >
> > > > There are 2 additional issues resolved for RC3:
> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > > HADOOP-16878
> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > > source
> > > > and destination are the same
> > > >
> > > > There are 4 issues resolved for RC2:
> > > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
> (#3055)
> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > > >
> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> > extra
> > > > fixes compared to hadoop-thirdparty 1.1.0:
> > > > * HADOOP-17707. Remove jaeger document from site index.
> > > > * HADOOP-17730. Add back error_prone
> > > >
> > > > *RC tag is release-3.3.1-RC3
> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > > >
> > > > *The RC3 artifacts are at*:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > > ARM artifacts:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > >
> > > > Things I've verified:
> > > > * all blocker issues targeting 3.3.1 have been resolved.
> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > > * LICENSE and NOTICE files checked
> > > > * RELEASENOTES and CHANGELOG
> > > > * rat check passed.
> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> > RC2.
> > > > Had to patch some of them due to commons-lang migration (Hadoop
> 3.2.0)
> > > and
> > > > dependency divergence. Issues are being identified but so far nothing
> > > > blocker for Hadoop itself.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > My +1 to start,
> > > >
> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Xiaoqiao He <he...@apache.org>.
+0.

- Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
- Verified signature and checksum
- Deployed pseudo-distributed cluster with 3 nodes
- Ran basic HDFS shell commands and sample MR Jobs, it worked well.
- Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*

NOTE: attach configuration for YARN. Please correct me if I missed
something.

Thanks Wei-Chiu for your great work!

- He Xiaoqiao


On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma <ta...@apache.org>
wrote:

> +1.
>  - Verified hashes
>  - Confirmed native build on CentOS7
>  - Started kerberized cluster (using docker)
>  - Checked NN/RBF Web UI
>  - Ran basic Erasure Coding shell commands
>
> Thanks for the great work, Wei-Chiu.
>
> - Takanobu
>
> 2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:
>
> > +1 (Binding)
> >
> > 1. Built from Tag.
> > 2. Successful Native Build on Ubuntu 20.04
> > 3. Verified Checksums
> > 4. Deployed the docker cluster with 3 nodes
> > 5. Ran sample MR Jobs
> >
> > -Vinay
> >
> >
> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > +1,
> > > Built from Source.
> > > Successful Native Build on Ubuntu 20.04
> > > Verified Checksums
> > > Ran basic hdfs shell commands.
> > > Ran simple MR jobs.
> > > Browsed NN,DN,RM and NM UI.
> > >
> > > Thanx Wei-Chiu for driving the release.
> > >
> > > -Ayush
> > >
> > >
> > > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > > >
> > > > +1 (binding)
> > > > Eric
> > > >
> > > >
> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > > weichiu@apache.org> wrote:
> > > >
> > > > Hi community,
> > > >
> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > > blocker
> > > > issues have been resolved [1] again.
> > > >
> > > > There are 2 additional issues resolved for RC3:
> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > > HADOOP-16878
> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > > source
> > > > and destination are the same
> > > >
> > > > There are 4 issues resolved for RC2:
> > > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
> (#3055)
> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > > >
> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> > extra
> > > > fixes compared to hadoop-thirdparty 1.1.0:
> > > > * HADOOP-17707. Remove jaeger document from site index.
> > > > * HADOOP-17730. Add back error_prone
> > > >
> > > > *RC tag is release-3.3.1-RC3
> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > > >
> > > > *The RC3 artifacts are at*:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > > ARM artifacts:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > >
> > > > Things I've verified:
> > > > * all blocker issues targeting 3.3.1 have been resolved.
> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > > * LICENSE and NOTICE files checked
> > > > * RELEASENOTES and CHANGELOG
> > > > * rat check passed.
> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> > RC2.
> > > > Had to patch some of them due to commons-lang migration (Hadoop
> 3.2.0)
> > > and
> > > > dependency divergence. Issues are being identified but so far nothing
> > > > blocker for Hadoop itself.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > My +1 to start,
> > > >
> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Takanobu Asanuma <ta...@apache.org>.
+1.
 - Verified hashes
 - Confirmed native build on CentOS7
 - Started kerberized cluster (using docker)
 - Checked NN/RBF Web UI
 - Ran basic Erasure Coding shell commands

Thanks for the great work, Wei-Chiu.

- Takanobu

2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:

> +1 (Binding)
>
> 1. Built from Tag.
> 2. Successful Native Build on Ubuntu 20.04
> 3. Verified Checksums
> 4. Deployed the docker cluster with 3 nodes
> 5. Ran sample MR Jobs
>
> -Vinay
>
>
> On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > +1,
> > Built from Source.
> > Successful Native Build on Ubuntu 20.04
> > Verified Checksums
> > Ran basic hdfs shell commands.
> > Ran simple MR jobs.
> > Browsed NN,DN,RM and NM UI.
> >
> > Thanx Wei-Chiu for driving the release.
> >
> > -Ayush
> >
> >
> > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > >
> > > +1 (binding)
> > > Eric
> > >
> > >
> > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > weichiu@apache.org> wrote:
> > >
> > > Hi community,
> > >
> > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > > issues have been resolved [1] again.
> > >
> > > There are 2 additional issues resolved for RC3:
> > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > HADOOP-16878
> > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > > and destination are the same
> > >
> > > There are 4 issues resolved for RC2:
> > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >
> > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > > fixes compared to hadoop-thirdparty 1.1.0:
> > > * HADOOP-17707. Remove jaeger document from site index.
> > > * HADOOP-17730. Add back error_prone
> > >
> > > *RC tag is release-3.3.1-RC3
> > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >
> > > *The RC3 artifacts are at*:
> > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >
> > > *The maven artifacts are hosted here:*
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >
> > > *My public key is available here:*
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > >
> > > Things I've verified:
> > > * all blocker issues targeting 3.3.1 have been resolved.
> > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > * LICENSE and NOTICE files checked
> > > * RELEASENOTES and CHANGELOG
> > > * rat check passed.
> > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > > dependency divergence. Issues are being identified but so far nothing
> > > blocker for Hadoop itself.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > My +1 to start,
> > >
> > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > [2]
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Takanobu Asanuma <ta...@apache.org>.
+1.
 - Verified hashes
 - Confirmed native build on CentOS7
 - Started kerberized cluster (using docker)
 - Checked NN/RBF Web UI
 - Ran basic Erasure Coding shell commands

Thanks for the great work, Wei-Chiu.

- Takanobu

2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:

> +1 (Binding)
>
> 1. Built from Tag.
> 2. Successful Native Build on Ubuntu 20.04
> 3. Verified Checksums
> 4. Deployed the docker cluster with 3 nodes
> 5. Ran sample MR Jobs
>
> -Vinay
>
>
> On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > +1,
> > Built from Source.
> > Successful Native Build on Ubuntu 20.04
> > Verified Checksums
> > Ran basic hdfs shell commands.
> > Ran simple MR jobs.
> > Browsed NN,DN,RM and NM UI.
> >
> > Thanx Wei-Chiu for driving the release.
> >
> > -Ayush
> >
> >
> > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > >
> > > +1 (binding)
> > > Eric
> > >
> > >
> > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > weichiu@apache.org> wrote:
> > >
> > > Hi community,
> > >
> > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > > issues have been resolved [1] again.
> > >
> > > There are 2 additional issues resolved for RC3:
> > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > HADOOP-16878
> > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > > and destination are the same
> > >
> > > There are 4 issues resolved for RC2:
> > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >
> > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > > fixes compared to hadoop-thirdparty 1.1.0:
> > > * HADOOP-17707. Remove jaeger document from site index.
> > > * HADOOP-17730. Add back error_prone
> > >
> > > *RC tag is release-3.3.1-RC3
> > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >
> > > *The RC3 artifacts are at*:
> > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >
> > > *The maven artifacts are hosted here:*
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >
> > > *My public key is available here:*
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > >
> > > Things I've verified:
> > > * all blocker issues targeting 3.3.1 have been resolved.
> > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > * LICENSE and NOTICE files checked
> > > * RELEASENOTES and CHANGELOG
> > > * rat check passed.
> > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > > dependency divergence. Issues are being identified but so far nothing
> > > blocker for Hadoop itself.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > My +1 to start,
> > >
> > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > [2]
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Takanobu Asanuma <ta...@apache.org>.
+1.
 - Verified hashes
 - Confirmed native build on CentOS7
 - Started kerberized cluster (using docker)
 - Checked NN/RBF Web UI
 - Ran basic Erasure Coding shell commands

Thanks for the great work, Wei-Chiu.

- Takanobu

2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:

> +1 (Binding)
>
> 1. Built from Tag.
> 2. Successful Native Build on Ubuntu 20.04
> 3. Verified Checksums
> 4. Deployed the docker cluster with 3 nodes
> 5. Ran sample MR Jobs
>
> -Vinay
>
>
> On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > +1,
> > Built from Source.
> > Successful Native Build on Ubuntu 20.04
> > Verified Checksums
> > Ran basic hdfs shell commands.
> > Ran simple MR jobs.
> > Browsed NN,DN,RM and NM UI.
> >
> > Thanx Wei-Chiu for driving the release.
> >
> > -Ayush
> >
> >
> > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > >
> > > +1 (binding)
> > > Eric
> > >
> > >
> > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > weichiu@apache.org> wrote:
> > >
> > > Hi community,
> > >
> > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > > issues have been resolved [1] again.
> > >
> > > There are 2 additional issues resolved for RC3:
> > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > HADOOP-16878
> > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > > and destination are the same
> > >
> > > There are 4 issues resolved for RC2:
> > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >
> > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > > fixes compared to hadoop-thirdparty 1.1.0:
> > > * HADOOP-17707. Remove jaeger document from site index.
> > > * HADOOP-17730. Add back error_prone
> > >
> > > *RC tag is release-3.3.1-RC3
> > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >
> > > *The RC3 artifacts are at*:
> > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >
> > > *The maven artifacts are hosted here:*
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >
> > > *My public key is available here:*
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > >
> > > Things I've verified:
> > > * all blocker issues targeting 3.3.1 have been resolved.
> > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > * LICENSE and NOTICE files checked
> > > * RELEASENOTES and CHANGELOG
> > > * rat check passed.
> > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > > dependency divergence. Issues are being identified but so far nothing
> > > blocker for Hadoop itself.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > My +1 to start,
> > >
> > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > [2]
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Takanobu Asanuma <ta...@apache.org>.
+1.
 - Verified hashes
 - Confirmed native build on CentOS7
 - Started kerberized cluster (using docker)
 - Checked NN/RBF Web UI
 - Ran basic Erasure Coding shell commands

Thanks for the great work, Wei-Chiu.

- Takanobu

2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:

> +1 (Binding)
>
> 1. Built from Tag.
> 2. Successful Native Build on Ubuntu 20.04
> 3. Verified Checksums
> 4. Deployed the docker cluster with 3 nodes
> 5. Ran sample MR Jobs
>
> -Vinay
>
>
> On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > +1,
> > Built from Source.
> > Successful Native Build on Ubuntu 20.04
> > Verified Checksums
> > Ran basic hdfs shell commands.
> > Ran simple MR jobs.
> > Browsed NN,DN,RM and NM UI.
> >
> > Thanx Wei-Chiu for driving the release.
> >
> > -Ayush
> >
> >
> > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > >
> > > +1 (binding)
> > > Eric
> > >
> > >
> > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > weichiu@apache.org> wrote:
> > >
> > > Hi community,
> > >
> > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > > issues have been resolved [1] again.
> > >
> > > There are 2 additional issues resolved for RC3:
> > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > HADOOP-16878
> > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > > and destination are the same
> > >
> > > There are 4 issues resolved for RC2:
> > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >
> > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > > fixes compared to hadoop-thirdparty 1.1.0:
> > > * HADOOP-17707. Remove jaeger document from site index.
> > > * HADOOP-17730. Add back error_prone
> > >
> > > *RC tag is release-3.3.1-RC3
> > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >
> > > *The RC3 artifacts are at*:
> > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >
> > > *The maven artifacts are hosted here:*
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >
> > > *My public key is available here:*
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > >
> > > Things I've verified:
> > > * all blocker issues targeting 3.3.1 have been resolved.
> > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > * LICENSE and NOTICE files checked
> > > * RELEASENOTES and CHANGELOG
> > > * rat check passed.
> > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > > dependency divergence. Issues are being identified but so far nothing
> > > blocker for Hadoop itself.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > My +1 to start,
> > >
> > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > [2]
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Takanobu Asanuma <ta...@apache.org>.
+1.
 - Verified hashes
 - Confirmed native build on CentOS7
 - Started kerberized cluster (using docker)
 - Checked NN/RBF Web UI
 - Ran basic Erasure Coding shell commands

Thanks for the great work, Wei-Chiu.

- Takanobu

2021年6月13日(日) 3:25 Vinayakumar B <vi...@apache.org>:

> +1 (Binding)
>
> 1. Built from Tag.
> 2. Successful Native Build on Ubuntu 20.04
> 3. Verified Checksums
> 4. Deployed the docker cluster with 3 nodes
> 5. Ran sample MR Jobs
>
> -Vinay
>
>
> On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > +1,
> > Built from Source.
> > Successful Native Build on Ubuntu 20.04
> > Verified Checksums
> > Ran basic hdfs shell commands.
> > Ran simple MR jobs.
> > Browsed NN,DN,RM and NM UI.
> >
> > Thanx Wei-Chiu for driving the release.
> >
> > -Ayush
> >
> >
> > > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> > >
> > > +1 (binding)
> > > Eric
> > >
> > >
> > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > weichiu@apache.org> wrote:
> > >
> > > Hi community,
> > >
> > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > blocker
> > > issues have been resolved [1] again.
> > >
> > > There are 2 additional issues resolved for RC3:
> > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > HADOOP-16878
> > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > source
> > > and destination are the same
> > >
> > > There are 4 issues resolved for RC2:
> > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > >
> > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> extra
> > > fixes compared to hadoop-thirdparty 1.1.0:
> > > * HADOOP-17707. Remove jaeger document from site index.
> > > * HADOOP-17730. Add back error_prone
> > >
> > > *RC tag is release-3.3.1-RC3
> > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > >
> > > *The RC3 artifacts are at*:
> > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > >
> > > *The maven artifacts are hosted here:*
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > >
> > > *My public key is available here:*
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > >
> > > Things I've verified:
> > > * all blocker issues targeting 3.3.1 have been resolved.
> > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > * LICENSE and NOTICE files checked
> > > * RELEASENOTES and CHANGELOG
> > > * rat check passed.
> > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> RC2.
> > > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> > and
> > > dependency divergence. Issues are being identified but so far nothing
> > > blocker for Hadoop itself.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > My +1 to start,
> > >
> > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > [2]
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Vinayakumar B <vi...@apache.org>.
+1 (Binding)

1. Built from Tag.
2. Successful Native Build on Ubuntu 20.04
3. Verified Checksums
4. Deployed the docker cluster with 3 nodes
5. Ran sample MR Jobs

-Vinay


On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:

> +1,
> Built from Source.
> Successful Native Build on Ubuntu 20.04
> Verified Checksums
> Ran basic hdfs shell commands.
> Ran simple MR jobs.
> Browsed NN,DN,RM and NM UI.
>
> Thanx Wei-Chiu for driving the release.
>
> -Ayush
>
>
> > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> >
> > +1 (binding)
> > Eric
> >
> >
> > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> weichiu@apache.org> wrote:
> >
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Vinayakumar B <vi...@apache.org>.
+1 (Binding)

1. Built from Tag.
2. Successful Native Build on Ubuntu 20.04
3. Verified Checksums
4. Deployed the docker cluster with 3 nodes
5. Ran sample MR Jobs

-Vinay


On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:

> +1,
> Built from Source.
> Successful Native Build on Ubuntu 20.04
> Verified Checksums
> Ran basic hdfs shell commands.
> Ran simple MR jobs.
> Browsed NN,DN,RM and NM UI.
>
> Thanx Wei-Chiu for driving the release.
>
> -Ayush
>
>
> > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> >
> > +1 (binding)
> > Eric
> >
> >
> > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> weichiu@apache.org> wrote:
> >
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Vinayakumar B <vi...@apache.org>.
+1 (Binding)

1. Built from Tag.
2. Successful Native Build on Ubuntu 20.04
3. Verified Checksums
4. Deployed the docker cluster with 3 nodes
5. Ran sample MR Jobs

-Vinay


On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:

> +1,
> Built from Source.
> Successful Native Build on Ubuntu 20.04
> Verified Checksums
> Ran basic hdfs shell commands.
> Ran simple MR jobs.
> Browsed NN,DN,RM and NM UI.
>
> Thanx Wei-Chiu for driving the release.
>
> -Ayush
>
>
> > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> >
> > +1 (binding)
> > Eric
> >
> >
> > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> weichiu@apache.org> wrote:
> >
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Vinayakumar B <vi...@apache.org>.
+1 (Binding)

1. Built from Tag.
2. Successful Native Build on Ubuntu 20.04
3. Verified Checksums
4. Deployed the docker cluster with 3 nodes
5. Ran sample MR Jobs

-Vinay


On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:

> +1,
> Built from Source.
> Successful Native Build on Ubuntu 20.04
> Verified Checksums
> Ran basic hdfs shell commands.
> Ran simple MR jobs.
> Browsed NN,DN,RM and NM UI.
>
> Thanx Wei-Chiu for driving the release.
>
> -Ayush
>
>
> > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> >
> > +1 (binding)
> > Eric
> >
> >
> > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> weichiu@apache.org> wrote:
> >
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Vinayakumar B <vi...@apache.org>.
+1 (Binding)

1. Built from Tag.
2. Successful Native Build on Ubuntu 20.04
3. Verified Checksums
4. Deployed the docker cluster with 3 nodes
5. Ran sample MR Jobs

-Vinay


On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena <ay...@gmail.com> wrote:

> +1,
> Built from Source.
> Successful Native Build on Ubuntu 20.04
> Verified Checksums
> Ran basic hdfs shell commands.
> Ran simple MR jobs.
> Browsed NN,DN,RM and NM UI.
>
> Thanx Wei-Chiu for driving the release.
>
> -Ayush
>
>
> > On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> >
> > +1 (binding)
> > Eric
> >
> >
> > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> weichiu@apache.org> wrote:
> >
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Ayush Saxena <ay...@gmail.com>.
+1,
Built from Source.
Successful Native Build on Ubuntu 20.04
Verified Checksums
Ran basic hdfs shell commands.
Ran simple MR jobs.
Browsed NN,DN,RM and NM UI.

Thanx Wei-Chiu for driving the release. 

-Ayush


> On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> 
> +1 (binding)
> Eric
> 
> 
> On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
> 
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Ayush Saxena <ay...@gmail.com>.
+1,
Built from Source.
Successful Native Build on Ubuntu 20.04
Verified Checksums
Ran basic hdfs shell commands.
Ran simple MR jobs.
Browsed NN,DN,RM and NM UI.

Thanx Wei-Chiu for driving the release. 

-Ayush


> On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> 
> +1 (binding)
> Eric
> 
> 
> On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
> 
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Ayush Saxena <ay...@gmail.com>.
+1,
Built from Source.
Successful Native Build on Ubuntu 20.04
Verified Checksums
Ran basic hdfs shell commands.
Ran simple MR jobs.
Browsed NN,DN,RM and NM UI.

Thanx Wei-Chiu for driving the release. 

-Ayush


> On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> 
> +1 (binding)
> Eric
> 
> 
> On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
> 
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Ayush Saxena <ay...@gmail.com>.
+1,
Built from Source.
Successful Native Build on Ubuntu 20.04
Verified Checksums
Ran basic hdfs shell commands.
Ran simple MR jobs.
Browsed NN,DN,RM and NM UI.

Thanx Wei-Chiu for driving the release. 

-Ayush


> On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> 
> +1 (binding)
> Eric
> 
> 
> On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
> 
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Ayush Saxena <ay...@gmail.com>.
+1,
Built from Source.
Successful Native Build on Ubuntu 20.04
Verified Checksums
Ran basic hdfs shell commands.
Ran simple MR jobs.
Browsed NN,DN,RM and NM UI.

Thanx Wei-Chiu for driving the release. 

-Ayush


> On 12-Jun-2021, at 1:45 AM, epayne@apache.org wrote:
> 
> +1 (binding)
> Eric
> 
> 
> On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
> 
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by "epayne@apache.org" <ep...@apache.org>.
+1 (binding)
Eric


   On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
 
 Hi community,

This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
issues have been resolved [1] again.

There are 2 additional issues resolved for RC3:
* Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
HADOOP-16878
* Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
and destination are the same

There are 4 issues resolved for RC2:
* HADOOP-17666. Update LICENSE for 3.3.1
* MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
* Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
* HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)

The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
fixes compared to hadoop-thirdparty 1.1.0:
* HADOOP-17707. Remove jaeger document from site index.
* HADOOP-17730. Add back error_prone

*RC tag is release-3.3.1-RC3
https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3

*The RC3 artifacts are at*:
https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/

*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1320/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS


Things I've verified:
* all blocker issues targeting 3.3.1 have been resolved.
* stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
* LICENSE and NOTICE files checked
* RELEASENOTES and CHANGELOG
* rat check passed.
* Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
* Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
* Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
dependency divergence. Issues are being identified but so far nothing
blocker for Hadoop itself.

Please try the release and vote. The vote will run for 5 days.

My +1 to start,

[1] https://issues.apache.org/jira/issues/?filter=12350491
[2]
https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
+1

Thanks for the great work, Wei-Chiu Chuang.

* verified signature and checksum.
* built site documentation by `mvn site` and skimmed the contents.
   # found that top-level index.html is not updated.
* built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.
   * launched pseudo cluster with security enabled and ran sample MR jobs.
   * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
* built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.

* built Hive with the patch of HIVE-24484 against hadoop-3.3.1
   and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).

Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@apache.org>.
Hello,
do we want to extend the release vote? I understand a big release like this
takes time to validate.

I am aware a number of people are testing it: Attila tested Ozone on Hadoop
3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
over Hadoop 3.2 library.

Looks like we may need some more time to test. How about extending it by a
week?

On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Stack <st...@duboce.net>.
+1



        * Signature: ok

        * Checksum : ok

        * Rat check (1.8.0_191): ok

         - mvn clean apache-rat:check

        * Built from source (1.8.0_191): ok

         - mvn clean install -DskipTests


Ran a ten node cluster w/ hbase on top running its verification loadings w/
(gentle) chaos. Had trouble getting the rig running but mostly pilot error
and none that I could particularly attribute to hdfs after poking in logs.

Messed in UI and shell some. Nothing untoward.

Wei-Chiu fixed broke tests over in hbase and complete runs are pretty much
there (a classic flakie seems more-so on 3.3.1... will dig in more on why).


Thanks,

S


On Tue, Jun 1, 2021 at 3:29 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
>
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1, binding.

Awesome piece of work!

I've done three forms of qualification, all related to s3 and azure storage

   1. tarball validate, CLI use
   2. build/test of downstream modules off maven artifacts; mine and some
   other ASF ones. I  (and it its very much me) have broken some downstream
   modules tests, as I will discuss below. PRs submitted to the relevant
   projects
   3. local rerun of the hadoop-aws and hadoop-azure test suites


*Regarding issues which surfaced*

Wei-Chiu: can you register your private GPG key with the public keystores?
The gpg client apps let you do this? Then we can coordinate signing each
other's keys

Filed PRs for the test regressions:
https://github.com/apache/hbase-filesystem/pull/23
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

*Artifact validation*

SHA checksum good:


shasum -a 512 hadoop-3.3.1-RC3.tar.gz
b80e0a8785b0f3d75d9db54340123872e39bad72cc60de5d263ae22024720e6e824e022090f01e248bf105e03b0f06163729adbe15b5b0978bae0447571e22eb
 hadoop-3.3.1-RC3.tar.gz


GPG: trickier, because Wei-Chiu wasn't trusted

> gpg --verify hadoop-3.3.1-RC3.tar.gz.asc

gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: requesting key 0xB362E1C021854B9D from hkps server
hkps.pool.sks-keyservers.net
gpg: Can't check signature: No public key


*Wei-Chiu: can you add your public keys to the GPG key servers*

To validate the keys I went to the directory where I have our site under
svn (https://dist.apache.org/repos/dist/release/hadoop/common) , and, after
reinstalling svn (where did it go? when did it go?) did an svn update to
get the keys

Did a gpg import of the KEYS file, added

gpg: key 0x386D80EF81E7469A: public key "Brahma Reddy Battula (CODE SIGNING
KEY) <br...@apache.org>" imported
gpg: key 0xFC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <
sammichen@apache.org>" imported
gpg: key 0x36243EECE206BB0D: public key "Masatake Iwasaki (CODE SIGNING
KEY) <iw...@apache.org>" imported
*gpg: key 0xB362E1C021854B9D: public key "Wei-Chiu Chuang
<weichiu@apache.org <we...@apache.org>>" imported*

This time an import did work, but Wei-Chiu isn't trusted by anyone yet

gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: Good signature from "Wei-Chiu Chuang <we...@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the
owner.
Primary key fingerprint: CD32 D773 FF41 C3F9 E74B  DB7F B362 E1C0 2185 4B9D

(Wei-Chiu, let's coordinate signing each other's public keys via a slack
channel; you need to be in the apache web of trust)


> time gunzip hadoop-3.3.1-RC3.tar.gz

(5 seconds)

cd into the hadoop dir;
cp my confs in: cp ~/(somewhere)/hadoop-conf/*  etc/hadoop/
cp the hadoop-azure dependencies from share/hadoop/tools/lib/ to
share/hadoop/common/lib (products built targeting Azure put things there)

run: all the s3a "qualifying an AWS SDK update" commands
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update

run: basic abfs:// FS operations; again no problems.
FWIW I think we should consider having the hadoop-aws module and
dependencies, and the aws ones in hadoop-common/lib. I can get them there
through env vars and the s3guard shell sets things up, but azure is fiddly.

*Build and test cloudstore JAR; invoke from CLI*

This is my cloud-storage extension library
https://github.com/steveloughran/cloudstore

I've always intended to put it into hadoop, but as it is where a lot of
diagnostics and quick way to put together fixes "here's a faster du ("dux"")

https://github.com/steveloughran/cloudstore.git

modify the hadoop-3.3 profile to use 3.3.1 artifacts, then build with
snapshots enabled. Because I'd not (yet) built any 3.3.1 artifacts locally,
this fetched them from maven staging

mvn package -Phadoop-3.3 -Pextra -Psnapshots-and-staging


Set up env var $CLOUDSTORE to point to JAR; $BUCKET to s3a bucket, run
various commands (storediag, cloudup, ...). As an example, here's the "dux"
command, which is "hadoop fs -du" with parallel scan underneath the dir for
better scaling


bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose
s3a://stevel-london/

output is in
https://gist.github.com/steveloughran/664d30cef20f605f3164ad01f92a458a

*Build and (unit test) google GCS: *


Two test failures, one of which was classpath related and the other just a
new rename contract test needing a new setting in gs.xml to declare what
rename of file over file does.

Everything is covered in:
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

Classpath: assertJ not coming through hadoop-common-test JAR dependencies.

[ERROR]
com.google.cloud.hadoop.fs.gcs.contract.TestInMemoryGoogleContractRootDirectory.testSimpleRootListing
 Time elapsed: 0.093 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/assertj/core/api/Assertions
Caused by: java.lang.ClassNotFoundException: org.assertj.core.api.Assertions


Happens because I added some tests to the AbstractContractRenameTest which
use assertJ assertions.
Assertj is declared in test scope for hadoop-common test JAR, it's somehow
not propagating. HBoss has the same issue.


    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <scope>test</scope>
    </dependency>

I really don't understand what is up with our declared exports; just
reviewed them. Nothing we can do about it that I can see.

Rename test failure is from a new test, with the expected behaviour needing
definition.

[ERROR] Failures:
[ERROR]
TestInMemoryGoogleContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:89
expected
rename(gs://fake-in-memory-test-bucket/contract-test/source-256.txt,
gs://fake-in-memory-test-bucket/contract-test/dest-512.txt) to be rejected
with exception, but got false

Fix, add "fs.contract.rename-returns-false-if-dest-exists" = true to the
XML contract.

*Build and test HBoss*

This is  the HBase extension to use ZK to lock file accesses on S3

I've broken their build through to changes to the internal S3 client
factory as some new client options were passed down (HADOOP-13551).
That change moved to a new build parameter object, so we can add future
changes without breaking the signature again (mehakmeet already has in
HADOOP-17705)

https://issues.apache.org/jira/browse/HBASE-25900


Got an initial PR up, though will need to do more so that it will also
compile/test against older builds
https://github.com/apache/hbase-filesystem/pull/23

*Build spark, then test S3A Committers through it*

Build spark-3 against 3.3.1, then ran integration tests against S3 london

Test are in: https://github.com/hortonworks-spark/cloud-integration.git

Most of an afternoon was frittered away dealing with the fact that the
spark version move (2.4 to 3.2) meant scalatest upgrade from 3.0 to 3.20 **and
every single test failed to compile because the scalatest project moved the
foundational test suite into a new package**. I had to do that same upgrade
to test my WiP manifest committer (MAPREDUCE-7341) against ABFS, so it's
not completely wasted. It does mean that module and tests is scala 3+ only.

hadoop-aws and hadoop-azure test suites

For these I checked out branch-3.3.1, rebuilt it and ran the test suites in
the hadoop-azure and hadoop-aws modules. This triggered a rebuild of those
two modules.

I did this after doing all the other checks, so everything else was
qualified against the genuine RC3 artifacts.


hadoop-aws

run 1: -Dparallel-tests -DtestsThreadCount=5 -Dmarkers=keep
run 2: -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Dscale -Ddynamo

azure
 -Dparallel-tests=abfs -DtestsThreadCount=5 -Dscale

 [ERROR] Errors:
 [ERROR]
ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:642
» TestTimedOut
 [INFO]

This is https://issues.apache.org/jira/browse/HADOOP-17628


Overall then:

   1. All production code good.
   2. some expansion of filesystem tests require some changes downstream,
   and the change in the S3Client from HADOOP-13551 the HBoss tests using an
   internal interface from compiling. The move to a parameter object (and
   documenting this use) is intended to prevent this reoccurring.


>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by "epayne@apache.org" <ep...@apache.org>.
+1 (binding)
Eric


   On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
 
 Hi community,

This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
issues have been resolved [1] again.

There are 2 additional issues resolved for RC3:
* Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
HADOOP-16878
* Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
and destination are the same

There are 4 issues resolved for RC2:
* HADOOP-17666. Update LICENSE for 3.3.1
* MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
* Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
* HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)

The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
fixes compared to hadoop-thirdparty 1.1.0:
* HADOOP-17707. Remove jaeger document from site index.
* HADOOP-17730. Add back error_prone

*RC tag is release-3.3.1-RC3
https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3

*The RC3 artifacts are at*:
https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/

*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1320/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS


Things I've verified:
* all blocker issues targeting 3.3.1 have been resolved.
* stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
* LICENSE and NOTICE files checked
* RELEASENOTES and CHANGELOG
* rat check passed.
* Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
* Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
* Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
dependency divergence. Issues are being identified but so far nothing
blocker for Hadoop itself.

Please try the release and vote. The vote will run for 5 days.

My +1 to start,

[1] https://issues.apache.org/jira/issues/?filter=12350491
[2]
https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@apache.org>.
Hello,
do we want to extend the release vote? I understand a big release like this
takes time to validate.

I am aware a number of people are testing it: Attila tested Ozone on Hadoop
3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
over Hadoop 3.2 library.

Looks like we may need some more time to test. How about extending it by a
week?

On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
+1, binding.

Awesome piece of work!

I've done three forms of qualification, all related to s3 and azure storage

   1. tarball validate, CLI use
   2. build/test of downstream modules off maven artifacts; mine and some
   other ASF ones. I  (and it its very much me) have broken some downstream
   modules tests, as I will discuss below. PRs submitted to the relevant
   projects
   3. local rerun of the hadoop-aws and hadoop-azure test suites


*Regarding issues which surfaced*

Wei-Chiu: can you register your private GPG key with the public keystores?
The gpg client apps let you do this? Then we can coordinate signing each
other's keys

Filed PRs for the test regressions:
https://github.com/apache/hbase-filesystem/pull/23
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

*Artifact validation*

SHA checksum good:


shasum -a 512 hadoop-3.3.1-RC3.tar.gz
b80e0a8785b0f3d75d9db54340123872e39bad72cc60de5d263ae22024720e6e824e022090f01e248bf105e03b0f06163729adbe15b5b0978bae0447571e22eb
 hadoop-3.3.1-RC3.tar.gz


GPG: trickier, because Wei-Chiu wasn't trusted

> gpg --verify hadoop-3.3.1-RC3.tar.gz.asc

gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: requesting key 0xB362E1C021854B9D from hkps server
hkps.pool.sks-keyservers.net
gpg: Can't check signature: No public key


*Wei-Chiu: can you add your public keys to the GPG key servers*

To validate the keys I went to the directory where I have our site under
svn (https://dist.apache.org/repos/dist/release/hadoop/common) , and, after
reinstalling svn (where did it go? when did it go?) did an svn update to
get the keys

Did a gpg import of the KEYS file, added

gpg: key 0x386D80EF81E7469A: public key "Brahma Reddy Battula (CODE SIGNING
KEY) <br...@apache.org>" imported
gpg: key 0xFC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <
sammichen@apache.org>" imported
gpg: key 0x36243EECE206BB0D: public key "Masatake Iwasaki (CODE SIGNING
KEY) <iw...@apache.org>" imported
*gpg: key 0xB362E1C021854B9D: public key "Wei-Chiu Chuang
<weichiu@apache.org <we...@apache.org>>" imported*

This time an import did work, but Wei-Chiu isn't trusted by anyone yet

gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:                using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: Good signature from "Wei-Chiu Chuang <we...@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the
owner.
Primary key fingerprint: CD32 D773 FF41 C3F9 E74B  DB7F B362 E1C0 2185 4B9D

(Wei-Chiu, let's coordinate signing each other's public keys via a slack
channel; you need to be in the apache web of trust)


> time gunzip hadoop-3.3.1-RC3.tar.gz

(5 seconds)

cd into the hadoop dir;
cp my confs in: cp ~/(somewhere)/hadoop-conf/*  etc/hadoop/
cp the hadoop-azure dependencies from share/hadoop/tools/lib/ to
share/hadoop/common/lib (products built targeting Azure put things there)

run: all the s3a "qualifying an AWS SDK update" commands
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update

run: basic abfs:// FS operations; again no problems.
FWIW I think we should consider having the hadoop-aws module and
dependencies, and the aws ones in hadoop-common/lib. I can get them there
through env vars and the s3guard shell sets things up, but azure is fiddly.

*Build and test cloudstore JAR; invoke from CLI*

This is my cloud-storage extension library
https://github.com/steveloughran/cloudstore

I've always intended to put it into hadoop, but as it is where a lot of
diagnostics and quick way to put together fixes "here's a faster du ("dux"")

https://github.com/steveloughran/cloudstore.git

modify the hadoop-3.3 profile to use 3.3.1 artifacts, then build with
snapshots enabled. Because I'd not (yet) built any 3.3.1 artifacts locally,
this fetched them from maven staging

mvn package -Phadoop-3.3 -Pextra -Psnapshots-and-staging


Set up env var $CLOUDSTORE to point to JAR; $BUCKET to s3a bucket, run
various commands (storediag, cloudup, ...). As an example, here's the "dux"
command, which is "hadoop fs -du" with parallel scan underneath the dir for
better scaling


bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose
s3a://stevel-london/

output is in
https://gist.github.com/steveloughran/664d30cef20f605f3164ad01f92a458a

*Build and (unit test) google GCS: *


Two test failures, one of which was classpath related and the other just a
new rename contract test needing a new setting in gs.xml to declare what
rename of file over file does.

Everything is covered in:
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

Classpath: assertJ not coming through hadoop-common-test JAR dependencies.

[ERROR]
com.google.cloud.hadoop.fs.gcs.contract.TestInMemoryGoogleContractRootDirectory.testSimpleRootListing
 Time elapsed: 0.093 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/assertj/core/api/Assertions
Caused by: java.lang.ClassNotFoundException: org.assertj.core.api.Assertions


Happens because I added some tests to the AbstractContractRenameTest which
use assertJ assertions.
Assertj is declared in test scope for hadoop-common test JAR, it's somehow
not propagating. HBoss has the same issue.


    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <scope>test</scope>
    </dependency>

I really don't understand what is up with our declared exports; just
reviewed them. Nothing we can do about it that I can see.

Rename test failure is from a new test, with the expected behaviour needing
definition.

[ERROR] Failures:
[ERROR]
TestInMemoryGoogleContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:89
expected
rename(gs://fake-in-memory-test-bucket/contract-test/source-256.txt,
gs://fake-in-memory-test-bucket/contract-test/dest-512.txt) to be rejected
with exception, but got false

Fix, add "fs.contract.rename-returns-false-if-dest-exists" = true to the
XML contract.

*Build and test HBoss*

This is  the HBase extension to use ZK to lock file accesses on S3

I've broken their build through to changes to the internal S3 client
factory as some new client options were passed down (HADOOP-13551).
That change moved to a new build parameter object, so we can add future
changes without breaking the signature again (mehakmeet already has in
HADOOP-17705)

https://issues.apache.org/jira/browse/HBASE-25900


Got an initial PR up, though will need to do more so that it will also
compile/test against older builds
https://github.com/apache/hbase-filesystem/pull/23

*Build spark, then test S3A Committers through it*

Build spark-3 against 3.3.1, then ran integration tests against S3 london

Test are in: https://github.com/hortonworks-spark/cloud-integration.git

Most of an afternoon was frittered away dealing with the fact that the
spark version move (2.4 to 3.2) meant scalatest upgrade from 3.0 to 3.20 **and
every single test failed to compile because the scalatest project moved the
foundational test suite into a new package**. I had to do that same upgrade
to test my WiP manifest committer (MAPREDUCE-7341) against ABFS, so it's
not completely wasted. It does mean that module and tests is scala 3+ only.

hadoop-aws and hadoop-azure test suites

For these I checked out branch-3.3.1, rebuilt it and ran the test suites in
the hadoop-azure and hadoop-aws modules. This triggered a rebuild of those
two modules.

I did this after doing all the other checks, so everything else was
qualified against the genuine RC3 artifacts.


hadoop-aws

run 1: -Dparallel-tests -DtestsThreadCount=5 -Dmarkers=keep
run 2: -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Dscale -Ddynamo

azure
 -Dparallel-tests=abfs -DtestsThreadCount=5 -Dscale

 [ERROR] Errors:
 [ERROR]
ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:642
» TestTimedOut
 [INFO]

This is https://issues.apache.org/jira/browse/HADOOP-17628


Overall then:

   1. All production code good.
   2. some expansion of filesystem tests require some changes downstream,
   and the change in the S3Client from HADOOP-13551 the HBoss tests using an
   internal interface from compiling. The move to a parameter object (and
   documenting this use) is intended to prevent this reoccurring.


>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by "epayne@apache.org" <ep...@apache.org>.
+1 (binding)
Eric


   On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
 
 Hi community,

This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
issues have been resolved [1] again.

There are 2 additional issues resolved for RC3:
* Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
HADOOP-16878
* Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
and destination are the same

There are 4 issues resolved for RC2:
* HADOOP-17666. Update LICENSE for 3.3.1
* MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
* Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
* HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)

The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
fixes compared to hadoop-thirdparty 1.1.0:
* HADOOP-17707. Remove jaeger document from site index.
* HADOOP-17730. Add back error_prone

*RC tag is release-3.3.1-RC3
https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3

*The RC3 artifacts are at*:
https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/

*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1320/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS


Things I've verified:
* all blocker issues targeting 3.3.1 have been resolved.
* stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
* LICENSE and NOTICE files checked
* RELEASENOTES and CHANGELOG
* rat check passed.
* Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
* Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
* Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
dependency divergence. Issues are being identified but so far nothing
blocker for Hadoop itself.

Please try the release and vote. The vote will run for 5 days.

My +1 to start,

[1] https://issues.apache.org/jira/issues/?filter=12350491
[2]
https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by "epayne@apache.org" <ep...@apache.org>.
+1 (binding)
Eric


   On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
 
 Hi community,

This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
issues have been resolved [1] again.

There are 2 additional issues resolved for RC3:
* Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
HADOOP-16878
* Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
and destination are the same

There are 4 issues resolved for RC2:
* HADOOP-17666. Update LICENSE for 3.3.1
* MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
* Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
* HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)

The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
fixes compared to hadoop-thirdparty 1.1.0:
* HADOOP-17707. Remove jaeger document from site index.
* HADOOP-17730. Add back error_prone

*RC tag is release-3.3.1-RC3
https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3

*The RC3 artifacts are at*:
https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/

*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1320/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS


Things I've verified:
* all blocker issues targeting 3.3.1 have been resolved.
* stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
* LICENSE and NOTICE files checked
* RELEASENOTES and CHANGELOG
* rat check passed.
* Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
* Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
* Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
dependency divergence. Issues are being identified but so far nothing
blocker for Hadoop itself.

Please try the release and vote. The vote will run for 5 days.

My +1 to start,

[1] https://issues.apache.org/jira/issues/?filter=12350491
[2]
https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Wei-Chiu Chuang <we...@apache.org>.
Hello,
do we want to extend the release vote? I understand a big release like this
takes time to validate.

I am aware a number of people are testing it: Attila tested Ozone on Hadoop
3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
over Hadoop 3.2 library.

Looks like we may need some more time to test. How about extending it by a
week?

On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang <we...@apache.org> wrote:

> Hi community,
>
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
>
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
>
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
>
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
>
> Please try the release and vote. The vote will run for 5 days.
>
> My +1 to start,
>
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
I found that the top page of site documentation (index.html on the top directory)
has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/).

While I think it is not worth for sinking the RC3,
you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/)
if you need to cut RC4.

Thanks,
Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
+1

Thanks for the great work, Wei-Chiu Chuang.

* verified signature and checksum.
* built site documentation by `mvn site` and skimmed the contents.
   # found that top-level index.html is not updated.
* built on CentOS 8 (x86_64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.
   * launched pseudo cluster with security enabled and ran sample MR jobs.
   * launched 3-nodes cluster with NN-HA and RM-HA and ran sample MR jobs.
* built on CentOS 7 (aarch64) and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`.

* built Hive with the patch of HIVE-24484 against hadoop-3.3.1
   and ran TestMiniLlapCliDrivera (fixed by HDFS-15790).

Masatake Iwasaki

On 2021/06/01 19:29, Wei-Chiu Chuang wrote:
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

Posted by "epayne@apache.org" <ep...@apache.org>.
+1 (binding)
Eric


   On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <we...@apache.org> wrote:  
 
 Hi community,

This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
issues have been resolved [1] again.

There are 2 additional issues resolved for RC3:
* Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
HADOOP-16878
* Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
and destination are the same

There are 4 issues resolved for RC2:
* HADOOP-17666. Update LICENSE for 3.3.1
* MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
* Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
* HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)

The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
fixes compared to hadoop-thirdparty 1.1.0:
* HADOOP-17707. Remove jaeger document from site index.
* HADOOP-17730. Add back error_prone

*RC tag is release-3.3.1-RC3
https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3

*The RC3 artifacts are at*:
https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/

*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1320/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS


Things I've verified:
* all blocker issues targeting 3.3.1 have been resolved.
* stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
* LICENSE and NOTICE files checked
* RELEASENOTES and CHANGELOG
* rat check passed.
* Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
* Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
* Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
dependency divergence. Issues are being identified but so far nothing
blocker for Hadoop itself.

Please try the release and vote. The vote will run for 5 days.

My +1 to start,

[1] https://issues.apache.org/jira/issues/?filter=12350491
[2]
https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3