You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Masatake Iwasaki <iw...@oss.nttdata.com> on 2022/07/12 16:14:24 UTC
[VOTE] Release Apache Hadoop 3.2.4 - RC0
Hi all,
Here's Hadoop 3.2.4 release candidate #0:
The RC is available at:
https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
The RC tag is at:
https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
The Maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1354
You can find my public key at:
https://downloads.apache.org/hadoop/common/KEYS
Please evaluate the RC and vote.
The vote will be open for (at least) 5 days.
Thanks,
Masatake Iwasaki
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
The vote passed with the following result.
Binding PMC members:
+1 Chris Nauroth
+1 Ayush Saxena
+1 Masatake Iwasaki
Non binding votes:
+1 Ashutosh Gupta
I'm going to publish the artifacts, site docs, maven artifacts, then announce.
Thank you to all who helped to push this release out.
Masatake Iwasaki
On 2022/07/22 4:58, Ayush Saxena wrote:
> +1(Binding)
>
> * Built from Source
> * Successful native build on Ubuntu 18.04
> * Verified Checksums
> * Verified Signatures
> * Successful RAT check
> * Ran Basic HDFS shell commands
> * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
> * Browsed through UI(NN, DN, RM, NM & JHS)
> * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
>
> Thanx Masatake for driving the release, Good Luck!!!
>
> -Ayush
>
> On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
>
>> I'm changing my vote to +1 (binding).
>>
>> Masatake and Ashutosh, thank you for investigating.
>>
>> I reran tests without the parallel options, and that mostly addressed the
>> failures. Maybe the tests in question are just not sufficiently isolated to
>> support parallel execution. That looks to be the case for TestFsck, where
>> the failure was caused by missing audit log entries. This test works by
>> toggling global logging state, so I can see why multi-threaded execution
>> might confuse the test.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
>> ashutoshgupta118@gmail.com>
>> wrote:
>>
>>> +1(non-binding)
>>>
>>> * Builds from source look good.
>>> * Checksums and signatures are correct.
>>> * Running basic HDFS and MapReduce commands looks good.
>>>
>>>> * TestAMRMProxy - Not able to reproduce in local
>>>> * TestFsck - I can see failure only I can see is
>>> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
>>> HDFS-15038
>>>> * TestSLSStreamAMSynth - Not able to reproduce in local
>>>> * TestServiceAM - Not able to reproduce in local
>>>
>>> Thanks Masatake for driving this release.
>>>
>>> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
>>> iwasakims@oss.nttdata.com>
>>> wrote:
>>>
>>>> Hi developers,
>>>>
>>>> I'm still waiting for your vote.
>>>> I'm considering the intermittent test failures mentioned by Chris are
>> not
>>>> blocker.
>>>> Please file a JIRA and let me know if you find a blocker issue.
>>>>
>>>> I will appreciate your help for the release process.
>>>>
>>>> Regards,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/20 14:50, Masatake Iwasaki wrote:
>>>>>> TestServiceAM
>>>>>
>>>>> I can see the reported failure of TestServiceAM in some "Apache
>> Hadoop
>>>> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
>>>>> 3.3.0 and above might be fixed by YARN-8867 which added guard using
>>>> GenericTestUtils#waitFor for stabilizing the
>>>> testContainersReleasedWhenPreLaunchFails.
>>>>> YARN 8867 did not modified other code under hadoop-yarn-services.
>>>>> If it is the case, TestServiceAM can be tagged as flaky in
>> branch-3.2.
>>>>>
>>>>>
>>>>> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>>>>>> Thanks for testing the RC0, Chris.
>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>
>>>>>> I could not reproduce the test failures on my local.
>>>>>>
>>>>>> For TestFsck, if the failed test case is
>>>> testFsckListCorruptSnapshotFiles,
>>>>>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>>>>>
>>>>>> The failure of TestSLSStreamAMSynth looks frequently reported by
>>>>>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>>>>>> It could be tagged as known flaky test.
>>>>>>
>>>>>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>>>>>> -0 (binding)
>>>>>>>
>>>>>>> * Verified all checksums.
>>>>>>> * Verified all signatures.
>>>>>>> * Built from source, including native code on Linux.
>>>>>>> * mvn clean package -Pnative -Psrc -Drequire.openssl
>>>> -Drequire.snappy
>>>>>>> -Drequire.zstd -DskipTests
>>>>>>> * Tests mostly passed, but see below.
>>>>>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>>>>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>>>>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>>
>>>>>>> The following tests also failed, but they also fail for me on
>> 3.2.3,
>>> so
>>>>>>> they aren't likely to be related to this release candidate:
>>>>>>> * TestCapacitySchedulerNodeLabelUpdate
>>>>>>> * TestFrameworkUploader
>>>>>>> * TestSLSGenericSynth
>>>>>>> * TestSLSRunner
>>>>>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>>>>>
>>>>>>> I'm not voting a full -1, because I haven't done any root cause
>>>> analysis on
>>>>>>> these new test failures. I don't know if it's a quirk to my
>>>> environment,
>>>>>>> though I'm using the start-build-env.sh Docker container, so any
>>> build
>>>>>>> dependencies should be consistent. I'd be comfortable moving ahead
>> if
>>>>>>> others are seeing these tests pass.
>>>>>>>
>>>>>>> Chris Nauroth
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
>>>> iwasakims@oss.nttdata.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 from myself.
>>>>>>>>
>>>>>>>> * skimmed the contents of site documentation.
>>>>>>>>
>>>>>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
>>> with
>>>>>>>> `-Pnative`.
>>>>>>>>
>>>>>>>> * launched pseudo distributed cluster including kms and httpfs
>> with
>>>>>>>> Kerberos and SSL enabled.
>>>>>>>>
>>>>>>>> * created encryption zone, put and read files via httpfs.
>>>>>>>> * ran example MR wordcount over encryption zone.
>>>>>>>>
>>>>>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
>>> ran
>>>> some
>>>>>>>> example MR jobs.
>>>>>>>>
>>>>>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
>>> 3.2.4
>>>> RC0
>>>>>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
>>>> smoke-tests.
>>>>>>>> https://github.com/apache/bigtop/pull/942
>>>>>>>>
>>>>>>>> * Hive needs updating exclusion rule to address HADOOP-18088
>>>> (migration
>>>>>>>> to reload4j).
>>>>>>>>
>>>>>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
>>>> repository::
>>>>>>>>
>>>>>>>> <repository>
>>>>>>>> <id>staged</id>
>>>>>>>> <name>staged-releases</name>
>>>>>>>> <url>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>> </url>
>>>>>>>> <releases>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </releases>
>>>>>>>> <snapshots>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </snapshots>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Masatake Iwasaki
>>>>>>>>
>>>>>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>>>>>
>>>>>>>>> The RC is available at:
>>>>>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>>>>>
>>>>>>>>> The RC tag is at:
>>>>>>>>>
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>>>>>
>>>>>>>>> The Maven artifacts are staged at:
>>>>>>>>>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>>>
>>>>>>>>> You can find my public key at:
>>>>>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>>>>>
>>>>>>>>> Please evaluate the RC and vote.
>>>>>>>>> The vote will be open for (at least) 5 days.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Masatake Iwasaki
>>>>>>>>>
>>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>>>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
The vote passed with the following result.
Binding PMC members:
+1 Chris Nauroth
+1 Ayush Saxena
+1 Masatake Iwasaki
Non binding votes:
+1 Ashutosh Gupta
I'm going to publish the artifacts, site docs, maven artifacts, then announce.
Thank you to all who helped to push this release out.
Masatake Iwasaki
On 2022/07/22 4:58, Ayush Saxena wrote:
> +1(Binding)
>
> * Built from Source
> * Successful native build on Ubuntu 18.04
> * Verified Checksums
> * Verified Signatures
> * Successful RAT check
> * Ran Basic HDFS shell commands
> * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
> * Browsed through UI(NN, DN, RM, NM & JHS)
> * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
>
> Thanx Masatake for driving the release, Good Luck!!!
>
> -Ayush
>
> On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
>
>> I'm changing my vote to +1 (binding).
>>
>> Masatake and Ashutosh, thank you for investigating.
>>
>> I reran tests without the parallel options, and that mostly addressed the
>> failures. Maybe the tests in question are just not sufficiently isolated to
>> support parallel execution. That looks to be the case for TestFsck, where
>> the failure was caused by missing audit log entries. This test works by
>> toggling global logging state, so I can see why multi-threaded execution
>> might confuse the test.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
>> ashutoshgupta118@gmail.com>
>> wrote:
>>
>>> +1(non-binding)
>>>
>>> * Builds from source look good.
>>> * Checksums and signatures are correct.
>>> * Running basic HDFS and MapReduce commands looks good.
>>>
>>>> * TestAMRMProxy - Not able to reproduce in local
>>>> * TestFsck - I can see failure only I can see is
>>> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
>>> HDFS-15038
>>>> * TestSLSStreamAMSynth - Not able to reproduce in local
>>>> * TestServiceAM - Not able to reproduce in local
>>>
>>> Thanks Masatake for driving this release.
>>>
>>> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
>>> iwasakims@oss.nttdata.com>
>>> wrote:
>>>
>>>> Hi developers,
>>>>
>>>> I'm still waiting for your vote.
>>>> I'm considering the intermittent test failures mentioned by Chris are
>> not
>>>> blocker.
>>>> Please file a JIRA and let me know if you find a blocker issue.
>>>>
>>>> I will appreciate your help for the release process.
>>>>
>>>> Regards,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/20 14:50, Masatake Iwasaki wrote:
>>>>>> TestServiceAM
>>>>>
>>>>> I can see the reported failure of TestServiceAM in some "Apache
>> Hadoop
>>>> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
>>>>> 3.3.0 and above might be fixed by YARN-8867 which added guard using
>>>> GenericTestUtils#waitFor for stabilizing the
>>>> testContainersReleasedWhenPreLaunchFails.
>>>>> YARN 8867 did not modified other code under hadoop-yarn-services.
>>>>> If it is the case, TestServiceAM can be tagged as flaky in
>> branch-3.2.
>>>>>
>>>>>
>>>>> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>>>>>> Thanks for testing the RC0, Chris.
>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>
>>>>>> I could not reproduce the test failures on my local.
>>>>>>
>>>>>> For TestFsck, if the failed test case is
>>>> testFsckListCorruptSnapshotFiles,
>>>>>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>>>>>
>>>>>> The failure of TestSLSStreamAMSynth looks frequently reported by
>>>>>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>>>>>> It could be tagged as known flaky test.
>>>>>>
>>>>>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>>>>>> -0 (binding)
>>>>>>>
>>>>>>> * Verified all checksums.
>>>>>>> * Verified all signatures.
>>>>>>> * Built from source, including native code on Linux.
>>>>>>> * mvn clean package -Pnative -Psrc -Drequire.openssl
>>>> -Drequire.snappy
>>>>>>> -Drequire.zstd -DskipTests
>>>>>>> * Tests mostly passed, but see below.
>>>>>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>>>>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>>>>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>>
>>>>>>> The following tests also failed, but they also fail for me on
>> 3.2.3,
>>> so
>>>>>>> they aren't likely to be related to this release candidate:
>>>>>>> * TestCapacitySchedulerNodeLabelUpdate
>>>>>>> * TestFrameworkUploader
>>>>>>> * TestSLSGenericSynth
>>>>>>> * TestSLSRunner
>>>>>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>>>>>
>>>>>>> I'm not voting a full -1, because I haven't done any root cause
>>>> analysis on
>>>>>>> these new test failures. I don't know if it's a quirk to my
>>>> environment,
>>>>>>> though I'm using the start-build-env.sh Docker container, so any
>>> build
>>>>>>> dependencies should be consistent. I'd be comfortable moving ahead
>> if
>>>>>>> others are seeing these tests pass.
>>>>>>>
>>>>>>> Chris Nauroth
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
>>>> iwasakims@oss.nttdata.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 from myself.
>>>>>>>>
>>>>>>>> * skimmed the contents of site documentation.
>>>>>>>>
>>>>>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
>>> with
>>>>>>>> `-Pnative`.
>>>>>>>>
>>>>>>>> * launched pseudo distributed cluster including kms and httpfs
>> with
>>>>>>>> Kerberos and SSL enabled.
>>>>>>>>
>>>>>>>> * created encryption zone, put and read files via httpfs.
>>>>>>>> * ran example MR wordcount over encryption zone.
>>>>>>>>
>>>>>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
>>> ran
>>>> some
>>>>>>>> example MR jobs.
>>>>>>>>
>>>>>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
>>> 3.2.4
>>>> RC0
>>>>>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
>>>> smoke-tests.
>>>>>>>> https://github.com/apache/bigtop/pull/942
>>>>>>>>
>>>>>>>> * Hive needs updating exclusion rule to address HADOOP-18088
>>>> (migration
>>>>>>>> to reload4j).
>>>>>>>>
>>>>>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
>>>> repository::
>>>>>>>>
>>>>>>>> <repository>
>>>>>>>> <id>staged</id>
>>>>>>>> <name>staged-releases</name>
>>>>>>>> <url>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>> </url>
>>>>>>>> <releases>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </releases>
>>>>>>>> <snapshots>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </snapshots>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Masatake Iwasaki
>>>>>>>>
>>>>>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>>>>>
>>>>>>>>> The RC is available at:
>>>>>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>>>>>
>>>>>>>>> The RC tag is at:
>>>>>>>>>
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>>>>>
>>>>>>>>> The Maven artifacts are staged at:
>>>>>>>>>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>>>
>>>>>>>>> You can find my public key at:
>>>>>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>>>>>
>>>>>>>>> Please evaluate the RC and vote.
>>>>>>>>> The vote will be open for (at least) 5 days.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Masatake Iwasaki
>>>>>>>>>
>>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>>>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
The vote passed with the following result.
Binding PMC members:
+1 Chris Nauroth
+1 Ayush Saxena
+1 Masatake Iwasaki
Non binding votes:
+1 Ashutosh Gupta
I'm going to publish the artifacts, site docs, maven artifacts, then announce.
Thank you to all who helped to push this release out.
Masatake Iwasaki
On 2022/07/22 4:58, Ayush Saxena wrote:
> +1(Binding)
>
> * Built from Source
> * Successful native build on Ubuntu 18.04
> * Verified Checksums
> * Verified Signatures
> * Successful RAT check
> * Ran Basic HDFS shell commands
> * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
> * Browsed through UI(NN, DN, RM, NM & JHS)
> * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
>
> Thanx Masatake for driving the release, Good Luck!!!
>
> -Ayush
>
> On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
>
>> I'm changing my vote to +1 (binding).
>>
>> Masatake and Ashutosh, thank you for investigating.
>>
>> I reran tests without the parallel options, and that mostly addressed the
>> failures. Maybe the tests in question are just not sufficiently isolated to
>> support parallel execution. That looks to be the case for TestFsck, where
>> the failure was caused by missing audit log entries. This test works by
>> toggling global logging state, so I can see why multi-threaded execution
>> might confuse the test.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
>> ashutoshgupta118@gmail.com>
>> wrote:
>>
>>> +1(non-binding)
>>>
>>> * Builds from source look good.
>>> * Checksums and signatures are correct.
>>> * Running basic HDFS and MapReduce commands looks good.
>>>
>>>> * TestAMRMProxy - Not able to reproduce in local
>>>> * TestFsck - I can see failure only I can see is
>>> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
>>> HDFS-15038
>>>> * TestSLSStreamAMSynth - Not able to reproduce in local
>>>> * TestServiceAM - Not able to reproduce in local
>>>
>>> Thanks Masatake for driving this release.
>>>
>>> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
>>> iwasakims@oss.nttdata.com>
>>> wrote:
>>>
>>>> Hi developers,
>>>>
>>>> I'm still waiting for your vote.
>>>> I'm considering the intermittent test failures mentioned by Chris are
>> not
>>>> blocker.
>>>> Please file a JIRA and let me know if you find a blocker issue.
>>>>
>>>> I will appreciate your help for the release process.
>>>>
>>>> Regards,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/20 14:50, Masatake Iwasaki wrote:
>>>>>> TestServiceAM
>>>>>
>>>>> I can see the reported failure of TestServiceAM in some "Apache
>> Hadoop
>>>> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
>>>>> 3.3.0 and above might be fixed by YARN-8867 which added guard using
>>>> GenericTestUtils#waitFor for stabilizing the
>>>> testContainersReleasedWhenPreLaunchFails.
>>>>> YARN 8867 did not modified other code under hadoop-yarn-services.
>>>>> If it is the case, TestServiceAM can be tagged as flaky in
>> branch-3.2.
>>>>>
>>>>>
>>>>> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>>>>>> Thanks for testing the RC0, Chris.
>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>
>>>>>> I could not reproduce the test failures on my local.
>>>>>>
>>>>>> For TestFsck, if the failed test case is
>>>> testFsckListCorruptSnapshotFiles,
>>>>>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>>>>>
>>>>>> The failure of TestSLSStreamAMSynth looks frequently reported by
>>>>>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>>>>>> It could be tagged as known flaky test.
>>>>>>
>>>>>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>>>>>> -0 (binding)
>>>>>>>
>>>>>>> * Verified all checksums.
>>>>>>> * Verified all signatures.
>>>>>>> * Built from source, including native code on Linux.
>>>>>>> * mvn clean package -Pnative -Psrc -Drequire.openssl
>>>> -Drequire.snappy
>>>>>>> -Drequire.zstd -DskipTests
>>>>>>> * Tests mostly passed, but see below.
>>>>>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>>>>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>>>>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>>
>>>>>>> The following tests also failed, but they also fail for me on
>> 3.2.3,
>>> so
>>>>>>> they aren't likely to be related to this release candidate:
>>>>>>> * TestCapacitySchedulerNodeLabelUpdate
>>>>>>> * TestFrameworkUploader
>>>>>>> * TestSLSGenericSynth
>>>>>>> * TestSLSRunner
>>>>>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>>>>>
>>>>>>> I'm not voting a full -1, because I haven't done any root cause
>>>> analysis on
>>>>>>> these new test failures. I don't know if it's a quirk to my
>>>> environment,
>>>>>>> though I'm using the start-build-env.sh Docker container, so any
>>> build
>>>>>>> dependencies should be consistent. I'd be comfortable moving ahead
>> if
>>>>>>> others are seeing these tests pass.
>>>>>>>
>>>>>>> Chris Nauroth
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
>>>> iwasakims@oss.nttdata.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 from myself.
>>>>>>>>
>>>>>>>> * skimmed the contents of site documentation.
>>>>>>>>
>>>>>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
>>> with
>>>>>>>> `-Pnative`.
>>>>>>>>
>>>>>>>> * launched pseudo distributed cluster including kms and httpfs
>> with
>>>>>>>> Kerberos and SSL enabled.
>>>>>>>>
>>>>>>>> * created encryption zone, put and read files via httpfs.
>>>>>>>> * ran example MR wordcount over encryption zone.
>>>>>>>>
>>>>>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
>>> ran
>>>> some
>>>>>>>> example MR jobs.
>>>>>>>>
>>>>>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
>>> 3.2.4
>>>> RC0
>>>>>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
>>>> smoke-tests.
>>>>>>>> https://github.com/apache/bigtop/pull/942
>>>>>>>>
>>>>>>>> * Hive needs updating exclusion rule to address HADOOP-18088
>>>> (migration
>>>>>>>> to reload4j).
>>>>>>>>
>>>>>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
>>>> repository::
>>>>>>>>
>>>>>>>> <repository>
>>>>>>>> <id>staged</id>
>>>>>>>> <name>staged-releases</name>
>>>>>>>> <url>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>> </url>
>>>>>>>> <releases>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </releases>
>>>>>>>> <snapshots>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </snapshots>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Masatake Iwasaki
>>>>>>>>
>>>>>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>>>>>
>>>>>>>>> The RC is available at:
>>>>>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>>>>>
>>>>>>>>> The RC tag is at:
>>>>>>>>>
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>>>>>
>>>>>>>>> The Maven artifacts are staged at:
>>>>>>>>>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>>>
>>>>>>>>> You can find my public key at:
>>>>>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>>>>>
>>>>>>>>> Please evaluate the RC and vote.
>>>>>>>>> The vote will be open for (at least) 5 days.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Masatake Iwasaki
>>>>>>>>>
>>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>>>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
The vote passed with the following result.
Binding PMC members:
+1 Chris Nauroth
+1 Ayush Saxena
+1 Masatake Iwasaki
Non binding votes:
+1 Ashutosh Gupta
I'm going to publish the artifacts, site docs, maven artifacts, then announce.
Thank you to all who helped to push this release out.
Masatake Iwasaki
On 2022/07/22 4:58, Ayush Saxena wrote:
> +1(Binding)
>
> * Built from Source
> * Successful native build on Ubuntu 18.04
> * Verified Checksums
> * Verified Signatures
> * Successful RAT check
> * Ran Basic HDFS shell commands
> * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
> * Browsed through UI(NN, DN, RM, NM & JHS)
> * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
>
> Thanx Masatake for driving the release, Good Luck!!!
>
> -Ayush
>
> On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
>
>> I'm changing my vote to +1 (binding).
>>
>> Masatake and Ashutosh, thank you for investigating.
>>
>> I reran tests without the parallel options, and that mostly addressed the
>> failures. Maybe the tests in question are just not sufficiently isolated to
>> support parallel execution. That looks to be the case for TestFsck, where
>> the failure was caused by missing audit log entries. This test works by
>> toggling global logging state, so I can see why multi-threaded execution
>> might confuse the test.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
>> ashutoshgupta118@gmail.com>
>> wrote:
>>
>>> +1(non-binding)
>>>
>>> * Builds from source look good.
>>> * Checksums and signatures are correct.
>>> * Running basic HDFS and MapReduce commands looks good.
>>>
>>>> * TestAMRMProxy - Not able to reproduce in local
>>>> * TestFsck - I can see failure only I can see is
>>> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
>>> HDFS-15038
>>>> * TestSLSStreamAMSynth - Not able to reproduce in local
>>>> * TestServiceAM - Not able to reproduce in local
>>>
>>> Thanks Masatake for driving this release.
>>>
>>> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
>>> iwasakims@oss.nttdata.com>
>>> wrote:
>>>
>>>> Hi developers,
>>>>
>>>> I'm still waiting for your vote.
>>>> I'm considering the intermittent test failures mentioned by Chris are
>> not
>>>> blocker.
>>>> Please file a JIRA and let me know if you find a blocker issue.
>>>>
>>>> I will appreciate your help for the release process.
>>>>
>>>> Regards,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/20 14:50, Masatake Iwasaki wrote:
>>>>>> TestServiceAM
>>>>>
>>>>> I can see the reported failure of TestServiceAM in some "Apache
>> Hadoop
>>>> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
>>>>> 3.3.0 and above might be fixed by YARN-8867 which added guard using
>>>> GenericTestUtils#waitFor for stabilizing the
>>>> testContainersReleasedWhenPreLaunchFails.
>>>>> YARN 8867 did not modified other code under hadoop-yarn-services.
>>>>> If it is the case, TestServiceAM can be tagged as flaky in
>> branch-3.2.
>>>>>
>>>>>
>>>>> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>>>>>> Thanks for testing the RC0, Chris.
>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>
>>>>>> I could not reproduce the test failures on my local.
>>>>>>
>>>>>> For TestFsck, if the failed test case is
>>>> testFsckListCorruptSnapshotFiles,
>>>>>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>>>>>
>>>>>> The failure of TestSLSStreamAMSynth looks frequently reported by
>>>>>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>>>>>> It could be tagged as known flaky test.
>>>>>>
>>>>>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>>>>>> -0 (binding)
>>>>>>>
>>>>>>> * Verified all checksums.
>>>>>>> * Verified all signatures.
>>>>>>> * Built from source, including native code on Linux.
>>>>>>> * mvn clean package -Pnative -Psrc -Drequire.openssl
>>>> -Drequire.snappy
>>>>>>> -Drequire.zstd -DskipTests
>>>>>>> * Tests mostly passed, but see below.
>>>>>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>>>>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>>>>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>>>>>
>>>>>>> The following are new test failures for me on 3.2.4:
>>>>>>> * TestAMRMProxy
>>>>>>> * TestFsck
>>>>>>> * TestSLSStreamAMSynth
>>>>>>> * TestServiceAM
>>>>>>>
>>>>>>> The following tests also failed, but they also fail for me on
>> 3.2.3,
>>> so
>>>>>>> they aren't likely to be related to this release candidate:
>>>>>>> * TestCapacitySchedulerNodeLabelUpdate
>>>>>>> * TestFrameworkUploader
>>>>>>> * TestSLSGenericSynth
>>>>>>> * TestSLSRunner
>>>>>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>>>>>
>>>>>>> I'm not voting a full -1, because I haven't done any root cause
>>>> analysis on
>>>>>>> these new test failures. I don't know if it's a quirk to my
>>>> environment,
>>>>>>> though I'm using the start-build-env.sh Docker container, so any
>>> build
>>>>>>> dependencies should be consistent. I'd be comfortable moving ahead
>> if
>>>>>>> others are seeing these tests pass.
>>>>>>>
>>>>>>> Chris Nauroth
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
>>>> iwasakims@oss.nttdata.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 from myself.
>>>>>>>>
>>>>>>>> * skimmed the contents of site documentation.
>>>>>>>>
>>>>>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
>>> with
>>>>>>>> `-Pnative`.
>>>>>>>>
>>>>>>>> * launched pseudo distributed cluster including kms and httpfs
>> with
>>>>>>>> Kerberos and SSL enabled.
>>>>>>>>
>>>>>>>> * created encryption zone, put and read files via httpfs.
>>>>>>>> * ran example MR wordcount over encryption zone.
>>>>>>>>
>>>>>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
>>> ran
>>>> some
>>>>>>>> example MR jobs.
>>>>>>>>
>>>>>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
>>> 3.2.4
>>>> RC0
>>>>>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
>>>> smoke-tests.
>>>>>>>> https://github.com/apache/bigtop/pull/942
>>>>>>>>
>>>>>>>> * Hive needs updating exclusion rule to address HADOOP-18088
>>>> (migration
>>>>>>>> to reload4j).
>>>>>>>>
>>>>>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
>>>> repository::
>>>>>>>>
>>>>>>>> <repository>
>>>>>>>> <id>staged</id>
>>>>>>>> <name>staged-releases</name>
>>>>>>>> <url>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>> </url>
>>>>>>>> <releases>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </releases>
>>>>>>>> <snapshots>
>>>>>>>> <enabled>true</enabled>
>>>>>>>> </snapshots>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Masatake Iwasaki
>>>>>>>>
>>>>>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>>>>>
>>>>>>>>> The RC is available at:
>>>>>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>>>>>
>>>>>>>>> The RC tag is at:
>>>>>>>>>
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>>>>>
>>>>>>>>> The Maven artifacts are staged at:
>>>>>>>>>
>>>>>>>>
>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>>>>>
>>>>>>>>> You can find my public key at:
>>>>>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>>>>>
>>>>>>>>> Please evaluate the RC and vote.
>>>>>>>>> The vote will be open for (at least) 5 days.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Masatake Iwasaki
>>>>>>>>>
>>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>>>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ayush Saxena <ay...@gmail.com>.
+1(Binding)
* Built from Source
* Successful native build on Ubuntu 18.04
* Verified Checksums
* Verified Signatures
* Successful RAT check
* Ran Basic HDFS shell commands
* Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
* Browsed through UI(NN, DN, RM, NM & JHS)
* Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
Thanx Masatake for driving the release, Good Luck!!!
-Ayush
On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
> I'm changing my vote to +1 (binding).
>
> Masatake and Ashutosh, thank you for investigating.
>
> I reran tests without the parallel options, and that mostly addressed the
> failures. Maybe the tests in question are just not sufficiently isolated to
> support parallel execution. That looks to be the case for TestFsck, where
> the failure was caused by missing audit log entries. This test works by
> toggling global logging state, so I can see why multi-threaded execution
> might confuse the test.
>
> Chris Nauroth
>
>
> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
> ashutoshgupta118@gmail.com>
> wrote:
>
> > +1(non-binding)
> >
> > * Builds from source look good.
> > * Checksums and signatures are correct.
> > * Running basic HDFS and MapReduce commands looks good.
> >
> > > * TestAMRMProxy - Not able to reproduce in local
> > > * TestFsck - I can see failure only I can see is
> > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> > HDFS-15038
> > > * TestSLSStreamAMSynth - Not able to reproduce in local
> > > * TestServiceAM - Not able to reproduce in local
> >
> > Thanks Masatake for driving this release.
> >
> > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > wrote:
> >
> > > Hi developers,
> > >
> > > I'm still waiting for your vote.
> > > I'm considering the intermittent test failures mentioned by Chris are
> not
> > > blocker.
> > > Please file a JIRA and let me know if you find a blocker issue.
> > >
> > > I will appreciate your help for the release process.
> > >
> > > Regards,
> > > Masatake Iwasaki
> > >
> > > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > > >> TestServiceAM
> > > >
> > > > I can see the reported failure of TestServiceAM in some "Apache
> Hadoop
> > > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > > GenericTestUtils#waitFor for stabilizing the
> > > testContainersReleasedWhenPreLaunchFails.
> > > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > > If it is the case, TestServiceAM can be tagged as flaky in
> branch-3.2.
> > > >
> > > >
> > > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > > >> Thanks for testing the RC0, Chris.
> > > >>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>
> > > >> I could not reproduce the test failures on my local.
> > > >>
> > > >> For TestFsck, if the failed test case is
> > > testFsckListCorruptSnapshotFiles,
> > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > > >>
> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > > >> It could be tagged as known flaky test.
> > > >>
> > > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > > >>> -0 (binding)
> > > >>>
> > > >>> * Verified all checksums.
> > > >>> * Verified all signatures.
> > > >>> * Built from source, including native code on Linux.
> > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > > -Drequire.snappy
> > > >>> -Drequire.zstd -DskipTests
> > > >>> * Tests mostly passed, but see below.
> > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > > >>>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>>
> > > >>> The following tests also failed, but they also fail for me on
> 3.2.3,
> > so
> > > >>> they aren't likely to be related to this release candidate:
> > > >>> * TestCapacitySchedulerNodeLabelUpdate
> > > >>> * TestFrameworkUploader
> > > >>> * TestSLSGenericSynth
> > > >>> * TestSLSRunner
> > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > > >>>
> > > >>> I'm not voting a full -1, because I haven't done any root cause
> > > analysis on
> > > >>> these new test failures. I don't know if it's a quirk to my
> > > environment,
> > > >>> though I'm using the start-build-env.sh Docker container, so any
> > build
> > > >>> dependencies should be consistent. I'd be comfortable moving ahead
> if
> > > >>> others are seeing these tests pass.
> > > >>>
> > > >>> Chris Nauroth
> > > >>>
> > > >>>
> > > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > > iwasakims@oss.nttdata.com>
> > > >>> wrote:
> > > >>>
> > > >>>> +1 from myself.
> > > >>>>
> > > >>>> * skimmed the contents of site documentation.
> > > >>>>
> > > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> > with
> > > >>>> `-Pnative`.
> > > >>>>
> > > >>>> * launched pseudo distributed cluster including kms and httpfs
> with
> > > >>>> Kerberos and SSL enabled.
> > > >>>>
> > > >>>> * created encryption zone, put and read files via httpfs.
> > > >>>> * ran example MR wordcount over encryption zone.
> > > >>>>
> > > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> > ran
> > > some
> > > >>>> example MR jobs.
> > > >>>>
> > > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> > 3.2.4
> > > RC0
> > > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > > smoke-tests.
> > > >>>> https://github.com/apache/bigtop/pull/942
> > > >>>>
> > > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > > (migration
> > > >>>> to reload4j).
> > > >>>>
> > > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > > repository::
> > > >>>>
> > > >>>> <repository>
> > > >>>> <id>staged</id>
> > > >>>> <name>staged-releases</name>
> > > >>>> <url>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>> </url>
> > > >>>> <releases>
> > > >>>> <enabled>true</enabled>
> > > >>>> </releases>
> > > >>>> <snapshots>
> > > >>>> <enabled>true</enabled>
> > > >>>> </snapshots>
> > > >>>> </repository>
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Masatake Iwasaki
> > > >>>>
> > > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > > >>>>>
> > > >>>>> The RC is available at:
> > > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > > >>>>>
> > > >>>>> The RC tag is at:
> > > >>>>>
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > > >>>>>
> > > >>>>> The Maven artifacts are staged at:
> > > >>>>>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>>>
> > > >>>>> You can find my public key at:
> > > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > > >>>>>
> > > >>>>> Please evaluate the RC and vote.
> > > >>>>> The vote will be open for (at least) 5 days.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Masatake Iwasaki
> > > >>>>>
> > > >>>>>
> > ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ayush Saxena <ay...@gmail.com>.
+1(Binding)
* Built from Source
* Successful native build on Ubuntu 18.04
* Verified Checksums
* Verified Signatures
* Successful RAT check
* Ran Basic HDFS shell commands
* Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
* Browsed through UI(NN, DN, RM, NM & JHS)
* Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
Thanx Masatake for driving the release, Good Luck!!!
-Ayush
On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
> I'm changing my vote to +1 (binding).
>
> Masatake and Ashutosh, thank you for investigating.
>
> I reran tests without the parallel options, and that mostly addressed the
> failures. Maybe the tests in question are just not sufficiently isolated to
> support parallel execution. That looks to be the case for TestFsck, where
> the failure was caused by missing audit log entries. This test works by
> toggling global logging state, so I can see why multi-threaded execution
> might confuse the test.
>
> Chris Nauroth
>
>
> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
> ashutoshgupta118@gmail.com>
> wrote:
>
> > +1(non-binding)
> >
> > * Builds from source look good.
> > * Checksums and signatures are correct.
> > * Running basic HDFS and MapReduce commands looks good.
> >
> > > * TestAMRMProxy - Not able to reproduce in local
> > > * TestFsck - I can see failure only I can see is
> > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> > HDFS-15038
> > > * TestSLSStreamAMSynth - Not able to reproduce in local
> > > * TestServiceAM - Not able to reproduce in local
> >
> > Thanks Masatake for driving this release.
> >
> > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > wrote:
> >
> > > Hi developers,
> > >
> > > I'm still waiting for your vote.
> > > I'm considering the intermittent test failures mentioned by Chris are
> not
> > > blocker.
> > > Please file a JIRA and let me know if you find a blocker issue.
> > >
> > > I will appreciate your help for the release process.
> > >
> > > Regards,
> > > Masatake Iwasaki
> > >
> > > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > > >> TestServiceAM
> > > >
> > > > I can see the reported failure of TestServiceAM in some "Apache
> Hadoop
> > > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > > GenericTestUtils#waitFor for stabilizing the
> > > testContainersReleasedWhenPreLaunchFails.
> > > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > > If it is the case, TestServiceAM can be tagged as flaky in
> branch-3.2.
> > > >
> > > >
> > > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > > >> Thanks for testing the RC0, Chris.
> > > >>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>
> > > >> I could not reproduce the test failures on my local.
> > > >>
> > > >> For TestFsck, if the failed test case is
> > > testFsckListCorruptSnapshotFiles,
> > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > > >>
> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > > >> It could be tagged as known flaky test.
> > > >>
> > > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > > >>> -0 (binding)
> > > >>>
> > > >>> * Verified all checksums.
> > > >>> * Verified all signatures.
> > > >>> * Built from source, including native code on Linux.
> > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > > -Drequire.snappy
> > > >>> -Drequire.zstd -DskipTests
> > > >>> * Tests mostly passed, but see below.
> > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > > >>>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>>
> > > >>> The following tests also failed, but they also fail for me on
> 3.2.3,
> > so
> > > >>> they aren't likely to be related to this release candidate:
> > > >>> * TestCapacitySchedulerNodeLabelUpdate
> > > >>> * TestFrameworkUploader
> > > >>> * TestSLSGenericSynth
> > > >>> * TestSLSRunner
> > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > > >>>
> > > >>> I'm not voting a full -1, because I haven't done any root cause
> > > analysis on
> > > >>> these new test failures. I don't know if it's a quirk to my
> > > environment,
> > > >>> though I'm using the start-build-env.sh Docker container, so any
> > build
> > > >>> dependencies should be consistent. I'd be comfortable moving ahead
> if
> > > >>> others are seeing these tests pass.
> > > >>>
> > > >>> Chris Nauroth
> > > >>>
> > > >>>
> > > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > > iwasakims@oss.nttdata.com>
> > > >>> wrote:
> > > >>>
> > > >>>> +1 from myself.
> > > >>>>
> > > >>>> * skimmed the contents of site documentation.
> > > >>>>
> > > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> > with
> > > >>>> `-Pnative`.
> > > >>>>
> > > >>>> * launched pseudo distributed cluster including kms and httpfs
> with
> > > >>>> Kerberos and SSL enabled.
> > > >>>>
> > > >>>> * created encryption zone, put and read files via httpfs.
> > > >>>> * ran example MR wordcount over encryption zone.
> > > >>>>
> > > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> > ran
> > > some
> > > >>>> example MR jobs.
> > > >>>>
> > > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> > 3.2.4
> > > RC0
> > > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > > smoke-tests.
> > > >>>> https://github.com/apache/bigtop/pull/942
> > > >>>>
> > > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > > (migration
> > > >>>> to reload4j).
> > > >>>>
> > > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > > repository::
> > > >>>>
> > > >>>> <repository>
> > > >>>> <id>staged</id>
> > > >>>> <name>staged-releases</name>
> > > >>>> <url>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>> </url>
> > > >>>> <releases>
> > > >>>> <enabled>true</enabled>
> > > >>>> </releases>
> > > >>>> <snapshots>
> > > >>>> <enabled>true</enabled>
> > > >>>> </snapshots>
> > > >>>> </repository>
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Masatake Iwasaki
> > > >>>>
> > > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > > >>>>>
> > > >>>>> The RC is available at:
> > > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > > >>>>>
> > > >>>>> The RC tag is at:
> > > >>>>>
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > > >>>>>
> > > >>>>> The Maven artifacts are staged at:
> > > >>>>>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>>>
> > > >>>>> You can find my public key at:
> > > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > > >>>>>
> > > >>>>> Please evaluate the RC and vote.
> > > >>>>> The vote will be open for (at least) 5 days.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Masatake Iwasaki
> > > >>>>>
> > > >>>>>
> > ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ayush Saxena <ay...@gmail.com>.
+1(Binding)
* Built from Source
* Successful native build on Ubuntu 18.04
* Verified Checksums
* Verified Signatures
* Successful RAT check
* Ran Basic HDFS shell commands
* Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
* Browsed through UI(NN, DN, RM, NM & JHS)
* Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
Thanx Masatake for driving the release, Good Luck!!!
-Ayush
On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
> I'm changing my vote to +1 (binding).
>
> Masatake and Ashutosh, thank you for investigating.
>
> I reran tests without the parallel options, and that mostly addressed the
> failures. Maybe the tests in question are just not sufficiently isolated to
> support parallel execution. That looks to be the case for TestFsck, where
> the failure was caused by missing audit log entries. This test works by
> toggling global logging state, so I can see why multi-threaded execution
> might confuse the test.
>
> Chris Nauroth
>
>
> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
> ashutoshgupta118@gmail.com>
> wrote:
>
> > +1(non-binding)
> >
> > * Builds from source look good.
> > * Checksums and signatures are correct.
> > * Running basic HDFS and MapReduce commands looks good.
> >
> > > * TestAMRMProxy - Not able to reproduce in local
> > > * TestFsck - I can see failure only I can see is
> > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> > HDFS-15038
> > > * TestSLSStreamAMSynth - Not able to reproduce in local
> > > * TestServiceAM - Not able to reproduce in local
> >
> > Thanks Masatake for driving this release.
> >
> > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > wrote:
> >
> > > Hi developers,
> > >
> > > I'm still waiting for your vote.
> > > I'm considering the intermittent test failures mentioned by Chris are
> not
> > > blocker.
> > > Please file a JIRA and let me know if you find a blocker issue.
> > >
> > > I will appreciate your help for the release process.
> > >
> > > Regards,
> > > Masatake Iwasaki
> > >
> > > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > > >> TestServiceAM
> > > >
> > > > I can see the reported failure of TestServiceAM in some "Apache
> Hadoop
> > > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > > GenericTestUtils#waitFor for stabilizing the
> > > testContainersReleasedWhenPreLaunchFails.
> > > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > > If it is the case, TestServiceAM can be tagged as flaky in
> branch-3.2.
> > > >
> > > >
> > > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > > >> Thanks for testing the RC0, Chris.
> > > >>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>
> > > >> I could not reproduce the test failures on my local.
> > > >>
> > > >> For TestFsck, if the failed test case is
> > > testFsckListCorruptSnapshotFiles,
> > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > > >>
> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > > >> It could be tagged as known flaky test.
> > > >>
> > > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > > >>> -0 (binding)
> > > >>>
> > > >>> * Verified all checksums.
> > > >>> * Verified all signatures.
> > > >>> * Built from source, including native code on Linux.
> > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > > -Drequire.snappy
> > > >>> -Drequire.zstd -DskipTests
> > > >>> * Tests mostly passed, but see below.
> > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > > >>>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>>
> > > >>> The following tests also failed, but they also fail for me on
> 3.2.3,
> > so
> > > >>> they aren't likely to be related to this release candidate:
> > > >>> * TestCapacitySchedulerNodeLabelUpdate
> > > >>> * TestFrameworkUploader
> > > >>> * TestSLSGenericSynth
> > > >>> * TestSLSRunner
> > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > > >>>
> > > >>> I'm not voting a full -1, because I haven't done any root cause
> > > analysis on
> > > >>> these new test failures. I don't know if it's a quirk to my
> > > environment,
> > > >>> though I'm using the start-build-env.sh Docker container, so any
> > build
> > > >>> dependencies should be consistent. I'd be comfortable moving ahead
> if
> > > >>> others are seeing these tests pass.
> > > >>>
> > > >>> Chris Nauroth
> > > >>>
> > > >>>
> > > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > > iwasakims@oss.nttdata.com>
> > > >>> wrote:
> > > >>>
> > > >>>> +1 from myself.
> > > >>>>
> > > >>>> * skimmed the contents of site documentation.
> > > >>>>
> > > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> > with
> > > >>>> `-Pnative`.
> > > >>>>
> > > >>>> * launched pseudo distributed cluster including kms and httpfs
> with
> > > >>>> Kerberos and SSL enabled.
> > > >>>>
> > > >>>> * created encryption zone, put and read files via httpfs.
> > > >>>> * ran example MR wordcount over encryption zone.
> > > >>>>
> > > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> > ran
> > > some
> > > >>>> example MR jobs.
> > > >>>>
> > > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> > 3.2.4
> > > RC0
> > > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > > smoke-tests.
> > > >>>> https://github.com/apache/bigtop/pull/942
> > > >>>>
> > > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > > (migration
> > > >>>> to reload4j).
> > > >>>>
> > > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > > repository::
> > > >>>>
> > > >>>> <repository>
> > > >>>> <id>staged</id>
> > > >>>> <name>staged-releases</name>
> > > >>>> <url>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>> </url>
> > > >>>> <releases>
> > > >>>> <enabled>true</enabled>
> > > >>>> </releases>
> > > >>>> <snapshots>
> > > >>>> <enabled>true</enabled>
> > > >>>> </snapshots>
> > > >>>> </repository>
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Masatake Iwasaki
> > > >>>>
> > > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > > >>>>>
> > > >>>>> The RC is available at:
> > > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > > >>>>>
> > > >>>>> The RC tag is at:
> > > >>>>>
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > > >>>>>
> > > >>>>> The Maven artifacts are staged at:
> > > >>>>>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>>>
> > > >>>>> You can find my public key at:
> > > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > > >>>>>
> > > >>>>> Please evaluate the RC and vote.
> > > >>>>> The vote will be open for (at least) 5 days.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Masatake Iwasaki
> > > >>>>>
> > > >>>>>
> > ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ayush Saxena <ay...@gmail.com>.
+1(Binding)
* Built from Source
* Successful native build on Ubuntu 18.04
* Verified Checksums
* Verified Signatures
* Successful RAT check
* Ran Basic HDFS shell commands
* Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
* Browsed through UI(NN, DN, RM, NM & JHS)
* Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good
Thanx Masatake for driving the release, Good Luck!!!
-Ayush
On Thu, 21 Jul 2022 at 23:28, Chris Nauroth <cn...@apache.org> wrote:
> I'm changing my vote to +1 (binding).
>
> Masatake and Ashutosh, thank you for investigating.
>
> I reran tests without the parallel options, and that mostly addressed the
> failures. Maybe the tests in question are just not sufficiently isolated to
> support parallel execution. That looks to be the case for TestFsck, where
> the failure was caused by missing audit log entries. This test works by
> toggling global logging state, so I can see why multi-threaded execution
> might confuse the test.
>
> Chris Nauroth
>
>
> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
> ashutoshgupta118@gmail.com>
> wrote:
>
> > +1(non-binding)
> >
> > * Builds from source look good.
> > * Checksums and signatures are correct.
> > * Running basic HDFS and MapReduce commands looks good.
> >
> > > * TestAMRMProxy - Not able to reproduce in local
> > > * TestFsck - I can see failure only I can see is
> > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> > HDFS-15038
> > > * TestSLSStreamAMSynth - Not able to reproduce in local
> > > * TestServiceAM - Not able to reproduce in local
> >
> > Thanks Masatake for driving this release.
> >
> > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > wrote:
> >
> > > Hi developers,
> > >
> > > I'm still waiting for your vote.
> > > I'm considering the intermittent test failures mentioned by Chris are
> not
> > > blocker.
> > > Please file a JIRA and let me know if you find a blocker issue.
> > >
> > > I will appreciate your help for the release process.
> > >
> > > Regards,
> > > Masatake Iwasaki
> > >
> > > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > > >> TestServiceAM
> > > >
> > > > I can see the reported failure of TestServiceAM in some "Apache
> Hadoop
> > > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > > GenericTestUtils#waitFor for stabilizing the
> > > testContainersReleasedWhenPreLaunchFails.
> > > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > > If it is the case, TestServiceAM can be tagged as flaky in
> branch-3.2.
> > > >
> > > >
> > > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > > >> Thanks for testing the RC0, Chris.
> > > >>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>
> > > >> I could not reproduce the test failures on my local.
> > > >>
> > > >> For TestFsck, if the failed test case is
> > > testFsckListCorruptSnapshotFiles,
> > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > > >>
> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > > >> It could be tagged as known flaky test.
> > > >>
> > > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > > >>> -0 (binding)
> > > >>>
> > > >>> * Verified all checksums.
> > > >>> * Verified all signatures.
> > > >>> * Built from source, including native code on Linux.
> > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > > -Drequire.snappy
> > > >>> -Drequire.zstd -DskipTests
> > > >>> * Tests mostly passed, but see below.
> > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > > >>>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>>
> > > >>> The following tests also failed, but they also fail for me on
> 3.2.3,
> > so
> > > >>> they aren't likely to be related to this release candidate:
> > > >>> * TestCapacitySchedulerNodeLabelUpdate
> > > >>> * TestFrameworkUploader
> > > >>> * TestSLSGenericSynth
> > > >>> * TestSLSRunner
> > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > > >>>
> > > >>> I'm not voting a full -1, because I haven't done any root cause
> > > analysis on
> > > >>> these new test failures. I don't know if it's a quirk to my
> > > environment,
> > > >>> though I'm using the start-build-env.sh Docker container, so any
> > build
> > > >>> dependencies should be consistent. I'd be comfortable moving ahead
> if
> > > >>> others are seeing these tests pass.
> > > >>>
> > > >>> Chris Nauroth
> > > >>>
> > > >>>
> > > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > > iwasakims@oss.nttdata.com>
> > > >>> wrote:
> > > >>>
> > > >>>> +1 from myself.
> > > >>>>
> > > >>>> * skimmed the contents of site documentation.
> > > >>>>
> > > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> > with
> > > >>>> `-Pnative`.
> > > >>>>
> > > >>>> * launched pseudo distributed cluster including kms and httpfs
> with
> > > >>>> Kerberos and SSL enabled.
> > > >>>>
> > > >>>> * created encryption zone, put and read files via httpfs.
> > > >>>> * ran example MR wordcount over encryption zone.
> > > >>>>
> > > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> > ran
> > > some
> > > >>>> example MR jobs.
> > > >>>>
> > > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> > 3.2.4
> > > RC0
> > > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > > smoke-tests.
> > > >>>> https://github.com/apache/bigtop/pull/942
> > > >>>>
> > > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > > (migration
> > > >>>> to reload4j).
> > > >>>>
> > > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > > repository::
> > > >>>>
> > > >>>> <repository>
> > > >>>> <id>staged</id>
> > > >>>> <name>staged-releases</name>
> > > >>>> <url>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>> </url>
> > > >>>> <releases>
> > > >>>> <enabled>true</enabled>
> > > >>>> </releases>
> > > >>>> <snapshots>
> > > >>>> <enabled>true</enabled>
> > > >>>> </snapshots>
> > > >>>> </repository>
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Masatake Iwasaki
> > > >>>>
> > > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > > >>>>>
> > > >>>>> The RC is available at:
> > > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > > >>>>>
> > > >>>>> The RC tag is at:
> > > >>>>>
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > > >>>>>
> > > >>>>> The Maven artifacts are staged at:
> > > >>>>>
> > > >>>>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > > >>>>>
> > > >>>>> You can find my public key at:
> > > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > > >>>>>
> > > >>>>> Please evaluate the RC and vote.
> > > >>>>> The vote will be open for (at least) 5 days.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Masatake Iwasaki
> > > >>>>>
> > > >>>>>
> > ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
I'm changing my vote to +1 (binding).
Masatake and Ashutosh, thank you for investigating.
I reran tests without the parallel options, and that mostly addressed the
failures. Maybe the tests in question are just not sufficiently isolated to
support parallel execution. That looks to be the case for TestFsck, where
the failure was caused by missing audit log entries. This test works by
toggling global logging state, so I can see why multi-threaded execution
might confuse the test.
Chris Nauroth
On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <as...@gmail.com>
wrote:
> +1(non-binding)
>
> * Builds from source look good.
> * Checksums and signatures are correct.
> * Running basic HDFS and MapReduce commands looks good.
>
> > * TestAMRMProxy - Not able to reproduce in local
> > * TestFsck - I can see failure only I can see is
> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> HDFS-15038
> > * TestSLSStreamAMSynth - Not able to reproduce in local
> > * TestServiceAM - Not able to reproduce in local
>
> Thanks Masatake for driving this release.
>
> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> wrote:
>
> > Hi developers,
> >
> > I'm still waiting for your vote.
> > I'm considering the intermittent test failures mentioned by Chris are not
> > blocker.
> > Please file a JIRA and let me know if you find a blocker issue.
> >
> > I will appreciate your help for the release process.
> >
> > Regards,
> > Masatake Iwasaki
> >
> > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > >> TestServiceAM
> > >
> > > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > GenericTestUtils#waitFor for stabilizing the
> > testContainersReleasedWhenPreLaunchFails.
> > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> > >
> > >
> > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > >> Thanks for testing the RC0, Chris.
> > >>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>
> > >> I could not reproduce the test failures on my local.
> > >>
> > >> For TestFsck, if the failed test case is
> > testFsckListCorruptSnapshotFiles,
> > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > >>
> > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > >> It could be tagged as known flaky test.
> > >>
> > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > >>> -0 (binding)
> > >>>
> > >>> * Verified all checksums.
> > >>> * Verified all signatures.
> > >>> * Built from source, including native code on Linux.
> > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > -Drequire.snappy
> > >>> -Drequire.zstd -DskipTests
> > >>> * Tests mostly passed, but see below.
> > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > >>>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>>
> > >>> The following tests also failed, but they also fail for me on 3.2.3,
> so
> > >>> they aren't likely to be related to this release candidate:
> > >>> * TestCapacitySchedulerNodeLabelUpdate
> > >>> * TestFrameworkUploader
> > >>> * TestSLSGenericSynth
> > >>> * TestSLSRunner
> > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > >>>
> > >>> I'm not voting a full -1, because I haven't done any root cause
> > analysis on
> > >>> these new test failures. I don't know if it's a quirk to my
> > environment,
> > >>> though I'm using the start-build-env.sh Docker container, so any
> build
> > >>> dependencies should be consistent. I'd be comfortable moving ahead if
> > >>> others are seeing these tests pass.
> > >>>
> > >>> Chris Nauroth
> > >>>
> > >>>
> > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > >>> wrote:
> > >>>
> > >>>> +1 from myself.
> > >>>>
> > >>>> * skimmed the contents of site documentation.
> > >>>>
> > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> with
> > >>>> `-Pnative`.
> > >>>>
> > >>>> * launched pseudo distributed cluster including kms and httpfs with
> > >>>> Kerberos and SSL enabled.
> > >>>>
> > >>>> * created encryption zone, put and read files via httpfs.
> > >>>> * ran example MR wordcount over encryption zone.
> > >>>>
> > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> ran
> > some
> > >>>> example MR jobs.
> > >>>>
> > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> 3.2.4
> > RC0
> > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > smoke-tests.
> > >>>> https://github.com/apache/bigtop/pull/942
> > >>>>
> > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > (migration
> > >>>> to reload4j).
> > >>>>
> > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > repository::
> > >>>>
> > >>>> <repository>
> > >>>> <id>staged</id>
> > >>>> <name>staged-releases</name>
> > >>>> <url>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>> </url>
> > >>>> <releases>
> > >>>> <enabled>true</enabled>
> > >>>> </releases>
> > >>>> <snapshots>
> > >>>> <enabled>true</enabled>
> > >>>> </snapshots>
> > >>>> </repository>
> > >>>>
> > >>>> Thanks,
> > >>>> Masatake Iwasaki
> > >>>>
> > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > >>>>>
> > >>>>> The RC tag is at:
> > >>>>>
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > >>>>>
> > >>>>> The Maven artifacts are staged at:
> > >>>>>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > >>>>>
> > >>>>> Please evaluate the RC and vote.
> > >>>>> The vote will be open for (at least) 5 days.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Masatake Iwasaki
> > >>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >>>>
> > >>>>
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
I'm changing my vote to +1 (binding).
Masatake and Ashutosh, thank you for investigating.
I reran tests without the parallel options, and that mostly addressed the
failures. Maybe the tests in question are just not sufficiently isolated to
support parallel execution. That looks to be the case for TestFsck, where
the failure was caused by missing audit log entries. This test works by
toggling global logging state, so I can see why multi-threaded execution
might confuse the test.
Chris Nauroth
On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <as...@gmail.com>
wrote:
> +1(non-binding)
>
> * Builds from source look good.
> * Checksums and signatures are correct.
> * Running basic HDFS and MapReduce commands looks good.
>
> > * TestAMRMProxy - Not able to reproduce in local
> > * TestFsck - I can see failure only I can see is
> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> HDFS-15038
> > * TestSLSStreamAMSynth - Not able to reproduce in local
> > * TestServiceAM - Not able to reproduce in local
>
> Thanks Masatake for driving this release.
>
> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> wrote:
>
> > Hi developers,
> >
> > I'm still waiting for your vote.
> > I'm considering the intermittent test failures mentioned by Chris are not
> > blocker.
> > Please file a JIRA and let me know if you find a blocker issue.
> >
> > I will appreciate your help for the release process.
> >
> > Regards,
> > Masatake Iwasaki
> >
> > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > >> TestServiceAM
> > >
> > > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > GenericTestUtils#waitFor for stabilizing the
> > testContainersReleasedWhenPreLaunchFails.
> > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> > >
> > >
> > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > >> Thanks for testing the RC0, Chris.
> > >>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>
> > >> I could not reproduce the test failures on my local.
> > >>
> > >> For TestFsck, if the failed test case is
> > testFsckListCorruptSnapshotFiles,
> > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > >>
> > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > >> It could be tagged as known flaky test.
> > >>
> > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > >>> -0 (binding)
> > >>>
> > >>> * Verified all checksums.
> > >>> * Verified all signatures.
> > >>> * Built from source, including native code on Linux.
> > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > -Drequire.snappy
> > >>> -Drequire.zstd -DskipTests
> > >>> * Tests mostly passed, but see below.
> > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > >>>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>>
> > >>> The following tests also failed, but they also fail for me on 3.2.3,
> so
> > >>> they aren't likely to be related to this release candidate:
> > >>> * TestCapacitySchedulerNodeLabelUpdate
> > >>> * TestFrameworkUploader
> > >>> * TestSLSGenericSynth
> > >>> * TestSLSRunner
> > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > >>>
> > >>> I'm not voting a full -1, because I haven't done any root cause
> > analysis on
> > >>> these new test failures. I don't know if it's a quirk to my
> > environment,
> > >>> though I'm using the start-build-env.sh Docker container, so any
> build
> > >>> dependencies should be consistent. I'd be comfortable moving ahead if
> > >>> others are seeing these tests pass.
> > >>>
> > >>> Chris Nauroth
> > >>>
> > >>>
> > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > >>> wrote:
> > >>>
> > >>>> +1 from myself.
> > >>>>
> > >>>> * skimmed the contents of site documentation.
> > >>>>
> > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> with
> > >>>> `-Pnative`.
> > >>>>
> > >>>> * launched pseudo distributed cluster including kms and httpfs with
> > >>>> Kerberos and SSL enabled.
> > >>>>
> > >>>> * created encryption zone, put and read files via httpfs.
> > >>>> * ran example MR wordcount over encryption zone.
> > >>>>
> > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> ran
> > some
> > >>>> example MR jobs.
> > >>>>
> > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> 3.2.4
> > RC0
> > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > smoke-tests.
> > >>>> https://github.com/apache/bigtop/pull/942
> > >>>>
> > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > (migration
> > >>>> to reload4j).
> > >>>>
> > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > repository::
> > >>>>
> > >>>> <repository>
> > >>>> <id>staged</id>
> > >>>> <name>staged-releases</name>
> > >>>> <url>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>> </url>
> > >>>> <releases>
> > >>>> <enabled>true</enabled>
> > >>>> </releases>
> > >>>> <snapshots>
> > >>>> <enabled>true</enabled>
> > >>>> </snapshots>
> > >>>> </repository>
> > >>>>
> > >>>> Thanks,
> > >>>> Masatake Iwasaki
> > >>>>
> > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > >>>>>
> > >>>>> The RC tag is at:
> > >>>>>
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > >>>>>
> > >>>>> The Maven artifacts are staged at:
> > >>>>>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > >>>>>
> > >>>>> Please evaluate the RC and vote.
> > >>>>> The vote will be open for (at least) 5 days.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Masatake Iwasaki
> > >>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >>>>
> > >>>>
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
I'm changing my vote to +1 (binding).
Masatake and Ashutosh, thank you for investigating.
I reran tests without the parallel options, and that mostly addressed the
failures. Maybe the tests in question are just not sufficiently isolated to
support parallel execution. That looks to be the case for TestFsck, where
the failure was caused by missing audit log entries. This test works by
toggling global logging state, so I can see why multi-threaded execution
might confuse the test.
Chris Nauroth
On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <as...@gmail.com>
wrote:
> +1(non-binding)
>
> * Builds from source look good.
> * Checksums and signatures are correct.
> * Running basic HDFS and MapReduce commands looks good.
>
> > * TestAMRMProxy - Not able to reproduce in local
> > * TestFsck - I can see failure only I can see is
> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> HDFS-15038
> > * TestSLSStreamAMSynth - Not able to reproduce in local
> > * TestServiceAM - Not able to reproduce in local
>
> Thanks Masatake for driving this release.
>
> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> wrote:
>
> > Hi developers,
> >
> > I'm still waiting for your vote.
> > I'm considering the intermittent test failures mentioned by Chris are not
> > blocker.
> > Please file a JIRA and let me know if you find a blocker issue.
> >
> > I will appreciate your help for the release process.
> >
> > Regards,
> > Masatake Iwasaki
> >
> > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > >> TestServiceAM
> > >
> > > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > GenericTestUtils#waitFor for stabilizing the
> > testContainersReleasedWhenPreLaunchFails.
> > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> > >
> > >
> > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > >> Thanks for testing the RC0, Chris.
> > >>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>
> > >> I could not reproduce the test failures on my local.
> > >>
> > >> For TestFsck, if the failed test case is
> > testFsckListCorruptSnapshotFiles,
> > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > >>
> > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > >> It could be tagged as known flaky test.
> > >>
> > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > >>> -0 (binding)
> > >>>
> > >>> * Verified all checksums.
> > >>> * Verified all signatures.
> > >>> * Built from source, including native code on Linux.
> > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > -Drequire.snappy
> > >>> -Drequire.zstd -DskipTests
> > >>> * Tests mostly passed, but see below.
> > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > >>>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>>
> > >>> The following tests also failed, but they also fail for me on 3.2.3,
> so
> > >>> they aren't likely to be related to this release candidate:
> > >>> * TestCapacitySchedulerNodeLabelUpdate
> > >>> * TestFrameworkUploader
> > >>> * TestSLSGenericSynth
> > >>> * TestSLSRunner
> > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > >>>
> > >>> I'm not voting a full -1, because I haven't done any root cause
> > analysis on
> > >>> these new test failures. I don't know if it's a quirk to my
> > environment,
> > >>> though I'm using the start-build-env.sh Docker container, so any
> build
> > >>> dependencies should be consistent. I'd be comfortable moving ahead if
> > >>> others are seeing these tests pass.
> > >>>
> > >>> Chris Nauroth
> > >>>
> > >>>
> > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > >>> wrote:
> > >>>
> > >>>> +1 from myself.
> > >>>>
> > >>>> * skimmed the contents of site documentation.
> > >>>>
> > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> with
> > >>>> `-Pnative`.
> > >>>>
> > >>>> * launched pseudo distributed cluster including kms and httpfs with
> > >>>> Kerberos and SSL enabled.
> > >>>>
> > >>>> * created encryption zone, put and read files via httpfs.
> > >>>> * ran example MR wordcount over encryption zone.
> > >>>>
> > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> ran
> > some
> > >>>> example MR jobs.
> > >>>>
> > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> 3.2.4
> > RC0
> > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > smoke-tests.
> > >>>> https://github.com/apache/bigtop/pull/942
> > >>>>
> > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > (migration
> > >>>> to reload4j).
> > >>>>
> > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > repository::
> > >>>>
> > >>>> <repository>
> > >>>> <id>staged</id>
> > >>>> <name>staged-releases</name>
> > >>>> <url>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>> </url>
> > >>>> <releases>
> > >>>> <enabled>true</enabled>
> > >>>> </releases>
> > >>>> <snapshots>
> > >>>> <enabled>true</enabled>
> > >>>> </snapshots>
> > >>>> </repository>
> > >>>>
> > >>>> Thanks,
> > >>>> Masatake Iwasaki
> > >>>>
> > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > >>>>>
> > >>>>> The RC tag is at:
> > >>>>>
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > >>>>>
> > >>>>> The Maven artifacts are staged at:
> > >>>>>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > >>>>>
> > >>>>> Please evaluate the RC and vote.
> > >>>>> The vote will be open for (at least) 5 days.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Masatake Iwasaki
> > >>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >>>>
> > >>>>
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
I'm changing my vote to +1 (binding).
Masatake and Ashutosh, thank you for investigating.
I reran tests without the parallel options, and that mostly addressed the
failures. Maybe the tests in question are just not sufficiently isolated to
support parallel execution. That looks to be the case for TestFsck, where
the failure was caused by missing audit log entries. This test works by
toggling global logging state, so I can see why multi-threaded execution
might confuse the test.
Chris Nauroth
On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <as...@gmail.com>
wrote:
> +1(non-binding)
>
> * Builds from source look good.
> * Checksums and signatures are correct.
> * Running basic HDFS and MapReduce commands looks good.
>
> > * TestAMRMProxy - Not able to reproduce in local
> > * TestFsck - I can see failure only I can see is
> TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> HDFS-15038
> > * TestSLSStreamAMSynth - Not able to reproduce in local
> > * TestServiceAM - Not able to reproduce in local
>
> Thanks Masatake for driving this release.
>
> On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> wrote:
>
> > Hi developers,
> >
> > I'm still waiting for your vote.
> > I'm considering the intermittent test failures mentioned by Chris are not
> > blocker.
> > Please file a JIRA and let me know if you find a blocker issue.
> >
> > I will appreciate your help for the release process.
> >
> > Regards,
> > Masatake Iwasaki
> >
> > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > >> TestServiceAM
> > >
> > > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > GenericTestUtils#waitFor for stabilizing the
> > testContainersReleasedWhenPreLaunchFails.
> > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> > >
> > >
> > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > >> Thanks for testing the RC0, Chris.
> > >>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>
> > >> I could not reproduce the test failures on my local.
> > >>
> > >> For TestFsck, if the failed test case is
> > testFsckListCorruptSnapshotFiles,
> > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > >>
> > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > >> It could be tagged as known flaky test.
> > >>
> > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > >>> -0 (binding)
> > >>>
> > >>> * Verified all checksums.
> > >>> * Verified all signatures.
> > >>> * Built from source, including native code on Linux.
> > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> > -Drequire.snappy
> > >>> -Drequire.zstd -DskipTests
> > >>> * Tests mostly passed, but see below.
> > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > >>>
> > >>> The following are new test failures for me on 3.2.4:
> > >>> * TestAMRMProxy
> > >>> * TestFsck
> > >>> * TestSLSStreamAMSynth
> > >>> * TestServiceAM
> > >>>
> > >>> The following tests also failed, but they also fail for me on 3.2.3,
> so
> > >>> they aren't likely to be related to this release candidate:
> > >>> * TestCapacitySchedulerNodeLabelUpdate
> > >>> * TestFrameworkUploader
> > >>> * TestSLSGenericSynth
> > >>> * TestSLSRunner
> > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > >>>
> > >>> I'm not voting a full -1, because I haven't done any root cause
> > analysis on
> > >>> these new test failures. I don't know if it's a quirk to my
> > environment,
> > >>> though I'm using the start-build-env.sh Docker container, so any
> build
> > >>> dependencies should be consistent. I'd be comfortable moving ahead if
> > >>> others are seeing these tests pass.
> > >>>
> > >>> Chris Nauroth
> > >>>
> > >>>
> > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> > iwasakims@oss.nttdata.com>
> > >>> wrote:
> > >>>
> > >>>> +1 from myself.
> > >>>>
> > >>>> * skimmed the contents of site documentation.
> > >>>>
> > >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8
> with
> > >>>> `-Pnative`.
> > >>>>
> > >>>> * launched pseudo distributed cluster including kms and httpfs with
> > >>>> Kerberos and SSL enabled.
> > >>>>
> > >>>> * created encryption zone, put and read files via httpfs.
> > >>>> * ran example MR wordcount over encryption zone.
> > >>>>
> > >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and
> ran
> > some
> > >>>> example MR jobs.
> > >>>>
> > >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop
> 3.2.4
> > RC0
> > >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> > smoke-tests.
> > >>>> https://github.com/apache/bigtop/pull/942
> > >>>>
> > >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> > (migration
> > >>>> to reload4j).
> > >>>>
> > >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> > repository::
> > >>>>
> > >>>> <repository>
> > >>>> <id>staged</id>
> > >>>> <name>staged-releases</name>
> > >>>> <url>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>> </url>
> > >>>> <releases>
> > >>>> <enabled>true</enabled>
> > >>>> </releases>
> > >>>> <snapshots>
> > >>>> <enabled>true</enabled>
> > >>>> </snapshots>
> > >>>> </repository>
> > >>>>
> > >>>> Thanks,
> > >>>> Masatake Iwasaki
> > >>>>
> > >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Here's Hadoop 3.2.4 release candidate #0:
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> > >>>>>
> > >>>>> The RC tag is at:
> > >>>>>
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> > >>>>>
> > >>>>> The Maven artifacts are staged at:
> > >>>>>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1354
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://downloads.apache.org/hadoop/common/KEYS
> > >>>>>
> > >>>>> Please evaluate the RC and vote.
> > >>>>> The vote will be open for (at least) 5 days.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Masatake Iwasaki
> > >>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >>>>
> > >>>>
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ashutosh Gupta <as...@gmail.com>.
+1(non-binding)
* Builds from source look good.
* Checksums and signatures are correct.
* Running basic HDFS and MapReduce commands looks good.
> * TestAMRMProxy - Not able to reproduce in local
> * TestFsck - I can see failure only I can see is
TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
HDFS-15038
> * TestSLSStreamAMSynth - Not able to reproduce in local
> * TestServiceAM - Not able to reproduce in local
Thanks Masatake for driving this release.
On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> Hi developers,
>
> I'm still waiting for your vote.
> I'm considering the intermittent test failures mentioned by Chris are not
> blocker.
> Please file a JIRA and let me know if you find a blocker issue.
>
> I will appreciate your help for the release process.
>
> Regards,
> Masatake Iwasaki
>
> On 2022/07/20 14:50, Masatake Iwasaki wrote:
> >> TestServiceAM
> >
> > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> GenericTestUtils#waitFor for stabilizing the
> testContainersReleasedWhenPreLaunchFails.
> > YARN 8867 did not modified other code under hadoop-yarn-services.
> > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> >
> >
> > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> >> Thanks for testing the RC0, Chris.
> >>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>
> >> I could not reproduce the test failures on my local.
> >>
> >> For TestFsck, if the failed test case is
> testFsckListCorruptSnapshotFiles,
> >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> >>
> >> The failure of TestSLSStreamAMSynth looks frequently reported by
> >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> >> It could be tagged as known flaky test.
> >>
> >> On 2022/07/20 9:15, Chris Nauroth wrote:
> >>> -0 (binding)
> >>>
> >>> * Verified all checksums.
> >>> * Verified all signatures.
> >>> * Built from source, including native code on Linux.
> >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >>> -Drequire.zstd -DskipTests
> >>> * Tests mostly passed, but see below.
> >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >>>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>>
> >>> The following tests also failed, but they also fail for me on 3.2.3, so
> >>> they aren't likely to be related to this release candidate:
> >>> * TestCapacitySchedulerNodeLabelUpdate
> >>> * TestFrameworkUploader
> >>> * TestSLSGenericSynth
> >>> * TestSLSRunner
> >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> >>>
> >>> I'm not voting a full -1, because I haven't done any root cause
> analysis on
> >>> these new test failures. I don't know if it's a quirk to my
> environment,
> >>> though I'm using the start-build-env.sh Docker container, so any build
> >>> dependencies should be consistent. I'd be comfortable moving ahead if
> >>> others are seeing these tests pass.
> >>>
> >>> Chris Nauroth
> >>>
> >>>
> >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> >>> wrote:
> >>>
> >>>> +1 from myself.
> >>>>
> >>>> * skimmed the contents of site documentation.
> >>>>
> >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> >>>> `-Pnative`.
> >>>>
> >>>> * launched pseudo distributed cluster including kms and httpfs with
> >>>> Kerberos and SSL enabled.
> >>>>
> >>>> * created encryption zone, put and read files via httpfs.
> >>>> * ran example MR wordcount over encryption zone.
> >>>>
> >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran
> some
> >>>> example MR jobs.
> >>>>
> >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4
> RC0
> >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> smoke-tests.
> >>>> https://github.com/apache/bigtop/pull/942
> >>>>
> >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> (migration
> >>>> to reload4j).
> >>>>
> >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> repository::
> >>>>
> >>>> <repository>
> >>>> <id>staged</id>
> >>>> <name>staged-releases</name>
> >>>> <url>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>> </url>
> >>>> <releases>
> >>>> <enabled>true</enabled>
> >>>> </releases>
> >>>> <snapshots>
> >>>> <enabled>true</enabled>
> >>>> </snapshots>
> >>>> </repository>
> >>>>
> >>>> Thanks,
> >>>> Masatake Iwasaki
> >>>>
> >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> Here's Hadoop 3.2.4 release candidate #0:
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >>>>>
> >>>>> The RC tag is at:
> >>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >>>>>
> >>>>> The Maven artifacts are staged at:
> >>>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://downloads.apache.org/hadoop/common/KEYS
> >>>>>
> >>>>> Please evaluate the RC and vote.
> >>>>> The vote will be open for (at least) 5 days.
> >>>>>
> >>>>> Thanks,
> >>>>> Masatake Iwasaki
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ashutosh Gupta <as...@gmail.com>.
+1(non-binding)
* Builds from source look good.
* Checksums and signatures are correct.
* Running basic HDFS and MapReduce commands looks good.
> * TestAMRMProxy - Not able to reproduce in local
> * TestFsck - I can see failure only I can see is
TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
HDFS-15038
> * TestSLSStreamAMSynth - Not able to reproduce in local
> * TestServiceAM - Not able to reproduce in local
Thanks Masatake for driving this release.
On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> Hi developers,
>
> I'm still waiting for your vote.
> I'm considering the intermittent test failures mentioned by Chris are not
> blocker.
> Please file a JIRA and let me know if you find a blocker issue.
>
> I will appreciate your help for the release process.
>
> Regards,
> Masatake Iwasaki
>
> On 2022/07/20 14:50, Masatake Iwasaki wrote:
> >> TestServiceAM
> >
> > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> GenericTestUtils#waitFor for stabilizing the
> testContainersReleasedWhenPreLaunchFails.
> > YARN 8867 did not modified other code under hadoop-yarn-services.
> > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> >
> >
> > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> >> Thanks for testing the RC0, Chris.
> >>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>
> >> I could not reproduce the test failures on my local.
> >>
> >> For TestFsck, if the failed test case is
> testFsckListCorruptSnapshotFiles,
> >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> >>
> >> The failure of TestSLSStreamAMSynth looks frequently reported by
> >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> >> It could be tagged as known flaky test.
> >>
> >> On 2022/07/20 9:15, Chris Nauroth wrote:
> >>> -0 (binding)
> >>>
> >>> * Verified all checksums.
> >>> * Verified all signatures.
> >>> * Built from source, including native code on Linux.
> >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >>> -Drequire.zstd -DskipTests
> >>> * Tests mostly passed, but see below.
> >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >>>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>>
> >>> The following tests also failed, but they also fail for me on 3.2.3, so
> >>> they aren't likely to be related to this release candidate:
> >>> * TestCapacitySchedulerNodeLabelUpdate
> >>> * TestFrameworkUploader
> >>> * TestSLSGenericSynth
> >>> * TestSLSRunner
> >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> >>>
> >>> I'm not voting a full -1, because I haven't done any root cause
> analysis on
> >>> these new test failures. I don't know if it's a quirk to my
> environment,
> >>> though I'm using the start-build-env.sh Docker container, so any build
> >>> dependencies should be consistent. I'd be comfortable moving ahead if
> >>> others are seeing these tests pass.
> >>>
> >>> Chris Nauroth
> >>>
> >>>
> >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> >>> wrote:
> >>>
> >>>> +1 from myself.
> >>>>
> >>>> * skimmed the contents of site documentation.
> >>>>
> >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> >>>> `-Pnative`.
> >>>>
> >>>> * launched pseudo distributed cluster including kms and httpfs with
> >>>> Kerberos and SSL enabled.
> >>>>
> >>>> * created encryption zone, put and read files via httpfs.
> >>>> * ran example MR wordcount over encryption zone.
> >>>>
> >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran
> some
> >>>> example MR jobs.
> >>>>
> >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4
> RC0
> >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> smoke-tests.
> >>>> https://github.com/apache/bigtop/pull/942
> >>>>
> >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> (migration
> >>>> to reload4j).
> >>>>
> >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> repository::
> >>>>
> >>>> <repository>
> >>>> <id>staged</id>
> >>>> <name>staged-releases</name>
> >>>> <url>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>> </url>
> >>>> <releases>
> >>>> <enabled>true</enabled>
> >>>> </releases>
> >>>> <snapshots>
> >>>> <enabled>true</enabled>
> >>>> </snapshots>
> >>>> </repository>
> >>>>
> >>>> Thanks,
> >>>> Masatake Iwasaki
> >>>>
> >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> Here's Hadoop 3.2.4 release candidate #0:
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >>>>>
> >>>>> The RC tag is at:
> >>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >>>>>
> >>>>> The Maven artifacts are staged at:
> >>>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://downloads.apache.org/hadoop/common/KEYS
> >>>>>
> >>>>> Please evaluate the RC and vote.
> >>>>> The vote will be open for (at least) 5 days.
> >>>>>
> >>>>> Thanks,
> >>>>> Masatake Iwasaki
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ashutosh Gupta <as...@gmail.com>.
+1(non-binding)
* Builds from source look good.
* Checksums and signatures are correct.
* Running basic HDFS and MapReduce commands looks good.
> * TestAMRMProxy - Not able to reproduce in local
> * TestFsck - I can see failure only I can see is
TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
HDFS-15038
> * TestSLSStreamAMSynth - Not able to reproduce in local
> * TestServiceAM - Not able to reproduce in local
Thanks Masatake for driving this release.
On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> Hi developers,
>
> I'm still waiting for your vote.
> I'm considering the intermittent test failures mentioned by Chris are not
> blocker.
> Please file a JIRA and let me know if you find a blocker issue.
>
> I will appreciate your help for the release process.
>
> Regards,
> Masatake Iwasaki
>
> On 2022/07/20 14:50, Masatake Iwasaki wrote:
> >> TestServiceAM
> >
> > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> GenericTestUtils#waitFor for stabilizing the
> testContainersReleasedWhenPreLaunchFails.
> > YARN 8867 did not modified other code under hadoop-yarn-services.
> > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> >
> >
> > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> >> Thanks for testing the RC0, Chris.
> >>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>
> >> I could not reproduce the test failures on my local.
> >>
> >> For TestFsck, if the failed test case is
> testFsckListCorruptSnapshotFiles,
> >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> >>
> >> The failure of TestSLSStreamAMSynth looks frequently reported by
> >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> >> It could be tagged as known flaky test.
> >>
> >> On 2022/07/20 9:15, Chris Nauroth wrote:
> >>> -0 (binding)
> >>>
> >>> * Verified all checksums.
> >>> * Verified all signatures.
> >>> * Built from source, including native code on Linux.
> >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >>> -Drequire.zstd -DskipTests
> >>> * Tests mostly passed, but see below.
> >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >>>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>>
> >>> The following tests also failed, but they also fail for me on 3.2.3, so
> >>> they aren't likely to be related to this release candidate:
> >>> * TestCapacitySchedulerNodeLabelUpdate
> >>> * TestFrameworkUploader
> >>> * TestSLSGenericSynth
> >>> * TestSLSRunner
> >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> >>>
> >>> I'm not voting a full -1, because I haven't done any root cause
> analysis on
> >>> these new test failures. I don't know if it's a quirk to my
> environment,
> >>> though I'm using the start-build-env.sh Docker container, so any build
> >>> dependencies should be consistent. I'd be comfortable moving ahead if
> >>> others are seeing these tests pass.
> >>>
> >>> Chris Nauroth
> >>>
> >>>
> >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> >>> wrote:
> >>>
> >>>> +1 from myself.
> >>>>
> >>>> * skimmed the contents of site documentation.
> >>>>
> >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> >>>> `-Pnative`.
> >>>>
> >>>> * launched pseudo distributed cluster including kms and httpfs with
> >>>> Kerberos and SSL enabled.
> >>>>
> >>>> * created encryption zone, put and read files via httpfs.
> >>>> * ran example MR wordcount over encryption zone.
> >>>>
> >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran
> some
> >>>> example MR jobs.
> >>>>
> >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4
> RC0
> >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> smoke-tests.
> >>>> https://github.com/apache/bigtop/pull/942
> >>>>
> >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> (migration
> >>>> to reload4j).
> >>>>
> >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> repository::
> >>>>
> >>>> <repository>
> >>>> <id>staged</id>
> >>>> <name>staged-releases</name>
> >>>> <url>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>> </url>
> >>>> <releases>
> >>>> <enabled>true</enabled>
> >>>> </releases>
> >>>> <snapshots>
> >>>> <enabled>true</enabled>
> >>>> </snapshots>
> >>>> </repository>
> >>>>
> >>>> Thanks,
> >>>> Masatake Iwasaki
> >>>>
> >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> Here's Hadoop 3.2.4 release candidate #0:
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >>>>>
> >>>>> The RC tag is at:
> >>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >>>>>
> >>>>> The Maven artifacts are staged at:
> >>>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://downloads.apache.org/hadoop/common/KEYS
> >>>>>
> >>>>> Please evaluate the RC and vote.
> >>>>> The vote will be open for (at least) 5 days.
> >>>>>
> >>>>> Thanks,
> >>>>> Masatake Iwasaki
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Ashutosh Gupta <as...@gmail.com>.
+1(non-binding)
* Builds from source look good.
* Checksums and signatures are correct.
* Running basic HDFS and MapReduce commands looks good.
> * TestAMRMProxy - Not able to reproduce in local
> * TestFsck - I can see failure only I can see is
TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
HDFS-15038
> * TestSLSStreamAMSynth - Not able to reproduce in local
> * TestServiceAM - Not able to reproduce in local
Thanks Masatake for driving this release.
On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> Hi developers,
>
> I'm still waiting for your vote.
> I'm considering the intermittent test failures mentioned by Chris are not
> blocker.
> Please file a JIRA and let me know if you find a blocker issue.
>
> I will appreciate your help for the release process.
>
> Regards,
> Masatake Iwasaki
>
> On 2022/07/20 14:50, Masatake Iwasaki wrote:
> >> TestServiceAM
> >
> > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> GenericTestUtils#waitFor for stabilizing the
> testContainersReleasedWhenPreLaunchFails.
> > YARN 8867 did not modified other code under hadoop-yarn-services.
> > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> >
> >
> > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> >> Thanks for testing the RC0, Chris.
> >>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>
> >> I could not reproduce the test failures on my local.
> >>
> >> For TestFsck, if the failed test case is
> testFsckListCorruptSnapshotFiles,
> >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> >>
> >> The failure of TestSLSStreamAMSynth looks frequently reported by
> >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> >> It could be tagged as known flaky test.
> >>
> >> On 2022/07/20 9:15, Chris Nauroth wrote:
> >>> -0 (binding)
> >>>
> >>> * Verified all checksums.
> >>> * Verified all signatures.
> >>> * Built from source, including native code on Linux.
> >>> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >>> -Drequire.zstd -DskipTests
> >>> * Tests mostly passed, but see below.
> >>> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >>>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>>
> >>> The following tests also failed, but they also fail for me on 3.2.3, so
> >>> they aren't likely to be related to this release candidate:
> >>> * TestCapacitySchedulerNodeLabelUpdate
> >>> * TestFrameworkUploader
> >>> * TestSLSGenericSynth
> >>> * TestSLSRunner
> >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> >>>
> >>> I'm not voting a full -1, because I haven't done any root cause
> analysis on
> >>> these new test failures. I don't know if it's a quirk to my
> environment,
> >>> though I'm using the start-build-env.sh Docker container, so any build
> >>> dependencies should be consistent. I'd be comfortable moving ahead if
> >>> others are seeing these tests pass.
> >>>
> >>> Chris Nauroth
> >>>
> >>>
> >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> iwasakims@oss.nttdata.com>
> >>> wrote:
> >>>
> >>>> +1 from myself.
> >>>>
> >>>> * skimmed the contents of site documentation.
> >>>>
> >>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> >>>> `-Pnative`.
> >>>>
> >>>> * launched pseudo distributed cluster including kms and httpfs with
> >>>> Kerberos and SSL enabled.
> >>>>
> >>>> * created encryption zone, put and read files via httpfs.
> >>>> * ran example MR wordcount over encryption zone.
> >>>>
> >>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran
> some
> >>>> example MR jobs.
> >>>>
> >>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4
> RC0
> >>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> smoke-tests.
> >>>> https://github.com/apache/bigtop/pull/942
> >>>>
> >>>> * Hive needs updating exclusion rule to address HADOOP-18088
> (migration
> >>>> to reload4j).
> >>>>
> >>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> repository::
> >>>>
> >>>> <repository>
> >>>> <id>staged</id>
> >>>> <name>staged-releases</name>
> >>>> <url>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>> </url>
> >>>> <releases>
> >>>> <enabled>true</enabled>
> >>>> </releases>
> >>>> <snapshots>
> >>>> <enabled>true</enabled>
> >>>> </snapshots>
> >>>> </repository>
> >>>>
> >>>> Thanks,
> >>>> Masatake Iwasaki
> >>>>
> >>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> Here's Hadoop 3.2.4 release candidate #0:
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >>>>>
> >>>>> The RC tag is at:
> >>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >>>>>
> >>>>> The Maven artifacts are staged at:
> >>>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://downloads.apache.org/hadoop/common/KEYS
> >>>>>
> >>>>> Please evaluate the RC and vote.
> >>>>> The vote will be open for (at least) 5 days.
> >>>>>
> >>>>> Thanks,
> >>>>> Masatake Iwasaki
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi developers,
I'm still waiting for your vote.
I'm considering the intermittent test failures mentioned by Chris are not blocker.
Please file a JIRA and let me know if you find a blocker issue.
I will appreciate your help for the release process.
Regards,
Masatake Iwasaki
On 2022/07/20 14:50, Masatake Iwasaki wrote:
>> TestServiceAM
>
> I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
> YARN 8867 did not modified other code under hadoop-yarn-services.
> If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
>
>
> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>> Thanks for testing the RC0, Chris.
>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>
>> I could not reproduce the test failures on my local.
>>
>> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>
>> The failure of TestSLSStreamAMSynth looks frequently reported by
>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>> It could be tagged as known flaky test.
>>
>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>> -0 (binding)
>>>
>>> * Verified all checksums.
>>> * Verified all signatures.
>>> * Built from source, including native code on Linux.
>>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>>> -Drequire.zstd -DskipTests
>>> * Tests mostly passed, but see below.
>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>>
>>> The following tests also failed, but they also fail for me on 3.2.3, so
>>> they aren't likely to be related to this release candidate:
>>> * TestCapacitySchedulerNodeLabelUpdate
>>> * TestFrameworkUploader
>>> * TestSLSGenericSynth
>>> * TestSLSRunner
>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>
>>> I'm not voting a full -1, because I haven't done any root cause analysis on
>>> these new test failures. I don't know if it's a quirk to my environment,
>>> though I'm using the start-build-env.sh Docker container, so any build
>>> dependencies should be consistent. I'd be comfortable moving ahead if
>>> others are seeing these tests pass.
>>>
>>> Chris Nauroth
>>>
>>>
>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>>> wrote:
>>>
>>>> +1 from myself.
>>>>
>>>> * skimmed the contents of site documentation.
>>>>
>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>>> `-Pnative`.
>>>>
>>>> * launched pseudo distributed cluster including kms and httpfs with
>>>> Kerberos and SSL enabled.
>>>>
>>>> * created encryption zone, put and read files via httpfs.
>>>> * ran example MR wordcount over encryption zone.
>>>>
>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>>> example MR jobs.
>>>>
>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>>> https://github.com/apache/bigtop/pull/942
>>>>
>>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>>> to reload4j).
>>>>
>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>>
>>>> <repository>
>>>> <id>staged</id>
>>>> <name>staged-releases</name>
>>>> <url>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>> </url>
>>>> <releases>
>>>> <enabled>true</enabled>
>>>> </releases>
>>>> <snapshots>
>>>> <enabled>true</enabled>
>>>> </snapshots>
>>>> </repository>
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>> Hi all,
>>>>>
>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>
>>>>> The RC is available at:
>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>
>>>>> The RC tag is at:
>>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>
>>>>> The Maven artifacts are staged at:
>>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>
>>>>> You can find my public key at:
>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>
>>>>> Please evaluate the RC and vote.
>>>>> The vote will be open for (at least) 5 days.
>>>>>
>>>>> Thanks,
>>>>> Masatake Iwasaki
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi developers,
I'm still waiting for your vote.
I'm considering the intermittent test failures mentioned by Chris are not blocker.
Please file a JIRA and let me know if you find a blocker issue.
I will appreciate your help for the release process.
Regards,
Masatake Iwasaki
On 2022/07/20 14:50, Masatake Iwasaki wrote:
>> TestServiceAM
>
> I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
> YARN 8867 did not modified other code under hadoop-yarn-services.
> If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
>
>
> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>> Thanks for testing the RC0, Chris.
>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>
>> I could not reproduce the test failures on my local.
>>
>> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>
>> The failure of TestSLSStreamAMSynth looks frequently reported by
>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>> It could be tagged as known flaky test.
>>
>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>> -0 (binding)
>>>
>>> * Verified all checksums.
>>> * Verified all signatures.
>>> * Built from source, including native code on Linux.
>>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>>> -Drequire.zstd -DskipTests
>>> * Tests mostly passed, but see below.
>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>>
>>> The following tests also failed, but they also fail for me on 3.2.3, so
>>> they aren't likely to be related to this release candidate:
>>> * TestCapacitySchedulerNodeLabelUpdate
>>> * TestFrameworkUploader
>>> * TestSLSGenericSynth
>>> * TestSLSRunner
>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>
>>> I'm not voting a full -1, because I haven't done any root cause analysis on
>>> these new test failures. I don't know if it's a quirk to my environment,
>>> though I'm using the start-build-env.sh Docker container, so any build
>>> dependencies should be consistent. I'd be comfortable moving ahead if
>>> others are seeing these tests pass.
>>>
>>> Chris Nauroth
>>>
>>>
>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>>> wrote:
>>>
>>>> +1 from myself.
>>>>
>>>> * skimmed the contents of site documentation.
>>>>
>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>>> `-Pnative`.
>>>>
>>>> * launched pseudo distributed cluster including kms and httpfs with
>>>> Kerberos and SSL enabled.
>>>>
>>>> * created encryption zone, put and read files via httpfs.
>>>> * ran example MR wordcount over encryption zone.
>>>>
>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>>> example MR jobs.
>>>>
>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>>> https://github.com/apache/bigtop/pull/942
>>>>
>>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>>> to reload4j).
>>>>
>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>>
>>>> <repository>
>>>> <id>staged</id>
>>>> <name>staged-releases</name>
>>>> <url>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>> </url>
>>>> <releases>
>>>> <enabled>true</enabled>
>>>> </releases>
>>>> <snapshots>
>>>> <enabled>true</enabled>
>>>> </snapshots>
>>>> </repository>
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>> Hi all,
>>>>>
>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>
>>>>> The RC is available at:
>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>
>>>>> The RC tag is at:
>>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>
>>>>> The Maven artifacts are staged at:
>>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>
>>>>> You can find my public key at:
>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>
>>>>> Please evaluate the RC and vote.
>>>>> The vote will be open for (at least) 5 days.
>>>>>
>>>>> Thanks,
>>>>> Masatake Iwasaki
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi developers,
I'm still waiting for your vote.
I'm considering the intermittent test failures mentioned by Chris are not blocker.
Please file a JIRA and let me know if you find a blocker issue.
I will appreciate your help for the release process.
Regards,
Masatake Iwasaki
On 2022/07/20 14:50, Masatake Iwasaki wrote:
>> TestServiceAM
>
> I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
> YARN 8867 did not modified other code under hadoop-yarn-services.
> If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
>
>
> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>> Thanks for testing the RC0, Chris.
>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>
>> I could not reproduce the test failures on my local.
>>
>> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>
>> The failure of TestSLSStreamAMSynth looks frequently reported by
>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>> It could be tagged as known flaky test.
>>
>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>> -0 (binding)
>>>
>>> * Verified all checksums.
>>> * Verified all signatures.
>>> * Built from source, including native code on Linux.
>>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>>> -Drequire.zstd -DskipTests
>>> * Tests mostly passed, but see below.
>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>>
>>> The following tests also failed, but they also fail for me on 3.2.3, so
>>> they aren't likely to be related to this release candidate:
>>> * TestCapacitySchedulerNodeLabelUpdate
>>> * TestFrameworkUploader
>>> * TestSLSGenericSynth
>>> * TestSLSRunner
>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>
>>> I'm not voting a full -1, because I haven't done any root cause analysis on
>>> these new test failures. I don't know if it's a quirk to my environment,
>>> though I'm using the start-build-env.sh Docker container, so any build
>>> dependencies should be consistent. I'd be comfortable moving ahead if
>>> others are seeing these tests pass.
>>>
>>> Chris Nauroth
>>>
>>>
>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>>> wrote:
>>>
>>>> +1 from myself.
>>>>
>>>> * skimmed the contents of site documentation.
>>>>
>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>>> `-Pnative`.
>>>>
>>>> * launched pseudo distributed cluster including kms and httpfs with
>>>> Kerberos and SSL enabled.
>>>>
>>>> * created encryption zone, put and read files via httpfs.
>>>> * ran example MR wordcount over encryption zone.
>>>>
>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>>> example MR jobs.
>>>>
>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>>> https://github.com/apache/bigtop/pull/942
>>>>
>>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>>> to reload4j).
>>>>
>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>>
>>>> <repository>
>>>> <id>staged</id>
>>>> <name>staged-releases</name>
>>>> <url>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>> </url>
>>>> <releases>
>>>> <enabled>true</enabled>
>>>> </releases>
>>>> <snapshots>
>>>> <enabled>true</enabled>
>>>> </snapshots>
>>>> </repository>
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>> Hi all,
>>>>>
>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>
>>>>> The RC is available at:
>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>
>>>>> The RC tag is at:
>>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>
>>>>> The Maven artifacts are staged at:
>>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>
>>>>> You can find my public key at:
>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>
>>>>> Please evaluate the RC and vote.
>>>>> The vote will be open for (at least) 5 days.
>>>>>
>>>>> Thanks,
>>>>> Masatake Iwasaki
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Hi developers,
I'm still waiting for your vote.
I'm considering the intermittent test failures mentioned by Chris are not blocker.
Please file a JIRA and let me know if you find a blocker issue.
I will appreciate your help for the release process.
Regards,
Masatake Iwasaki
On 2022/07/20 14:50, Masatake Iwasaki wrote:
>> TestServiceAM
>
> I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
> YARN 8867 did not modified other code under hadoop-yarn-services.
> If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
>
>
> On 2022/07/20 14:21, Masatake Iwasaki wrote:
>> Thanks for testing the RC0, Chris.
>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>
>> I could not reproduce the test failures on my local.
>>
>> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
>> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>>
>> The failure of TestSLSStreamAMSynth looks frequently reported by
>> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
>> It could be tagged as known flaky test.
>>
>> On 2022/07/20 9:15, Chris Nauroth wrote:
>>> -0 (binding)
>>>
>>> * Verified all checksums.
>>> * Verified all signatures.
>>> * Built from source, including native code on Linux.
>>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>>> -Drequire.zstd -DskipTests
>>> * Tests mostly passed, but see below.
>>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>>
>>> The following are new test failures for me on 3.2.4:
>>> * TestAMRMProxy
>>> * TestFsck
>>> * TestSLSStreamAMSynth
>>> * TestServiceAM
>>>
>>> The following tests also failed, but they also fail for me on 3.2.3, so
>>> they aren't likely to be related to this release candidate:
>>> * TestCapacitySchedulerNodeLabelUpdate
>>> * TestFrameworkUploader
>>> * TestSLSGenericSynth
>>> * TestSLSRunner
>>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>>
>>> I'm not voting a full -1, because I haven't done any root cause analysis on
>>> these new test failures. I don't know if it's a quirk to my environment,
>>> though I'm using the start-build-env.sh Docker container, so any build
>>> dependencies should be consistent. I'd be comfortable moving ahead if
>>> others are seeing these tests pass.
>>>
>>> Chris Nauroth
>>>
>>>
>>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>>> wrote:
>>>
>>>> +1 from myself.
>>>>
>>>> * skimmed the contents of site documentation.
>>>>
>>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>>> `-Pnative`.
>>>>
>>>> * launched pseudo distributed cluster including kms and httpfs with
>>>> Kerberos and SSL enabled.
>>>>
>>>> * created encryption zone, put and read files via httpfs.
>>>> * ran example MR wordcount over encryption zone.
>>>>
>>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>>> example MR jobs.
>>>>
>>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>>> https://github.com/apache/bigtop/pull/942
>>>>
>>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>>> to reload4j).
>>>>
>>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>>
>>>> <repository>
>>>> <id>staged</id>
>>>> <name>staged-releases</name>
>>>> <url>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>> </url>
>>>> <releases>
>>>> <enabled>true</enabled>
>>>> </releases>
>>>> <snapshots>
>>>> <enabled>true</enabled>
>>>> </snapshots>
>>>> </repository>
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>>> Hi all,
>>>>>
>>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>>
>>>>> The RC is available at:
>>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>>
>>>>> The RC tag is at:
>>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>>
>>>>> The Maven artifacts are staged at:
>>>>>
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>>
>>>>> You can find my public key at:
>>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>>
>>>>> Please evaluate the RC and vote.
>>>>> The vote will be open for (at least) 5 days.
>>>>>
>>>>> Thanks,
>>>>> Masatake Iwasaki
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> TestServiceAM
I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
YARN 8867 did not modified other code under hadoop-yarn-services.
If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
On 2022/07/20 14:21, Masatake Iwasaki wrote:
> Thanks for testing the RC0, Chris.
>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>
> I could not reproduce the test failures on my local.
>
> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>
> The failure of TestSLSStreamAMSynth looks frequently reported by
> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> It could be tagged as known flaky test.
>
> On 2022/07/20 9:15, Chris Nauroth wrote:
>> -0 (binding)
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests mostly passed, but see below.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>>
>> The following tests also failed, but they also fail for me on 3.2.3, so
>> they aren't likely to be related to this release candidate:
>> * TestCapacitySchedulerNodeLabelUpdate
>> * TestFrameworkUploader
>> * TestSLSGenericSynth
>> * TestSLSRunner
>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>
>> I'm not voting a full -1, because I haven't done any root cause analysis on
>> these new test failures. I don't know if it's a quirk to my environment,
>> though I'm using the start-build-env.sh Docker container, so any build
>> dependencies should be consistent. I'd be comfortable moving ahead if
>> others are seeing these tests pass.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>> wrote:
>>
>>> +1 from myself.
>>>
>>> * skimmed the contents of site documentation.
>>>
>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>> `-Pnative`.
>>>
>>> * launched pseudo distributed cluster including kms and httpfs with
>>> Kerberos and SSL enabled.
>>>
>>> * created encryption zone, put and read files via httpfs.
>>> * ran example MR wordcount over encryption zone.
>>>
>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>> example MR jobs.
>>>
>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>> https://github.com/apache/bigtop/pull/942
>>>
>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>> to reload4j).
>>>
>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>
>>> <repository>
>>> <id>staged</id>
>>> <name>staged-releases</name>
>>> <url>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>> </url>
>>> <releases>
>>> <enabled>true</enabled>
>>> </releases>
>>> <snapshots>
>>> <enabled>true</enabled>
>>> </snapshots>
>>> </repository>
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>> Hi all,
>>>>
>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>
>>>> The RC is available at:
>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>
>>>> The RC tag is at:
>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>
>>>> The Maven artifacts are staged at:
>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>
>>>> You can find my public key at:
>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>
>>>> Please evaluate the RC and vote.
>>>> The vote will be open for (at least) 5 days.
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> TestServiceAM
I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
YARN 8867 did not modified other code under hadoop-yarn-services.
If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
On 2022/07/20 14:21, Masatake Iwasaki wrote:
> Thanks for testing the RC0, Chris.
>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>
> I could not reproduce the test failures on my local.
>
> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>
> The failure of TestSLSStreamAMSynth looks frequently reported by
> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> It could be tagged as known flaky test.
>
> On 2022/07/20 9:15, Chris Nauroth wrote:
>> -0 (binding)
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests mostly passed, but see below.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>>
>> The following tests also failed, but they also fail for me on 3.2.3, so
>> they aren't likely to be related to this release candidate:
>> * TestCapacitySchedulerNodeLabelUpdate
>> * TestFrameworkUploader
>> * TestSLSGenericSynth
>> * TestSLSRunner
>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>
>> I'm not voting a full -1, because I haven't done any root cause analysis on
>> these new test failures. I don't know if it's a quirk to my environment,
>> though I'm using the start-build-env.sh Docker container, so any build
>> dependencies should be consistent. I'd be comfortable moving ahead if
>> others are seeing these tests pass.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>> wrote:
>>
>>> +1 from myself.
>>>
>>> * skimmed the contents of site documentation.
>>>
>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>> `-Pnative`.
>>>
>>> * launched pseudo distributed cluster including kms and httpfs with
>>> Kerberos and SSL enabled.
>>>
>>> * created encryption zone, put and read files via httpfs.
>>> * ran example MR wordcount over encryption zone.
>>>
>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>> example MR jobs.
>>>
>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>> https://github.com/apache/bigtop/pull/942
>>>
>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>> to reload4j).
>>>
>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>
>>> <repository>
>>> <id>staged</id>
>>> <name>staged-releases</name>
>>> <url>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>> </url>
>>> <releases>
>>> <enabled>true</enabled>
>>> </releases>
>>> <snapshots>
>>> <enabled>true</enabled>
>>> </snapshots>
>>> </repository>
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>> Hi all,
>>>>
>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>
>>>> The RC is available at:
>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>
>>>> The RC tag is at:
>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>
>>>> The Maven artifacts are staged at:
>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>
>>>> You can find my public key at:
>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>
>>>> Please evaluate the RC and vote.
>>>> The vote will be open for (at least) 5 days.
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> TestServiceAM
I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
YARN 8867 did not modified other code under hadoop-yarn-services.
If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
On 2022/07/20 14:21, Masatake Iwasaki wrote:
> Thanks for testing the RC0, Chris.
>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>
> I could not reproduce the test failures on my local.
>
> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>
> The failure of TestSLSStreamAMSynth looks frequently reported by
> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> It could be tagged as known flaky test.
>
> On 2022/07/20 9:15, Chris Nauroth wrote:
>> -0 (binding)
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests mostly passed, but see below.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>>
>> The following tests also failed, but they also fail for me on 3.2.3, so
>> they aren't likely to be related to this release candidate:
>> * TestCapacitySchedulerNodeLabelUpdate
>> * TestFrameworkUploader
>> * TestSLSGenericSynth
>> * TestSLSRunner
>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>
>> I'm not voting a full -1, because I haven't done any root cause analysis on
>> these new test failures. I don't know if it's a quirk to my environment,
>> though I'm using the start-build-env.sh Docker container, so any build
>> dependencies should be consistent. I'd be comfortable moving ahead if
>> others are seeing these tests pass.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>> wrote:
>>
>>> +1 from myself.
>>>
>>> * skimmed the contents of site documentation.
>>>
>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>> `-Pnative`.
>>>
>>> * launched pseudo distributed cluster including kms and httpfs with
>>> Kerberos and SSL enabled.
>>>
>>> * created encryption zone, put and read files via httpfs.
>>> * ran example MR wordcount over encryption zone.
>>>
>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>> example MR jobs.
>>>
>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>> https://github.com/apache/bigtop/pull/942
>>>
>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>> to reload4j).
>>>
>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>
>>> <repository>
>>> <id>staged</id>
>>> <name>staged-releases</name>
>>> <url>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>> </url>
>>> <releases>
>>> <enabled>true</enabled>
>>> </releases>
>>> <snapshots>
>>> <enabled>true</enabled>
>>> </snapshots>
>>> </repository>
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>> Hi all,
>>>>
>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>
>>>> The RC is available at:
>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>
>>>> The RC tag is at:
>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>
>>>> The Maven artifacts are staged at:
>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>
>>>> You can find my public key at:
>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>
>>>> Please evaluate the RC and vote.
>>>> The vote will be open for (at least) 5 days.
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> TestServiceAM
I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64".
3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails.
YARN 8867 did not modified other code under hadoop-yarn-services.
If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
On 2022/07/20 14:21, Masatake Iwasaki wrote:
> Thanks for testing the RC0, Chris.
>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>
> I could not reproduce the test failures on my local.
>
> For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
>
> The failure of TestSLSStreamAMSynth looks frequently reported by
> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> It could be tagged as known flaky test.
>
> On 2022/07/20 9:15, Chris Nauroth wrote:
>> -0 (binding)
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests mostly passed, but see below.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>>
>> The following are new test failures for me on 3.2.4:
>> * TestAMRMProxy
>> * TestFsck
>> * TestSLSStreamAMSynth
>> * TestServiceAM
>>
>> The following tests also failed, but they also fail for me on 3.2.3, so
>> they aren't likely to be related to this release candidate:
>> * TestCapacitySchedulerNodeLabelUpdate
>> * TestFrameworkUploader
>> * TestSLSGenericSynth
>> * TestSLSRunner
>> * test_libhdfs_threaded_hdfspp_test_shim_static
>>
>> I'm not voting a full -1, because I haven't done any root cause analysis on
>> these new test failures. I don't know if it's a quirk to my environment,
>> though I'm using the start-build-env.sh Docker container, so any build
>> dependencies should be consistent. I'd be comfortable moving ahead if
>> others are seeing these tests pass.
>>
>> Chris Nauroth
>>
>>
>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
>> wrote:
>>
>>> +1 from myself.
>>>
>>> * skimmed the contents of site documentation.
>>>
>>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>>> `-Pnative`.
>>>
>>> * launched pseudo distributed cluster including kms and httpfs with
>>> Kerberos and SSL enabled.
>>>
>>> * created encryption zone, put and read files via httpfs.
>>> * ran example MR wordcount over encryption zone.
>>>
>>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>>> example MR jobs.
>>>
>>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>>> https://github.com/apache/bigtop/pull/942
>>>
>>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>>> to reload4j).
>>>
>>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>>
>>> <repository>
>>> <id>staged</id>
>>> <name>staged-releases</name>
>>> <url>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>> </url>
>>> <releases>
>>> <enabled>true</enabled>
>>> </releases>
>>> <snapshots>
>>> <enabled>true</enabled>
>>> </snapshots>
>>> </repository>
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>>> Hi all,
>>>>
>>>> Here's Hadoop 3.2.4 release candidate #0:
>>>>
>>>> The RC is available at:
>>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>>
>>>> The RC tag is at:
>>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>>
>>>> The Maven artifacts are staged at:
>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>>
>>>> You can find my public key at:
>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>
>>>> Please evaluate the RC and vote.
>>>> The vote will be open for (at least) 5 days.
>>>>
>>>> Thanks,
>>>> Masatake Iwasaki
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for testing the RC0, Chris.
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
I could not reproduce the test failures on my local.
For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
cherry-picking HDFS-15038 (fixing only test code) could be the fix.
The failure of TestSLSStreamAMSynth looks frequently reported by
"Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
It could be tagged as known flaky test.
On 2022/07/20 9:15, Chris Nauroth wrote:
> -0 (binding)
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests mostly passed, but see below.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
>
> The following tests also failed, but they also fail for me on 3.2.3, so
> they aren't likely to be related to this release candidate:
> * TestCapacitySchedulerNodeLabelUpdate
> * TestFrameworkUploader
> * TestSLSGenericSynth
> * TestSLSRunner
> * test_libhdfs_threaded_hdfspp_test_shim_static
>
> I'm not voting a full -1, because I haven't done any root cause analysis on
> these new test failures. I don't know if it's a quirk to my environment,
> though I'm using the start-build-env.sh Docker container, so any build
> dependencies should be consistent. I'd be comfortable moving ahead if
> others are seeing these tests pass.
>
> Chris Nauroth
>
>
> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
> wrote:
>
>> +1 from myself.
>>
>> * skimmed the contents of site documentation.
>>
>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>> `-Pnative`.
>>
>> * launched pseudo distributed cluster including kms and httpfs with
>> Kerberos and SSL enabled.
>>
>> * created encryption zone, put and read files via httpfs.
>> * ran example MR wordcount over encryption zone.
>>
>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>> example MR jobs.
>>
>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>> https://github.com/apache/bigtop/pull/942
>>
>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>> to reload4j).
>>
>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>
>> <repository>
>> <id>staged</id>
>> <name>staged-releases</name>
>> <url>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>> </url>
>> <releases>
>> <enabled>true</enabled>
>> </releases>
>> <snapshots>
>> <enabled>true</enabled>
>> </snapshots>
>> </repository>
>>
>> Thanks,
>> Masatake Iwasaki
>>
>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>> Hi all,
>>>
>>> Here's Hadoop 3.2.4 release candidate #0:
>>>
>>> The RC is available at:
>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>
>>> The RC tag is at:
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>
>>> The Maven artifacts are staged at:
>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>
>>> You can find my public key at:
>>> https://downloads.apache.org/hadoop/common/KEYS
>>>
>>> Please evaluate the RC and vote.
>>> The vote will be open for (at least) 5 days.
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for testing the RC0, Chris.
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
I could not reproduce the test failures on my local.
For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
cherry-picking HDFS-15038 (fixing only test code) could be the fix.
The failure of TestSLSStreamAMSynth looks frequently reported by
"Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
It could be tagged as known flaky test.
On 2022/07/20 9:15, Chris Nauroth wrote:
> -0 (binding)
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests mostly passed, but see below.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
>
> The following tests also failed, but they also fail for me on 3.2.3, so
> they aren't likely to be related to this release candidate:
> * TestCapacitySchedulerNodeLabelUpdate
> * TestFrameworkUploader
> * TestSLSGenericSynth
> * TestSLSRunner
> * test_libhdfs_threaded_hdfspp_test_shim_static
>
> I'm not voting a full -1, because I haven't done any root cause analysis on
> these new test failures. I don't know if it's a quirk to my environment,
> though I'm using the start-build-env.sh Docker container, so any build
> dependencies should be consistent. I'd be comfortable moving ahead if
> others are seeing these tests pass.
>
> Chris Nauroth
>
>
> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
> wrote:
>
>> +1 from myself.
>>
>> * skimmed the contents of site documentation.
>>
>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>> `-Pnative`.
>>
>> * launched pseudo distributed cluster including kms and httpfs with
>> Kerberos and SSL enabled.
>>
>> * created encryption zone, put and read files via httpfs.
>> * ran example MR wordcount over encryption zone.
>>
>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>> example MR jobs.
>>
>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>> https://github.com/apache/bigtop/pull/942
>>
>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>> to reload4j).
>>
>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>
>> <repository>
>> <id>staged</id>
>> <name>staged-releases</name>
>> <url>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>> </url>
>> <releases>
>> <enabled>true</enabled>
>> </releases>
>> <snapshots>
>> <enabled>true</enabled>
>> </snapshots>
>> </repository>
>>
>> Thanks,
>> Masatake Iwasaki
>>
>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>> Hi all,
>>>
>>> Here's Hadoop 3.2.4 release candidate #0:
>>>
>>> The RC is available at:
>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>
>>> The RC tag is at:
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>
>>> The Maven artifacts are staged at:
>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>
>>> You can find my public key at:
>>> https://downloads.apache.org/hadoop/common/KEYS
>>>
>>> Please evaluate the RC and vote.
>>> The vote will be open for (at least) 5 days.
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for testing the RC0, Chris.
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
I could not reproduce the test failures on my local.
For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
cherry-picking HDFS-15038 (fixing only test code) could be the fix.
The failure of TestSLSStreamAMSynth looks frequently reported by
"Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
It could be tagged as known flaky test.
On 2022/07/20 9:15, Chris Nauroth wrote:
> -0 (binding)
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests mostly passed, but see below.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
>
> The following tests also failed, but they also fail for me on 3.2.3, so
> they aren't likely to be related to this release candidate:
> * TestCapacitySchedulerNodeLabelUpdate
> * TestFrameworkUploader
> * TestSLSGenericSynth
> * TestSLSRunner
> * test_libhdfs_threaded_hdfspp_test_shim_static
>
> I'm not voting a full -1, because I haven't done any root cause analysis on
> these new test failures. I don't know if it's a quirk to my environment,
> though I'm using the start-build-env.sh Docker container, so any build
> dependencies should be consistent. I'd be comfortable moving ahead if
> others are seeing these tests pass.
>
> Chris Nauroth
>
>
> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
> wrote:
>
>> +1 from myself.
>>
>> * skimmed the contents of site documentation.
>>
>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>> `-Pnative`.
>>
>> * launched pseudo distributed cluster including kms and httpfs with
>> Kerberos and SSL enabled.
>>
>> * created encryption zone, put and read files via httpfs.
>> * ran example MR wordcount over encryption zone.
>>
>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>> example MR jobs.
>>
>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>> https://github.com/apache/bigtop/pull/942
>>
>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>> to reload4j).
>>
>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>
>> <repository>
>> <id>staged</id>
>> <name>staged-releases</name>
>> <url>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>> </url>
>> <releases>
>> <enabled>true</enabled>
>> </releases>
>> <snapshots>
>> <enabled>true</enabled>
>> </snapshots>
>> </repository>
>>
>> Thanks,
>> Masatake Iwasaki
>>
>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>> Hi all,
>>>
>>> Here's Hadoop 3.2.4 release candidate #0:
>>>
>>> The RC is available at:
>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>
>>> The RC tag is at:
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>
>>> The Maven artifacts are staged at:
>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>
>>> You can find my public key at:
>>> https://downloads.apache.org/hadoop/common/KEYS
>>>
>>> Please evaluate the RC and vote.
>>> The vote will be open for (at least) 5 days.
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
Thanks for testing the RC0, Chris.
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
I could not reproduce the test failures on my local.
For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles,
cherry-picking HDFS-15038 (fixing only test code) could be the fix.
The failure of TestSLSStreamAMSynth looks frequently reported by
"Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
It could be tagged as known flaky test.
On 2022/07/20 9:15, Chris Nauroth wrote:
> -0 (binding)
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests mostly passed, but see below.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>
> The following are new test failures for me on 3.2.4:
> * TestAMRMProxy
> * TestFsck
> * TestSLSStreamAMSynth
> * TestServiceAM
>
> The following tests also failed, but they also fail for me on 3.2.3, so
> they aren't likely to be related to this release candidate:
> * TestCapacitySchedulerNodeLabelUpdate
> * TestFrameworkUploader
> * TestSLSGenericSynth
> * TestSLSRunner
> * test_libhdfs_threaded_hdfspp_test_shim_static
>
> I'm not voting a full -1, because I haven't done any root cause analysis on
> these new test failures. I don't know if it's a quirk to my environment,
> though I'm using the start-build-env.sh Docker container, so any build
> dependencies should be consistent. I'd be comfortable moving ahead if
> others are seeing these tests pass.
>
> Chris Nauroth
>
>
> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
> wrote:
>
>> +1 from myself.
>>
>> * skimmed the contents of site documentation.
>>
>> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>> `-Pnative`.
>>
>> * launched pseudo distributed cluster including kms and httpfs with
>> Kerberos and SSL enabled.
>>
>> * created encryption zone, put and read files via httpfs.
>> * ran example MR wordcount over encryption zone.
>>
>> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
>> example MR jobs.
>>
>> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
>> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
>> https://github.com/apache/bigtop/pull/942
>>
>> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
>> to reload4j).
>>
>> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>>
>> <repository>
>> <id>staged</id>
>> <name>staged-releases</name>
>> <url>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>> </url>
>> <releases>
>> <enabled>true</enabled>
>> </releases>
>> <snapshots>
>> <enabled>true</enabled>
>> </snapshots>
>> </repository>
>>
>> Thanks,
>> Masatake Iwasaki
>>
>> On 2022/07/13 1:14, Masatake Iwasaki wrote:
>>> Hi all,
>>>
>>> Here's Hadoop 3.2.4 release candidate #0:
>>>
>>> The RC is available at:
>>> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>>>
>>> The RC tag is at:
>>> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>>>
>>> The Maven artifacts are staged at:
>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>>>
>>> You can find my public key at:
>>> https://downloads.apache.org/hadoop/common/KEYS
>>>
>>> Please evaluate the RC and vote.
>>> The vote will be open for (at least) 5 days.
>>>
>>> Thanks,
>>> Masatake Iwasaki
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
-0 (binding)
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests mostly passed, but see below.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
The following are new test failures for me on 3.2.4:
* TestAMRMProxy
* TestFsck
* TestSLSStreamAMSynth
* TestServiceAM
The following tests also failed, but they also fail for me on 3.2.3, so
they aren't likely to be related to this release candidate:
* TestCapacitySchedulerNodeLabelUpdate
* TestFrameworkUploader
* TestSLSGenericSynth
* TestSLSRunner
* test_libhdfs_threaded_hdfspp_test_shim_static
I'm not voting a full -1, because I haven't done any root cause analysis on
these new test failures. I don't know if it's a quirk to my environment,
though I'm using the start-build-env.sh Docker container, so any build
dependencies should be consistent. I'd be comfortable moving ahead if
others are seeing these tests pass.
Chris Nauroth
On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> +1 from myself.
>
> * skimmed the contents of site documentation.
>
> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> `-Pnative`.
>
> * launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
>
> * created encryption zone, put and read files via httpfs.
> * ran example MR wordcount over encryption zone.
>
> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
> example MR jobs.
>
> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
> https://github.com/apache/bigtop/pull/942
>
> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
> to reload4j).
>
> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>
> <repository>
> <id>staged</id>
> <name>staged-releases</name>
> <url>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> </url>
> <releases>
> <enabled>true</enabled>
> </releases>
> <snapshots>
> <enabled>true</enabled>
> </snapshots>
> </repository>
>
> Thanks,
> Masatake Iwasaki
>
> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > Hi all,
> >
> > Here's Hadoop 3.2.4 release candidate #0:
> >
> > The RC is available at:
> > https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >
> > The RC tag is at:
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >
> > The Maven artifacts are staged at:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >
> > You can find my public key at:
> > https://downloads.apache.org/hadoop/common/KEYS
> >
> > Please evaluate the RC and vote.
> > The vote will be open for (at least) 5 days.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
-0 (binding)
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests mostly passed, but see below.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
The following are new test failures for me on 3.2.4:
* TestAMRMProxy
* TestFsck
* TestSLSStreamAMSynth
* TestServiceAM
The following tests also failed, but they also fail for me on 3.2.3, so
they aren't likely to be related to this release candidate:
* TestCapacitySchedulerNodeLabelUpdate
* TestFrameworkUploader
* TestSLSGenericSynth
* TestSLSRunner
* test_libhdfs_threaded_hdfspp_test_shim_static
I'm not voting a full -1, because I haven't done any root cause analysis on
these new test failures. I don't know if it's a quirk to my environment,
though I'm using the start-build-env.sh Docker container, so any build
dependencies should be consistent. I'd be comfortable moving ahead if
others are seeing these tests pass.
Chris Nauroth
On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> +1 from myself.
>
> * skimmed the contents of site documentation.
>
> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> `-Pnative`.
>
> * launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
>
> * created encryption zone, put and read files via httpfs.
> * ran example MR wordcount over encryption zone.
>
> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
> example MR jobs.
>
> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
> https://github.com/apache/bigtop/pull/942
>
> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
> to reload4j).
>
> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>
> <repository>
> <id>staged</id>
> <name>staged-releases</name>
> <url>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> </url>
> <releases>
> <enabled>true</enabled>
> </releases>
> <snapshots>
> <enabled>true</enabled>
> </snapshots>
> </repository>
>
> Thanks,
> Masatake Iwasaki
>
> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > Hi all,
> >
> > Here's Hadoop 3.2.4 release candidate #0:
> >
> > The RC is available at:
> > https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >
> > The RC tag is at:
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >
> > The Maven artifacts are staged at:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >
> > You can find my public key at:
> > https://downloads.apache.org/hadoop/common/KEYS
> >
> > Please evaluate the RC and vote.
> > The vote will be open for (at least) 5 days.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
-0 (binding)
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests mostly passed, but see below.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
The following are new test failures for me on 3.2.4:
* TestAMRMProxy
* TestFsck
* TestSLSStreamAMSynth
* TestServiceAM
The following tests also failed, but they also fail for me on 3.2.3, so
they aren't likely to be related to this release candidate:
* TestCapacitySchedulerNodeLabelUpdate
* TestFrameworkUploader
* TestSLSGenericSynth
* TestSLSRunner
* test_libhdfs_threaded_hdfspp_test_shim_static
I'm not voting a full -1, because I haven't done any root cause analysis on
these new test failures. I don't know if it's a quirk to my environment,
though I'm using the start-build-env.sh Docker container, so any build
dependencies should be consistent. I'd be comfortable moving ahead if
others are seeing these tests pass.
Chris Nauroth
On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> +1 from myself.
>
> * skimmed the contents of site documentation.
>
> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> `-Pnative`.
>
> * launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
>
> * created encryption zone, put and read files via httpfs.
> * ran example MR wordcount over encryption zone.
>
> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
> example MR jobs.
>
> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
> https://github.com/apache/bigtop/pull/942
>
> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
> to reload4j).
>
> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>
> <repository>
> <id>staged</id>
> <name>staged-releases</name>
> <url>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> </url>
> <releases>
> <enabled>true</enabled>
> </releases>
> <snapshots>
> <enabled>true</enabled>
> </snapshots>
> </repository>
>
> Thanks,
> Masatake Iwasaki
>
> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > Hi all,
> >
> > Here's Hadoop 3.2.4 release candidate #0:
> >
> > The RC is available at:
> > https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >
> > The RC tag is at:
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >
> > The Maven artifacts are staged at:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >
> > You can find my public key at:
> > https://downloads.apache.org/hadoop/common/KEYS
> >
> > Please evaluate the RC and vote.
> > The vote will be open for (at least) 5 days.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Chris Nauroth <cn...@apache.org>.
-0 (binding)
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests mostly passed, but see below.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
The following are new test failures for me on 3.2.4:
* TestAMRMProxy
* TestFsck
* TestSLSStreamAMSynth
* TestServiceAM
The following tests also failed, but they also fail for me on 3.2.3, so
they aren't likely to be related to this release candidate:
* TestCapacitySchedulerNodeLabelUpdate
* TestFrameworkUploader
* TestSLSGenericSynth
* TestSLSRunner
* test_libhdfs_threaded_hdfspp_test_shim_static
I'm not voting a full -1, because I haven't done any root cause analysis on
these new test failures. I don't know if it's a quirk to my environment,
though I'm using the start-build-env.sh Docker container, so any build
dependencies should be consistent. I'd be comfortable moving ahead if
others are seeing these tests pass.
Chris Nauroth
On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> +1 from myself.
>
> * skimmed the contents of site documentation.
>
> * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
> `-Pnative`.
>
> * launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
>
> * created encryption zone, put and read files via httpfs.
> * ran example MR wordcount over encryption zone.
>
> * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some
> example MR jobs.
>
> * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
> on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
> https://github.com/apache/bigtop/pull/942
>
> * Hive needs updating exclusion rule to address HADOOP-18088 (migration
> to reload4j).
>
> * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
>
> <repository>
> <id>staged</id>
> <name>staged-releases</name>
> <url>
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> </url>
> <releases>
> <enabled>true</enabled>
> </releases>
> <snapshots>
> <enabled>true</enabled>
> </snapshots>
> </repository>
>
> Thanks,
> Masatake Iwasaki
>
> On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > Hi all,
> >
> > Here's Hadoop 3.2.4 release candidate #0:
> >
> > The RC is available at:
> > https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
> >
> > The RC tag is at:
> > https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
> >
> > The Maven artifacts are staged at:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
> >
> > You can find my public key at:
> > https://downloads.apache.org/hadoop/common/KEYS
> >
> > Please evaluate the RC and vote.
> > The vote will be open for (at least) 5 days.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 from myself.
* skimmed the contents of site documentation.
* built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`.
* launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
* created encryption zone, put and read files via httpfs.
* ran example MR wordcount over encryption zone.
* launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs.
* built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
https://github.com/apache/bigtop/pull/942
* Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j).
* built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
<repository>
<id>staged</id>
<name>staged-releases</name>
<url>https://repository.apache.org/content/repositories/orgapachehadoop-1354</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
Thanks,
Masatake Iwasaki
On 2022/07/13 1:14, Masatake Iwasaki wrote:
> Hi all,
>
> Here's Hadoop 3.2.4 release candidate #0:
>
> The RC is available at:
> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 from myself.
* skimmed the contents of site documentation.
* built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`.
* launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
* created encryption zone, put and read files via httpfs.
* ran example MR wordcount over encryption zone.
* launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs.
* built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
https://github.com/apache/bigtop/pull/942
* Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j).
* built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
<repository>
<id>staged</id>
<name>staged-releases</name>
<url>https://repository.apache.org/content/repositories/orgapachehadoop-1354</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
Thanks,
Masatake Iwasaki
On 2022/07/13 1:14, Masatake Iwasaki wrote:
> Hi all,
>
> Here's Hadoop 3.2.4 release candidate #0:
>
> The RC is available at:
> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 from myself.
* skimmed the contents of site documentation.
* built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`.
* launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
* created encryption zone, put and read files via httpfs.
* ran example MR wordcount over encryption zone.
* launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs.
* built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
https://github.com/apache/bigtop/pull/942
* Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j).
* built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
<repository>
<id>staged</id>
<name>staged-releases</name>
<url>https://repository.apache.org/content/repositories/orgapachehadoop-1354</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
Thanks,
Masatake Iwasaki
On 2022/07/13 1:14, Masatake Iwasaki wrote:
> Hi all,
>
> Here's Hadoop 3.2.4 release candidate #0:
>
> The RC is available at:
> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
+1 from myself.
* skimmed the contents of site documentation.
* built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`.
* launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
* created encryption zone, put and read files via httpfs.
* ran example MR wordcount over encryption zone.
* launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs.
* built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0
on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests.
https://github.com/apache/bigtop/pull/942
* Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j).
* built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository::
<repository>
<id>staged</id>
<name>staged-releases</name>
<url>https://repository.apache.org/content/repositories/orgapachehadoop-1354</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
Thanks,
Masatake Iwasaki
On 2022/07/13 1:14, Masatake Iwasaki wrote:
> Hi all,
>
> Here's Hadoop 3.2.4 release candidate #0:
>
> The RC is available at:
> https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/
>
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0
>
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org