You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by Steve Loughran <st...@cloudera.com.INVALID> on 2022/05/03 11:18:55 UTC

[VOTE] Release Apache Hadoop 3.3.3

I have put together a release candidate (rc0) for Hadoop 3.3.3

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/

The git tag is release-3.3.3-RC0, commit d37586cbda3

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1348/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.


   - The critical fixes which shipped in the 3.2.3 release.
   -  CVEs in our code and dependencies
   - Shaded client packaging issues.
   - A switch from log4j to reload4j


reload4j is an active fork of the log4j 1.17 library with the classes which
contain CVEs removed. Even though hadoop never used those classes, they
regularly raised alerts on security scans and concen from users. Switching
to the forked project allows us to ship a secure logging framework. It will
complicate the builds of downstream maven/ivy/gradle projects which exclude
our log4j artifacts, as they need to cut the new dependency instead/as well.

See the release notes for details.

This is my first release through the new docker build process, do please
validate artifact signing &c to make sure it is good. I'll be trying builds
of downstream projects.

We know there are some outstanding issues with at least one library we are
shipping (okhttp), but I don't want to hold this release up for it. If the
docker based release process works smoothly enough we can do a followup
security release in a few weeks.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

had a report of that staging playing up too from Gautham

I'm going to have to treat this as a failure from the maven deploy process,
so I'll have to cancel this RC and try and work out WTF went wrong.

I'll also put together some minimal apps to fetch and compile against
everything. i did do some myself (storediag, hboss), and did try to make
sure I had cleaned up my local repo first, but maybe it was tainted.

oh well

-------

However, I get this error saying that it can’t find the maven plugins and
jars for Hadoop 3.3.3 –

[INFO] ------------------< org.apache.hadoop:hadoop-common
>-------------------

[INFO] Building Apache Hadoop Common 3.3.3
[5/9]

[INFO] --------------------------------[ jar
]---------------------------------

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.pom

[WARNING] The POM for org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 is
missing, no dependency information available

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.jar

[INFO]
------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop Common Project 3.3.3:

[INFO]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [
7.349 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
1.810 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [
7.922 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
0.375 s]

[INFO] Apache Hadoop Common ............................... FAILURE [
2.095 s]

[INFO] Apache Hadoop NFS .................................. SKIPPED

[INFO] Apache Hadoop KMS .................................. SKIPPED

[INFO] Apache Hadoop Registry ............................. SKIPPED

[INFO] Apache Hadoop Common Project ....................... SKIPPED

[INFO]
------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO]
------------------------------------------------------------------------

[INFO] Total time:  24.946 s

[INFO] Finished at: 2022-05-07T11:29:18+05:30

[INFO]
------------------------------------------------------------------------

[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.3.3 or one of its
dependencies could not be resolved: Could not find artifact
org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 in central (
https://repository.apache.org/content/repositories/orgapachehadoop-1348/)
-> [Help 1]

[ERROR]



Could you please let me know how I can consume the Hadoop 3 jars in maven?



Thanks,

--Gautham

On Sat, 7 May 2022 at 22:45, Akira Ajisaka <aa...@apache.org> wrote:

> Hi Chao,
>
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
>
> Akira
>
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm, I see the artifacts ideally should have got overwritten by the new
> > RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> > have any jars…
> > That is why it was picking the old jars. I think Steve needs to run mvn
> > deploy again…
> >
> > Sent from my iPhone
> >
> > > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >
> > > 
> > >>
> > >> Chao can you use the one that Steve mentioned in the mail?
> > >
> > > Hmm how do I do that? Typically after closing the RC in nexus the
> > > release bits will show up in
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > and Spark build will be able to pick them up for testing. However in
> > > this case I don't see any 3.3.3 jars in the URL.
> > >
> > >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> > >>
> > >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > date was also april 22, I archived that. Chao can you use the one that
> > Steve mentioned in the mail?
> > >>
> > >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> Seems there are some issues with the shaded client as I was not able
> > >>> to compile Apache Spark with the RC
> > >>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> > >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>> contain any class:
> > >>>
> > >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>> META-INF/
> > >>> META-INF/MANIFEST.MF
> > >>> META-INF/NOTICE.txt
> > >>> META-INF/LICENSE.txt
> > >>> META-INF/maven/
> > >>> META-INF/maven/org.apache.hadoop/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>
> > >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>>  * Signature: ok
> > >>>>  * Checksum : passed
> > >>>>  * Rat check (1.8.0_191): passed
> > >>>>   - mvn clean apache-rat:check
> > >>>>  * Built from source (1.8.0_191): failed
> > >>>>   - mvn clean install  -DskipTests
> > >>>>   - mvn -fae --no-transfer-progress -DskipTests
> > -Dmaven.javadoc.skip=true
> > >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>  * Unit tests pass (1.8.0_191):
> > >>>>    - HDFS Tests passed (Didn't run more than this).
> > >>>>
> > >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> > >>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> > >>>> bigger job with mild chaos. All seems to be working properly
> > (recoveries,
> > >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> > >>>> good. Poked at the hdfs cli. Seems good.
> > >>>>
> > >>>> S
> > >>>>
> > >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > >>>> wrote:
> > >>>>
> > >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>
> > >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>
> > >>>>> The maven artifacts are staged at
> > >>>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>
> > >>>>> Change log
> > >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>
> > >>>>> Release notes
> > >>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>
> > >>>>> There's a very small number of changes, primarily critical
> > code/packaging
> > >>>>> issues and security fixes.
> > >>>>>
> > >>>>>
> > >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>   -  CVEs in our code and dependencies
> > >>>>>   - Shaded client packaging issues.
> > >>>>>   - A switch from log4j to reload4j
> > >>>>>
> > >>>>>
> > >>>>> reload4j is an active fork of the log4j 1.17 library with the
> > classes which
> > >>>>> contain CVEs removed. Even though hadoop never used those classes,
> > they
> > >>>>> regularly raised alerts on security scans and concen from users.
> > Switching
> > >>>>> to the forked project allows us to ship a secure logging framework.
> > It will
> > >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > exclude
> > >>>>> our log4j artifacts, as they need to cut the new dependency
> > instead/as
> > >>>>> well.
> > >>>>>
> > >>>>> See the release notes for details.
> > >>>>>
> > >>>>> This is my first release through the new docker build process, do
> > please
> > >>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> > builds
> > >>>>> of downstream projects.
> > >>>>>
> > >>>>> We know there are some outstanding issues with at least one library
> > we are
> > >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > If the
> > >>>>> docker based release process works smoothly enough we can do a
> > followup
> > >>>>> security release in a few weeks.
> > >>>>>
> > >>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>
> > >>>>> -Steve
> > >>>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

had a report of that staging playing up too from Gautham

I'm going to have to treat this as a failure from the maven deploy process,
so I'll have to cancel this RC and try and work out WTF went wrong.

I'll also put together some minimal apps to fetch and compile against
everything. i did do some myself (storediag, hboss), and did try to make
sure I had cleaned up my local repo first, but maybe it was tainted.

oh well

-------

However, I get this error saying that it can’t find the maven plugins and
jars for Hadoop 3.3.3 –

[INFO] ------------------< org.apache.hadoop:hadoop-common
>-------------------

[INFO] Building Apache Hadoop Common 3.3.3
[5/9]

[INFO] --------------------------------[ jar
]---------------------------------

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.pom

[WARNING] The POM for org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 is
missing, no dependency information available

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.jar

[INFO]
------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop Common Project 3.3.3:

[INFO]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [
7.349 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
1.810 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [
7.922 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
0.375 s]

[INFO] Apache Hadoop Common ............................... FAILURE [
2.095 s]

[INFO] Apache Hadoop NFS .................................. SKIPPED

[INFO] Apache Hadoop KMS .................................. SKIPPED

[INFO] Apache Hadoop Registry ............................. SKIPPED

[INFO] Apache Hadoop Common Project ....................... SKIPPED

[INFO]
------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO]
------------------------------------------------------------------------

[INFO] Total time:  24.946 s

[INFO] Finished at: 2022-05-07T11:29:18+05:30

[INFO]
------------------------------------------------------------------------

[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.3.3 or one of its
dependencies could not be resolved: Could not find artifact
org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 in central (
https://repository.apache.org/content/repositories/orgapachehadoop-1348/)
-> [Help 1]

[ERROR]



Could you please let me know how I can consume the Hadoop 3 jars in maven?



Thanks,

--Gautham

On Sat, 7 May 2022 at 22:45, Akira Ajisaka <aa...@apache.org> wrote:

> Hi Chao,
>
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
>
> Akira
>
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm, I see the artifacts ideally should have got overwritten by the new
> > RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> > have any jars…
> > That is why it was picking the old jars. I think Steve needs to run mvn
> > deploy again…
> >
> > Sent from my iPhone
> >
> > > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >
> > > 
> > >>
> > >> Chao can you use the one that Steve mentioned in the mail?
> > >
> > > Hmm how do I do that? Typically after closing the RC in nexus the
> > > release bits will show up in
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > and Spark build will be able to pick them up for testing. However in
> > > this case I don't see any 3.3.3 jars in the URL.
> > >
> > >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> > >>
> > >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > date was also april 22, I archived that. Chao can you use the one that
> > Steve mentioned in the mail?
> > >>
> > >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> Seems there are some issues with the shaded client as I was not able
> > >>> to compile Apache Spark with the RC
> > >>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> > >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>> contain any class:
> > >>>
> > >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>> META-INF/
> > >>> META-INF/MANIFEST.MF
> > >>> META-INF/NOTICE.txt
> > >>> META-INF/LICENSE.txt
> > >>> META-INF/maven/
> > >>> META-INF/maven/org.apache.hadoop/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>
> > >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>>  * Signature: ok
> > >>>>  * Checksum : passed
> > >>>>  * Rat check (1.8.0_191): passed
> > >>>>   - mvn clean apache-rat:check
> > >>>>  * Built from source (1.8.0_191): failed
> > >>>>   - mvn clean install  -DskipTests
> > >>>>   - mvn -fae --no-transfer-progress -DskipTests
> > -Dmaven.javadoc.skip=true
> > >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>  * Unit tests pass (1.8.0_191):
> > >>>>    - HDFS Tests passed (Didn't run more than this).
> > >>>>
> > >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> > >>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> > >>>> bigger job with mild chaos. All seems to be working properly
> > (recoveries,
> > >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> > >>>> good. Poked at the hdfs cli. Seems good.
> > >>>>
> > >>>> S
> > >>>>
> > >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > >>>> wrote:
> > >>>>
> > >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>
> > >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>
> > >>>>> The maven artifacts are staged at
> > >>>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>
> > >>>>> Change log
> > >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>
> > >>>>> Release notes
> > >>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>
> > >>>>> There's a very small number of changes, primarily critical
> > code/packaging
> > >>>>> issues and security fixes.
> > >>>>>
> > >>>>>
> > >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>   -  CVEs in our code and dependencies
> > >>>>>   - Shaded client packaging issues.
> > >>>>>   - A switch from log4j to reload4j
> > >>>>>
> > >>>>>
> > >>>>> reload4j is an active fork of the log4j 1.17 library with the
> > classes which
> > >>>>> contain CVEs removed. Even though hadoop never used those classes,
> > they
> > >>>>> regularly raised alerts on security scans and concen from users.
> > Switching
> > >>>>> to the forked project allows us to ship a secure logging framework.
> > It will
> > >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > exclude
> > >>>>> our log4j artifacts, as they need to cut the new dependency
> > instead/as
> > >>>>> well.
> > >>>>>
> > >>>>> See the release notes for details.
> > >>>>>
> > >>>>> This is my first release through the new docker build process, do
> > please
> > >>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> > builds
> > >>>>> of downstream projects.
> > >>>>>
> > >>>>> We know there are some outstanding issues with at least one library
> > we are
> > >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > If the
> > >>>>> docker based release process works smoothly enough we can do a
> > followup
> > >>>>> security release in a few weeks.
> > >>>>>
> > >>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>
> > >>>>> -Steve
> > >>>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

had a report of that staging playing up too from Gautham

I'm going to have to treat this as a failure from the maven deploy process,
so I'll have to cancel this RC and try and work out WTF went wrong.

I'll also put together some minimal apps to fetch and compile against
everything. i did do some myself (storediag, hboss), and did try to make
sure I had cleaned up my local repo first, but maybe it was tainted.

oh well

-------

However, I get this error saying that it can’t find the maven plugins and
jars for Hadoop 3.3.3 –

[INFO] ------------------< org.apache.hadoop:hadoop-common
>-------------------

[INFO] Building Apache Hadoop Common 3.3.3
[5/9]

[INFO] --------------------------------[ jar
]---------------------------------

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.pom

[WARNING] The POM for org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 is
missing, no dependency information available

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.jar

[INFO]
------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop Common Project 3.3.3:

[INFO]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [
7.349 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
1.810 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [
7.922 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
0.375 s]

[INFO] Apache Hadoop Common ............................... FAILURE [
2.095 s]

[INFO] Apache Hadoop NFS .................................. SKIPPED

[INFO] Apache Hadoop KMS .................................. SKIPPED

[INFO] Apache Hadoop Registry ............................. SKIPPED

[INFO] Apache Hadoop Common Project ....................... SKIPPED

[INFO]
------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO]
------------------------------------------------------------------------

[INFO] Total time:  24.946 s

[INFO] Finished at: 2022-05-07T11:29:18+05:30

[INFO]
------------------------------------------------------------------------

[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.3.3 or one of its
dependencies could not be resolved: Could not find artifact
org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 in central (
https://repository.apache.org/content/repositories/orgapachehadoop-1348/)
-> [Help 1]

[ERROR]



Could you please let me know how I can consume the Hadoop 3 jars in maven?



Thanks,

--Gautham

On Sat, 7 May 2022 at 22:45, Akira Ajisaka <aa...@apache.org> wrote:

> Hi Chao,
>
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
>
> Akira
>
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm, I see the artifacts ideally should have got overwritten by the new
> > RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> > have any jars…
> > That is why it was picking the old jars. I think Steve needs to run mvn
> > deploy again…
> >
> > Sent from my iPhone
> >
> > > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >
> > > 
> > >>
> > >> Chao can you use the one that Steve mentioned in the mail?
> > >
> > > Hmm how do I do that? Typically after closing the RC in nexus the
> > > release bits will show up in
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > and Spark build will be able to pick them up for testing. However in
> > > this case I don't see any 3.3.3 jars in the URL.
> > >
> > >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> > >>
> > >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > date was also april 22, I archived that. Chao can you use the one that
> > Steve mentioned in the mail?
> > >>
> > >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> Seems there are some issues with the shaded client as I was not able
> > >>> to compile Apache Spark with the RC
> > >>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> > >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>> contain any class:
> > >>>
> > >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>> META-INF/
> > >>> META-INF/MANIFEST.MF
> > >>> META-INF/NOTICE.txt
> > >>> META-INF/LICENSE.txt
> > >>> META-INF/maven/
> > >>> META-INF/maven/org.apache.hadoop/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>
> > >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>>  * Signature: ok
> > >>>>  * Checksum : passed
> > >>>>  * Rat check (1.8.0_191): passed
> > >>>>   - mvn clean apache-rat:check
> > >>>>  * Built from source (1.8.0_191): failed
> > >>>>   - mvn clean install  -DskipTests
> > >>>>   - mvn -fae --no-transfer-progress -DskipTests
> > -Dmaven.javadoc.skip=true
> > >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>  * Unit tests pass (1.8.0_191):
> > >>>>    - HDFS Tests passed (Didn't run more than this).
> > >>>>
> > >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> > >>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> > >>>> bigger job with mild chaos. All seems to be working properly
> > (recoveries,
> > >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> > >>>> good. Poked at the hdfs cli. Seems good.
> > >>>>
> > >>>> S
> > >>>>
> > >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > >>>> wrote:
> > >>>>
> > >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>
> > >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>
> > >>>>> The maven artifacts are staged at
> > >>>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>
> > >>>>> Change log
> > >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>
> > >>>>> Release notes
> > >>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>
> > >>>>> There's a very small number of changes, primarily critical
> > code/packaging
> > >>>>> issues and security fixes.
> > >>>>>
> > >>>>>
> > >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>   -  CVEs in our code and dependencies
> > >>>>>   - Shaded client packaging issues.
> > >>>>>   - A switch from log4j to reload4j
> > >>>>>
> > >>>>>
> > >>>>> reload4j is an active fork of the log4j 1.17 library with the
> > classes which
> > >>>>> contain CVEs removed. Even though hadoop never used those classes,
> > they
> > >>>>> regularly raised alerts on security scans and concen from users.
> > Switching
> > >>>>> to the forked project allows us to ship a secure logging framework.
> > It will
> > >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > exclude
> > >>>>> our log4j artifacts, as they need to cut the new dependency
> > instead/as
> > >>>>> well.
> > >>>>>
> > >>>>> See the release notes for details.
> > >>>>>
> > >>>>> This is my first release through the new docker build process, do
> > please
> > >>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> > builds
> > >>>>> of downstream projects.
> > >>>>>
> > >>>>> We know there are some outstanding issues with at least one library
> > we are
> > >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > If the
> > >>>>> docker based release process works smoothly enough we can do a
> > followup
> > >>>>> security release in a few weeks.
> > >>>>>
> > >>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>
> > >>>>> -Steve
> > >>>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease#HowToRelease-Creatingthereleasecandidate(X.Y.Z-RC%3CN%3E)

On 2022/05/09 23:18, Masatake Iwasaki wrote:
> It seems to be caused by obsolete instruction in HowToRelease Wiki?
> 
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
> 
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> 
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
> 
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
> 
> Masatake Iwasaki
> 
> On 2022/05/08 6:45, Akira Ajisaka wrote:
>> Hi Chao,
>>
>> How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> instead of https://repository.apache.org/content/repositories/staging/ ?
>>
>> Akira
>>
>> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>>> Hmm, I see the artifacts ideally should have got overwritten by the new
>>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>>> have any jars…
>>> That is why it was picking the old jars. I think Steve needs to run mvn
>>> deploy again…
>>>
>>> Sent from my iPhone
>>>
>>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>>
>>>> 
>>>>>
>>>>> Chao can you use the one that Steve mentioned in the mail?
>>>>
>>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>>> release bits will show up in
>>>>
>>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>>> and Spark build will be able to pick them up for testing. However in
>>>> this case I don't see any 3.3.3 jars in the URL.
>>>>
>>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>>
>>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>>> date was also april 22, I archived that. Chao can you use the one that
>>> Steve mentioned in the mail?
>>>>>
>>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>>
>>>>>> Seems there are some issues with the shaded client as I was not able
>>>>>> to compile Apache Spark with the RC
>>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>>> contain any class:
>>>>>>
>>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>>> META-INF/
>>>>>> META-INF/MANIFEST.MF
>>>>>> META-INF/NOTICE.txt
>>>>>> META-INF/LICENSE.txt
>>>>>> META-INF/maven/
>>>>>> META-INF/maven/org.apache.hadoop/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>>
>>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>>   * Signature: ok
>>>>>>>   * Checksum : passed
>>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>>    - mvn clean apache-rat:check
>>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>>    - mvn clean install  -DskipTests
>>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>>> -Dmaven.javadoc.skip=true
>>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>>
>>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>>> bigger job with mild chaos. All seems to be working properly
>>> (recoveries,
>>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>>
>>>>>>> S
>>>>>>>
>>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>>> <st...@cloudera.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>>
>>>>>>>> The RC is available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>>
>>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>>
>>>>>>>> The maven artifacts are staged at
>>>>>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>>
>>>>>>>> You can find my public key at:
>>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>>
>>>>>>>> Change log
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>>
>>>>>>>> Release notes
>>>>>>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>>
>>>>>>>> There's a very small number of changes, primarily critical
>>> code/packaging
>>>>>>>> issues and security fixes.
>>>>>>>>
>>>>>>>>
>>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>>    - Shaded client packaging issues.
>>>>>>>>    - A switch from log4j to reload4j
>>>>>>>>
>>>>>>>>
>>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>>> classes which
>>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>>> they
>>>>>>>> regularly raised alerts on security scans and concen from users.
>>> Switching
>>>>>>>> to the forked project allows us to ship a secure logging framework.
>>> It will
>>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>>> exclude
>>>>>>>> our log4j artifacts, as they need to cut the new dependency
>>> instead/as
>>>>>>>> well.
>>>>>>>>
>>>>>>>> See the release notes for details.
>>>>>>>>
>>>>>>>> This is my first release through the new docker build process, do
>>> please
>>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>>> builds
>>>>>>>> of downstream projects.
>>>>>>>>
>>>>>>>> We know there are some outstanding issues with at least one library
>>> we are
>>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>>> If the
>>>>>>>> docker based release process works smoothly enough we can do a
>>> followup
>>>>>>>> security release in a few weeks.
>>>>>>>>
>>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>>
>>>>>>>> -Steve
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease#HowToRelease-Creatingthereleasecandidate(X.Y.Z-RC%3CN%3E)

On 2022/05/09 23:18, Masatake Iwasaki wrote:
> It seems to be caused by obsolete instruction in HowToRelease Wiki?
> 
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
> 
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> 
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
> 
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
> 
> Masatake Iwasaki
> 
> On 2022/05/08 6:45, Akira Ajisaka wrote:
>> Hi Chao,
>>
>> How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> instead of https://repository.apache.org/content/repositories/staging/ ?
>>
>> Akira
>>
>> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>>> Hmm, I see the artifacts ideally should have got overwritten by the new
>>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>>> have any jars…
>>> That is why it was picking the old jars. I think Steve needs to run mvn
>>> deploy again…
>>>
>>> Sent from my iPhone
>>>
>>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>>
>>>> 
>>>>>
>>>>> Chao can you use the one that Steve mentioned in the mail?
>>>>
>>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>>> release bits will show up in
>>>>
>>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>>> and Spark build will be able to pick them up for testing. However in
>>>> this case I don't see any 3.3.3 jars in the URL.
>>>>
>>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>>
>>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>>> date was also april 22, I archived that. Chao can you use the one that
>>> Steve mentioned in the mail?
>>>>>
>>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>>
>>>>>> Seems there are some issues with the shaded client as I was not able
>>>>>> to compile Apache Spark with the RC
>>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>>> contain any class:
>>>>>>
>>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>>> META-INF/
>>>>>> META-INF/MANIFEST.MF
>>>>>> META-INF/NOTICE.txt
>>>>>> META-INF/LICENSE.txt
>>>>>> META-INF/maven/
>>>>>> META-INF/maven/org.apache.hadoop/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>>
>>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>>   * Signature: ok
>>>>>>>   * Checksum : passed
>>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>>    - mvn clean apache-rat:check
>>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>>    - mvn clean install  -DskipTests
>>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>>> -Dmaven.javadoc.skip=true
>>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>>
>>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>>> bigger job with mild chaos. All seems to be working properly
>>> (recoveries,
>>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>>
>>>>>>> S
>>>>>>>
>>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>>> <st...@cloudera.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>>
>>>>>>>> The RC is available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>>
>>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>>
>>>>>>>> The maven artifacts are staged at
>>>>>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>>
>>>>>>>> You can find my public key at:
>>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>>
>>>>>>>> Change log
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>>
>>>>>>>> Release notes
>>>>>>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>>
>>>>>>>> There's a very small number of changes, primarily critical
>>> code/packaging
>>>>>>>> issues and security fixes.
>>>>>>>>
>>>>>>>>
>>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>>    - Shaded client packaging issues.
>>>>>>>>    - A switch from log4j to reload4j
>>>>>>>>
>>>>>>>>
>>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>>> classes which
>>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>>> they
>>>>>>>> regularly raised alerts on security scans and concen from users.
>>> Switching
>>>>>>>> to the forked project allows us to ship a secure logging framework.
>>> It will
>>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>>> exclude
>>>>>>>> our log4j artifacts, as they need to cut the new dependency
>>> instead/as
>>>>>>>> well.
>>>>>>>>
>>>>>>>> See the release notes for details.
>>>>>>>>
>>>>>>>> This is my first release through the new docker build process, do
>>> please
>>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>>> builds
>>>>>>>> of downstream projects.
>>>>>>>>
>>>>>>>> We know there are some outstanding issues with at least one library
>>> we are
>>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>>> If the
>>>>>>>> docker based release process works smoothly enough we can do a
>>> followup
>>>>>>>> security release in a few weeks.
>>>>>>>>
>>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>>
>>>>>>>> -Steve
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease#HowToRelease-Creatingthereleasecandidate(X.Y.Z-RC%3CN%3E)

On 2022/05/09 23:18, Masatake Iwasaki wrote:
> It seems to be caused by obsolete instruction in HowToRelease Wiki?
> 
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
> 
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> 
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
> 
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
> 
> Masatake Iwasaki
> 
> On 2022/05/08 6:45, Akira Ajisaka wrote:
>> Hi Chao,
>>
>> How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> instead of https://repository.apache.org/content/repositories/staging/ ?
>>
>> Akira
>>
>> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>>> Hmm, I see the artifacts ideally should have got overwritten by the new
>>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>>> have any jars…
>>> That is why it was picking the old jars. I think Steve needs to run mvn
>>> deploy again…
>>>
>>> Sent from my iPhone
>>>
>>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>>
>>>> 
>>>>>
>>>>> Chao can you use the one that Steve mentioned in the mail?
>>>>
>>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>>> release bits will show up in
>>>>
>>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>>> and Spark build will be able to pick them up for testing. However in
>>>> this case I don't see any 3.3.3 jars in the URL.
>>>>
>>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>>
>>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>>> date was also april 22, I archived that. Chao can you use the one that
>>> Steve mentioned in the mail?
>>>>>
>>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>>
>>>>>> Seems there are some issues with the shaded client as I was not able
>>>>>> to compile Apache Spark with the RC
>>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>>> contain any class:
>>>>>>
>>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>>> META-INF/
>>>>>> META-INF/MANIFEST.MF
>>>>>> META-INF/NOTICE.txt
>>>>>> META-INF/LICENSE.txt
>>>>>> META-INF/maven/
>>>>>> META-INF/maven/org.apache.hadoop/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>>
>>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>>   * Signature: ok
>>>>>>>   * Checksum : passed
>>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>>    - mvn clean apache-rat:check
>>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>>    - mvn clean install  -DskipTests
>>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>>> -Dmaven.javadoc.skip=true
>>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>>
>>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>>> bigger job with mild chaos. All seems to be working properly
>>> (recoveries,
>>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>>
>>>>>>> S
>>>>>>>
>>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>>> <st...@cloudera.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>>
>>>>>>>> The RC is available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>>
>>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>>
>>>>>>>> The maven artifacts are staged at
>>>>>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>>
>>>>>>>> You can find my public key at:
>>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>>
>>>>>>>> Change log
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>>
>>>>>>>> Release notes
>>>>>>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>>
>>>>>>>> There's a very small number of changes, primarily critical
>>> code/packaging
>>>>>>>> issues and security fixes.
>>>>>>>>
>>>>>>>>
>>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>>    - Shaded client packaging issues.
>>>>>>>>    - A switch from log4j to reload4j
>>>>>>>>
>>>>>>>>
>>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>>> classes which
>>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>>> they
>>>>>>>> regularly raised alerts on security scans and concen from users.
>>> Switching
>>>>>>>> to the forked project allows us to ship a secure logging framework.
>>> It will
>>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>>> exclude
>>>>>>>> our log4j artifacts, as they need to cut the new dependency
>>> instead/as
>>>>>>>> well.
>>>>>>>>
>>>>>>>> See the release notes for details.
>>>>>>>>
>>>>>>>> This is my first release through the new docker build process, do
>>> please
>>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>>> builds
>>>>>>>> of downstream projects.
>>>>>>>>
>>>>>>>> We know there are some outstanding issues with at least one library
>>> we are
>>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>>> If the
>>>>>>>> docker based release process works smoothly enough we can do a
>>> followup
>>>>>>>> security release in a few weeks.
>>>>>>>>
>>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>>
>>>>>>>> -Steve
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

update: got a stub project whose test run requires the client api and
shaded artifacts, and whose clean target wiil delete all artifacts of a
specific version from your local repo, so as to ensure all it untainted

https://github.com/steveloughran/validate-hadoop-client-artifacts

I do now have a version in staging with the files; still don't know how the
previous delete lacked them, I will do more testing tomorrow before putting
up the next RC. which will be off the same git commit as before, just a
rebuild, repackage and republish.

-Steve

On Tue, 10 May 2022 at 00:18, Steve Loughran <st...@cloudera.com> wrote:

> I've done another docker build and the client jars appear to be there.
> I'll test tomorrow before putting up another vote. it will be exactly the
> same commit as before, just recompiled/republished
>
> On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:
>
>> Agreed, that step #10 is out-dated and should be removed (I skipped
>> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>>
>> > How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>
>> Akira, I tried this too but it didn't work. I think we'd need the
>> artifacts to be properly pushed to the staging repository.
>>
>> > Could you please let me know how I can consume the Hadoop 3 jars in
>> maven?
>>
>> Gautham (if you are following this thread), you'll need to add the
>> following:
>>
>>     <repository>
>>        <id>staged</id>
>>        <name>staged-releases</name>
>>        <url>https://repository.apache.org/content/repositories/staging/
>> </url>
>>        <releases>
>>          <enabled>true</enabled>
>>        </releases>
>>        <snapshots>
>>          <enabled>true</enabled>
>>        </snapshots>
>>      </repository>
>>
>> to the `<repositories>` section in your Maven pom.xml file.
>>
>> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>> <st...@cloudera.com.invalid> wrote:
>> >
>> > I didn't do that as the docker image was doing it itself...I discussed
>> this
>> > with Akira and Ayush & they agreed. so whatever went wrong. it was
>> > something else.
>> >
>> > I have been building a list of things I'd like to change there; cutting
>> > that line was one of them. but I need to work out the correct workflow.
>> >
>> > trying again, and creating a stub module to verify the client is in
>> staging
>> >
>> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
>> iwasakims@oss.nttdata.co.jp>
>> > wrote:
>> >
>> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
>> > >
>> > > After HADOOP-15058, `mvn deploy` is kicked by
>> > > `dev-support/bin/create-release --asfrelease`.
>> > > https://issues.apache.org/jira/browse/HADOOP-15058
>> > >
>> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
>> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>> > >
>> > > 2 sets of artifact are deployed after creating RC based on the
>> instruction.
>> > > The latest one contains empty shaded jars.
>> > >
>> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
>> > > looks having same issue...
>> > >
>> > > Masatake Iwasaki
>> > >
>> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
>> > > > Hi Chao,
>> > > >
>> > > > How about using
>> > > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > > instead of
>> https://repository.apache.org/content/repositories/staging/ ?
>> > > >
>> > > > Akira
>> > > >
>> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hmm, I see the artifacts ideally should have got overwritten by
>> the new
>> > > >> RC, but they didn’t. The reason seems like the staging path shared
>> > > doesn’t
>> > > >> have any jars…
>> > > >> That is why it was picking the old jars. I think Steve needs to
>> run mvn
>> > > >> deploy again…
>> > > >>
>> > > >> Sent from my iPhone
>> > > >>
>> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>> > > >>>
>> > > >>> 
>> > > >>>>
>> > > >>>> Chao can you use the one that Steve mentioned in the mail?
>> > > >>>
>> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
>> > > >>> release bits will show up in
>> > > >>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>> > > >>> and Spark build will be able to pick them up for testing. However
>> in
>> > > >>> this case I don't see any 3.3.3 jars in the URL.
>> > > >>>
>> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
>> > > wrote:
>> > > >>>>
>> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
>> the
>> > > >> date was also april 22, I archived that. Chao can you use the one
>> that
>> > > >> Steve mentioned in the mail?
>> > > >>>>
>> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
>> wrote:
>> > > >>>>>
>> > > >>>>> Seems there are some issues with the shaded client as I was not
>> able
>> > > >>>>> to compile Apache Spark with the RC
>> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
>> > > compiled
>> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
>> doesn't
>> > > >>>>> contain any class:
>> > > >>>>>
>> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> > > >>>>> META-INF/
>> > > >>>>> META-INF/MANIFEST.MF
>> > > >>>>> META-INF/NOTICE.txt
>> > > >>>>> META-INF/LICENSE.txt
>> > > >>>>> META-INF/maven/
>> > > >>>>> META-INF/maven/org.apache.hadoop/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> > > >>>>>
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>> > > >>>>>
>> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> > > >>>>>>
>> > > >>>>>> +1 (binding)
>> > > >>>>>>
>> > > >>>>>>   * Signature: ok
>> > > >>>>>>   * Checksum : passed
>> > > >>>>>>   * Rat check (1.8.0_191): passed
>> > > >>>>>>    - mvn clean apache-rat:check
>> > > >>>>>>   * Built from source (1.8.0_191): failed
>> > > >>>>>>    - mvn clean install  -DskipTests
>> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> > > >> -Dmaven.javadoc.skip=true
>> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>> > > >>>>>>   * Unit tests pass (1.8.0_191):
>> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
>> > > >>>>>>
>> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
>> five
>> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
>> > > built
>> > > >>>>>> against 3.3.2) against it. Tried a small verification job.
>> Good.
>> > > Ran a
>> > > >>>>>> bigger job with mild chaos. All seems to be working properly
>> > > >> (recoveries,
>> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
>> UIs
>> > > look
>> > > >>>>>> good. Poked at the hdfs cli. Seems good.
>> > > >>>>>>
>> > > >>>>>> S
>> > > >>>>>>
>> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> > > >> <st...@cloudera.com.invalid>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > > >>>>>>>
>> > > >>>>>>> The RC is available at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > > >>>>>>>
>> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > > >>>>>>>
>> > > >>>>>>> The maven artifacts are staged at
>> > > >>>>>>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > >>>>>>>
>> > > >>>>>>> You can find my public key at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>>>>>>
>> > > >>>>>>> Change log
>> > > >>>>>>>
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > > >>>>>>>
>> > > >>>>>>> Release notes
>> > > >>>>>>>
>> > > >>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > > >>>>>>>
>> > > >>>>>>> There's a very small number of changes, primarily critical
>> > > >> code/packaging
>> > > >>>>>>> issues and security fixes.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>> > > >>>>>>>    -  CVEs in our code and dependencies
>> > > >>>>>>>    - Shaded client packaging issues.
>> > > >>>>>>>    - A switch from log4j to reload4j
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> > > >> classes which
>> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
>> classes,
>> > > >> they
>> > > >>>>>>> regularly raised alerts on security scans and concen from
>> users.
>> > > >> Switching
>> > > >>>>>>> to the forked project allows us to ship a secure logging
>> framework.
>> > > >> It will
>> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
>> which
>> > > >> exclude
>> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
>> > > >> instead/as
>> > > >>>>>>> well.
>> > > >>>>>>>
>> > > >>>>>>> See the release notes for details.
>> > > >>>>>>>
>> > > >>>>>>> This is my first release through the new docker build
>> process, do
>> > > >> please
>> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
>> > > trying
>> > > >> builds
>> > > >>>>>>> of downstream projects.
>> > > >>>>>>>
>> > > >>>>>>> We know there are some outstanding issues with at least one
>> library
>> > > >> we are
>> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
>> for it.
>> > > >> If the
>> > > >>>>>>> docker based release process works smoothly enough we can do a
>> > > >> followup
>> > > >>>>>>> security release in a few weeks.
>> > > >>>>>>>
>> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
>> > > >>>>>>>
>> > > >>>>>>> -Steve
>> > > >>>>>>>
>> > > >>>>>
>> > > >>>>>
>> ---------------------------------------------------------------------
>> > > >>>>> To unsubscribe, e-mail:
>> common-dev-unsubscribe@hadoop.apache.org
>> > > >>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>> > > >>>>>
>> > > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > > >>
>> > > >>
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

update: got a stub project whose test run requires the client api and
shaded artifacts, and whose clean target wiil delete all artifacts of a
specific version from your local repo, so as to ensure all it untainted

https://github.com/steveloughran/validate-hadoop-client-artifacts

I do now have a version in staging with the files; still don't know how the
previous delete lacked them, I will do more testing tomorrow before putting
up the next RC. which will be off the same git commit as before, just a
rebuild, repackage and republish.

-Steve

On Tue, 10 May 2022 at 00:18, Steve Loughran <st...@cloudera.com> wrote:

> I've done another docker build and the client jars appear to be there.
> I'll test tomorrow before putting up another vote. it will be exactly the
> same commit as before, just recompiled/republished
>
> On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:
>
>> Agreed, that step #10 is out-dated and should be removed (I skipped
>> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>>
>> > How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>
>> Akira, I tried this too but it didn't work. I think we'd need the
>> artifacts to be properly pushed to the staging repository.
>>
>> > Could you please let me know how I can consume the Hadoop 3 jars in
>> maven?
>>
>> Gautham (if you are following this thread), you'll need to add the
>> following:
>>
>>     <repository>
>>        <id>staged</id>
>>        <name>staged-releases</name>
>>        <url>https://repository.apache.org/content/repositories/staging/
>> </url>
>>        <releases>
>>          <enabled>true</enabled>
>>        </releases>
>>        <snapshots>
>>          <enabled>true</enabled>
>>        </snapshots>
>>      </repository>
>>
>> to the `<repositories>` section in your Maven pom.xml file.
>>
>> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>> <st...@cloudera.com.invalid> wrote:
>> >
>> > I didn't do that as the docker image was doing it itself...I discussed
>> this
>> > with Akira and Ayush & they agreed. so whatever went wrong. it was
>> > something else.
>> >
>> > I have been building a list of things I'd like to change there; cutting
>> > that line was one of them. but I need to work out the correct workflow.
>> >
>> > trying again, and creating a stub module to verify the client is in
>> staging
>> >
>> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
>> iwasakims@oss.nttdata.co.jp>
>> > wrote:
>> >
>> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
>> > >
>> > > After HADOOP-15058, `mvn deploy` is kicked by
>> > > `dev-support/bin/create-release --asfrelease`.
>> > > https://issues.apache.org/jira/browse/HADOOP-15058
>> > >
>> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
>> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>> > >
>> > > 2 sets of artifact are deployed after creating RC based on the
>> instruction.
>> > > The latest one contains empty shaded jars.
>> > >
>> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
>> > > looks having same issue...
>> > >
>> > > Masatake Iwasaki
>> > >
>> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
>> > > > Hi Chao,
>> > > >
>> > > > How about using
>> > > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > > instead of
>> https://repository.apache.org/content/repositories/staging/ ?
>> > > >
>> > > > Akira
>> > > >
>> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hmm, I see the artifacts ideally should have got overwritten by
>> the new
>> > > >> RC, but they didn’t. The reason seems like the staging path shared
>> > > doesn’t
>> > > >> have any jars…
>> > > >> That is why it was picking the old jars. I think Steve needs to
>> run mvn
>> > > >> deploy again…
>> > > >>
>> > > >> Sent from my iPhone
>> > > >>
>> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>> > > >>>
>> > > >>> 
>> > > >>>>
>> > > >>>> Chao can you use the one that Steve mentioned in the mail?
>> > > >>>
>> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
>> > > >>> release bits will show up in
>> > > >>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>> > > >>> and Spark build will be able to pick them up for testing. However
>> in
>> > > >>> this case I don't see any 3.3.3 jars in the URL.
>> > > >>>
>> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
>> > > wrote:
>> > > >>>>
>> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
>> the
>> > > >> date was also april 22, I archived that. Chao can you use the one
>> that
>> > > >> Steve mentioned in the mail?
>> > > >>>>
>> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
>> wrote:
>> > > >>>>>
>> > > >>>>> Seems there are some issues with the shaded client as I was not
>> able
>> > > >>>>> to compile Apache Spark with the RC
>> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
>> > > compiled
>> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
>> doesn't
>> > > >>>>> contain any class:
>> > > >>>>>
>> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> > > >>>>> META-INF/
>> > > >>>>> META-INF/MANIFEST.MF
>> > > >>>>> META-INF/NOTICE.txt
>> > > >>>>> META-INF/LICENSE.txt
>> > > >>>>> META-INF/maven/
>> > > >>>>> META-INF/maven/org.apache.hadoop/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> > > >>>>>
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>> > > >>>>>
>> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> > > >>>>>>
>> > > >>>>>> +1 (binding)
>> > > >>>>>>
>> > > >>>>>>   * Signature: ok
>> > > >>>>>>   * Checksum : passed
>> > > >>>>>>   * Rat check (1.8.0_191): passed
>> > > >>>>>>    - mvn clean apache-rat:check
>> > > >>>>>>   * Built from source (1.8.0_191): failed
>> > > >>>>>>    - mvn clean install  -DskipTests
>> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> > > >> -Dmaven.javadoc.skip=true
>> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>> > > >>>>>>   * Unit tests pass (1.8.0_191):
>> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
>> > > >>>>>>
>> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
>> five
>> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
>> > > built
>> > > >>>>>> against 3.3.2) against it. Tried a small verification job.
>> Good.
>> > > Ran a
>> > > >>>>>> bigger job with mild chaos. All seems to be working properly
>> > > >> (recoveries,
>> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
>> UIs
>> > > look
>> > > >>>>>> good. Poked at the hdfs cli. Seems good.
>> > > >>>>>>
>> > > >>>>>> S
>> > > >>>>>>
>> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> > > >> <st...@cloudera.com.invalid>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > > >>>>>>>
>> > > >>>>>>> The RC is available at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > > >>>>>>>
>> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > > >>>>>>>
>> > > >>>>>>> The maven artifacts are staged at
>> > > >>>>>>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > >>>>>>>
>> > > >>>>>>> You can find my public key at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>>>>>>
>> > > >>>>>>> Change log
>> > > >>>>>>>
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > > >>>>>>>
>> > > >>>>>>> Release notes
>> > > >>>>>>>
>> > > >>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > > >>>>>>>
>> > > >>>>>>> There's a very small number of changes, primarily critical
>> > > >> code/packaging
>> > > >>>>>>> issues and security fixes.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>> > > >>>>>>>    -  CVEs in our code and dependencies
>> > > >>>>>>>    - Shaded client packaging issues.
>> > > >>>>>>>    - A switch from log4j to reload4j
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> > > >> classes which
>> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
>> classes,
>> > > >> they
>> > > >>>>>>> regularly raised alerts on security scans and concen from
>> users.
>> > > >> Switching
>> > > >>>>>>> to the forked project allows us to ship a secure logging
>> framework.
>> > > >> It will
>> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
>> which
>> > > >> exclude
>> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
>> > > >> instead/as
>> > > >>>>>>> well.
>> > > >>>>>>>
>> > > >>>>>>> See the release notes for details.
>> > > >>>>>>>
>> > > >>>>>>> This is my first release through the new docker build
>> process, do
>> > > >> please
>> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
>> > > trying
>> > > >> builds
>> > > >>>>>>> of downstream projects.
>> > > >>>>>>>
>> > > >>>>>>> We know there are some outstanding issues with at least one
>> library
>> > > >> we are
>> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
>> for it.
>> > > >> If the
>> > > >>>>>>> docker based release process works smoothly enough we can do a
>> > > >> followup
>> > > >>>>>>> security release in a few weeks.
>> > > >>>>>>>
>> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
>> > > >>>>>>>
>> > > >>>>>>> -Steve
>> > > >>>>>>>
>> > > >>>>>
>> > > >>>>>
>> ---------------------------------------------------------------------
>> > > >>>>> To unsubscribe, e-mail:
>> common-dev-unsubscribe@hadoop.apache.org
>> > > >>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>> > > >>>>>
>> > > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > > >>
>> > > >>
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

update: got a stub project whose test run requires the client api and
shaded artifacts, and whose clean target wiil delete all artifacts of a
specific version from your local repo, so as to ensure all it untainted

https://github.com/steveloughran/validate-hadoop-client-artifacts

I do now have a version in staging with the files; still don't know how the
previous delete lacked them, I will do more testing tomorrow before putting
up the next RC. which will be off the same git commit as before, just a
rebuild, repackage and republish.

-Steve

On Tue, 10 May 2022 at 00:18, Steve Loughran <st...@cloudera.com> wrote:

> I've done another docker build and the client jars appear to be there.
> I'll test tomorrow before putting up another vote. it will be exactly the
> same commit as before, just recompiled/republished
>
> On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:
>
>> Agreed, that step #10 is out-dated and should be removed (I skipped
>> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>>
>> > How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>
>> Akira, I tried this too but it didn't work. I think we'd need the
>> artifacts to be properly pushed to the staging repository.
>>
>> > Could you please let me know how I can consume the Hadoop 3 jars in
>> maven?
>>
>> Gautham (if you are following this thread), you'll need to add the
>> following:
>>
>>     <repository>
>>        <id>staged</id>
>>        <name>staged-releases</name>
>>        <url>https://repository.apache.org/content/repositories/staging/
>> </url>
>>        <releases>
>>          <enabled>true</enabled>
>>        </releases>
>>        <snapshots>
>>          <enabled>true</enabled>
>>        </snapshots>
>>      </repository>
>>
>> to the `<repositories>` section in your Maven pom.xml file.
>>
>> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>> <st...@cloudera.com.invalid> wrote:
>> >
>> > I didn't do that as the docker image was doing it itself...I discussed
>> this
>> > with Akira and Ayush & they agreed. so whatever went wrong. it was
>> > something else.
>> >
>> > I have been building a list of things I'd like to change there; cutting
>> > that line was one of them. but I need to work out the correct workflow.
>> >
>> > trying again, and creating a stub module to verify the client is in
>> staging
>> >
>> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
>> iwasakims@oss.nttdata.co.jp>
>> > wrote:
>> >
>> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
>> > >
>> > > After HADOOP-15058, `mvn deploy` is kicked by
>> > > `dev-support/bin/create-release --asfrelease`.
>> > > https://issues.apache.org/jira/browse/HADOOP-15058
>> > >
>> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
>> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>> > >
>> > > 2 sets of artifact are deployed after creating RC based on the
>> instruction.
>> > > The latest one contains empty shaded jars.
>> > >
>> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
>> > > looks having same issue...
>> > >
>> > > Masatake Iwasaki
>> > >
>> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
>> > > > Hi Chao,
>> > > >
>> > > > How about using
>> > > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > > instead of
>> https://repository.apache.org/content/repositories/staging/ ?
>> > > >
>> > > > Akira
>> > > >
>> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hmm, I see the artifacts ideally should have got overwritten by
>> the new
>> > > >> RC, but they didn’t. The reason seems like the staging path shared
>> > > doesn’t
>> > > >> have any jars…
>> > > >> That is why it was picking the old jars. I think Steve needs to
>> run mvn
>> > > >> deploy again…
>> > > >>
>> > > >> Sent from my iPhone
>> > > >>
>> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>> > > >>>
>> > > >>> 
>> > > >>>>
>> > > >>>> Chao can you use the one that Steve mentioned in the mail?
>> > > >>>
>> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
>> > > >>> release bits will show up in
>> > > >>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>> > > >>> and Spark build will be able to pick them up for testing. However
>> in
>> > > >>> this case I don't see any 3.3.3 jars in the URL.
>> > > >>>
>> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
>> > > wrote:
>> > > >>>>
>> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
>> the
>> > > >> date was also april 22, I archived that. Chao can you use the one
>> that
>> > > >> Steve mentioned in the mail?
>> > > >>>>
>> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
>> wrote:
>> > > >>>>>
>> > > >>>>> Seems there are some issues with the shaded client as I was not
>> able
>> > > >>>>> to compile Apache Spark with the RC
>> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
>> > > compiled
>> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
>> doesn't
>> > > >>>>> contain any class:
>> > > >>>>>
>> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> > > >>>>> META-INF/
>> > > >>>>> META-INF/MANIFEST.MF
>> > > >>>>> META-INF/NOTICE.txt
>> > > >>>>> META-INF/LICENSE.txt
>> > > >>>>> META-INF/maven/
>> > > >>>>> META-INF/maven/org.apache.hadoop/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> > > >>>>>
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>> > > >>>>>
>> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> > > >>>>>>
>> > > >>>>>> +1 (binding)
>> > > >>>>>>
>> > > >>>>>>   * Signature: ok
>> > > >>>>>>   * Checksum : passed
>> > > >>>>>>   * Rat check (1.8.0_191): passed
>> > > >>>>>>    - mvn clean apache-rat:check
>> > > >>>>>>   * Built from source (1.8.0_191): failed
>> > > >>>>>>    - mvn clean install  -DskipTests
>> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> > > >> -Dmaven.javadoc.skip=true
>> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>> > > >>>>>>   * Unit tests pass (1.8.0_191):
>> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
>> > > >>>>>>
>> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
>> five
>> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
>> > > built
>> > > >>>>>> against 3.3.2) against it. Tried a small verification job.
>> Good.
>> > > Ran a
>> > > >>>>>> bigger job with mild chaos. All seems to be working properly
>> > > >> (recoveries,
>> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
>> UIs
>> > > look
>> > > >>>>>> good. Poked at the hdfs cli. Seems good.
>> > > >>>>>>
>> > > >>>>>> S
>> > > >>>>>>
>> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> > > >> <st...@cloudera.com.invalid>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > > >>>>>>>
>> > > >>>>>>> The RC is available at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > > >>>>>>>
>> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > > >>>>>>>
>> > > >>>>>>> The maven artifacts are staged at
>> > > >>>>>>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > >>>>>>>
>> > > >>>>>>> You can find my public key at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>>>>>>
>> > > >>>>>>> Change log
>> > > >>>>>>>
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > > >>>>>>>
>> > > >>>>>>> Release notes
>> > > >>>>>>>
>> > > >>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > > >>>>>>>
>> > > >>>>>>> There's a very small number of changes, primarily critical
>> > > >> code/packaging
>> > > >>>>>>> issues and security fixes.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>> > > >>>>>>>    -  CVEs in our code and dependencies
>> > > >>>>>>>    - Shaded client packaging issues.
>> > > >>>>>>>    - A switch from log4j to reload4j
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> > > >> classes which
>> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
>> classes,
>> > > >> they
>> > > >>>>>>> regularly raised alerts on security scans and concen from
>> users.
>> > > >> Switching
>> > > >>>>>>> to the forked project allows us to ship a secure logging
>> framework.
>> > > >> It will
>> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
>> which
>> > > >> exclude
>> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
>> > > >> instead/as
>> > > >>>>>>> well.
>> > > >>>>>>>
>> > > >>>>>>> See the release notes for details.
>> > > >>>>>>>
>> > > >>>>>>> This is my first release through the new docker build
>> process, do
>> > > >> please
>> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
>> > > trying
>> > > >> builds
>> > > >>>>>>> of downstream projects.
>> > > >>>>>>>
>> > > >>>>>>> We know there are some outstanding issues with at least one
>> library
>> > > >> we are
>> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
>> for it.
>> > > >> If the
>> > > >>>>>>> docker based release process works smoothly enough we can do a
>> > > >> followup
>> > > >>>>>>> security release in a few weeks.
>> > > >>>>>>>
>> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
>> > > >>>>>>>
>> > > >>>>>>> -Steve
>> > > >>>>>>>
>> > > >>>>>
>> > > >>>>>
>> ---------------------------------------------------------------------
>> > > >>>>> To unsubscribe, e-mail:
>> common-dev-unsubscribe@hadoop.apache.org
>> > > >>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>> > > >>>>>
>> > > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > > >>
>> > > >>
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

update: got a stub project whose test run requires the client api and
shaded artifacts, and whose clean target wiil delete all artifacts of a
specific version from your local repo, so as to ensure all it untainted

https://github.com/steveloughran/validate-hadoop-client-artifacts

I do now have a version in staging with the files; still don't know how the
previous delete lacked them, I will do more testing tomorrow before putting
up the next RC. which will be off the same git commit as before, just a
rebuild, repackage and republish.

-Steve

On Tue, 10 May 2022 at 00:18, Steve Loughran <st...@cloudera.com> wrote:

> I've done another docker build and the client jars appear to be there.
> I'll test tomorrow before putting up another vote. it will be exactly the
> same commit as before, just recompiled/republished
>
> On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:
>
>> Agreed, that step #10 is out-dated and should be removed (I skipped
>> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>>
>> > How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>
>> Akira, I tried this too but it didn't work. I think we'd need the
>> artifacts to be properly pushed to the staging repository.
>>
>> > Could you please let me know how I can consume the Hadoop 3 jars in
>> maven?
>>
>> Gautham (if you are following this thread), you'll need to add the
>> following:
>>
>>     <repository>
>>        <id>staged</id>
>>        <name>staged-releases</name>
>>        <url>https://repository.apache.org/content/repositories/staging/
>> </url>
>>        <releases>
>>          <enabled>true</enabled>
>>        </releases>
>>        <snapshots>
>>          <enabled>true</enabled>
>>        </snapshots>
>>      </repository>
>>
>> to the `<repositories>` section in your Maven pom.xml file.
>>
>> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>> <st...@cloudera.com.invalid> wrote:
>> >
>> > I didn't do that as the docker image was doing it itself...I discussed
>> this
>> > with Akira and Ayush & they agreed. so whatever went wrong. it was
>> > something else.
>> >
>> > I have been building a list of things I'd like to change there; cutting
>> > that line was one of them. but I need to work out the correct workflow.
>> >
>> > trying again, and creating a stub module to verify the client is in
>> staging
>> >
>> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
>> iwasakims@oss.nttdata.co.jp>
>> > wrote:
>> >
>> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
>> > >
>> > > After HADOOP-15058, `mvn deploy` is kicked by
>> > > `dev-support/bin/create-release --asfrelease`.
>> > > https://issues.apache.org/jira/browse/HADOOP-15058
>> > >
>> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
>> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>> > >
>> > > 2 sets of artifact are deployed after creating RC based on the
>> instruction.
>> > > The latest one contains empty shaded jars.
>> > >
>> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
>> > > looks having same issue...
>> > >
>> > > Masatake Iwasaki
>> > >
>> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
>> > > > Hi Chao,
>> > > >
>> > > > How about using
>> > > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > > instead of
>> https://repository.apache.org/content/repositories/staging/ ?
>> > > >
>> > > > Akira
>> > > >
>> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hmm, I see the artifacts ideally should have got overwritten by
>> the new
>> > > >> RC, but they didn’t. The reason seems like the staging path shared
>> > > doesn’t
>> > > >> have any jars…
>> > > >> That is why it was picking the old jars. I think Steve needs to
>> run mvn
>> > > >> deploy again…
>> > > >>
>> > > >> Sent from my iPhone
>> > > >>
>> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>> > > >>>
>> > > >>> 
>> > > >>>>
>> > > >>>> Chao can you use the one that Steve mentioned in the mail?
>> > > >>>
>> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
>> > > >>> release bits will show up in
>> > > >>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>> > > >>> and Spark build will be able to pick them up for testing. However
>> in
>> > > >>> this case I don't see any 3.3.3 jars in the URL.
>> > > >>>
>> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
>> > > wrote:
>> > > >>>>
>> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
>> the
>> > > >> date was also april 22, I archived that. Chao can you use the one
>> that
>> > > >> Steve mentioned in the mail?
>> > > >>>>
>> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
>> wrote:
>> > > >>>>>
>> > > >>>>> Seems there are some issues with the shaded client as I was not
>> able
>> > > >>>>> to compile Apache Spark with the RC
>> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
>> > > compiled
>> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
>> doesn't
>> > > >>>>> contain any class:
>> > > >>>>>
>> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> > > >>>>> META-INF/
>> > > >>>>> META-INF/MANIFEST.MF
>> > > >>>>> META-INF/NOTICE.txt
>> > > >>>>> META-INF/LICENSE.txt
>> > > >>>>> META-INF/maven/
>> > > >>>>> META-INF/maven/org.apache.hadoop/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> > > >>>>>
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>> > > >>>>>
>> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> > > >>>>>>
>> > > >>>>>> +1 (binding)
>> > > >>>>>>
>> > > >>>>>>   * Signature: ok
>> > > >>>>>>   * Checksum : passed
>> > > >>>>>>   * Rat check (1.8.0_191): passed
>> > > >>>>>>    - mvn clean apache-rat:check
>> > > >>>>>>   * Built from source (1.8.0_191): failed
>> > > >>>>>>    - mvn clean install  -DskipTests
>> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> > > >> -Dmaven.javadoc.skip=true
>> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>> > > >>>>>>   * Unit tests pass (1.8.0_191):
>> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
>> > > >>>>>>
>> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
>> five
>> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
>> > > built
>> > > >>>>>> against 3.3.2) against it. Tried a small verification job.
>> Good.
>> > > Ran a
>> > > >>>>>> bigger job with mild chaos. All seems to be working properly
>> > > >> (recoveries,
>> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
>> UIs
>> > > look
>> > > >>>>>> good. Poked at the hdfs cli. Seems good.
>> > > >>>>>>
>> > > >>>>>> S
>> > > >>>>>>
>> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> > > >> <st...@cloudera.com.invalid>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > > >>>>>>>
>> > > >>>>>>> The RC is available at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > > >>>>>>>
>> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > > >>>>>>>
>> > > >>>>>>> The maven artifacts are staged at
>> > > >>>>>>>
>> > > >>
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > >>>>>>>
>> > > >>>>>>> You can find my public key at:
>> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>>>>>>
>> > > >>>>>>> Change log
>> > > >>>>>>>
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > > >>>>>>>
>> > > >>>>>>> Release notes
>> > > >>>>>>>
>> > > >>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > > >>>>>>>
>> > > >>>>>>> There's a very small number of changes, primarily critical
>> > > >> code/packaging
>> > > >>>>>>> issues and security fixes.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>> > > >>>>>>>    -  CVEs in our code and dependencies
>> > > >>>>>>>    - Shaded client packaging issues.
>> > > >>>>>>>    - A switch from log4j to reload4j
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> > > >> classes which
>> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
>> classes,
>> > > >> they
>> > > >>>>>>> regularly raised alerts on security scans and concen from
>> users.
>> > > >> Switching
>> > > >>>>>>> to the forked project allows us to ship a secure logging
>> framework.
>> > > >> It will
>> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
>> which
>> > > >> exclude
>> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
>> > > >> instead/as
>> > > >>>>>>> well.
>> > > >>>>>>>
>> > > >>>>>>> See the release notes for details.
>> > > >>>>>>>
>> > > >>>>>>> This is my first release through the new docker build
>> process, do
>> > > >> please
>> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
>> > > trying
>> > > >> builds
>> > > >>>>>>> of downstream projects.
>> > > >>>>>>>
>> > > >>>>>>> We know there are some outstanding issues with at least one
>> library
>> > > >> we are
>> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
>> for it.
>> > > >> If the
>> > > >>>>>>> docker based release process works smoothly enough we can do a
>> > > >> followup
>> > > >>>>>>> security release in a few weeks.
>> > > >>>>>>>
>> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
>> > > >>>>>>>
>> > > >>>>>>> -Steve
>> > > >>>>>>>
>> > > >>>>>
>> > > >>>>>
>> ---------------------------------------------------------------------
>> > > >>>>> To unsubscribe, e-mail:
>> common-dev-unsubscribe@hadoop.apache.org
>> > > >>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>> > > >>>>>
>> > > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > > >>
>> > > >>
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> > >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I've done another docker build and the client jars appear to be there. I'll
test tomorrow before putting up another vote. it will be exactly the same
commit as before, just recompiled/republished

On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:

> Agreed, that step #10 is out-dated and should be removed (I skipped
> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>
> > How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> Akira, I tried this too but it didn't work. I think we'd need the
> artifacts to be properly pushed to the staging repository.
>
> > Could you please let me know how I can consume the Hadoop 3 jars in
> maven?
>
> Gautham (if you are following this thread), you'll need to add the
> following:
>
>     <repository>
>        <id>staged</id>
>        <name>staged-releases</name>
>        <url>https://repository.apache.org/content/repositories/staging/
> </url>
>        <releases>
>          <enabled>true</enabled>
>        </releases>
>        <snapshots>
>          <enabled>true</enabled>
>        </snapshots>
>      </repository>
>
> to the `<repositories>` section in your Maven pom.xml file.
>
> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
> <st...@cloudera.com.invalid> wrote:
> >
> > I didn't do that as the docker image was doing it itself...I discussed
> this
> > with Akira and Ayush & they agreed. so whatever went wrong. it was
> > something else.
> >
> > I have been building a list of things I'd like to change there; cutting
> > that line was one of them. but I need to work out the correct workflow.
> >
> > trying again, and creating a stub module to verify the client is in
> staging
> >
> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
> iwasakims@oss.nttdata.co.jp>
> > wrote:
> >
> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> > >
> > > After HADOOP-15058, `mvn deploy` is kicked by
> > > `dev-support/bin/create-release --asfrelease`.
> > > https://issues.apache.org/jira/browse/HADOOP-15058
> > >
> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> > >
> > > 2 sets of artifact are deployed after creating RC based on the
> instruction.
> > > The latest one contains empty shaded jars.
> > >
> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > > looks having same issue...
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > > Hi Chao,
> > > >
> > > > How about using
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > > instead of
> https://repository.apache.org/content/repositories/staging/ ?
> > > >
> > > > Akira
> > > >
> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >
> > > >> Hmm, I see the artifacts ideally should have got overwritten by the
> new
> > > >> RC, but they didn’t. The reason seems like the staging path shared
> > > doesn’t
> > > >> have any jars…
> > > >> That is why it was picking the old jars. I think Steve needs to run
> mvn
> > > >> deploy again…
> > > >>
> > > >> Sent from my iPhone
> > > >>
> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > > >>>
> > > >>> 
> > > >>>>
> > > >>>> Chao can you use the one that Steve mentioned in the mail?
> > > >>>
> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > > >>> release bits will show up in
> > > >>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > >>> and Spark build will be able to pick them up for testing. However
> in
> > > >>> this case I don't see any 3.3.3 jars in the URL.
> > > >>>
> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
> the
> > > >> date was also april 22, I archived that. Chao can you use the one
> that
> > > >> Steve mentioned in the mail?
> > > >>>>
> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
> wrote:
> > > >>>>>
> > > >>>>> Seems there are some issues with the shaded client as I was not
> able
> > > >>>>> to compile Apache Spark with the RC
> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > > compiled
> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
> doesn't
> > > >>>>> contain any class:
> > > >>>>>
> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > > >>>>> META-INF/
> > > >>>>> META-INF/MANIFEST.MF
> > > >>>>> META-INF/NOTICE.txt
> > > >>>>> META-INF/LICENSE.txt
> > > >>>>> META-INF/maven/
> > > >>>>> META-INF/maven/org.apache.hadoop/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > > >>>>>
> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > > >>>>>>
> > > >>>>>> +1 (binding)
> > > >>>>>>
> > > >>>>>>   * Signature: ok
> > > >>>>>>   * Checksum : passed
> > > >>>>>>   * Rat check (1.8.0_191): passed
> > > >>>>>>    - mvn clean apache-rat:check
> > > >>>>>>   * Built from source (1.8.0_191): failed
> > > >>>>>>    - mvn clean install  -DskipTests
> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > > >> -Dmaven.javadoc.skip=true
> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > > >>>>>>   * Unit tests pass (1.8.0_191):
> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > > >>>>>>
> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
> five
> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > > built
> > > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > > Ran a
> > > >>>>>> bigger job with mild chaos. All seems to be working properly
> > > >> (recoveries,
> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
> UIs
> > > look
> > > >>>>>> good. Poked at the hdfs cli. Seems good.
> > > >>>>>>
> > > >>>>>> S
> > > >>>>>>
> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > > >> <st...@cloudera.com.invalid>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > > >>>>>>>
> > > >>>>>>> The RC is available at:
> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > > >>>>>>>
> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > > >>>>>>>
> > > >>>>>>> The maven artifacts are staged at
> > > >>>>>>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > >>>>>>>
> > > >>>>>>> You can find my public key at:
> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>>>>>>
> > > >>>>>>> Change log
> > > >>>>>>>
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > > >>>>>>>
> > > >>>>>>> Release notes
> > > >>>>>>>
> > > >>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > > >>>>>>>
> > > >>>>>>> There's a very small number of changes, primarily critical
> > > >> code/packaging
> > > >>>>>>> issues and security fixes.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > > >>>>>>>    -  CVEs in our code and dependencies
> > > >>>>>>>    - Shaded client packaging issues.
> > > >>>>>>>    - A switch from log4j to reload4j
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > > >> classes which
> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
> classes,
> > > >> they
> > > >>>>>>> regularly raised alerts on security scans and concen from
> users.
> > > >> Switching
> > > >>>>>>> to the forked project allows us to ship a secure logging
> framework.
> > > >> It will
> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
> which
> > > >> exclude
> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > > >> instead/as
> > > >>>>>>> well.
> > > >>>>>>>
> > > >>>>>>> See the release notes for details.
> > > >>>>>>>
> > > >>>>>>> This is my first release through the new docker build process,
> do
> > > >> please
> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > > trying
> > > >> builds
> > > >>>>>>> of downstream projects.
> > > >>>>>>>
> > > >>>>>>> We know there are some outstanding issues with at least one
> library
> > > >> we are
> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
> for it.
> > > >> If the
> > > >>>>>>> docker based release process works smoothly enough we can do a
> > > >> followup
> > > >>>>>>> security release in a few weeks.
> > > >>>>>>>
> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > > >>>>>>>
> > > >>>>>>> -Steve
> > > >>>>>>>
> > > >>>>>
> > > >>>>>
> ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > > >>
> > > >>
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I've done another docker build and the client jars appear to be there. I'll
test tomorrow before putting up another vote. it will be exactly the same
commit as before, just recompiled/republished

On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:

> Agreed, that step #10 is out-dated and should be removed (I skipped
> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>
> > How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> Akira, I tried this too but it didn't work. I think we'd need the
> artifacts to be properly pushed to the staging repository.
>
> > Could you please let me know how I can consume the Hadoop 3 jars in
> maven?
>
> Gautham (if you are following this thread), you'll need to add the
> following:
>
>     <repository>
>        <id>staged</id>
>        <name>staged-releases</name>
>        <url>https://repository.apache.org/content/repositories/staging/
> </url>
>        <releases>
>          <enabled>true</enabled>
>        </releases>
>        <snapshots>
>          <enabled>true</enabled>
>        </snapshots>
>      </repository>
>
> to the `<repositories>` section in your Maven pom.xml file.
>
> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
> <st...@cloudera.com.invalid> wrote:
> >
> > I didn't do that as the docker image was doing it itself...I discussed
> this
> > with Akira and Ayush & they agreed. so whatever went wrong. it was
> > something else.
> >
> > I have been building a list of things I'd like to change there; cutting
> > that line was one of them. but I need to work out the correct workflow.
> >
> > trying again, and creating a stub module to verify the client is in
> staging
> >
> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
> iwasakims@oss.nttdata.co.jp>
> > wrote:
> >
> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> > >
> > > After HADOOP-15058, `mvn deploy` is kicked by
> > > `dev-support/bin/create-release --asfrelease`.
> > > https://issues.apache.org/jira/browse/HADOOP-15058
> > >
> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> > >
> > > 2 sets of artifact are deployed after creating RC based on the
> instruction.
> > > The latest one contains empty shaded jars.
> > >
> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > > looks having same issue...
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > > Hi Chao,
> > > >
> > > > How about using
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > > instead of
> https://repository.apache.org/content/repositories/staging/ ?
> > > >
> > > > Akira
> > > >
> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >
> > > >> Hmm, I see the artifacts ideally should have got overwritten by the
> new
> > > >> RC, but they didn’t. The reason seems like the staging path shared
> > > doesn’t
> > > >> have any jars…
> > > >> That is why it was picking the old jars. I think Steve needs to run
> mvn
> > > >> deploy again…
> > > >>
> > > >> Sent from my iPhone
> > > >>
> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > > >>>
> > > >>> 
> > > >>>>
> > > >>>> Chao can you use the one that Steve mentioned in the mail?
> > > >>>
> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > > >>> release bits will show up in
> > > >>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > >>> and Spark build will be able to pick them up for testing. However
> in
> > > >>> this case I don't see any 3.3.3 jars in the URL.
> > > >>>
> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
> the
> > > >> date was also april 22, I archived that. Chao can you use the one
> that
> > > >> Steve mentioned in the mail?
> > > >>>>
> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
> wrote:
> > > >>>>>
> > > >>>>> Seems there are some issues with the shaded client as I was not
> able
> > > >>>>> to compile Apache Spark with the RC
> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > > compiled
> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
> doesn't
> > > >>>>> contain any class:
> > > >>>>>
> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > > >>>>> META-INF/
> > > >>>>> META-INF/MANIFEST.MF
> > > >>>>> META-INF/NOTICE.txt
> > > >>>>> META-INF/LICENSE.txt
> > > >>>>> META-INF/maven/
> > > >>>>> META-INF/maven/org.apache.hadoop/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > > >>>>>
> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > > >>>>>>
> > > >>>>>> +1 (binding)
> > > >>>>>>
> > > >>>>>>   * Signature: ok
> > > >>>>>>   * Checksum : passed
> > > >>>>>>   * Rat check (1.8.0_191): passed
> > > >>>>>>    - mvn clean apache-rat:check
> > > >>>>>>   * Built from source (1.8.0_191): failed
> > > >>>>>>    - mvn clean install  -DskipTests
> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > > >> -Dmaven.javadoc.skip=true
> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > > >>>>>>   * Unit tests pass (1.8.0_191):
> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > > >>>>>>
> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
> five
> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > > built
> > > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > > Ran a
> > > >>>>>> bigger job with mild chaos. All seems to be working properly
> > > >> (recoveries,
> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
> UIs
> > > look
> > > >>>>>> good. Poked at the hdfs cli. Seems good.
> > > >>>>>>
> > > >>>>>> S
> > > >>>>>>
> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > > >> <st...@cloudera.com.invalid>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > > >>>>>>>
> > > >>>>>>> The RC is available at:
> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > > >>>>>>>
> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > > >>>>>>>
> > > >>>>>>> The maven artifacts are staged at
> > > >>>>>>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > >>>>>>>
> > > >>>>>>> You can find my public key at:
> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>>>>>>
> > > >>>>>>> Change log
> > > >>>>>>>
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > > >>>>>>>
> > > >>>>>>> Release notes
> > > >>>>>>>
> > > >>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > > >>>>>>>
> > > >>>>>>> There's a very small number of changes, primarily critical
> > > >> code/packaging
> > > >>>>>>> issues and security fixes.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > > >>>>>>>    -  CVEs in our code and dependencies
> > > >>>>>>>    - Shaded client packaging issues.
> > > >>>>>>>    - A switch from log4j to reload4j
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > > >> classes which
> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
> classes,
> > > >> they
> > > >>>>>>> regularly raised alerts on security scans and concen from
> users.
> > > >> Switching
> > > >>>>>>> to the forked project allows us to ship a secure logging
> framework.
> > > >> It will
> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
> which
> > > >> exclude
> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > > >> instead/as
> > > >>>>>>> well.
> > > >>>>>>>
> > > >>>>>>> See the release notes for details.
> > > >>>>>>>
> > > >>>>>>> This is my first release through the new docker build process,
> do
> > > >> please
> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > > trying
> > > >> builds
> > > >>>>>>> of downstream projects.
> > > >>>>>>>
> > > >>>>>>> We know there are some outstanding issues with at least one
> library
> > > >> we are
> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
> for it.
> > > >> If the
> > > >>>>>>> docker based release process works smoothly enough we can do a
> > > >> followup
> > > >>>>>>> security release in a few weeks.
> > > >>>>>>>
> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > > >>>>>>>
> > > >>>>>>> -Steve
> > > >>>>>>>
> > > >>>>>
> > > >>>>>
> ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > > >>
> > > >>
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I've done another docker build and the client jars appear to be there. I'll
test tomorrow before putting up another vote. it will be exactly the same
commit as before, just recompiled/republished

On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:

> Agreed, that step #10 is out-dated and should be removed (I skipped
> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>
> > How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> Akira, I tried this too but it didn't work. I think we'd need the
> artifacts to be properly pushed to the staging repository.
>
> > Could you please let me know how I can consume the Hadoop 3 jars in
> maven?
>
> Gautham (if you are following this thread), you'll need to add the
> following:
>
>     <repository>
>        <id>staged</id>
>        <name>staged-releases</name>
>        <url>https://repository.apache.org/content/repositories/staging/
> </url>
>        <releases>
>          <enabled>true</enabled>
>        </releases>
>        <snapshots>
>          <enabled>true</enabled>
>        </snapshots>
>      </repository>
>
> to the `<repositories>` section in your Maven pom.xml file.
>
> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
> <st...@cloudera.com.invalid> wrote:
> >
> > I didn't do that as the docker image was doing it itself...I discussed
> this
> > with Akira and Ayush & they agreed. so whatever went wrong. it was
> > something else.
> >
> > I have been building a list of things I'd like to change there; cutting
> > that line was one of them. but I need to work out the correct workflow.
> >
> > trying again, and creating a stub module to verify the client is in
> staging
> >
> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
> iwasakims@oss.nttdata.co.jp>
> > wrote:
> >
> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> > >
> > > After HADOOP-15058, `mvn deploy` is kicked by
> > > `dev-support/bin/create-release --asfrelease`.
> > > https://issues.apache.org/jira/browse/HADOOP-15058
> > >
> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> > >
> > > 2 sets of artifact are deployed after creating RC based on the
> instruction.
> > > The latest one contains empty shaded jars.
> > >
> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > > looks having same issue...
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > > Hi Chao,
> > > >
> > > > How about using
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > > instead of
> https://repository.apache.org/content/repositories/staging/ ?
> > > >
> > > > Akira
> > > >
> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >
> > > >> Hmm, I see the artifacts ideally should have got overwritten by the
> new
> > > >> RC, but they didn’t. The reason seems like the staging path shared
> > > doesn’t
> > > >> have any jars…
> > > >> That is why it was picking the old jars. I think Steve needs to run
> mvn
> > > >> deploy again…
> > > >>
> > > >> Sent from my iPhone
> > > >>
> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > > >>>
> > > >>> 
> > > >>>>
> > > >>>> Chao can you use the one that Steve mentioned in the mail?
> > > >>>
> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > > >>> release bits will show up in
> > > >>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > >>> and Spark build will be able to pick them up for testing. However
> in
> > > >>> this case I don't see any 3.3.3 jars in the URL.
> > > >>>
> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
> the
> > > >> date was also april 22, I archived that. Chao can you use the one
> that
> > > >> Steve mentioned in the mail?
> > > >>>>
> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
> wrote:
> > > >>>>>
> > > >>>>> Seems there are some issues with the shaded client as I was not
> able
> > > >>>>> to compile Apache Spark with the RC
> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > > compiled
> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
> doesn't
> > > >>>>> contain any class:
> > > >>>>>
> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > > >>>>> META-INF/
> > > >>>>> META-INF/MANIFEST.MF
> > > >>>>> META-INF/NOTICE.txt
> > > >>>>> META-INF/LICENSE.txt
> > > >>>>> META-INF/maven/
> > > >>>>> META-INF/maven/org.apache.hadoop/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > > >>>>>
> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > > >>>>>>
> > > >>>>>> +1 (binding)
> > > >>>>>>
> > > >>>>>>   * Signature: ok
> > > >>>>>>   * Checksum : passed
> > > >>>>>>   * Rat check (1.8.0_191): passed
> > > >>>>>>    - mvn clean apache-rat:check
> > > >>>>>>   * Built from source (1.8.0_191): failed
> > > >>>>>>    - mvn clean install  -DskipTests
> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > > >> -Dmaven.javadoc.skip=true
> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > > >>>>>>   * Unit tests pass (1.8.0_191):
> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > > >>>>>>
> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
> five
> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > > built
> > > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > > Ran a
> > > >>>>>> bigger job with mild chaos. All seems to be working properly
> > > >> (recoveries,
> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
> UIs
> > > look
> > > >>>>>> good. Poked at the hdfs cli. Seems good.
> > > >>>>>>
> > > >>>>>> S
> > > >>>>>>
> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > > >> <st...@cloudera.com.invalid>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > > >>>>>>>
> > > >>>>>>> The RC is available at:
> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > > >>>>>>>
> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > > >>>>>>>
> > > >>>>>>> The maven artifacts are staged at
> > > >>>>>>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > >>>>>>>
> > > >>>>>>> You can find my public key at:
> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>>>>>>
> > > >>>>>>> Change log
> > > >>>>>>>
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > > >>>>>>>
> > > >>>>>>> Release notes
> > > >>>>>>>
> > > >>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > > >>>>>>>
> > > >>>>>>> There's a very small number of changes, primarily critical
> > > >> code/packaging
> > > >>>>>>> issues and security fixes.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > > >>>>>>>    -  CVEs in our code and dependencies
> > > >>>>>>>    - Shaded client packaging issues.
> > > >>>>>>>    - A switch from log4j to reload4j
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > > >> classes which
> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
> classes,
> > > >> they
> > > >>>>>>> regularly raised alerts on security scans and concen from
> users.
> > > >> Switching
> > > >>>>>>> to the forked project allows us to ship a secure logging
> framework.
> > > >> It will
> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
> which
> > > >> exclude
> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > > >> instead/as
> > > >>>>>>> well.
> > > >>>>>>>
> > > >>>>>>> See the release notes for details.
> > > >>>>>>>
> > > >>>>>>> This is my first release through the new docker build process,
> do
> > > >> please
> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > > trying
> > > >> builds
> > > >>>>>>> of downstream projects.
> > > >>>>>>>
> > > >>>>>>> We know there are some outstanding issues with at least one
> library
> > > >> we are
> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
> for it.
> > > >> If the
> > > >>>>>>> docker based release process works smoothly enough we can do a
> > > >> followup
> > > >>>>>>> security release in a few weeks.
> > > >>>>>>>
> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > > >>>>>>>
> > > >>>>>>> -Steve
> > > >>>>>>>
> > > >>>>>
> > > >>>>>
> ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > > >>
> > > >>
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I've done another docker build and the client jars appear to be there. I'll
test tomorrow before putting up another vote. it will be exactly the same
commit as before, just recompiled/republished

On Mon, 9 May 2022 at 17:45, Chao Sun <su...@apache.org> wrote:

> Agreed, that step #10 is out-dated and should be removed (I skipped
> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>
> > How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> Akira, I tried this too but it didn't work. I think we'd need the
> artifacts to be properly pushed to the staging repository.
>
> > Could you please let me know how I can consume the Hadoop 3 jars in
> maven?
>
> Gautham (if you are following this thread), you'll need to add the
> following:
>
>     <repository>
>        <id>staged</id>
>        <name>staged-releases</name>
>        <url>https://repository.apache.org/content/repositories/staging/
> </url>
>        <releases>
>          <enabled>true</enabled>
>        </releases>
>        <snapshots>
>          <enabled>true</enabled>
>        </snapshots>
>      </repository>
>
> to the `<repositories>` section in your Maven pom.xml file.
>
> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
> <st...@cloudera.com.invalid> wrote:
> >
> > I didn't do that as the docker image was doing it itself...I discussed
> this
> > with Akira and Ayush & they agreed. so whatever went wrong. it was
> > something else.
> >
> > I have been building a list of things I'd like to change there; cutting
> > that line was one of them. but I need to work out the correct workflow.
> >
> > trying again, and creating a stub module to verify the client is in
> staging
> >
> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
> iwasakims@oss.nttdata.co.jp>
> > wrote:
> >
> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> > >
> > > After HADOOP-15058, `mvn deploy` is kicked by
> > > `dev-support/bin/create-release --asfrelease`.
> > > https://issues.apache.org/jira/browse/HADOOP-15058
> > >
> > > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> > >
> > > 2 sets of artifact are deployed after creating RC based on the
> instruction.
> > > The latest one contains empty shaded jars.
> > >
> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > > looks having same issue...
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > > Hi Chao,
> > > >
> > > > How about using
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > > instead of
> https://repository.apache.org/content/repositories/staging/ ?
> > > >
> > > > Akira
> > > >
> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >
> > > >> Hmm, I see the artifacts ideally should have got overwritten by the
> new
> > > >> RC, but they didn’t. The reason seems like the staging path shared
> > > doesn’t
> > > >> have any jars…
> > > >> That is why it was picking the old jars. I think Steve needs to run
> mvn
> > > >> deploy again…
> > > >>
> > > >> Sent from my iPhone
> > > >>
> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > > >>>
> > > >>> 
> > > >>>>
> > > >>>> Chao can you use the one that Steve mentioned in the mail?
> > > >>>
> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > > >>> release bits will show up in
> > > >>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > >>> and Spark build will be able to pick them up for testing. However
> in
> > > >>> this case I don't see any 3.3.3 jars in the URL.
> > > >>>
> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
> the
> > > >> date was also april 22, I archived that. Chao can you use the one
> that
> > > >> Steve mentioned in the mail?
> > > >>>>
> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org>
> wrote:
> > > >>>>>
> > > >>>>> Seems there are some issues with the shaded client as I was not
> able
> > > >>>>> to compile Apache Spark with the RC
> > > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > > compiled
> > > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR
> doesn't
> > > >>>>> contain any class:
> > > >>>>>
> > > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > > >>>>> META-INF/
> > > >>>>> META-INF/MANIFEST.MF
> > > >>>>> META-INF/NOTICE.txt
> > > >>>>> META-INF/LICENSE.txt
> > > >>>>> META-INF/maven/
> > > >>>>> META-INF/maven/org.apache.hadoop/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > > >>>>>
> > > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > > >>>>>>
> > > >>>>>> +1 (binding)
> > > >>>>>>
> > > >>>>>>   * Signature: ok
> > > >>>>>>   * Checksum : passed
> > > >>>>>>   * Rat check (1.8.0_191): passed
> > > >>>>>>    - mvn clean apache-rat:check
> > > >>>>>>   * Built from source (1.8.0_191): failed
> > > >>>>>>    - mvn clean install  -DskipTests
> > > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > > >> -Dmaven.javadoc.skip=true
> > > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > > >>>>>>   * Unit tests pass (1.8.0_191):
> > > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > > >>>>>>
> > > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and
> five
> > > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > > built
> > > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > > Ran a
> > > >>>>>> bigger job with mild chaos. All seems to be working properly
> > > >> (recoveries,
> > > >>>>>> logs look fine). Killed a namenode. Failover worked promptly.
> UIs
> > > look
> > > >>>>>> good. Poked at the hdfs cli. Seems good.
> > > >>>>>>
> > > >>>>>> S
> > > >>>>>>
> > > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > > >> <st...@cloudera.com.invalid>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > > >>>>>>>
> > > >>>>>>> The RC is available at:
> > > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > > >>>>>>>
> > > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > > >>>>>>>
> > > >>>>>>> The maven artifacts are staged at
> > > >>>>>>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > >>>>>>>
> > > >>>>>>> You can find my public key at:
> > > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>>>>>>
> > > >>>>>>> Change log
> > > >>>>>>>
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > > >>>>>>>
> > > >>>>>>> Release notes
> > > >>>>>>>
> > > >>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > > >>>>>>>
> > > >>>>>>> There's a very small number of changes, primarily critical
> > > >> code/packaging
> > > >>>>>>> issues and security fixes.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > > >>>>>>>    -  CVEs in our code and dependencies
> > > >>>>>>>    - Shaded client packaging issues.
> > > >>>>>>>    - A switch from log4j to reload4j
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > > >> classes which
> > > >>>>>>> contain CVEs removed. Even though hadoop never used those
> classes,
> > > >> they
> > > >>>>>>> regularly raised alerts on security scans and concen from
> users.
> > > >> Switching
> > > >>>>>>> to the forked project allows us to ship a secure logging
> framework.
> > > >> It will
> > > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects
> which
> > > >> exclude
> > > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > > >> instead/as
> > > >>>>>>> well.
> > > >>>>>>>
> > > >>>>>>> See the release notes for details.
> > > >>>>>>>
> > > >>>>>>> This is my first release through the new docker build process,
> do
> > > >> please
> > > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > > trying
> > > >> builds
> > > >>>>>>> of downstream projects.
> > > >>>>>>>
> > > >>>>>>> We know there are some outstanding issues with at least one
> library
> > > >> we are
> > > >>>>>>> shipping (okhttp), but I don't want to hold this release up
> for it.
> > > >> If the
> > > >>>>>>> docker based release process works smoothly enough we can do a
> > > >> followup
> > > >>>>>>> security release in a few weeks.
> > > >>>>>>>
> > > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > > >>>>>>>
> > > >>>>>>> -Steve
> > > >>>>>>>
> > > >>>>>
> > > >>>>>
> ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > >>>>> For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >>>>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > > >>
> > > >>
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Agreed, that step #10 is out-dated and should be removed (I skipped
that when releasing Hadoop 3.3.2 but didn't update it, sorry).

> How about using https://repository.apache.org/content/repositories/orgapachehadoop-1348/

Akira, I tried this too but it didn't work. I think we'd need the
artifacts to be properly pushed to the staging repository.

> Could you please let me know how I can consume the Hadoop 3 jars in maven?

Gautham (if you are following this thread), you'll need to add the following:

    <repository>
       <id>staged</id>
       <name>staged-releases</name>
       <url>https://repository.apache.org/content/repositories/staging/</url>
       <releases>
         <enabled>true</enabled>
       </releases>
       <snapshots>
         <enabled>true</enabled>
       </snapshots>
     </repository>

to the `<repositories>` section in your Maven pom.xml file.

On Mon, May 9, 2022 at 8:52 AM Steve Loughran
<st...@cloudera.com.invalid> wrote:
>
> I didn't do that as the docker image was doing it itself...I discussed this
> with Akira and Ayush & they agreed. so whatever went wrong. it was
> something else.
>
> I have been building a list of things I'd like to change there; cutting
> that line was one of them. but I need to work out the correct workflow.
>
> trying again, and creating a stub module to verify the client is in staging
>
> On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
>
> > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> >
> > After HADOOP-15058, `mvn deploy` is kicked by
> > `dev-support/bin/create-release --asfrelease`.
> > https://issues.apache.org/jira/browse/HADOOP-15058
> >
> > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> >
> > 2 sets of artifact are deployed after creating RC based on the instruction.
> > The latest one contains empty shaded jars.
> >
> > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > looks having same issue...
> >
> > Masatake Iwasaki
> >
> > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > Hi Chao,
> > >
> > > How about using
> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > instead of https://repository.apache.org/content/repositories/staging/ ?
> > >
> > > Akira
> > >
> > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> > >
> > >> Hmm, I see the artifacts ideally should have got overwritten by the new
> > >> RC, but they didn’t. The reason seems like the staging path shared
> > doesn’t
> > >> have any jars…
> > >> That is why it was picking the old jars. I think Steve needs to run mvn
> > >> deploy again…
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> 
> > >>>>
> > >>>> Chao can you use the one that Steve mentioned in the mail?
> > >>>
> > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > >>> release bits will show up in
> > >>>
> > >>
> > https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > >>> and Spark build will be able to pick them up for testing. However in
> > >>> this case I don't see any 3.3.3 jars in the URL.
> > >>>
> > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > wrote:
> > >>>>
> > >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > >> date was also april 22, I archived that. Chao can you use the one that
> > >> Steve mentioned in the mail?
> > >>>>
> > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>>>
> > >>>>> Seems there are some issues with the shaded client as I was not able
> > >>>>> to compile Apache Spark with the RC
> > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > compiled
> > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>>>> contain any class:
> > >>>>>
> > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>>>> META-INF/
> > >>>>> META-INF/MANIFEST.MF
> > >>>>> META-INF/NOTICE.txt
> > >>>>> META-INF/LICENSE.txt
> > >>>>> META-INF/maven/
> > >>>>> META-INF/maven/org.apache.hadoop/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>>>
> > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>>>
> > >>>>>> +1 (binding)
> > >>>>>>
> > >>>>>>   * Signature: ok
> > >>>>>>   * Checksum : passed
> > >>>>>>   * Rat check (1.8.0_191): passed
> > >>>>>>    - mvn clean apache-rat:check
> > >>>>>>   * Built from source (1.8.0_191): failed
> > >>>>>>    - mvn clean install  -DskipTests
> > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > >> -Dmaven.javadoc.skip=true
> > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>>>   * Unit tests pass (1.8.0_191):
> > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > >>>>>>
> > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > built
> > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > Ran a
> > >>>>>> bigger job with mild chaos. All seems to be working properly
> > >> (recoveries,
> > >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> > look
> > >>>>>> good. Poked at the hdfs cli. Seems good.
> > >>>>>>
> > >>>>>> S
> > >>>>>>
> > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > >> <st...@cloudera.com.invalid>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>>>
> > >>>>>>> The RC is available at:
> > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>>>
> > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>>>
> > >>>>>>> The maven artifacts are staged at
> > >>>>>>>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>>>
> > >>>>>>> You can find my public key at:
> > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>>>
> > >>>>>>> Change log
> > >>>>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>>>
> > >>>>>>> Release notes
> > >>>>>>>
> > >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>>>
> > >>>>>>> There's a very small number of changes, primarily critical
> > >> code/packaging
> > >>>>>>> issues and security fixes.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>>>    -  CVEs in our code and dependencies
> > >>>>>>>    - Shaded client packaging issues.
> > >>>>>>>    - A switch from log4j to reload4j
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > >> classes which
> > >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> > >> they
> > >>>>>>> regularly raised alerts on security scans and concen from users.
> > >> Switching
> > >>>>>>> to the forked project allows us to ship a secure logging framework.
> > >> It will
> > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > >> exclude
> > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > >> instead/as
> > >>>>>>> well.
> > >>>>>>>
> > >>>>>>> See the release notes for details.
> > >>>>>>>
> > >>>>>>> This is my first release through the new docker build process, do
> > >> please
> > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > trying
> > >> builds
> > >>>>>>> of downstream projects.
> > >>>>>>>
> > >>>>>>> We know there are some outstanding issues with at least one library
> > >> we are
> > >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > >> If the
> > >>>>>>> docker based release process works smoothly enough we can do a
> > >> followup
> > >>>>>>> security release in a few weeks.
> > >>>>>>>
> > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>>>
> > >>>>>>> -Steve
> > >>>>>>>
> > >>>>>
> > >>>>> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Agreed, that step #10 is out-dated and should be removed (I skipped
that when releasing Hadoop 3.3.2 but didn't update it, sorry).

> How about using https://repository.apache.org/content/repositories/orgapachehadoop-1348/

Akira, I tried this too but it didn't work. I think we'd need the
artifacts to be properly pushed to the staging repository.

> Could you please let me know how I can consume the Hadoop 3 jars in maven?

Gautham (if you are following this thread), you'll need to add the following:

    <repository>
       <id>staged</id>
       <name>staged-releases</name>
       <url>https://repository.apache.org/content/repositories/staging/</url>
       <releases>
         <enabled>true</enabled>
       </releases>
       <snapshots>
         <enabled>true</enabled>
       </snapshots>
     </repository>

to the `<repositories>` section in your Maven pom.xml file.

On Mon, May 9, 2022 at 8:52 AM Steve Loughran
<st...@cloudera.com.invalid> wrote:
>
> I didn't do that as the docker image was doing it itself...I discussed this
> with Akira and Ayush & they agreed. so whatever went wrong. it was
> something else.
>
> I have been building a list of things I'd like to change there; cutting
> that line was one of them. but I need to work out the correct workflow.
>
> trying again, and creating a stub module to verify the client is in staging
>
> On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
>
> > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> >
> > After HADOOP-15058, `mvn deploy` is kicked by
> > `dev-support/bin/create-release --asfrelease`.
> > https://issues.apache.org/jira/browse/HADOOP-15058
> >
> > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> >
> > 2 sets of artifact are deployed after creating RC based on the instruction.
> > The latest one contains empty shaded jars.
> >
> > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > looks having same issue...
> >
> > Masatake Iwasaki
> >
> > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > Hi Chao,
> > >
> > > How about using
> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > instead of https://repository.apache.org/content/repositories/staging/ ?
> > >
> > > Akira
> > >
> > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> > >
> > >> Hmm, I see the artifacts ideally should have got overwritten by the new
> > >> RC, but they didn’t. The reason seems like the staging path shared
> > doesn’t
> > >> have any jars…
> > >> That is why it was picking the old jars. I think Steve needs to run mvn
> > >> deploy again…
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> 
> > >>>>
> > >>>> Chao can you use the one that Steve mentioned in the mail?
> > >>>
> > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > >>> release bits will show up in
> > >>>
> > >>
> > https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > >>> and Spark build will be able to pick them up for testing. However in
> > >>> this case I don't see any 3.3.3 jars in the URL.
> > >>>
> > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > wrote:
> > >>>>
> > >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > >> date was also april 22, I archived that. Chao can you use the one that
> > >> Steve mentioned in the mail?
> > >>>>
> > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>>>
> > >>>>> Seems there are some issues with the shaded client as I was not able
> > >>>>> to compile Apache Spark with the RC
> > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > compiled
> > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>>>> contain any class:
> > >>>>>
> > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>>>> META-INF/
> > >>>>> META-INF/MANIFEST.MF
> > >>>>> META-INF/NOTICE.txt
> > >>>>> META-INF/LICENSE.txt
> > >>>>> META-INF/maven/
> > >>>>> META-INF/maven/org.apache.hadoop/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>>>
> > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>>>
> > >>>>>> +1 (binding)
> > >>>>>>
> > >>>>>>   * Signature: ok
> > >>>>>>   * Checksum : passed
> > >>>>>>   * Rat check (1.8.0_191): passed
> > >>>>>>    - mvn clean apache-rat:check
> > >>>>>>   * Built from source (1.8.0_191): failed
> > >>>>>>    - mvn clean install  -DskipTests
> > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > >> -Dmaven.javadoc.skip=true
> > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>>>   * Unit tests pass (1.8.0_191):
> > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > >>>>>>
> > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > built
> > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > Ran a
> > >>>>>> bigger job with mild chaos. All seems to be working properly
> > >> (recoveries,
> > >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> > look
> > >>>>>> good. Poked at the hdfs cli. Seems good.
> > >>>>>>
> > >>>>>> S
> > >>>>>>
> > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > >> <st...@cloudera.com.invalid>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>>>
> > >>>>>>> The RC is available at:
> > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>>>
> > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>>>
> > >>>>>>> The maven artifacts are staged at
> > >>>>>>>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>>>
> > >>>>>>> You can find my public key at:
> > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>>>
> > >>>>>>> Change log
> > >>>>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>>>
> > >>>>>>> Release notes
> > >>>>>>>
> > >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>>>
> > >>>>>>> There's a very small number of changes, primarily critical
> > >> code/packaging
> > >>>>>>> issues and security fixes.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>>>    -  CVEs in our code and dependencies
> > >>>>>>>    - Shaded client packaging issues.
> > >>>>>>>    - A switch from log4j to reload4j
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > >> classes which
> > >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> > >> they
> > >>>>>>> regularly raised alerts on security scans and concen from users.
> > >> Switching
> > >>>>>>> to the forked project allows us to ship a secure logging framework.
> > >> It will
> > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > >> exclude
> > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > >> instead/as
> > >>>>>>> well.
> > >>>>>>>
> > >>>>>>> See the release notes for details.
> > >>>>>>>
> > >>>>>>> This is my first release through the new docker build process, do
> > >> please
> > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > trying
> > >> builds
> > >>>>>>> of downstream projects.
> > >>>>>>>
> > >>>>>>> We know there are some outstanding issues with at least one library
> > >> we are
> > >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > >> If the
> > >>>>>>> docker based release process works smoothly enough we can do a
> > >> followup
> > >>>>>>> security release in a few weeks.
> > >>>>>>>
> > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>>>
> > >>>>>>> -Steve
> > >>>>>>>
> > >>>>>
> > >>>>> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Agreed, that step #10 is out-dated and should be removed (I skipped
that when releasing Hadoop 3.3.2 but didn't update it, sorry).

> How about using https://repository.apache.org/content/repositories/orgapachehadoop-1348/

Akira, I tried this too but it didn't work. I think we'd need the
artifacts to be properly pushed to the staging repository.

> Could you please let me know how I can consume the Hadoop 3 jars in maven?

Gautham (if you are following this thread), you'll need to add the following:

    <repository>
       <id>staged</id>
       <name>staged-releases</name>
       <url>https://repository.apache.org/content/repositories/staging/</url>
       <releases>
         <enabled>true</enabled>
       </releases>
       <snapshots>
         <enabled>true</enabled>
       </snapshots>
     </repository>

to the `<repositories>` section in your Maven pom.xml file.

On Mon, May 9, 2022 at 8:52 AM Steve Loughran
<st...@cloudera.com.invalid> wrote:
>
> I didn't do that as the docker image was doing it itself...I discussed this
> with Akira and Ayush & they agreed. so whatever went wrong. it was
> something else.
>
> I have been building a list of things I'd like to change there; cutting
> that line was one of them. but I need to work out the correct workflow.
>
> trying again, and creating a stub module to verify the client is in staging
>
> On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
>
> > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> >
> > After HADOOP-15058, `mvn deploy` is kicked by
> > `dev-support/bin/create-release --asfrelease`.
> > https://issues.apache.org/jira/browse/HADOOP-15058
> >
> > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> >
> > 2 sets of artifact are deployed after creating RC based on the instruction.
> > The latest one contains empty shaded jars.
> >
> > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > looks having same issue...
> >
> > Masatake Iwasaki
> >
> > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > Hi Chao,
> > >
> > > How about using
> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > instead of https://repository.apache.org/content/repositories/staging/ ?
> > >
> > > Akira
> > >
> > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> > >
> > >> Hmm, I see the artifacts ideally should have got overwritten by the new
> > >> RC, but they didn’t. The reason seems like the staging path shared
> > doesn’t
> > >> have any jars…
> > >> That is why it was picking the old jars. I think Steve needs to run mvn
> > >> deploy again…
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> 
> > >>>>
> > >>>> Chao can you use the one that Steve mentioned in the mail?
> > >>>
> > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > >>> release bits will show up in
> > >>>
> > >>
> > https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > >>> and Spark build will be able to pick them up for testing. However in
> > >>> this case I don't see any 3.3.3 jars in the URL.
> > >>>
> > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > wrote:
> > >>>>
> > >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > >> date was also april 22, I archived that. Chao can you use the one that
> > >> Steve mentioned in the mail?
> > >>>>
> > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>>>
> > >>>>> Seems there are some issues with the shaded client as I was not able
> > >>>>> to compile Apache Spark with the RC
> > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > compiled
> > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>>>> contain any class:
> > >>>>>
> > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>>>> META-INF/
> > >>>>> META-INF/MANIFEST.MF
> > >>>>> META-INF/NOTICE.txt
> > >>>>> META-INF/LICENSE.txt
> > >>>>> META-INF/maven/
> > >>>>> META-INF/maven/org.apache.hadoop/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>>>
> > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>>>
> > >>>>>> +1 (binding)
> > >>>>>>
> > >>>>>>   * Signature: ok
> > >>>>>>   * Checksum : passed
> > >>>>>>   * Rat check (1.8.0_191): passed
> > >>>>>>    - mvn clean apache-rat:check
> > >>>>>>   * Built from source (1.8.0_191): failed
> > >>>>>>    - mvn clean install  -DskipTests
> > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > >> -Dmaven.javadoc.skip=true
> > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>>>   * Unit tests pass (1.8.0_191):
> > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > >>>>>>
> > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > built
> > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > Ran a
> > >>>>>> bigger job with mild chaos. All seems to be working properly
> > >> (recoveries,
> > >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> > look
> > >>>>>> good. Poked at the hdfs cli. Seems good.
> > >>>>>>
> > >>>>>> S
> > >>>>>>
> > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > >> <st...@cloudera.com.invalid>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>>>
> > >>>>>>> The RC is available at:
> > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>>>
> > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>>>
> > >>>>>>> The maven artifacts are staged at
> > >>>>>>>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>>>
> > >>>>>>> You can find my public key at:
> > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>>>
> > >>>>>>> Change log
> > >>>>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>>>
> > >>>>>>> Release notes
> > >>>>>>>
> > >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>>>
> > >>>>>>> There's a very small number of changes, primarily critical
> > >> code/packaging
> > >>>>>>> issues and security fixes.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>>>    -  CVEs in our code and dependencies
> > >>>>>>>    - Shaded client packaging issues.
> > >>>>>>>    - A switch from log4j to reload4j
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > >> classes which
> > >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> > >> they
> > >>>>>>> regularly raised alerts on security scans and concen from users.
> > >> Switching
> > >>>>>>> to the forked project allows us to ship a secure logging framework.
> > >> It will
> > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > >> exclude
> > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > >> instead/as
> > >>>>>>> well.
> > >>>>>>>
> > >>>>>>> See the release notes for details.
> > >>>>>>>
> > >>>>>>> This is my first release through the new docker build process, do
> > >> please
> > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > trying
> > >> builds
> > >>>>>>> of downstream projects.
> > >>>>>>>
> > >>>>>>> We know there are some outstanding issues with at least one library
> > >> we are
> > >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > >> If the
> > >>>>>>> docker based release process works smoothly enough we can do a
> > >> followup
> > >>>>>>> security release in a few weeks.
> > >>>>>>>
> > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>>>
> > >>>>>>> -Steve
> > >>>>>>>
> > >>>>>
> > >>>>> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Agreed, that step #10 is out-dated and should be removed (I skipped
that when releasing Hadoop 3.3.2 but didn't update it, sorry).

> How about using https://repository.apache.org/content/repositories/orgapachehadoop-1348/

Akira, I tried this too but it didn't work. I think we'd need the
artifacts to be properly pushed to the staging repository.

> Could you please let me know how I can consume the Hadoop 3 jars in maven?

Gautham (if you are following this thread), you'll need to add the following:

    <repository>
       <id>staged</id>
       <name>staged-releases</name>
       <url>https://repository.apache.org/content/repositories/staging/</url>
       <releases>
         <enabled>true</enabled>
       </releases>
       <snapshots>
         <enabled>true</enabled>
       </snapshots>
     </repository>

to the `<repositories>` section in your Maven pom.xml file.

On Mon, May 9, 2022 at 8:52 AM Steve Loughran
<st...@cloudera.com.invalid> wrote:
>
> I didn't do that as the docker image was doing it itself...I discussed this
> with Akira and Ayush & they agreed. so whatever went wrong. it was
> something else.
>
> I have been building a list of things I'd like to change there; cutting
> that line was one of them. but I need to work out the correct workflow.
>
> trying again, and creating a stub module to verify the client is in staging
>
> On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
>
> > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> >
> > After HADOOP-15058, `mvn deploy` is kicked by
> > `dev-support/bin/create-release --asfrelease`.
> > https://issues.apache.org/jira/browse/HADOOP-15058
> >
> > Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> >
> > 2 sets of artifact are deployed after creating RC based on the instruction.
> > The latest one contains empty shaded jars.
> >
> > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > looks having same issue...
> >
> > Masatake Iwasaki
> >
> > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > Hi Chao,
> > >
> > > How about using
> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > instead of https://repository.apache.org/content/repositories/staging/ ?
> > >
> > > Akira
> > >
> > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> > >
> > >> Hmm, I see the artifacts ideally should have got overwritten by the new
> > >> RC, but they didn’t. The reason seems like the staging path shared
> > doesn’t
> > >> have any jars…
> > >> That is why it was picking the old jars. I think Steve needs to run mvn
> > >> deploy again…
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> 
> > >>>>
> > >>>> Chao can you use the one that Steve mentioned in the mail?
> > >>>
> > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > >>> release bits will show up in
> > >>>
> > >>
> > https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > >>> and Spark build will be able to pick them up for testing. However in
> > >>> this case I don't see any 3.3.3 jars in the URL.
> > >>>
> > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> > wrote:
> > >>>>
> > >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > >> date was also april 22, I archived that. Chao can you use the one that
> > >> Steve mentioned in the mail?
> > >>>>
> > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>>>
> > >>>>> Seems there are some issues with the shaded client as I was not able
> > >>>>> to compile Apache Spark with the RC
> > >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> > compiled
> > >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>>>> contain any class:
> > >>>>>
> > >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>>>> META-INF/
> > >>>>> META-INF/MANIFEST.MF
> > >>>>> META-INF/NOTICE.txt
> > >>>>> META-INF/LICENSE.txt
> > >>>>> META-INF/maven/
> > >>>>> META-INF/maven/org.apache.hadoop/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>>>
> > >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>>>
> > >>>>>> +1 (binding)
> > >>>>>>
> > >>>>>>   * Signature: ok
> > >>>>>>   * Checksum : passed
> > >>>>>>   * Rat check (1.8.0_191): passed
> > >>>>>>    - mvn clean apache-rat:check
> > >>>>>>   * Built from source (1.8.0_191): failed
> > >>>>>>    - mvn clean install  -DskipTests
> > >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> > >> -Dmaven.javadoc.skip=true
> > >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>>>   * Unit tests pass (1.8.0_191):
> > >>>>>>     - HDFS Tests passed (Didn't run more than this).
> > >>>>>>
> > >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> > built
> > >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> > Ran a
> > >>>>>> bigger job with mild chaos. All seems to be working properly
> > >> (recoveries,
> > >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> > look
> > >>>>>> good. Poked at the hdfs cli. Seems good.
> > >>>>>>
> > >>>>>> S
> > >>>>>>
> > >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > >> <st...@cloudera.com.invalid>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>>>
> > >>>>>>> The RC is available at:
> > >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>>>
> > >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>>>
> > >>>>>>> The maven artifacts are staged at
> > >>>>>>>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>>>
> > >>>>>>> You can find my public key at:
> > >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>>>
> > >>>>>>> Change log
> > >>>>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>>>
> > >>>>>>> Release notes
> > >>>>>>>
> > >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>>>
> > >>>>>>> There's a very small number of changes, primarily critical
> > >> code/packaging
> > >>>>>>> issues and security fixes.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>>>    -  CVEs in our code and dependencies
> > >>>>>>>    - Shaded client packaging issues.
> > >>>>>>>    - A switch from log4j to reload4j
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> > >> classes which
> > >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> > >> they
> > >>>>>>> regularly raised alerts on security scans and concen from users.
> > >> Switching
> > >>>>>>> to the forked project allows us to ship a secure logging framework.
> > >> It will
> > >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > >> exclude
> > >>>>>>> our log4j artifacts, as they need to cut the new dependency
> > >> instead/as
> > >>>>>>> well.
> > >>>>>>>
> > >>>>>>> See the release notes for details.
> > >>>>>>>
> > >>>>>>> This is my first release through the new docker build process, do
> > >> please
> > >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> > trying
> > >> builds
> > >>>>>>> of downstream projects.
> > >>>>>>>
> > >>>>>>> We know there are some outstanding issues with at least one library
> > >> we are
> > >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > >> If the
> > >>>>>>> docker based release process works smoothly enough we can do a
> > >> followup
> > >>>>>>> security release in a few weeks.
> > >>>>>>>
> > >>>>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>>>
> > >>>>>>> -Steve
> > >>>>>>>
> > >>>>>
> > >>>>> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I didn't do that as the docker image was doing it itself...I discussed this
with Akira and Ayush & they agreed. so whatever went wrong. it was
something else.

I have been building a list of things I'd like to change there; cutting
that line was one of them. but I need to work out the correct workflow.

trying again, and creating a stub module to verify the client is in staging

On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
wrote:

> It seems to be caused by obsolete instruction in HowToRelease Wiki?
>
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
>
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
>
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
>
> Masatake Iwasaki
>
> On 2022/05/08 6:45, Akira Ajisaka wrote:
> > Hi Chao,
> >
> > How about using
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > instead of https://repository.apache.org/content/repositories/staging/ ?
> >
> > Akira
> >
> > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Hmm, I see the artifacts ideally should have got overwritten by the new
> >> RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> >> have any jars…
> >> That is why it was picking the old jars. I think Steve needs to run mvn
> >> deploy again…
> >>
> >> Sent from my iPhone
> >>
> >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> 
> >>>>
> >>>> Chao can you use the one that Steve mentioned in the mail?
> >>>
> >>> Hmm how do I do that? Typically after closing the RC in nexus the
> >>> release bits will show up in
> >>>
> >>
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> >>> and Spark build will be able to pick them up for testing. However in
> >>> this case I don't see any 3.3.3 jars in the URL.
> >>>
> >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >>>>
> >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> >> date was also april 22, I archived that. Chao can you use the one that
> >> Steve mentioned in the mail?
> >>>>
> >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>>>
> >>>>> Seems there are some issues with the shaded client as I was not able
> >>>>> to compile Apache Spark with the RC
> >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>>>> contain any class:
> >>>>>
> >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>>>> META-INF/
> >>>>> META-INF/MANIFEST.MF
> >>>>> META-INF/NOTICE.txt
> >>>>> META-INF/LICENSE.txt
> >>>>> META-INF/maven/
> >>>>> META-INF/maven/org.apache.hadoop/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>>>
> >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>>   * Signature: ok
> >>>>>>   * Checksum : passed
> >>>>>>   * Rat check (1.8.0_191): passed
> >>>>>>    - mvn clean apache-rat:check
> >>>>>>   * Built from source (1.8.0_191): failed
> >>>>>>    - mvn clean install  -DskipTests
> >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> >> -Dmaven.javadoc.skip=true
> >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>>>   * Unit tests pass (1.8.0_191):
> >>>>>>     - HDFS Tests passed (Didn't run more than this).
> >>>>>>
> >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> >>>>>> bigger job with mild chaos. All seems to be working properly
> >> (recoveries,
> >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> >>>>>> good. Poked at the hdfs cli. Seems good.
> >>>>>>
> >>>>>> S
> >>>>>>
> >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>>>
> >>>>>>> The RC is available at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>>>
> >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>>>
> >>>>>>> The maven artifacts are staged at
> >>>>>>>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>>>
> >>>>>>> You can find my public key at:
> >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>>>
> >>>>>>> Change log
> >>>>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>>>
> >>>>>>> Release notes
> >>>>>>>
> >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>>>
> >>>>>>> There's a very small number of changes, primarily critical
> >> code/packaging
> >>>>>>> issues and security fixes.
> >>>>>>>
> >>>>>>>
> >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> >>>>>>>    -  CVEs in our code and dependencies
> >>>>>>>    - Shaded client packaging issues.
> >>>>>>>    - A switch from log4j to reload4j
> >>>>>>>
> >>>>>>>
> >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> >> classes which
> >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> >> they
> >>>>>>> regularly raised alerts on security scans and concen from users.
> >> Switching
> >>>>>>> to the forked project allows us to ship a secure logging framework.
> >> It will
> >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> >> exclude
> >>>>>>> our log4j artifacts, as they need to cut the new dependency
> >> instead/as
> >>>>>>> well.
> >>>>>>>
> >>>>>>> See the release notes for details.
> >>>>>>>
> >>>>>>> This is my first release through the new docker build process, do
> >> please
> >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> >> builds
> >>>>>>> of downstream projects.
> >>>>>>>
> >>>>>>> We know there are some outstanding issues with at least one library
> >> we are
> >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> >> If the
> >>>>>>> docker based release process works smoothly enough we can do a
> >> followup
> >>>>>>> security release in a few weeks.
> >>>>>>>
> >>>>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>>>
> >>>>>>> -Steve
> >>>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I didn't do that as the docker image was doing it itself...I discussed this
with Akira and Ayush & they agreed. so whatever went wrong. it was
something else.

I have been building a list of things I'd like to change there; cutting
that line was one of them. but I need to work out the correct workflow.

trying again, and creating a stub module to verify the client is in staging

On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
wrote:

> It seems to be caused by obsolete instruction in HowToRelease Wiki?
>
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
>
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
>
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
>
> Masatake Iwasaki
>
> On 2022/05/08 6:45, Akira Ajisaka wrote:
> > Hi Chao,
> >
> > How about using
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > instead of https://repository.apache.org/content/repositories/staging/ ?
> >
> > Akira
> >
> > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Hmm, I see the artifacts ideally should have got overwritten by the new
> >> RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> >> have any jars…
> >> That is why it was picking the old jars. I think Steve needs to run mvn
> >> deploy again…
> >>
> >> Sent from my iPhone
> >>
> >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> 
> >>>>
> >>>> Chao can you use the one that Steve mentioned in the mail?
> >>>
> >>> Hmm how do I do that? Typically after closing the RC in nexus the
> >>> release bits will show up in
> >>>
> >>
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> >>> and Spark build will be able to pick them up for testing. However in
> >>> this case I don't see any 3.3.3 jars in the URL.
> >>>
> >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >>>>
> >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> >> date was also april 22, I archived that. Chao can you use the one that
> >> Steve mentioned in the mail?
> >>>>
> >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>>>
> >>>>> Seems there are some issues with the shaded client as I was not able
> >>>>> to compile Apache Spark with the RC
> >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>>>> contain any class:
> >>>>>
> >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>>>> META-INF/
> >>>>> META-INF/MANIFEST.MF
> >>>>> META-INF/NOTICE.txt
> >>>>> META-INF/LICENSE.txt
> >>>>> META-INF/maven/
> >>>>> META-INF/maven/org.apache.hadoop/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>>>
> >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>>   * Signature: ok
> >>>>>>   * Checksum : passed
> >>>>>>   * Rat check (1.8.0_191): passed
> >>>>>>    - mvn clean apache-rat:check
> >>>>>>   * Built from source (1.8.0_191): failed
> >>>>>>    - mvn clean install  -DskipTests
> >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> >> -Dmaven.javadoc.skip=true
> >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>>>   * Unit tests pass (1.8.0_191):
> >>>>>>     - HDFS Tests passed (Didn't run more than this).
> >>>>>>
> >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> >>>>>> bigger job with mild chaos. All seems to be working properly
> >> (recoveries,
> >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> >>>>>> good. Poked at the hdfs cli. Seems good.
> >>>>>>
> >>>>>> S
> >>>>>>
> >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>>>
> >>>>>>> The RC is available at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>>>
> >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>>>
> >>>>>>> The maven artifacts are staged at
> >>>>>>>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>>>
> >>>>>>> You can find my public key at:
> >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>>>
> >>>>>>> Change log
> >>>>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>>>
> >>>>>>> Release notes
> >>>>>>>
> >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>>>
> >>>>>>> There's a very small number of changes, primarily critical
> >> code/packaging
> >>>>>>> issues and security fixes.
> >>>>>>>
> >>>>>>>
> >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> >>>>>>>    -  CVEs in our code and dependencies
> >>>>>>>    - Shaded client packaging issues.
> >>>>>>>    - A switch from log4j to reload4j
> >>>>>>>
> >>>>>>>
> >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> >> classes which
> >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> >> they
> >>>>>>> regularly raised alerts on security scans and concen from users.
> >> Switching
> >>>>>>> to the forked project allows us to ship a secure logging framework.
> >> It will
> >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> >> exclude
> >>>>>>> our log4j artifacts, as they need to cut the new dependency
> >> instead/as
> >>>>>>> well.
> >>>>>>>
> >>>>>>> See the release notes for details.
> >>>>>>>
> >>>>>>> This is my first release through the new docker build process, do
> >> please
> >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> >> builds
> >>>>>>> of downstream projects.
> >>>>>>>
> >>>>>>> We know there are some outstanding issues with at least one library
> >> we are
> >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> >> If the
> >>>>>>> docker based release process works smoothly enough we can do a
> >> followup
> >>>>>>> security release in a few weeks.
> >>>>>>>
> >>>>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>>>
> >>>>>>> -Steve
> >>>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I didn't do that as the docker image was doing it itself...I discussed this
with Akira and Ayush & they agreed. so whatever went wrong. it was
something else.

I have been building a list of things I'd like to change there; cutting
that line was one of them. but I need to work out the correct workflow.

trying again, and creating a stub module to verify the client is in staging

On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
wrote:

> It seems to be caused by obsolete instruction in HowToRelease Wiki?
>
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
>
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
>
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
>
> Masatake Iwasaki
>
> On 2022/05/08 6:45, Akira Ajisaka wrote:
> > Hi Chao,
> >
> > How about using
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > instead of https://repository.apache.org/content/repositories/staging/ ?
> >
> > Akira
> >
> > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Hmm, I see the artifacts ideally should have got overwritten by the new
> >> RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> >> have any jars…
> >> That is why it was picking the old jars. I think Steve needs to run mvn
> >> deploy again…
> >>
> >> Sent from my iPhone
> >>
> >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> 
> >>>>
> >>>> Chao can you use the one that Steve mentioned in the mail?
> >>>
> >>> Hmm how do I do that? Typically after closing the RC in nexus the
> >>> release bits will show up in
> >>>
> >>
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> >>> and Spark build will be able to pick them up for testing. However in
> >>> this case I don't see any 3.3.3 jars in the URL.
> >>>
> >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >>>>
> >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> >> date was also april 22, I archived that. Chao can you use the one that
> >> Steve mentioned in the mail?
> >>>>
> >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>>>
> >>>>> Seems there are some issues with the shaded client as I was not able
> >>>>> to compile Apache Spark with the RC
> >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>>>> contain any class:
> >>>>>
> >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>>>> META-INF/
> >>>>> META-INF/MANIFEST.MF
> >>>>> META-INF/NOTICE.txt
> >>>>> META-INF/LICENSE.txt
> >>>>> META-INF/maven/
> >>>>> META-INF/maven/org.apache.hadoop/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>>>
> >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>>   * Signature: ok
> >>>>>>   * Checksum : passed
> >>>>>>   * Rat check (1.8.0_191): passed
> >>>>>>    - mvn clean apache-rat:check
> >>>>>>   * Built from source (1.8.0_191): failed
> >>>>>>    - mvn clean install  -DskipTests
> >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> >> -Dmaven.javadoc.skip=true
> >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>>>   * Unit tests pass (1.8.0_191):
> >>>>>>     - HDFS Tests passed (Didn't run more than this).
> >>>>>>
> >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> >>>>>> bigger job with mild chaos. All seems to be working properly
> >> (recoveries,
> >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> >>>>>> good. Poked at the hdfs cli. Seems good.
> >>>>>>
> >>>>>> S
> >>>>>>
> >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>>>
> >>>>>>> The RC is available at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>>>
> >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>>>
> >>>>>>> The maven artifacts are staged at
> >>>>>>>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>>>
> >>>>>>> You can find my public key at:
> >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>>>
> >>>>>>> Change log
> >>>>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>>>
> >>>>>>> Release notes
> >>>>>>>
> >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>>>
> >>>>>>> There's a very small number of changes, primarily critical
> >> code/packaging
> >>>>>>> issues and security fixes.
> >>>>>>>
> >>>>>>>
> >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> >>>>>>>    -  CVEs in our code and dependencies
> >>>>>>>    - Shaded client packaging issues.
> >>>>>>>    - A switch from log4j to reload4j
> >>>>>>>
> >>>>>>>
> >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> >> classes which
> >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> >> they
> >>>>>>> regularly raised alerts on security scans and concen from users.
> >> Switching
> >>>>>>> to the forked project allows us to ship a secure logging framework.
> >> It will
> >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> >> exclude
> >>>>>>> our log4j artifacts, as they need to cut the new dependency
> >> instead/as
> >>>>>>> well.
> >>>>>>>
> >>>>>>> See the release notes for details.
> >>>>>>>
> >>>>>>> This is my first release through the new docker build process, do
> >> please
> >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> >> builds
> >>>>>>> of downstream projects.
> >>>>>>>
> >>>>>>> We know there are some outstanding issues with at least one library
> >> we are
> >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> >> If the
> >>>>>>> docker based release process works smoothly enough we can do a
> >> followup
> >>>>>>> security release in a few weeks.
> >>>>>>>
> >>>>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>>>
> >>>>>>> -Steve
> >>>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

I didn't do that as the docker image was doing it itself...I discussed this
with Akira and Ayush & they agreed. so whatever went wrong. it was
something else.

I have been building a list of things I'd like to change there; cutting
that line was one of them. but I need to work out the correct workflow.

trying again, and creating a stub module to verify the client is in staging

On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
wrote:

> It seems to be caused by obsolete instruction in HowToRelease Wiki?
>
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
>
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
>
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
>
> Masatake Iwasaki
>
> On 2022/05/08 6:45, Akira Ajisaka wrote:
> > Hi Chao,
> >
> > How about using
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > instead of https://repository.apache.org/content/repositories/staging/ ?
> >
> > Akira
> >
> > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Hmm, I see the artifacts ideally should have got overwritten by the new
> >> RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> >> have any jars…
> >> That is why it was picking the old jars. I think Steve needs to run mvn
> >> deploy again…
> >>
> >> Sent from my iPhone
> >>
> >>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> 
> >>>>
> >>>> Chao can you use the one that Steve mentioned in the mail?
> >>>
> >>> Hmm how do I do that? Typically after closing the RC in nexus the
> >>> release bits will show up in
> >>>
> >>
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> >>> and Spark build will be able to pick them up for testing. However in
> >>> this case I don't see any 3.3.3 jars in the URL.
> >>>
> >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >>>>
> >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> >> date was also april 22, I archived that. Chao can you use the one that
> >> Steve mentioned in the mail?
> >>>>
> >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>>>
> >>>>> Seems there are some issues with the shaded client as I was not able
> >>>>> to compile Apache Spark with the RC
> >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>>>> contain any class:
> >>>>>
> >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>>>> META-INF/
> >>>>> META-INF/MANIFEST.MF
> >>>>> META-INF/NOTICE.txt
> >>>>> META-INF/LICENSE.txt
> >>>>> META-INF/maven/
> >>>>> META-INF/maven/org.apache.hadoop/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>>>
> >>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>>   * Signature: ok
> >>>>>>   * Checksum : passed
> >>>>>>   * Rat check (1.8.0_191): passed
> >>>>>>    - mvn clean apache-rat:check
> >>>>>>   * Built from source (1.8.0_191): failed
> >>>>>>    - mvn clean install  -DskipTests
> >>>>>>    - mvn -fae --no-transfer-progress -DskipTests
> >> -Dmaven.javadoc.skip=true
> >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>>>   * Unit tests pass (1.8.0_191):
> >>>>>>     - HDFS Tests passed (Didn't run more than this).
> >>>>>>
> >>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> >>>>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> >>>>>> bigger job with mild chaos. All seems to be working properly
> >> (recoveries,
> >>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> >>>>>> good. Poked at the hdfs cli. Seems good.
> >>>>>>
> >>>>>> S
> >>>>>>
> >>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>>>
> >>>>>>> The RC is available at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>>>
> >>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>>>
> >>>>>>> The maven artifacts are staged at
> >>>>>>>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>>>
> >>>>>>> You can find my public key at:
> >>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>>>
> >>>>>>> Change log
> >>>>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>>>
> >>>>>>> Release notes
> >>>>>>>
> >> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>>>
> >>>>>>> There's a very small number of changes, primarily critical
> >> code/packaging
> >>>>>>> issues and security fixes.
> >>>>>>>
> >>>>>>>
> >>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
> >>>>>>>    -  CVEs in our code and dependencies
> >>>>>>>    - Shaded client packaging issues.
> >>>>>>>    - A switch from log4j to reload4j
> >>>>>>>
> >>>>>>>
> >>>>>>> reload4j is an active fork of the log4j 1.17 library with the
> >> classes which
> >>>>>>> contain CVEs removed. Even though hadoop never used those classes,
> >> they
> >>>>>>> regularly raised alerts on security scans and concen from users.
> >> Switching
> >>>>>>> to the forked project allows us to ship a secure logging framework.
> >> It will
> >>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
> >> exclude
> >>>>>>> our log4j artifacts, as they need to cut the new dependency
> >> instead/as
> >>>>>>> well.
> >>>>>>>
> >>>>>>> See the release notes for details.
> >>>>>>>
> >>>>>>> This is my first release through the new docker build process, do
> >> please
> >>>>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> >> builds
> >>>>>>> of downstream projects.
> >>>>>>>
> >>>>>>> We know there are some outstanding issues with at least one library
> >> we are
> >>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
> >> If the
> >>>>>>> docker based release process works smoothly enough we can do a
> >> followup
> >>>>>>> security release in a few weeks.
> >>>>>>>
> >>>>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>>>
> >>>>>>> -Steve
> >>>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease#HowToRelease-Creatingthereleasecandidate(X.Y.Z-RC%3CN%3E)

On 2022/05/09 23:18, Masatake Iwasaki wrote:
> It seems to be caused by obsolete instruction in HowToRelease Wiki?
> 
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
> 
> Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> 
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
> 
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
> 
> Masatake Iwasaki
> 
> On 2022/05/08 6:45, Akira Ajisaka wrote:
>> Hi Chao,
>>
>> How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> instead of https://repository.apache.org/content/repositories/staging/ ?
>>
>> Akira
>>
>> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>>> Hmm, I see the artifacts ideally should have got overwritten by the new
>>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>>> have any jars…
>>> That is why it was picking the old jars. I think Steve needs to run mvn
>>> deploy again…
>>>
>>> Sent from my iPhone
>>>
>>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>>
>>>> 
>>>>>
>>>>> Chao can you use the one that Steve mentioned in the mail?
>>>>
>>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>>> release bits will show up in
>>>>
>>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>>> and Spark build will be able to pick them up for testing. However in
>>>> this case I don't see any 3.3.3 jars in the URL.
>>>>
>>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>>
>>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>>> date was also april 22, I archived that. Chao can you use the one that
>>> Steve mentioned in the mail?
>>>>>
>>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>>
>>>>>> Seems there are some issues with the shaded client as I was not able
>>>>>> to compile Apache Spark with the RC
>>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>>> contain any class:
>>>>>>
>>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>>> META-INF/
>>>>>> META-INF/MANIFEST.MF
>>>>>> META-INF/NOTICE.txt
>>>>>> META-INF/LICENSE.txt
>>>>>> META-INF/maven/
>>>>>> META-INF/maven/org.apache.hadoop/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>>
>>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>>   * Signature: ok
>>>>>>>   * Checksum : passed
>>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>>    - mvn clean apache-rat:check
>>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>>    - mvn clean install  -DskipTests
>>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>>> -Dmaven.javadoc.skip=true
>>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>>
>>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>>> bigger job with mild chaos. All seems to be working properly
>>> (recoveries,
>>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>>
>>>>>>> S
>>>>>>>
>>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>>> <st...@cloudera.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>>
>>>>>>>> The RC is available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>>
>>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>>
>>>>>>>> The maven artifacts are staged at
>>>>>>>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>>
>>>>>>>> You can find my public key at:
>>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>>
>>>>>>>> Change log
>>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>>
>>>>>>>> Release notes
>>>>>>>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>>
>>>>>>>> There's a very small number of changes, primarily critical
>>> code/packaging
>>>>>>>> issues and security fixes.
>>>>>>>>
>>>>>>>>
>>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>>    - Shaded client packaging issues.
>>>>>>>>    - A switch from log4j to reload4j
>>>>>>>>
>>>>>>>>
>>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>>> classes which
>>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>>> they
>>>>>>>> regularly raised alerts on security scans and concen from users.
>>> Switching
>>>>>>>> to the forked project allows us to ship a secure logging framework.
>>> It will
>>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>>> exclude
>>>>>>>> our log4j artifacts, as they need to cut the new dependency
>>> instead/as
>>>>>>>> well.
>>>>>>>>
>>>>>>>> See the release notes for details.
>>>>>>>>
>>>>>>>> This is my first release through the new docker build process, do
>>> please
>>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>>> builds
>>>>>>>> of downstream projects.
>>>>>>>>
>>>>>>>> We know there are some outstanding issues with at least one library
>>> we are
>>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>>> If the
>>>>>>>> docker based release process works smoothly enough we can do a
>>> followup
>>>>>>>> security release in a few weeks.
>>>>>>>>
>>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>>
>>>>>>>> -Steve
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

It seems to be caused by obsolete instruction in HowToRelease Wiki?

After HADOOP-15058, `mvn deploy` is kicked by
`dev-support/bin/create-release --asfrelease`.
https://issues.apache.org/jira/browse/HADOOP-15058

Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

2 sets of artifact are deployed after creating RC based on the instruction.
The latest one contains empty shaded jars.

hadoop-client-api and hadoop-client-runtime of already released 3.2.3
looks having same issue...

Masatake Iwasaki

On 2022/05/08 6:45, Akira Ajisaka wrote:
> Hi Chao,
> 
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
> 
> Akira
> 
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> 
>> Hmm, I see the artifacts ideally should have got overwritten by the new
>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>> have any jars…
>> That is why it was picking the old jars. I think Steve needs to run mvn
>> deploy again…
>>
>> Sent from my iPhone
>>
>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>
>>> 
>>>>
>>>> Chao can you use the one that Steve mentioned in the mail?
>>>
>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>> release bits will show up in
>>>
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>> and Spark build will be able to pick them up for testing. However in
>>> this case I don't see any 3.3.3 jars in the URL.
>>>
>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>
>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>> date was also april 22, I archived that. Chao can you use the one that
>> Steve mentioned in the mail?
>>>>
>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>
>>>>> Seems there are some issues with the shaded client as I was not able
>>>>> to compile Apache Spark with the RC
>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>> contain any class:
>>>>>
>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> META-INF/NOTICE.txt
>>>>> META-INF/LICENSE.txt
>>>>> META-INF/maven/
>>>>> META-INF/maven/org.apache.hadoop/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>
>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>>   * Signature: ok
>>>>>>   * Checksum : passed
>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>    - mvn clean apache-rat:check
>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>    - mvn clean install  -DskipTests
>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> -Dmaven.javadoc.skip=true
>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>
>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>> bigger job with mild chaos. All seems to be working properly
>> (recoveries,
>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>
>>>>>> S
>>>>>>
>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>
>>>>>>> The RC is available at:
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>
>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>
>>>>>>> The maven artifacts are staged at
>>>>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>
>>>>>>> You can find my public key at:
>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>
>>>>>>> Change log
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>
>>>>>>> Release notes
>>>>>>>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>
>>>>>>> There's a very small number of changes, primarily critical
>> code/packaging
>>>>>>> issues and security fixes.
>>>>>>>
>>>>>>>
>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>    - Shaded client packaging issues.
>>>>>>>    - A switch from log4j to reload4j
>>>>>>>
>>>>>>>
>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> classes which
>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>> they
>>>>>>> regularly raised alerts on security scans and concen from users.
>> Switching
>>>>>>> to the forked project allows us to ship a secure logging framework.
>> It will
>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>> exclude
>>>>>>> our log4j artifacts, as they need to cut the new dependency
>> instead/as
>>>>>>> well.
>>>>>>>
>>>>>>> See the release notes for details.
>>>>>>>
>>>>>>> This is my first release through the new docker build process, do
>> please
>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>> builds
>>>>>>> of downstream projects.
>>>>>>>
>>>>>>> We know there are some outstanding issues with at least one library
>> we are
>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>> If the
>>>>>>> docker based release process works smoothly enough we can do a
>> followup
>>>>>>> security release in a few weeks.
>>>>>>>
>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>
>>>>>>> -Steve
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

It seems to be caused by obsolete instruction in HowToRelease Wiki?

After HADOOP-15058, `mvn deploy` is kicked by
`dev-support/bin/create-release --asfrelease`.
https://issues.apache.org/jira/browse/HADOOP-15058

Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

2 sets of artifact are deployed after creating RC based on the instruction.
The latest one contains empty shaded jars.

hadoop-client-api and hadoop-client-runtime of already released 3.2.3
looks having same issue...

Masatake Iwasaki

On 2022/05/08 6:45, Akira Ajisaka wrote:
> Hi Chao,
> 
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
> 
> Akira
> 
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> 
>> Hmm, I see the artifacts ideally should have got overwritten by the new
>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>> have any jars…
>> That is why it was picking the old jars. I think Steve needs to run mvn
>> deploy again…
>>
>> Sent from my iPhone
>>
>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>
>>> 
>>>>
>>>> Chao can you use the one that Steve mentioned in the mail?
>>>
>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>> release bits will show up in
>>>
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>> and Spark build will be able to pick them up for testing. However in
>>> this case I don't see any 3.3.3 jars in the URL.
>>>
>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>
>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>> date was also april 22, I archived that. Chao can you use the one that
>> Steve mentioned in the mail?
>>>>
>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>
>>>>> Seems there are some issues with the shaded client as I was not able
>>>>> to compile Apache Spark with the RC
>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>> contain any class:
>>>>>
>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> META-INF/NOTICE.txt
>>>>> META-INF/LICENSE.txt
>>>>> META-INF/maven/
>>>>> META-INF/maven/org.apache.hadoop/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>
>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>>   * Signature: ok
>>>>>>   * Checksum : passed
>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>    - mvn clean apache-rat:check
>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>    - mvn clean install  -DskipTests
>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> -Dmaven.javadoc.skip=true
>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>
>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>> bigger job with mild chaos. All seems to be working properly
>> (recoveries,
>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>
>>>>>> S
>>>>>>
>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>
>>>>>>> The RC is available at:
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>
>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>
>>>>>>> The maven artifacts are staged at
>>>>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>
>>>>>>> You can find my public key at:
>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>
>>>>>>> Change log
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>
>>>>>>> Release notes
>>>>>>>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>
>>>>>>> There's a very small number of changes, primarily critical
>> code/packaging
>>>>>>> issues and security fixes.
>>>>>>>
>>>>>>>
>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>    - Shaded client packaging issues.
>>>>>>>    - A switch from log4j to reload4j
>>>>>>>
>>>>>>>
>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> classes which
>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>> they
>>>>>>> regularly raised alerts on security scans and concen from users.
>> Switching
>>>>>>> to the forked project allows us to ship a secure logging framework.
>> It will
>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>> exclude
>>>>>>> our log4j artifacts, as they need to cut the new dependency
>> instead/as
>>>>>>> well.
>>>>>>>
>>>>>>> See the release notes for details.
>>>>>>>
>>>>>>> This is my first release through the new docker build process, do
>> please
>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>> builds
>>>>>>> of downstream projects.
>>>>>>>
>>>>>>> We know there are some outstanding issues with at least one library
>> we are
>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>> If the
>>>>>>> docker based release process works smoothly enough we can do a
>> followup
>>>>>>> security release in a few weeks.
>>>>>>>
>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>
>>>>>>> -Steve
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

It seems to be caused by obsolete instruction in HowToRelease Wiki?

After HADOOP-15058, `mvn deploy` is kicked by
`dev-support/bin/create-release --asfrelease`.
https://issues.apache.org/jira/browse/HADOOP-15058

Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

2 sets of artifact are deployed after creating RC based on the instruction.
The latest one contains empty shaded jars.

hadoop-client-api and hadoop-client-runtime of already released 3.2.3
looks having same issue...

Masatake Iwasaki

On 2022/05/08 6:45, Akira Ajisaka wrote:
> Hi Chao,
> 
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
> 
> Akira
> 
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> 
>> Hmm, I see the artifacts ideally should have got overwritten by the new
>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>> have any jars…
>> That is why it was picking the old jars. I think Steve needs to run mvn
>> deploy again…
>>
>> Sent from my iPhone
>>
>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>
>>> 
>>>>
>>>> Chao can you use the one that Steve mentioned in the mail?
>>>
>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>> release bits will show up in
>>>
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>> and Spark build will be able to pick them up for testing. However in
>>> this case I don't see any 3.3.3 jars in the URL.
>>>
>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>
>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>> date was also april 22, I archived that. Chao can you use the one that
>> Steve mentioned in the mail?
>>>>
>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>
>>>>> Seems there are some issues with the shaded client as I was not able
>>>>> to compile Apache Spark with the RC
>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>> contain any class:
>>>>>
>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> META-INF/NOTICE.txt
>>>>> META-INF/LICENSE.txt
>>>>> META-INF/maven/
>>>>> META-INF/maven/org.apache.hadoop/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>
>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>>   * Signature: ok
>>>>>>   * Checksum : passed
>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>    - mvn clean apache-rat:check
>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>    - mvn clean install  -DskipTests
>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> -Dmaven.javadoc.skip=true
>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>
>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>> bigger job with mild chaos. All seems to be working properly
>> (recoveries,
>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>
>>>>>> S
>>>>>>
>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>
>>>>>>> The RC is available at:
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>
>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>
>>>>>>> The maven artifacts are staged at
>>>>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>
>>>>>>> You can find my public key at:
>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>
>>>>>>> Change log
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>
>>>>>>> Release notes
>>>>>>>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>
>>>>>>> There's a very small number of changes, primarily critical
>> code/packaging
>>>>>>> issues and security fixes.
>>>>>>>
>>>>>>>
>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>    - Shaded client packaging issues.
>>>>>>>    - A switch from log4j to reload4j
>>>>>>>
>>>>>>>
>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> classes which
>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>> they
>>>>>>> regularly raised alerts on security scans and concen from users.
>> Switching
>>>>>>> to the forked project allows us to ship a secure logging framework.
>> It will
>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>> exclude
>>>>>>> our log4j artifacts, as they need to cut the new dependency
>> instead/as
>>>>>>> well.
>>>>>>>
>>>>>>> See the release notes for details.
>>>>>>>
>>>>>>> This is my first release through the new docker build process, do
>> please
>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>> builds
>>>>>>> of downstream projects.
>>>>>>>
>>>>>>> We know there are some outstanding issues with at least one library
>> we are
>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>> If the
>>>>>>> docker based release process works smoothly enough we can do a
>> followup
>>>>>>> security release in a few weeks.
>>>>>>>
>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>
>>>>>>> -Steve
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

It seems to be caused by obsolete instruction in HowToRelease Wiki?

After HADOOP-15058, `mvn deploy` is kicked by
`dev-support/bin/create-release --asfrelease`.
https://issues.apache.org/jira/browse/HADOOP-15058

Step #10 in "Creating the release candidate (X.Y.Z-RC<N>)" section
of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.

2 sets of artifact are deployed after creating RC based on the instruction.
The latest one contains empty shaded jars.

hadoop-client-api and hadoop-client-runtime of already released 3.2.3
looks having same issue...

Masatake Iwasaki

On 2022/05/08 6:45, Akira Ajisaka wrote:
> Hi Chao,
> 
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
> 
> Akira
> 
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> 
>> Hmm, I see the artifacts ideally should have got overwritten by the new
>> RC, but they didn’t. The reason seems like the staging path shared doesn’t
>> have any jars…
>> That is why it was picking the old jars. I think Steve needs to run mvn
>> deploy again…
>>
>> Sent from my iPhone
>>
>>> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
>>>
>>> 
>>>>
>>>> Chao can you use the one that Steve mentioned in the mail?
>>>
>>> Hmm how do I do that? Typically after closing the RC in nexus the
>>> release bits will show up in
>>>
>> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
>>> and Spark build will be able to pick them up for testing. However in
>>> this case I don't see any 3.3.3 jars in the URL.
>>>
>>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>>>>
>>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
>> date was also april 22, I archived that. Chao can you use the one that
>> Steve mentioned in the mail?
>>>>
>>>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>>>>
>>>>> Seems there are some issues with the shaded client as I was not able
>>>>> to compile Apache Spark with the RC
>>>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>>>> contain any class:
>>>>>
>>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> META-INF/NOTICE.txt
>>>>> META-INF/LICENSE.txt
>>>>> META-INF/maven/
>>>>> META-INF/maven/org.apache.hadoop/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>>>>
>>>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>>   * Signature: ok
>>>>>>   * Checksum : passed
>>>>>>   * Rat check (1.8.0_191): passed
>>>>>>    - mvn clean apache-rat:check
>>>>>>   * Built from source (1.8.0_191): failed
>>>>>>    - mvn clean install  -DskipTests
>>>>>>    - mvn -fae --no-transfer-progress -DskipTests
>> -Dmaven.javadoc.skip=true
>>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>>>   * Unit tests pass (1.8.0_191):
>>>>>>     - HDFS Tests passed (Didn't run more than this).
>>>>>>
>>>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>>>> bigger job with mild chaos. All seems to be working properly
>> (recoveries,
>>>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>>>> good. Poked at the hdfs cli. Seems good.
>>>>>>
>>>>>> S
>>>>>>
>>>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>>>>
>>>>>>> The RC is available at:
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>>>>
>>>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>>>>
>>>>>>> The maven artifacts are staged at
>>>>>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>>>>
>>>>>>> You can find my public key at:
>>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>>>>
>>>>>>> Change log
>>>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>>>>
>>>>>>> Release notes
>>>>>>>
>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>>>>
>>>>>>> There's a very small number of changes, primarily critical
>> code/packaging
>>>>>>> issues and security fixes.
>>>>>>>
>>>>>>>
>>>>>>>    - The critical fixes which shipped in the 3.2.3 release.
>>>>>>>    -  CVEs in our code and dependencies
>>>>>>>    - Shaded client packaging issues.
>>>>>>>    - A switch from log4j to reload4j
>>>>>>>
>>>>>>>
>>>>>>> reload4j is an active fork of the log4j 1.17 library with the
>> classes which
>>>>>>> contain CVEs removed. Even though hadoop never used those classes,
>> they
>>>>>>> regularly raised alerts on security scans and concen from users.
>> Switching
>>>>>>> to the forked project allows us to ship a secure logging framework.
>> It will
>>>>>>> complicate the builds of downstream maven/ivy/gradle projects which
>> exclude
>>>>>>> our log4j artifacts, as they need to cut the new dependency
>> instead/as
>>>>>>> well.
>>>>>>>
>>>>>>> See the release notes for details.
>>>>>>>
>>>>>>> This is my first release through the new docker build process, do
>> please
>>>>>>> validate artifact signing &c to make sure it is good. I'll be trying
>> builds
>>>>>>> of downstream projects.
>>>>>>>
>>>>>>> We know there are some outstanding issues with at least one library
>> we are
>>>>>>> shipping (okhttp), but I don't want to hold this release up for it.
>> If the
>>>>>>> docker based release process works smoothly enough we can do a
>> followup
>>>>>>> security release in a few weeks.
>>>>>>>
>>>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>>>>
>>>>>>> -Steve
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Steve Loughran <st...@cloudera.com.INVALID>.

had a report of that staging playing up too from Gautham

I'm going to have to treat this as a failure from the maven deploy process,
so I'll have to cancel this RC and try and work out WTF went wrong.

I'll also put together some minimal apps to fetch and compile against
everything. i did do some myself (storediag, hboss), and did try to make
sure I had cleaned up my local repo first, but maybe it was tainted.

oh well

-------

However, I get this error saying that it can’t find the maven plugins and
jars for Hadoop 3.3.3 –

[INFO] ------------------< org.apache.hadoop:hadoop-common
>-------------------

[INFO] Building Apache Hadoop Common 3.3.3
[5/9]

[INFO] --------------------------------[ jar
]---------------------------------

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.pom

[WARNING] The POM for org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 is
missing, no dependency information available

Downloading from central:
https://repository.apache.org/content/repositories/orgapachehadoop-1348/org/apache/hadoop/hadoop-maven-plugins/3.3.3/hadoop-maven-plugins-3.3.3.jar

[INFO]
------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop Common Project 3.3.3:

[INFO]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [
7.349 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [
1.810 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [
7.922 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [
0.375 s]

[INFO] Apache Hadoop Common ............................... FAILURE [
2.095 s]

[INFO] Apache Hadoop NFS .................................. SKIPPED

[INFO] Apache Hadoop KMS .................................. SKIPPED

[INFO] Apache Hadoop Registry ............................. SKIPPED

[INFO] Apache Hadoop Common Project ....................... SKIPPED

[INFO]
------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO]
------------------------------------------------------------------------

[INFO] Total time:  24.946 s

[INFO] Finished at: 2022-05-07T11:29:18+05:30

[INFO]
------------------------------------------------------------------------

[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.3.3 or one of its
dependencies could not be resolved: Could not find artifact
org.apache.hadoop:hadoop-maven-plugins:jar:3.3.3 in central (
https://repository.apache.org/content/repositories/orgapachehadoop-1348/)
-> [Help 1]

[ERROR]



Could you please let me know how I can consume the Hadoop 3 jars in maven?



Thanks,

--Gautham

On Sat, 7 May 2022 at 22:45, Akira Ajisaka <aa...@apache.org> wrote:

> Hi Chao,
>
> How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> instead of https://repository.apache.org/content/repositories/staging/ ?
>
> Akira
>
> On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm, I see the artifacts ideally should have got overwritten by the new
> > RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> > have any jars…
> > That is why it was picking the old jars. I think Steve needs to run mvn
> > deploy again…
> >
> > Sent from my iPhone
> >
> > > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> > >
> > > 
> > >>
> > >> Chao can you use the one that Steve mentioned in the mail?
> > >
> > > Hmm how do I do that? Typically after closing the RC in nexus the
> > > release bits will show up in
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > and Spark build will be able to pick them up for testing. However in
> > > this case I don't see any 3.3.3 jars in the URL.
> > >
> > >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> > >>
> > >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> > date was also april 22, I archived that. Chao can you use the one that
> > Steve mentioned in the mail?
> > >>
> > >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> > >>>
> > >>> Seems there are some issues with the shaded client as I was not able
> > >>> to compile Apache Spark with the RC
> > >>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> > >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> > >>> contain any class:
> > >>>
> > >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> > >>> META-INF/
> > >>> META-INF/MANIFEST.MF
> > >>> META-INF/NOTICE.txt
> > >>> META-INF/LICENSE.txt
> > >>> META-INF/maven/
> > >>> META-INF/maven/org.apache.hadoop/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>
> > >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>>  * Signature: ok
> > >>>>  * Checksum : passed
> > >>>>  * Rat check (1.8.0_191): passed
> > >>>>   - mvn clean apache-rat:check
> > >>>>  * Built from source (1.8.0_191): failed
> > >>>>   - mvn clean install  -DskipTests
> > >>>>   - mvn -fae --no-transfer-progress -DskipTests
> > -Dmaven.javadoc.skip=true
> > >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>  * Unit tests pass (1.8.0_191):
> > >>>>    - HDFS Tests passed (Didn't run more than this).
> > >>>>
> > >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> > >>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> > >>>> bigger job with mild chaos. All seems to be working properly
> > (recoveries,
> > >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> > >>>> good. Poked at the hdfs cli. Seems good.
> > >>>>
> > >>>> S
> > >>>>
> > >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > >>>> wrote:
> > >>>>
> > >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>
> > >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>
> > >>>>> The maven artifacts are staged at
> > >>>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>
> > >>>>> Change log
> > >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>
> > >>>>> Release notes
> > >>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>
> > >>>>> There's a very small number of changes, primarily critical
> > code/packaging
> > >>>>> issues and security fixes.
> > >>>>>
> > >>>>>
> > >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>   -  CVEs in our code and dependencies
> > >>>>>   - Shaded client packaging issues.
> > >>>>>   - A switch from log4j to reload4j
> > >>>>>
> > >>>>>
> > >>>>> reload4j is an active fork of the log4j 1.17 library with the
> > classes which
> > >>>>> contain CVEs removed. Even though hadoop never used those classes,
> > they
> > >>>>> regularly raised alerts on security scans and concen from users.
> > Switching
> > >>>>> to the forked project allows us to ship a secure logging framework.
> > It will
> > >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > exclude
> > >>>>> our log4j artifacts, as they need to cut the new dependency
> > instead/as
> > >>>>> well.
> > >>>>>
> > >>>>> See the release notes for details.
> > >>>>>
> > >>>>> This is my first release through the new docker build process, do
> > please
> > >>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> > builds
> > >>>>> of downstream projects.
> > >>>>>
> > >>>>> We know there are some outstanding issues with at least one library
> > we are
> > >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > If the
> > >>>>> docker based release process works smoothly enough we can do a
> > followup
> > >>>>> security release in a few weeks.
> > >>>>>
> > >>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>
> > >>>>> -Steve
> > >>>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Akira Ajisaka <aa...@apache.org>.

Hi Chao,

How about using
https://repository.apache.org/content/repositories/orgapachehadoop-1348/
instead of https://repository.apache.org/content/repositories/staging/ ?

Akira

On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:

> Hmm, I see the artifacts ideally should have got overwritten by the new
> RC, but they didn’t. The reason seems like the staging path shared doesn’t
> have any jars…
> That is why it was picking the old jars. I think Steve needs to run mvn
> deploy again…
>
> Sent from my iPhone
>
> > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >
> > 
> >>
> >> Chao can you use the one that Steve mentioned in the mail?
> >
> > Hmm how do I do that? Typically after closing the RC in nexus the
> > release bits will show up in
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > and Spark build will be able to pick them up for testing. However in
> > this case I don't see any 3.3.3 jars in the URL.
> >
> >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> date was also april 22, I archived that. Chao can you use the one that
> Steve mentioned in the mail?
> >>
> >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> Seems there are some issues with the shaded client as I was not able
> >>> to compile Apache Spark with the RC
> >>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>> contain any class:
> >>>
> >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>> META-INF/
> >>> META-INF/MANIFEST.MF
> >>> META-INF/NOTICE.txt
> >>> META-INF/LICENSE.txt
> >>> META-INF/maven/
> >>> META-INF/maven/org.apache.hadoop/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>
> >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>>  * Signature: ok
> >>>>  * Checksum : passed
> >>>>  * Rat check (1.8.0_191): passed
> >>>>   - mvn clean apache-rat:check
> >>>>  * Built from source (1.8.0_191): failed
> >>>>   - mvn clean install  -DskipTests
> >>>>   - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>  * Unit tests pass (1.8.0_191):
> >>>>    - HDFS Tests passed (Didn't run more than this).
> >>>>
> >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> >>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> >>>> bigger job with mild chaos. All seems to be working properly
> (recoveries,
> >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> >>>> good. Poked at the hdfs cli. Seems good.
> >>>>
> >>>> S
> >>>>
> >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> >>>> wrote:
> >>>>
> >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>
> >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>
> >>>>> The maven artifacts are staged at
> >>>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>
> >>>>> Change log
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>
> >>>>> Release notes
> >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>
> >>>>> There's a very small number of changes, primarily critical
> code/packaging
> >>>>> issues and security fixes.
> >>>>>
> >>>>>
> >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> >>>>>   -  CVEs in our code and dependencies
> >>>>>   - Shaded client packaging issues.
> >>>>>   - A switch from log4j to reload4j
> >>>>>
> >>>>>
> >>>>> reload4j is an active fork of the log4j 1.17 library with the
> classes which
> >>>>> contain CVEs removed. Even though hadoop never used those classes,
> they
> >>>>> regularly raised alerts on security scans and concen from users.
> Switching
> >>>>> to the forked project allows us to ship a secure logging framework.
> It will
> >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> >>>>> our log4j artifacts, as they need to cut the new dependency
> instead/as
> >>>>> well.
> >>>>>
> >>>>> See the release notes for details.
> >>>>>
> >>>>> This is my first release through the new docker build process, do
> please
> >>>>> validate artifact signing &c to make sure it is good. I'll be trying
> builds
> >>>>> of downstream projects.
> >>>>>
> >>>>> We know there are some outstanding issues with at least one library
> we are
> >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> If the
> >>>>> docker based release process works smoothly enough we can do a
> followup
> >>>>> security release in a few weeks.
> >>>>>
> >>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>
> >>>>> -Steve
> >>>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Akira Ajisaka <aa...@apache.org>.

Hi Chao,

How about using
https://repository.apache.org/content/repositories/orgapachehadoop-1348/
instead of https://repository.apache.org/content/repositories/staging/ ?

Akira

On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:

> Hmm, I see the artifacts ideally should have got overwritten by the new
> RC, but they didn’t. The reason seems like the staging path shared doesn’t
> have any jars…
> That is why it was picking the old jars. I think Steve needs to run mvn
> deploy again…
>
> Sent from my iPhone
>
> > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >
> > 
> >>
> >> Chao can you use the one that Steve mentioned in the mail?
> >
> > Hmm how do I do that? Typically after closing the RC in nexus the
> > release bits will show up in
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > and Spark build will be able to pick them up for testing. However in
> > this case I don't see any 3.3.3 jars in the URL.
> >
> >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> date was also april 22, I archived that. Chao can you use the one that
> Steve mentioned in the mail?
> >>
> >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> Seems there are some issues with the shaded client as I was not able
> >>> to compile Apache Spark with the RC
> >>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>> contain any class:
> >>>
> >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>> META-INF/
> >>> META-INF/MANIFEST.MF
> >>> META-INF/NOTICE.txt
> >>> META-INF/LICENSE.txt
> >>> META-INF/maven/
> >>> META-INF/maven/org.apache.hadoop/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>
> >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>>  * Signature: ok
> >>>>  * Checksum : passed
> >>>>  * Rat check (1.8.0_191): passed
> >>>>   - mvn clean apache-rat:check
> >>>>  * Built from source (1.8.0_191): failed
> >>>>   - mvn clean install  -DskipTests
> >>>>   - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>  * Unit tests pass (1.8.0_191):
> >>>>    - HDFS Tests passed (Didn't run more than this).
> >>>>
> >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> >>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> >>>> bigger job with mild chaos. All seems to be working properly
> (recoveries,
> >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> >>>> good. Poked at the hdfs cli. Seems good.
> >>>>
> >>>> S
> >>>>
> >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> >>>> wrote:
> >>>>
> >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>
> >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>
> >>>>> The maven artifacts are staged at
> >>>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>
> >>>>> Change log
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>
> >>>>> Release notes
> >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>
> >>>>> There's a very small number of changes, primarily critical
> code/packaging
> >>>>> issues and security fixes.
> >>>>>
> >>>>>
> >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> >>>>>   -  CVEs in our code and dependencies
> >>>>>   - Shaded client packaging issues.
> >>>>>   - A switch from log4j to reload4j
> >>>>>
> >>>>>
> >>>>> reload4j is an active fork of the log4j 1.17 library with the
> classes which
> >>>>> contain CVEs removed. Even though hadoop never used those classes,
> they
> >>>>> regularly raised alerts on security scans and concen from users.
> Switching
> >>>>> to the forked project allows us to ship a secure logging framework.
> It will
> >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> >>>>> our log4j artifacts, as they need to cut the new dependency
> instead/as
> >>>>> well.
> >>>>>
> >>>>> See the release notes for details.
> >>>>>
> >>>>> This is my first release through the new docker build process, do
> please
> >>>>> validate artifact signing &c to make sure it is good. I'll be trying
> builds
> >>>>> of downstream projects.
> >>>>>
> >>>>> We know there are some outstanding issues with at least one library
> we are
> >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> If the
> >>>>> docker based release process works smoothly enough we can do a
> followup
> >>>>> security release in a few weeks.
> >>>>>
> >>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>
> >>>>> -Steve
> >>>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Akira Ajisaka <aa...@apache.org>.

Hi Chao,

How about using
https://repository.apache.org/content/repositories/orgapachehadoop-1348/
instead of https://repository.apache.org/content/repositories/staging/ ?

Akira

On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:

> Hmm, I see the artifacts ideally should have got overwritten by the new
> RC, but they didn’t. The reason seems like the staging path shared doesn’t
> have any jars…
> That is why it was picking the old jars. I think Steve needs to run mvn
> deploy again…
>
> Sent from my iPhone
>
> > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >
> > 
> >>
> >> Chao can you use the one that Steve mentioned in the mail?
> >
> > Hmm how do I do that? Typically after closing the RC in nexus the
> > release bits will show up in
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > and Spark build will be able to pick them up for testing. However in
> > this case I don't see any 3.3.3 jars in the URL.
> >
> >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> date was also april 22, I archived that. Chao can you use the one that
> Steve mentioned in the mail?
> >>
> >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> Seems there are some issues with the shaded client as I was not able
> >>> to compile Apache Spark with the RC
> >>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>> contain any class:
> >>>
> >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>> META-INF/
> >>> META-INF/MANIFEST.MF
> >>> META-INF/NOTICE.txt
> >>> META-INF/LICENSE.txt
> >>> META-INF/maven/
> >>> META-INF/maven/org.apache.hadoop/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>
> >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>>  * Signature: ok
> >>>>  * Checksum : passed
> >>>>  * Rat check (1.8.0_191): passed
> >>>>   - mvn clean apache-rat:check
> >>>>  * Built from source (1.8.0_191): failed
> >>>>   - mvn clean install  -DskipTests
> >>>>   - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>  * Unit tests pass (1.8.0_191):
> >>>>    - HDFS Tests passed (Didn't run more than this).
> >>>>
> >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> >>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> >>>> bigger job with mild chaos. All seems to be working properly
> (recoveries,
> >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> >>>> good. Poked at the hdfs cli. Seems good.
> >>>>
> >>>> S
> >>>>
> >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> >>>> wrote:
> >>>>
> >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>
> >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>
> >>>>> The maven artifacts are staged at
> >>>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>
> >>>>> Change log
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>
> >>>>> Release notes
> >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>
> >>>>> There's a very small number of changes, primarily critical
> code/packaging
> >>>>> issues and security fixes.
> >>>>>
> >>>>>
> >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> >>>>>   -  CVEs in our code and dependencies
> >>>>>   - Shaded client packaging issues.
> >>>>>   - A switch from log4j to reload4j
> >>>>>
> >>>>>
> >>>>> reload4j is an active fork of the log4j 1.17 library with the
> classes which
> >>>>> contain CVEs removed. Even though hadoop never used those classes,
> they
> >>>>> regularly raised alerts on security scans and concen from users.
> Switching
> >>>>> to the forked project allows us to ship a secure logging framework.
> It will
> >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> >>>>> our log4j artifacts, as they need to cut the new dependency
> instead/as
> >>>>> well.
> >>>>>
> >>>>> See the release notes for details.
> >>>>>
> >>>>> This is my first release through the new docker build process, do
> please
> >>>>> validate artifact signing &c to make sure it is good. I'll be trying
> builds
> >>>>> of downstream projects.
> >>>>>
> >>>>> We know there are some outstanding issues with at least one library
> we are
> >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> If the
> >>>>> docker based release process works smoothly enough we can do a
> followup
> >>>>> security release in a few weeks.
> >>>>>
> >>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>
> >>>>> -Steve
> >>>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Akira Ajisaka <aa...@apache.org>.

Hi Chao,

How about using
https://repository.apache.org/content/repositories/orgapachehadoop-1348/
instead of https://repository.apache.org/content/repositories/staging/ ?

Akira

On Sat, May 7, 2022 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:

> Hmm, I see the artifacts ideally should have got overwritten by the new
> RC, but they didn’t. The reason seems like the staging path shared doesn’t
> have any jars…
> That is why it was picking the old jars. I think Steve needs to run mvn
> deploy again…
>
> Sent from my iPhone
>
> > On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> >
> > 
> >>
> >> Chao can you use the one that Steve mentioned in the mail?
> >
> > Hmm how do I do that? Typically after closing the RC in nexus the
> > release bits will show up in
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > and Spark build will be able to pick them up for testing. However in
> > this case I don't see any 3.3.3 jars in the URL.
> >
> >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> date was also april 22, I archived that. Chao can you use the one that
> Steve mentioned in the mail?
> >>
> >>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
> >>>
> >>> Seems there are some issues with the shaded client as I was not able
> >>> to compile Apache Spark with the RC
> >>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>> contain any class:
> >>>
> >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>> META-INF/
> >>> META-INF/MANIFEST.MF
> >>> META-INF/NOTICE.txt
> >>> META-INF/LICENSE.txt
> >>> META-INF/maven/
> >>> META-INF/maven/org.apache.hadoop/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>
> >>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>>  * Signature: ok
> >>>>  * Checksum : passed
> >>>>  * Rat check (1.8.0_191): passed
> >>>>   - mvn clean apache-rat:check
> >>>>  * Built from source (1.8.0_191): failed
> >>>>   - mvn clean install  -DskipTests
> >>>>   - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>  * Unit tests pass (1.8.0_191):
> >>>>    - HDFS Tests passed (Didn't run more than this).
> >>>>
> >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> >>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> >>>> bigger job with mild chaos. All seems to be working properly
> (recoveries,
> >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> >>>> good. Poked at the hdfs cli. Seems good.
> >>>>
> >>>> S
> >>>>
> >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> >>>> wrote:
> >>>>
> >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> >>>>>
> >>>>> The RC is available at:
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >>>>>
> >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> >>>>>
> >>>>> The maven artifacts are staged at
> >>>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >>>>>
> >>>>> You can find my public key at:
> >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>>>
> >>>>> Change log
> >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >>>>>
> >>>>> Release notes
> >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >>>>>
> >>>>> There's a very small number of changes, primarily critical
> code/packaging
> >>>>> issues and security fixes.
> >>>>>
> >>>>>
> >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> >>>>>   -  CVEs in our code and dependencies
> >>>>>   - Shaded client packaging issues.
> >>>>>   - A switch from log4j to reload4j
> >>>>>
> >>>>>
> >>>>> reload4j is an active fork of the log4j 1.17 library with the
> classes which
> >>>>> contain CVEs removed. Even though hadoop never used those classes,
> they
> >>>>> regularly raised alerts on security scans and concen from users.
> Switching
> >>>>> to the forked project allows us to ship a secure logging framework.
> It will
> >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> >>>>> our log4j artifacts, as they need to cut the new dependency
> instead/as
> >>>>> well.
> >>>>>
> >>>>> See the release notes for details.
> >>>>>
> >>>>> This is my first release through the new docker build process, do
> please
> >>>>> validate artifact signing &c to make sure it is good. I'll be trying
> builds
> >>>>> of downstream projects.
> >>>>>
> >>>>> We know there are some outstanding issues with at least one library
> we are
> >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> If the
> >>>>> docker based release process works smoothly enough we can do a
> followup
> >>>>> security release in a few weeks.
> >>>>>
> >>>>> Please try the release and vote. The vote will run for 5 days.
> >>>>>
> >>>>> -Steve
> >>>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

Hmm, I see the artifacts ideally should have got overwritten by the new RC, but they didn’t. The reason seems like the staging path shared doesn’t have any jars…
That is why it was picking the old jars. I think Steve needs to run mvn deploy again…

Sent from my iPhone

> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> 
> 
>> 
>> Chao can you use the one that Steve mentioned in the mail?
> 
> Hmm how do I do that? Typically after closing the RC in nexus the
> release bits will show up in
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> and Spark build will be able to pick them up for testing. However in
> this case I don't see any 3.3.3 jars in the URL.
> 
>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>> 
>> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>> 
>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>> 
>>> Seems there are some issues with the shaded client as I was not able
>>> to compile Apache Spark with the RC
>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>> contain any class:
>>> 
>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>> META-INF/
>>> META-INF/MANIFEST.MF
>>> META-INF/NOTICE.txt
>>> META-INF/LICENSE.txt
>>> META-INF/maven/
>>> META-INF/maven/org.apache.hadoop/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>> 
>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>> 
>>>> +1 (binding)
>>>> 
>>>>  * Signature: ok
>>>>  * Checksum : passed
>>>>  * Rat check (1.8.0_191): passed
>>>>   - mvn clean apache-rat:check
>>>>  * Built from source (1.8.0_191): failed
>>>>   - mvn clean install  -DskipTests
>>>>   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>  * Unit tests pass (1.8.0_191):
>>>>    - HDFS Tests passed (Didn't run more than this).
>>>> 
>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>> bigger job with mild chaos. All seems to be working properly (recoveries,
>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>> good. Poked at the hdfs cli. Seems good.
>>>> 
>>>> S
>>>> 
>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>>>> wrote:
>>>> 
>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>> 
>>>>> The RC is available at:
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>> 
>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>> 
>>>>> The maven artifacts are staged at
>>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>> 
>>>>> You can find my public key at:
>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>> 
>>>>> Change log
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>> 
>>>>> Release notes
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>> 
>>>>> There's a very small number of changes, primarily critical code/packaging
>>>>> issues and security fixes.
>>>>> 
>>>>> 
>>>>>   - The critical fixes which shipped in the 3.2.3 release.
>>>>>   -  CVEs in our code and dependencies
>>>>>   - Shaded client packaging issues.
>>>>>   - A switch from log4j to reload4j
>>>>> 
>>>>> 
>>>>> reload4j is an active fork of the log4j 1.17 library with the classes which
>>>>> contain CVEs removed. Even though hadoop never used those classes, they
>>>>> regularly raised alerts on security scans and concen from users. Switching
>>>>> to the forked project allows us to ship a secure logging framework. It will
>>>>> complicate the builds of downstream maven/ivy/gradle projects which exclude
>>>>> our log4j artifacts, as they need to cut the new dependency instead/as
>>>>> well.
>>>>> 
>>>>> See the release notes for details.
>>>>> 
>>>>> This is my first release through the new docker build process, do please
>>>>> validate artifact signing &c to make sure it is good. I'll be trying builds
>>>>> of downstream projects.
>>>>> 
>>>>> We know there are some outstanding issues with at least one library we are
>>>>> shipping (okhttp), but I don't want to hold this release up for it. If the
>>>>> docker based release process works smoothly enough we can do a followup
>>>>> security release in a few weeks.
>>>>> 
>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>> 
>>>>> -Steve
>>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

Hmm, I see the artifacts ideally should have got overwritten by the new RC, but they didn’t. The reason seems like the staging path shared doesn’t have any jars…
That is why it was picking the old jars. I think Steve needs to run mvn deploy again…

Sent from my iPhone

> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> 
> 
>> 
>> Chao can you use the one that Steve mentioned in the mail?
> 
> Hmm how do I do that? Typically after closing the RC in nexus the
> release bits will show up in
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> and Spark build will be able to pick them up for testing. However in
> this case I don't see any 3.3.3 jars in the URL.
> 
>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>> 
>> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>> 
>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>> 
>>> Seems there are some issues with the shaded client as I was not able
>>> to compile Apache Spark with the RC
>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>> contain any class:
>>> 
>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>> META-INF/
>>> META-INF/MANIFEST.MF
>>> META-INF/NOTICE.txt
>>> META-INF/LICENSE.txt
>>> META-INF/maven/
>>> META-INF/maven/org.apache.hadoop/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>> 
>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>> 
>>>> +1 (binding)
>>>> 
>>>>  * Signature: ok
>>>>  * Checksum : passed
>>>>  * Rat check (1.8.0_191): passed
>>>>   - mvn clean apache-rat:check
>>>>  * Built from source (1.8.0_191): failed
>>>>   - mvn clean install  -DskipTests
>>>>   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>  * Unit tests pass (1.8.0_191):
>>>>    - HDFS Tests passed (Didn't run more than this).
>>>> 
>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>> bigger job with mild chaos. All seems to be working properly (recoveries,
>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>> good. Poked at the hdfs cli. Seems good.
>>>> 
>>>> S
>>>> 
>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>>>> wrote:
>>>> 
>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>> 
>>>>> The RC is available at:
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>> 
>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>> 
>>>>> The maven artifacts are staged at
>>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>> 
>>>>> You can find my public key at:
>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>> 
>>>>> Change log
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>> 
>>>>> Release notes
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>> 
>>>>> There's a very small number of changes, primarily critical code/packaging
>>>>> issues and security fixes.
>>>>> 
>>>>> 
>>>>>   - The critical fixes which shipped in the 3.2.3 release.
>>>>>   -  CVEs in our code and dependencies
>>>>>   - Shaded client packaging issues.
>>>>>   - A switch from log4j to reload4j
>>>>> 
>>>>> 
>>>>> reload4j is an active fork of the log4j 1.17 library with the classes which
>>>>> contain CVEs removed. Even though hadoop never used those classes, they
>>>>> regularly raised alerts on security scans and concen from users. Switching
>>>>> to the forked project allows us to ship a secure logging framework. It will
>>>>> complicate the builds of downstream maven/ivy/gradle projects which exclude
>>>>> our log4j artifacts, as they need to cut the new dependency instead/as
>>>>> well.
>>>>> 
>>>>> See the release notes for details.
>>>>> 
>>>>> This is my first release through the new docker build process, do please
>>>>> validate artifact signing &c to make sure it is good. I'll be trying builds
>>>>> of downstream projects.
>>>>> 
>>>>> We know there are some outstanding issues with at least one library we are
>>>>> shipping (okhttp), but I don't want to hold this release up for it. If the
>>>>> docker based release process works smoothly enough we can do a followup
>>>>> security release in a few weeks.
>>>>> 
>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>> 
>>>>> -Steve
>>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

Hmm, I see the artifacts ideally should have got overwritten by the new RC, but they didn’t. The reason seems like the staging path shared doesn’t have any jars…
That is why it was picking the old jars. I think Steve needs to run mvn deploy again…

Sent from my iPhone

> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> 
> 
>> 
>> Chao can you use the one that Steve mentioned in the mail?
> 
> Hmm how do I do that? Typically after closing the RC in nexus the
> release bits will show up in
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> and Spark build will be able to pick them up for testing. However in
> this case I don't see any 3.3.3 jars in the URL.
> 
>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>> 
>> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>> 
>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>> 
>>> Seems there are some issues with the shaded client as I was not able
>>> to compile Apache Spark with the RC
>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>> contain any class:
>>> 
>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>> META-INF/
>>> META-INF/MANIFEST.MF
>>> META-INF/NOTICE.txt
>>> META-INF/LICENSE.txt
>>> META-INF/maven/
>>> META-INF/maven/org.apache.hadoop/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>> 
>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>> 
>>>> +1 (binding)
>>>> 
>>>>  * Signature: ok
>>>>  * Checksum : passed
>>>>  * Rat check (1.8.0_191): passed
>>>>   - mvn clean apache-rat:check
>>>>  * Built from source (1.8.0_191): failed
>>>>   - mvn clean install  -DskipTests
>>>>   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>  * Unit tests pass (1.8.0_191):
>>>>    - HDFS Tests passed (Didn't run more than this).
>>>> 
>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>> bigger job with mild chaos. All seems to be working properly (recoveries,
>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>> good. Poked at the hdfs cli. Seems good.
>>>> 
>>>> S
>>>> 
>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>>>> wrote:
>>>> 
>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>> 
>>>>> The RC is available at:
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>> 
>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>> 
>>>>> The maven artifacts are staged at
>>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>> 
>>>>> You can find my public key at:
>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>> 
>>>>> Change log
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>> 
>>>>> Release notes
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>> 
>>>>> There's a very small number of changes, primarily critical code/packaging
>>>>> issues and security fixes.
>>>>> 
>>>>> 
>>>>>   - The critical fixes which shipped in the 3.2.3 release.
>>>>>   -  CVEs in our code and dependencies
>>>>>   - Shaded client packaging issues.
>>>>>   - A switch from log4j to reload4j
>>>>> 
>>>>> 
>>>>> reload4j is an active fork of the log4j 1.17 library with the classes which
>>>>> contain CVEs removed. Even though hadoop never used those classes, they
>>>>> regularly raised alerts on security scans and concen from users. Switching
>>>>> to the forked project allows us to ship a secure logging framework. It will
>>>>> complicate the builds of downstream maven/ivy/gradle projects which exclude
>>>>> our log4j artifacts, as they need to cut the new dependency instead/as
>>>>> well.
>>>>> 
>>>>> See the release notes for details.
>>>>> 
>>>>> This is my first release through the new docker build process, do please
>>>>> validate artifact signing &c to make sure it is good. I'll be trying builds
>>>>> of downstream projects.
>>>>> 
>>>>> We know there are some outstanding issues with at least one library we are
>>>>> shipping (okhttp), but I don't want to hold this release up for it. If the
>>>>> docker based release process works smoothly enough we can do a followup
>>>>> security release in a few weeks.
>>>>> 
>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>> 
>>>>> -Steve
>>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

Hmm, I see the artifacts ideally should have got overwritten by the new RC, but they didn’t. The reason seems like the staging path shared doesn’t have any jars…
That is why it was picking the old jars. I think Steve needs to run mvn deploy again…

Sent from my iPhone

> On 07-May-2022, at 7:12 AM, Chao Sun <su...@apache.org> wrote:
> 
> 
>> 
>> Chao can you use the one that Steve mentioned in the mail?
> 
> Hmm how do I do that? Typically after closing the RC in nexus the
> release bits will show up in
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> and Spark build will be able to pick them up for testing. However in
> this case I don't see any 3.3.3 jars in the URL.
> 
>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>> 
>> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>> 
>>> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>> 
>>> Seems there are some issues with the shaded client as I was not able
>>> to compile Apache Spark with the RC
>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>> contain any class:
>>> 
>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>> META-INF/
>>> META-INF/MANIFEST.MF
>>> META-INF/NOTICE.txt
>>> META-INF/LICENSE.txt
>>> META-INF/maven/
>>> META-INF/maven/org.apache.hadoop/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>> 
>>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>>>> 
>>>> +1 (binding)
>>>> 
>>>>  * Signature: ok
>>>>  * Checksum : passed
>>>>  * Rat check (1.8.0_191): passed
>>>>   - mvn clean apache-rat:check
>>>>  * Built from source (1.8.0_191): failed
>>>>   - mvn clean install  -DskipTests
>>>>   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>  * Unit tests pass (1.8.0_191):
>>>>    - HDFS Tests passed (Didn't run more than this).
>>>> 
>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>> bigger job with mild chaos. All seems to be working properly (recoveries,
>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>> good. Poked at the hdfs cli. Seems good.
>>>> 
>>>> S
>>>> 
>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>>>> wrote:
>>>> 
>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>> 
>>>>> The RC is available at:
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>> 
>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>> 
>>>>> The maven artifacts are staged at
>>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>> 
>>>>> You can find my public key at:
>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>> 
>>>>> Change log
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>> 
>>>>> Release notes
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>> 
>>>>> There's a very small number of changes, primarily critical code/packaging
>>>>> issues and security fixes.
>>>>> 
>>>>> 
>>>>>   - The critical fixes which shipped in the 3.2.3 release.
>>>>>   -  CVEs in our code and dependencies
>>>>>   - Shaded client packaging issues.
>>>>>   - A switch from log4j to reload4j
>>>>> 
>>>>> 
>>>>> reload4j is an active fork of the log4j 1.17 library with the classes which
>>>>> contain CVEs removed. Even though hadoop never used those classes, they
>>>>> regularly raised alerts on security scans and concen from users. Switching
>>>>> to the forked project allows us to ship a secure logging framework. It will
>>>>> complicate the builds of downstream maven/ivy/gradle projects which exclude
>>>>> our log4j artifacts, as they need to cut the new dependency instead/as
>>>>> well.
>>>>> 
>>>>> See the release notes for details.
>>>>> 
>>>>> This is my first release through the new docker build process, do please
>>>>> validate artifact signing &c to make sure it is good. I'll be trying builds
>>>>> of downstream projects.
>>>>> 
>>>>> We know there are some outstanding issues with at least one library we are
>>>>> shipping (okhttp), but I don't want to hold this release up for it. If the
>>>>> docker based release process works smoothly enough we can do a followup
>>>>> security release in a few weeks.
>>>>> 
>>>>> Please try the release and vote. The vote will run for 5 days.
>>>>> 
>>>>> -Steve
>>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

> Chao can you use the one that Steve mentioned in the mail?

Hmm how do I do that? Typically after closing the RC in nexus the
release bits will show up in
https://repository.apache.org/content/repositories/staging/org/apache/hadoop
and Spark build will be able to pick them up for testing. However in
this case I don't see any 3.3.3 jars in the URL.

On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>
> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>
>> Seems there are some issues with the shaded client as I was not able
>> to compile Apache Spark with the RC
>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>> contain any class:
>>
>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> META-INF/
>> META-INF/MANIFEST.MF
>> META-INF/NOTICE.txt
>> META-INF/LICENSE.txt
>> META-INF/maven/
>> META-INF/maven/org.apache.hadoop/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>
>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> >
>> > +1 (binding)
>> >
>> >   * Signature: ok
>> >   * Checksum : passed
>> >   * Rat check (1.8.0_191): passed
>> >    - mvn clean apache-rat:check
>> >   * Built from source (1.8.0_191): failed
>> >    - mvn clean install  -DskipTests
>> >    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > -Drequire.zstd -Drequire.test.libhadoop clean install
>> >   * Unit tests pass (1.8.0_191):
>> >     - HDFS Tests passed (Didn't run more than this).
>> >
>> > Deployed a ten node ha hdfs cluster with three namenodes and five
>> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
>> > bigger job with mild chaos. All seems to be working properly (recoveries,
>> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
>> > good. Poked at the hdfs cli. Seems good.
>> >
>> > S
>> >
>> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > >
>> > > The RC is available at:
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > >
>> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > >
>> > > The maven artifacts are staged at
>> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > >
>> > > You can find my public key at:
>> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > >
>> > > Change log
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > >
>> > > Release notes
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > >
>> > > There's a very small number of changes, primarily critical code/packaging
>> > > issues and security fixes.
>> > >
>> > >
>> > >    - The critical fixes which shipped in the 3.2.3 release.
>> > >    -  CVEs in our code and dependencies
>> > >    - Shaded client packaging issues.
>> > >    - A switch from log4j to reload4j
>> > >
>> > >
>> > > reload4j is an active fork of the log4j 1.17 library with the classes which
>> > > contain CVEs removed. Even though hadoop never used those classes, they
>> > > regularly raised alerts on security scans and concen from users. Switching
>> > > to the forked project allows us to ship a secure logging framework. It will
>> > > complicate the builds of downstream maven/ivy/gradle projects which exclude
>> > > our log4j artifacts, as they need to cut the new dependency instead/as
>> > > well.
>> > >
>> > > See the release notes for details.
>> > >
>> > > This is my first release through the new docker build process, do please
>> > > validate artifact signing &c to make sure it is good. I'll be trying builds
>> > > of downstream projects.
>> > >
>> > > We know there are some outstanding issues with at least one library we are
>> > > shipping (okhttp), but I don't want to hold this release up for it. If the
>> > > docker based release process works smoothly enough we can do a followup
>> > > security release in a few weeks.
>> > >
>> > > Please try the release and vote. The vote will run for 5 days.
>> > >
>> > > -Steve
>> > >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

> Chao can you use the one that Steve mentioned in the mail?

Hmm how do I do that? Typically after closing the RC in nexus the
release bits will show up in
https://repository.apache.org/content/repositories/staging/org/apache/hadoop
and Spark build will be able to pick them up for testing. However in
this case I don't see any 3.3.3 jars in the URL.

On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>
> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>
>> Seems there are some issues with the shaded client as I was not able
>> to compile Apache Spark with the RC
>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>> contain any class:
>>
>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> META-INF/
>> META-INF/MANIFEST.MF
>> META-INF/NOTICE.txt
>> META-INF/LICENSE.txt
>> META-INF/maven/
>> META-INF/maven/org.apache.hadoop/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>
>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> >
>> > +1 (binding)
>> >
>> >   * Signature: ok
>> >   * Checksum : passed
>> >   * Rat check (1.8.0_191): passed
>> >    - mvn clean apache-rat:check
>> >   * Built from source (1.8.0_191): failed
>> >    - mvn clean install  -DskipTests
>> >    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > -Drequire.zstd -Drequire.test.libhadoop clean install
>> >   * Unit tests pass (1.8.0_191):
>> >     - HDFS Tests passed (Didn't run more than this).
>> >
>> > Deployed a ten node ha hdfs cluster with three namenodes and five
>> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
>> > bigger job with mild chaos. All seems to be working properly (recoveries,
>> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
>> > good. Poked at the hdfs cli. Seems good.
>> >
>> > S
>> >
>> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > >
>> > > The RC is available at:
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > >
>> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > >
>> > > The maven artifacts are staged at
>> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > >
>> > > You can find my public key at:
>> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > >
>> > > Change log
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > >
>> > > Release notes
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > >
>> > > There's a very small number of changes, primarily critical code/packaging
>> > > issues and security fixes.
>> > >
>> > >
>> > >    - The critical fixes which shipped in the 3.2.3 release.
>> > >    -  CVEs in our code and dependencies
>> > >    - Shaded client packaging issues.
>> > >    - A switch from log4j to reload4j
>> > >
>> > >
>> > > reload4j is an active fork of the log4j 1.17 library with the classes which
>> > > contain CVEs removed. Even though hadoop never used those classes, they
>> > > regularly raised alerts on security scans and concen from users. Switching
>> > > to the forked project allows us to ship a secure logging framework. It will
>> > > complicate the builds of downstream maven/ivy/gradle projects which exclude
>> > > our log4j artifacts, as they need to cut the new dependency instead/as
>> > > well.
>> > >
>> > > See the release notes for details.
>> > >
>> > > This is my first release through the new docker build process, do please
>> > > validate artifact signing &c to make sure it is good. I'll be trying builds
>> > > of downstream projects.
>> > >
>> > > We know there are some outstanding issues with at least one library we are
>> > > shipping (okhttp), but I don't want to hold this release up for it. If the
>> > > docker based release process works smoothly enough we can do a followup
>> > > security release in a few weeks.
>> > >
>> > > Please try the release and vote. The vote will run for 5 days.
>> > >
>> > > -Steve
>> > >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

> Chao can you use the one that Steve mentioned in the mail?

Hmm how do I do that? Typically after closing the RC in nexus the
release bits will show up in
https://repository.apache.org/content/repositories/staging/org/apache/hadoop
and Spark build will be able to pick them up for testing. However in
this case I don't see any 3.3.3 jars in the URL.

On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>
> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>
>> Seems there are some issues with the shaded client as I was not able
>> to compile Apache Spark with the RC
>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>> contain any class:
>>
>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> META-INF/
>> META-INF/MANIFEST.MF
>> META-INF/NOTICE.txt
>> META-INF/LICENSE.txt
>> META-INF/maven/
>> META-INF/maven/org.apache.hadoop/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>
>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> >
>> > +1 (binding)
>> >
>> >   * Signature: ok
>> >   * Checksum : passed
>> >   * Rat check (1.8.0_191): passed
>> >    - mvn clean apache-rat:check
>> >   * Built from source (1.8.0_191): failed
>> >    - mvn clean install  -DskipTests
>> >    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > -Drequire.zstd -Drequire.test.libhadoop clean install
>> >   * Unit tests pass (1.8.0_191):
>> >     - HDFS Tests passed (Didn't run more than this).
>> >
>> > Deployed a ten node ha hdfs cluster with three namenodes and five
>> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
>> > bigger job with mild chaos. All seems to be working properly (recoveries,
>> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
>> > good. Poked at the hdfs cli. Seems good.
>> >
>> > S
>> >
>> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > >
>> > > The RC is available at:
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > >
>> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > >
>> > > The maven artifacts are staged at
>> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > >
>> > > You can find my public key at:
>> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > >
>> > > Change log
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > >
>> > > Release notes
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > >
>> > > There's a very small number of changes, primarily critical code/packaging
>> > > issues and security fixes.
>> > >
>> > >
>> > >    - The critical fixes which shipped in the 3.2.3 release.
>> > >    -  CVEs in our code and dependencies
>> > >    - Shaded client packaging issues.
>> > >    - A switch from log4j to reload4j
>> > >
>> > >
>> > > reload4j is an active fork of the log4j 1.17 library with the classes which
>> > > contain CVEs removed. Even though hadoop never used those classes, they
>> > > regularly raised alerts on security scans and concen from users. Switching
>> > > to the forked project allows us to ship a secure logging framework. It will
>> > > complicate the builds of downstream maven/ivy/gradle projects which exclude
>> > > our log4j artifacts, as they need to cut the new dependency instead/as
>> > > well.
>> > >
>> > > See the release notes for details.
>> > >
>> > > This is my first release through the new docker build process, do please
>> > > validate artifact signing &c to make sure it is good. I'll be trying builds
>> > > of downstream projects.
>> > >
>> > > We know there are some outstanding issues with at least one library we are
>> > > shipping (okhttp), but I don't want to hold this release up for it. If the
>> > > docker based release process works smoothly enough we can do a followup
>> > > security release in a few weeks.
>> > >
>> > > Please try the release and vote. The vote will run for 5 days.
>> > >
>> > > -Steve
>> > >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

> Chao can you use the one that Steve mentioned in the mail?

Hmm how do I do that? Typically after closing the RC in nexus the
release bits will show up in
https://repository.apache.org/content/repositories/staging/org/apache/hadoop
and Spark build will be able to pick them up for testing. However in
this case I don't see any 3.3.3 jars in the URL.

On Fri, May 6, 2022 at 6:24 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> There were two 3.3.3 staged. The earlier one was with skipShade, the date was also april 22, I archived that. Chao can you use the one that Steve mentioned in the mail?
>
> On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:
>>
>> Seems there are some issues with the shaded client as I was not able
>> to compile Apache Spark with the RC
>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>> contain any class:
>>
>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>> META-INF/
>> META-INF/MANIFEST.MF
>> META-INF/NOTICE.txt
>> META-INF/LICENSE.txt
>> META-INF/maven/
>> META-INF/maven/org.apache.hadoop/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>
>> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>> >
>> > +1 (binding)
>> >
>> >   * Signature: ok
>> >   * Checksum : passed
>> >   * Rat check (1.8.0_191): passed
>> >    - mvn clean apache-rat:check
>> >   * Built from source (1.8.0_191): failed
>> >    - mvn clean install  -DskipTests
>> >    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>> > -Drequire.zstd -Drequire.test.libhadoop clean install
>> >   * Unit tests pass (1.8.0_191):
>> >     - HDFS Tests passed (Didn't run more than this).
>> >
>> > Deployed a ten node ha hdfs cluster with three namenodes and five
>> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
>> > bigger job with mild chaos. All seems to be working properly (recoveries,
>> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
>> > good. Poked at the hdfs cli. Seems good.
>> >
>> > S
>> >
>> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
>> > wrote:
>> >
>> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
>> > >
>> > > The RC is available at:
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>> > >
>> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
>> > >
>> > > The maven artifacts are staged at
>> > > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > >
>> > > You can find my public key at:
>> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > >
>> > > Change log
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>> > >
>> > > Release notes
>> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>> > >
>> > > There's a very small number of changes, primarily critical code/packaging
>> > > issues and security fixes.
>> > >
>> > >
>> > >    - The critical fixes which shipped in the 3.2.3 release.
>> > >    -  CVEs in our code and dependencies
>> > >    - Shaded client packaging issues.
>> > >    - A switch from log4j to reload4j
>> > >
>> > >
>> > > reload4j is an active fork of the log4j 1.17 library with the classes which
>> > > contain CVEs removed. Even though hadoop never used those classes, they
>> > > regularly raised alerts on security scans and concen from users. Switching
>> > > to the forked project allows us to ship a secure logging framework. It will
>> > > complicate the builds of downstream maven/ivy/gradle projects which exclude
>> > > our log4j artifacts, as they need to cut the new dependency instead/as
>> > > well.
>> > >
>> > > See the release notes for details.
>> > >
>> > > This is my first release through the new docker build process, do please
>> > > validate artifact signing &c to make sure it is good. I'll be trying builds
>> > > of downstream projects.
>> > >
>> > > We know there are some outstanding issues with at least one library we are
>> > > shipping (okhttp), but I don't want to hold this release up for it. If the
>> > > docker based release process works smoothly enough we can do a followup
>> > > security release in a few weeks.
>> > >
>> > > Please try the release and vote. The vote will run for 5 days.
>> > >
>> > > -Steve
>> > >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

There were two 3.3.3 staged. The earlier one was with skipShade, the date
was also april 22, I archived that. Chao can you use the one that Steve
mentioned in the mail?

On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:

> Seems there are some issues with the shaded client as I was not able
> to compile Apache Spark with the RC
> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> contain any class:
>
> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> META-INF/
> META-INF/MANIFEST.MF
> META-INF/NOTICE.txt
> META-INF/LICENSE.txt
> META-INF/maven/
> META-INF/maven/org.apache.hadoop/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>
> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >
> > +1 (binding)
> >
> >   * Signature: ok
> >   * Checksum : passed
> >   * Rat check (1.8.0_191): passed
> >    - mvn clean apache-rat:check
> >   * Built from source (1.8.0_191): failed
> >    - mvn clean install  -DskipTests
> >    - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > -Drequire.zstd -Drequire.test.libhadoop clean install
> >   * Unit tests pass (1.8.0_191):
> >     - HDFS Tests passed (Didn't run more than this).
> >
> > Deployed a ten node ha hdfs cluster with three namenodes and five
> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
> > bigger job with mild chaos. All seems to be working properly (recoveries,
> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
> > good. Poked at the hdfs cli. Seems good.
> >
> > S
> >
> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >
> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >
> > > Release notes
> > >
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >
> > > There's a very small number of changes, primarily critical
> code/packaging
> > > issues and security fixes.
> > >
> > >
> > >    - The critical fixes which shipped in the 3.2.3 release.
> > >    -  CVEs in our code and dependencies
> > >    - Shaded client packaging issues.
> > >    - A switch from log4j to reload4j
> > >
> > >
> > > reload4j is an active fork of the log4j 1.17 library with the classes
> which
> > > contain CVEs removed. Even though hadoop never used those classes, they
> > > regularly raised alerts on security scans and concen from users.
> Switching
> > > to the forked project allows us to ship a secure logging framework. It
> will
> > > complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> > > our log4j artifacts, as they need to cut the new dependency instead/as
> > > well.
> > >
> > > See the release notes for details.
> > >
> > > This is my first release through the new docker build process, do
> please
> > > validate artifact signing &c to make sure it is good. I'll be trying
> builds
> > > of downstream projects.
> > >
> > > We know there are some outstanding issues with at least one library we
> are
> > > shipping (okhttp), but I don't want to hold this release up for it. If
> the
> > > docker based release process works smoothly enough we can do a followup
> > > security release in a few weeks.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

There were two 3.3.3 staged. The earlier one was with skipShade, the date
was also april 22, I archived that. Chao can you use the one that Steve
mentioned in the mail?

On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:

> Seems there are some issues with the shaded client as I was not able
> to compile Apache Spark with the RC
> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> contain any class:
>
> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> META-INF/
> META-INF/MANIFEST.MF
> META-INF/NOTICE.txt
> META-INF/LICENSE.txt
> META-INF/maven/
> META-INF/maven/org.apache.hadoop/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>
> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >
> > +1 (binding)
> >
> >   * Signature: ok
> >   * Checksum : passed
> >   * Rat check (1.8.0_191): passed
> >    - mvn clean apache-rat:check
> >   * Built from source (1.8.0_191): failed
> >    - mvn clean install  -DskipTests
> >    - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > -Drequire.zstd -Drequire.test.libhadoop clean install
> >   * Unit tests pass (1.8.0_191):
> >     - HDFS Tests passed (Didn't run more than this).
> >
> > Deployed a ten node ha hdfs cluster with three namenodes and five
> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
> > bigger job with mild chaos. All seems to be working properly (recoveries,
> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
> > good. Poked at the hdfs cli. Seems good.
> >
> > S
> >
> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >
> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >
> > > Release notes
> > >
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >
> > > There's a very small number of changes, primarily critical
> code/packaging
> > > issues and security fixes.
> > >
> > >
> > >    - The critical fixes which shipped in the 3.2.3 release.
> > >    -  CVEs in our code and dependencies
> > >    - Shaded client packaging issues.
> > >    - A switch from log4j to reload4j
> > >
> > >
> > > reload4j is an active fork of the log4j 1.17 library with the classes
> which
> > > contain CVEs removed. Even though hadoop never used those classes, they
> > > regularly raised alerts on security scans and concen from users.
> Switching
> > > to the forked project allows us to ship a secure logging framework. It
> will
> > > complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> > > our log4j artifacts, as they need to cut the new dependency instead/as
> > > well.
> > >
> > > See the release notes for details.
> > >
> > > This is my first release through the new docker build process, do
> please
> > > validate artifact signing &c to make sure it is good. I'll be trying
> builds
> > > of downstream projects.
> > >
> > > We know there are some outstanding issues with at least one library we
> are
> > > shipping (okhttp), but I don't want to hold this release up for it. If
> the
> > > docker based release process works smoothly enough we can do a followup
> > > security release in a few weeks.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

There were two 3.3.3 staged. The earlier one was with skipShade, the date
was also april 22, I archived that. Chao can you use the one that Steve
mentioned in the mail?

On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:

> Seems there are some issues with the shaded client as I was not able
> to compile Apache Spark with the RC
> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> contain any class:
>
> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> META-INF/
> META-INF/MANIFEST.MF
> META-INF/NOTICE.txt
> META-INF/LICENSE.txt
> META-INF/maven/
> META-INF/maven/org.apache.hadoop/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>
> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >
> > +1 (binding)
> >
> >   * Signature: ok
> >   * Checksum : passed
> >   * Rat check (1.8.0_191): passed
> >    - mvn clean apache-rat:check
> >   * Built from source (1.8.0_191): failed
> >    - mvn clean install  -DskipTests
> >    - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > -Drequire.zstd -Drequire.test.libhadoop clean install
> >   * Unit tests pass (1.8.0_191):
> >     - HDFS Tests passed (Didn't run more than this).
> >
> > Deployed a ten node ha hdfs cluster with three namenodes and five
> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
> > bigger job with mild chaos. All seems to be working properly (recoveries,
> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
> > good. Poked at the hdfs cli. Seems good.
> >
> > S
> >
> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >
> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >
> > > Release notes
> > >
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >
> > > There's a very small number of changes, primarily critical
> code/packaging
> > > issues and security fixes.
> > >
> > >
> > >    - The critical fixes which shipped in the 3.2.3 release.
> > >    -  CVEs in our code and dependencies
> > >    - Shaded client packaging issues.
> > >    - A switch from log4j to reload4j
> > >
> > >
> > > reload4j is an active fork of the log4j 1.17 library with the classes
> which
> > > contain CVEs removed. Even though hadoop never used those classes, they
> > > regularly raised alerts on security scans and concen from users.
> Switching
> > > to the forked project allows us to ship a secure logging framework. It
> will
> > > complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> > > our log4j artifacts, as they need to cut the new dependency instead/as
> > > well.
> > >
> > > See the release notes for details.
> > >
> > > This is my first release through the new docker build process, do
> please
> > > validate artifact signing &c to make sure it is good. I'll be trying
> builds
> > > of downstream projects.
> > >
> > > We know there are some outstanding issues with at least one library we
> are
> > > shipping (okhttp), but I don't want to hold this release up for it. If
> the
> > > docker based release process works smoothly enough we can do a followup
> > > security release in a few weeks.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

There were two 3.3.3 staged. The earlier one was with skipShade, the date
was also april 22, I archived that. Chao can you use the one that Steve
mentioned in the mail?

On Sat, 7 May 2022 at 06:18, Chao Sun <su...@apache.org> wrote:

> Seems there are some issues with the shaded client as I was not able
> to compile Apache Spark with the RC
> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> contain any class:
>
> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> META-INF/
> META-INF/MANIFEST.MF
> META-INF/NOTICE.txt
> META-INF/LICENSE.txt
> META-INF/maven/
> META-INF/maven/org.apache.hadoop/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>
> On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
> >
> > +1 (binding)
> >
> >   * Signature: ok
> >   * Checksum : passed
> >   * Rat check (1.8.0_191): passed
> >    - mvn clean apache-rat:check
> >   * Built from source (1.8.0_191): failed
> >    - mvn clean install  -DskipTests
> >    - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > -Drequire.zstd -Drequire.test.libhadoop clean install
> >   * Unit tests pass (1.8.0_191):
> >     - HDFS Tests passed (Didn't run more than this).
> >
> > Deployed a ten node ha hdfs cluster with three namenodes and five
> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
> > bigger job with mild chaos. All seems to be working properly (recoveries,
> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
> > good. Poked at the hdfs cli. Seems good.
> >
> > S
> >
> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >
> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >
> > > Release notes
> > >
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >
> > > There's a very small number of changes, primarily critical
> code/packaging
> > > issues and security fixes.
> > >
> > >
> > >    - The critical fixes which shipped in the 3.2.3 release.
> > >    -  CVEs in our code and dependencies
> > >    - Shaded client packaging issues.
> > >    - A switch from log4j to reload4j
> > >
> > >
> > > reload4j is an active fork of the log4j 1.17 library with the classes
> which
> > > contain CVEs removed. Even though hadoop never used those classes, they
> > > regularly raised alerts on security scans and concen from users.
> Switching
> > > to the forked project allows us to ship a secure logging framework. It
> will
> > > complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> > > our log4j artifacts, as they need to cut the new dependency instead/as
> > > well.
> > >
> > > See the release notes for details.
> > >
> > > This is my first release through the new docker build process, do
> please
> > > validate artifact signing &c to make sure it is good. I'll be trying
> builds
> > > of downstream projects.
> > >
> > > We know there are some outstanding issues with at least one library we
> are
> > > shipping (okhttp), but I don't want to hold this release up for it. If
> the
> > > docker based release process works smoothly enough we can do a followup
> > > security release in a few weeks.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Seems there are some issues with the shaded client as I was not able
to compile Apache Spark with the RC
(https://github.com/apache/spark/pull/36474). Looks like it's compiled
with the `-DskipShade` option and the hadoop-client-api JAR doesn't
contain any class:

➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
META-INF/
META-INF/MANIFEST.MF
META-INF/NOTICE.txt
META-INF/LICENSE.txt
META-INF/maven/
META-INF/maven/org.apache.hadoop/
META-INF/maven/org.apache.hadoop/hadoop-client-api/
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties

On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>
> +1 (binding)
>
>   * Signature: ok
>   * Checksum : passed
>   * Rat check (1.8.0_191): passed
>    - mvn clean apache-rat:check
>   * Built from source (1.8.0_191): failed
>    - mvn clean install  -DskipTests
>    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> -Drequire.zstd -Drequire.test.libhadoop clean install
>   * Unit tests pass (1.8.0_191):
>     - HDFS Tests passed (Didn't run more than this).
>
> Deployed a ten node ha hdfs cluster with three namenodes and five
> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> bigger job with mild chaos. All seems to be working properly (recoveries,
> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> good. Poked at the hdfs cli. Seems good.
>
> S
>
> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
> > I have put together a release candidate (rc0) for Hadoop 3.3.3
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >
> > The git tag is release-3.3.3-RC0, commit d37586cbda3
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >
> > Release notes
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >
> > There's a very small number of changes, primarily critical code/packaging
> > issues and security fixes.
> >
> >
> >    - The critical fixes which shipped in the 3.2.3 release.
> >    -  CVEs in our code and dependencies
> >    - Shaded client packaging issues.
> >    - A switch from log4j to reload4j
> >
> >
> > reload4j is an active fork of the log4j 1.17 library with the classes which
> > contain CVEs removed. Even though hadoop never used those classes, they
> > regularly raised alerts on security scans and concen from users. Switching
> > to the forked project allows us to ship a secure logging framework. It will
> > complicate the builds of downstream maven/ivy/gradle projects which exclude
> > our log4j artifacts, as they need to cut the new dependency instead/as
> > well.
> >
> > See the release notes for details.
> >
> > This is my first release through the new docker build process, do please
> > validate artifact signing &c to make sure it is good. I'll be trying builds
> > of downstream projects.
> >
> > We know there are some outstanding issues with at least one library we are
> > shipping (okhttp), but I don't want to hold this release up for it. If the
> > docker based release process works smoothly enough we can do a followup
> > security release in a few weeks.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Seems there are some issues with the shaded client as I was not able
to compile Apache Spark with the RC
(https://github.com/apache/spark/pull/36474). Looks like it's compiled
with the `-DskipShade` option and the hadoop-client-api JAR doesn't
contain any class:

➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
META-INF/
META-INF/MANIFEST.MF
META-INF/NOTICE.txt
META-INF/LICENSE.txt
META-INF/maven/
META-INF/maven/org.apache.hadoop/
META-INF/maven/org.apache.hadoop/hadoop-client-api/
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties

On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>
> +1 (binding)
>
>   * Signature: ok
>   * Checksum : passed
>   * Rat check (1.8.0_191): passed
>    - mvn clean apache-rat:check
>   * Built from source (1.8.0_191): failed
>    - mvn clean install  -DskipTests
>    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> -Drequire.zstd -Drequire.test.libhadoop clean install
>   * Unit tests pass (1.8.0_191):
>     - HDFS Tests passed (Didn't run more than this).
>
> Deployed a ten node ha hdfs cluster with three namenodes and five
> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> bigger job with mild chaos. All seems to be working properly (recoveries,
> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> good. Poked at the hdfs cli. Seems good.
>
> S
>
> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
> > I have put together a release candidate (rc0) for Hadoop 3.3.3
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >
> > The git tag is release-3.3.3-RC0, commit d37586cbda3
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >
> > Release notes
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >
> > There's a very small number of changes, primarily critical code/packaging
> > issues and security fixes.
> >
> >
> >    - The critical fixes which shipped in the 3.2.3 release.
> >    -  CVEs in our code and dependencies
> >    - Shaded client packaging issues.
> >    - A switch from log4j to reload4j
> >
> >
> > reload4j is an active fork of the log4j 1.17 library with the classes which
> > contain CVEs removed. Even though hadoop never used those classes, they
> > regularly raised alerts on security scans and concen from users. Switching
> > to the forked project allows us to ship a secure logging framework. It will
> > complicate the builds of downstream maven/ivy/gradle projects which exclude
> > our log4j artifacts, as they need to cut the new dependency instead/as
> > well.
> >
> > See the release notes for details.
> >
> > This is my first release through the new docker build process, do please
> > validate artifact signing &c to make sure it is good. I'll be trying builds
> > of downstream projects.
> >
> > We know there are some outstanding issues with at least one library we are
> > shipping (okhttp), but I don't want to hold this release up for it. If the
> > docker based release process works smoothly enough we can do a followup
> > security release in a few weeks.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Seems there are some issues with the shaded client as I was not able
to compile Apache Spark with the RC
(https://github.com/apache/spark/pull/36474). Looks like it's compiled
with the `-DskipShade` option and the hadoop-client-api JAR doesn't
contain any class:

➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
META-INF/
META-INF/MANIFEST.MF
META-INF/NOTICE.txt
META-INF/LICENSE.txt
META-INF/maven/
META-INF/maven/org.apache.hadoop/
META-INF/maven/org.apache.hadoop/hadoop-client-api/
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties

On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>
> +1 (binding)
>
>   * Signature: ok
>   * Checksum : passed
>   * Rat check (1.8.0_191): passed
>    - mvn clean apache-rat:check
>   * Built from source (1.8.0_191): failed
>    - mvn clean install  -DskipTests
>    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> -Drequire.zstd -Drequire.test.libhadoop clean install
>   * Unit tests pass (1.8.0_191):
>     - HDFS Tests passed (Didn't run more than this).
>
> Deployed a ten node ha hdfs cluster with three namenodes and five
> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> bigger job with mild chaos. All seems to be working properly (recoveries,
> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> good. Poked at the hdfs cli. Seems good.
>
> S
>
> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
> > I have put together a release candidate (rc0) for Hadoop 3.3.3
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >
> > The git tag is release-3.3.3-RC0, commit d37586cbda3
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >
> > Release notes
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >
> > There's a very small number of changes, primarily critical code/packaging
> > issues and security fixes.
> >
> >
> >    - The critical fixes which shipped in the 3.2.3 release.
> >    -  CVEs in our code and dependencies
> >    - Shaded client packaging issues.
> >    - A switch from log4j to reload4j
> >
> >
> > reload4j is an active fork of the log4j 1.17 library with the classes which
> > contain CVEs removed. Even though hadoop never used those classes, they
> > regularly raised alerts on security scans and concen from users. Switching
> > to the forked project allows us to ship a secure logging framework. It will
> > complicate the builds of downstream maven/ivy/gradle projects which exclude
> > our log4j artifacts, as they need to cut the new dependency instead/as
> > well.
> >
> > See the release notes for details.
> >
> > This is my first release through the new docker build process, do please
> > validate artifact signing &c to make sure it is good. I'll be trying builds
> > of downstream projects.
> >
> > We know there are some outstanding issues with at least one library we are
> > shipping (okhttp), but I don't want to hold this release up for it. If the
> > docker based release process works smoothly enough we can do a followup
> > security release in a few weeks.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Chao Sun <su...@apache.org>.

Seems there are some issues with the shaded client as I was not able
to compile Apache Spark with the RC
(https://github.com/apache/spark/pull/36474). Looks like it's compiled
with the `-DskipShade` option and the hadoop-client-api JAR doesn't
contain any class:

➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
META-INF/
META-INF/MANIFEST.MF
META-INF/NOTICE.txt
META-INF/LICENSE.txt
META-INF/maven/
META-INF/maven/org.apache.hadoop/
META-INF/maven/org.apache.hadoop/hadoop-client-api/
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties

On Fri, May 6, 2022 at 4:24 PM Stack <st...@duboce.net> wrote:
>
> +1 (binding)
>
>   * Signature: ok
>   * Checksum : passed
>   * Rat check (1.8.0_191): passed
>    - mvn clean apache-rat:check
>   * Built from source (1.8.0_191): failed
>    - mvn clean install  -DskipTests
>    - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> -Drequire.zstd -Drequire.test.libhadoop clean install
>   * Unit tests pass (1.8.0_191):
>     - HDFS Tests passed (Didn't run more than this).
>
> Deployed a ten node ha hdfs cluster with three namenodes and five
> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> against 3.3.2) against it. Tried a small verification job. Good. Ran a
> bigger job with mild chaos. All seems to be working properly (recoveries,
> logs look fine). Killed a namenode. Failover worked promptly. UIs look
> good. Poked at the hdfs cli. Seems good.
>
> S
>
> On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
> > I have put together a release candidate (rc0) for Hadoop 3.3.3
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >
> > The git tag is release-3.3.3-RC0, commit d37586cbda3
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >
> > Release notes
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >
> > There's a very small number of changes, primarily critical code/packaging
> > issues and security fixes.
> >
> >
> >    - The critical fixes which shipped in the 3.2.3 release.
> >    -  CVEs in our code and dependencies
> >    - Shaded client packaging issues.
> >    - A switch from log4j to reload4j
> >
> >
> > reload4j is an active fork of the log4j 1.17 library with the classes which
> > contain CVEs removed. Even though hadoop never used those classes, they
> > regularly raised alerts on security scans and concen from users. Switching
> > to the forked project allows us to ship a secure logging framework. It will
> > complicate the builds of downstream maven/ivy/gradle projects which exclude
> > our log4j artifacts, as they need to cut the new dependency instead/as
> > well.
> >
> > See the release notes for details.
> >
> > This is my first release through the new docker build process, do please
> > validate artifact signing &c to make sure it is good. I'll be trying builds
> > of downstream projects.
> >
> > We know there are some outstanding issues with at least one library we are
> > shipping (okhttp), but I don't want to hold this release up for it. If the
> > docker based release process works smoothly enough we can do a followup
> > security release in a few weeks.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Stack <st...@duboce.net>.

+1 (binding)

  * Signature: ok
  * Checksum : passed
  * Rat check (1.8.0_191): passed
   - mvn clean apache-rat:check
  * Built from source (1.8.0_191): failed
   - mvn clean install  -DskipTests
   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
-Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop clean install
  * Unit tests pass (1.8.0_191):
    - HDFS Tests passed (Didn't run more than this).

Deployed a ten node ha hdfs cluster with three namenodes and five
journalnodes. Ran a ten node hbase (older version of 2.5 branch built
against 3.3.2) against it. Tried a small verification job. Good. Ran a
bigger job with mild chaos. All seems to be working properly (recoveries,
logs look fine). Killed a namenode. Failover worked promptly. UIs look
good. Poked at the hdfs cli. Seems good.

S

On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.

+1 (binding)
Built tar from source in mac os and 1.8.0_161.   good

mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true


Able to access cloud s3 and abfs storage through command line.

Ran S3 and Abfs integration tests. All good.



On Tue, May 3, 2022 at 6:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

+1,
* Built from source
* Successful native build on Ubuntu 18.04
* Verified Checksums
(CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
* Successful RAT check
* Ran some basic HDFS shell commands
* Ran some basic YARN shell commands
* Browsed through UI (NN ,DN, RM, NM & JHS)
* Tried some commands on Hive using Hive built on hive-master
* Verified Signature: Says Good Signature but "This key is not certified
with a trusted signature!"
* Ran some MR example jobs(TeraGen, TeraSort, TeraValidate, WordCount,
WordMean & Pi)
* Version & commit hash seems correct in UI as well as in hadoop version
output.
* Browsed through the ChangeLog & Release Notes (One place mentions hadoop
3.4.0 though, but we can survive I suppose)
* Browsed through the documentation.

Thanx Steve for driving the release, Good Luck!!!

-Ayush



On Tue, 3 May 2022 at 16:54, Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Viraj Jasani <vj...@apache.org>.

+1 (non-binding),

With a minor change <https://github.com/apache/hadoop/pull/4272> in
hadoop-vote,

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

HDFS and MapReduce functional testing looks good.

As per PR#4268 <https://github.com/apache/hadoop/pull/4268>, except for a
few flakes, TestDistributedShell and TestCsiClient are consistently failing.


On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.

+1 (binding)
Built tar from source in mac os and 1.8.0_161.   good

mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true


Able to access cloud s3 and abfs storage through command line.

Ran S3 and Abfs integration tests. All good.



On Tue, May 3, 2022 at 6:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by "Jones, Daniel Carl" <dj...@amazon.co.uk.INVALID>.

+1 (advisory)

* Verified sha512 checksum was correct for source tarball
* Verified signature was correct for source tarball (not verified trust)
* Built source code from tarball on Ubuntu 20.04 (x86) and OpenJDK 8 in Amazon EC2
* Verified source tarball matches content of RC0 tag on GitHub
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing from src
* Verified S3A (hadoop-tools/hadoop-aws) integ tests passing from src against Amazon S3 in eu-west-1
* There is a single failure, already known and described in https://issues.apache.org/jira/browse/HADOOP-18168

- Danny

On 03/05/2022, 12:25, "Steve Loughran" <st...@cloudera.com> wrote:

I have put together a release candidate (rc0) for Hadoop 3.3.3

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/

The git tag is release-3.3.3-RC0, commit d37586cbda3

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1348/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.

- The critical fixes which shipped in the 3.2.3 release.
- CVEs in our code and dependencies
- Shaded client packaging issues.
- A switch from log4j to reload4j

reload4j is an active fork of the log4j 1.17 library with the classes which
contain CVEs removed. Even though hadoop never used those classes, they
regularly raised alerts on security scans and concen from users. Switching
to the forked project allows us to ship a secure logging framework. It will
complicate the builds of downstream maven/ivy/gradle projects which exclude
our log4j artifacts, as they need to cut the new dependency instead/as well.

See the release notes for details.

This is my first release through the new docker build process, do please
validate artifact signing &c to make sure it is good. I'll be trying builds
of downstream projects.

We know there are some outstanding issues with at least one library we are
shipping (okhttp), but I don't want to hold this release up for it. If the
docker based release process works smoothly enough we can do a followup
security release in a few weeks.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

+1,
* Built from source
* Successful native build on Ubuntu 18.04
* Verified Checksums
(CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
* Successful RAT check
* Ran some basic HDFS shell commands
* Ran some basic YARN shell commands
* Browsed through UI (NN ,DN, RM, NM & JHS)
* Tried some commands on Hive using Hive built on hive-master
* Verified Signature: Says Good Signature but "This key is not certified
with a trusted signature!"
* Ran some MR example jobs(TeraGen, TeraSort, TeraValidate, WordCount,
WordMean & Pi)
* Version & commit hash seems correct in UI as well as in hadoop version
output.
* Browsed through the ChangeLog & Release Notes (One place mentions hadoop
3.4.0 though, but we can survive I suppose)
* Browsed through the documentation.

Thanx Steve for driving the release, Good Luck!!!

-Ayush



On Tue, 3 May 2022 at 16:54, Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Viraj Jasani <vj...@apache.org>.

+1 (non-binding),

With a minor change <https://github.com/apache/hadoop/pull/4272> in
hadoop-vote,

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

HDFS and MapReduce functional testing looks good.

As per PR#4268 <https://github.com/apache/hadoop/pull/4268>, except for a
few flakes, TestDistributedShell and TestCsiClient are consistently failing.


On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

+1,
* Built from source
* Successful native build on Ubuntu 18.04
* Verified Checksums
(CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
* Successful RAT check
* Ran some basic HDFS shell commands
* Ran some basic YARN shell commands
* Browsed through UI (NN ,DN, RM, NM & JHS)
* Tried some commands on Hive using Hive built on hive-master
* Verified Signature: Says Good Signature but "This key is not certified
with a trusted signature!"
* Ran some MR example jobs(TeraGen, TeraSort, TeraValidate, WordCount,
WordMean & Pi)
* Version & commit hash seems correct in UI as well as in hadoop version
output.
* Browsed through the ChangeLog & Release Notes (One place mentions hadoop
3.4.0 though, but we can survive I suppose)
* Browsed through the documentation.

Thanx Steve for driving the release, Good Luck!!!

-Ayush



On Tue, 3 May 2022 at 16:54, Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.

+1 (binding)
Built tar from source in mac os and 1.8.0_161.   good

mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true


Able to access cloud s3 and abfs storage through command line.

Ran S3 and Abfs integration tests. All good.



On Tue, May 3, 2022 at 6:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Stack <st...@duboce.net>.

+1 (binding)

  * Signature: ok
  * Checksum : passed
  * Rat check (1.8.0_191): passed
   - mvn clean apache-rat:check
  * Built from source (1.8.0_191): failed
   - mvn clean install  -DskipTests
   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
-Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop clean install
  * Unit tests pass (1.8.0_191):
    - HDFS Tests passed (Didn't run more than this).

Deployed a ten node ha hdfs cluster with three namenodes and five
journalnodes. Ran a ten node hbase (older version of 2.5 branch built
against 3.3.2) against it. Tried a small verification job. Good. Ran a
bigger job with mild chaos. All seems to be working properly (recoveries,
logs look fine). Killed a namenode. Failover worked promptly. UIs look
good. Poked at the hdfs cli. Seems good.

S

On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

+1 (binding)

* verified signature and checksum of the source tarball.
* built the source code on Rocky Linux 8 (x86_64) and OpenJDK 8
   by `mvn install -DskipTests -Pnative -Pdist`.
* launched pseudo distributed cluster with Kerberos security enabled and ran sample MR jobs.
* launched HA enabled 3-nodes docker cluster and ran sample MR jobs.
* launched pseudo distributed cluster and `spark-shell --master yarn`
   with spark-3.2.1-bin-without-hadoop and ran some tutorial code.
* built site documentation by `mvn site site:stage -Preleasedocs` and skimmed the contents.

Thanks,
Masatake Iwasaki

On 2022/05/03 20:18, Steve Loughran wrote:
> I have put together a release candidate (rc0) for Hadoop 3.3.3
> 
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> 
> The git tag is release-3.3.3-RC0, commit d37586cbda3
> 
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> 
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> 
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> 
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
> 
> 
>     - The critical fixes which shipped in the 3.2.3 release.
>     -  CVEs in our code and dependencies
>     - Shaded client packaging issues.
>     - A switch from log4j to reload4j
> 
> 
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as well.
> 
> See the release notes for details.
> 
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
> 
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> -Steve
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Viraj Jasani <vj...@apache.org>.

+1 (non-binding),

With a minor change <https://github.com/apache/hadoop/pull/4272> in
hadoop-vote,

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

HDFS and MapReduce functional testing looks good.

As per PR#4268 <https://github.com/apache/hadoop/pull/4268>, except for a
few flakes, TestDistributedShell and TestCsiClient are consistently failing.


On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Stack <st...@duboce.net>.

+1 (binding)

  * Signature: ok
  * Checksum : passed
  * Rat check (1.8.0_191): passed
   - mvn clean apache-rat:check
  * Built from source (1.8.0_191): failed
   - mvn clean install  -DskipTests
   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
-Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop clean install
  * Unit tests pass (1.8.0_191):
    - HDFS Tests passed (Didn't run more than this).

Deployed a ten node ha hdfs cluster with three namenodes and five
journalnodes. Ran a ten node hbase (older version of 2.5 branch built
against 3.3.2) against it. Tried a small verification job. Good. Ran a
bigger job with mild chaos. All seems to be working properly (recoveries,
logs look fine). Killed a namenode. Failover worked promptly. UIs look
good. Poked at the hdfs cli. Seems good.

S

On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

+1 (binding)

* verified signature and checksum of the source tarball.
* built the source code on Rocky Linux 8 (x86_64) and OpenJDK 8
   by `mvn install -DskipTests -Pnative -Pdist`.
* launched pseudo distributed cluster with Kerberos security enabled and ran sample MR jobs.
* launched HA enabled 3-nodes docker cluster and ran sample MR jobs.
* launched pseudo distributed cluster and `spark-shell --master yarn`
   with spark-3.2.1-bin-without-hadoop and ran some tutorial code.
* built site documentation by `mvn site site:stage -Preleasedocs` and skimmed the contents.

Thanks,
Masatake Iwasaki

On 2022/05/03 20:18, Steve Loughran wrote:
> I have put together a release candidate (rc0) for Hadoop 3.3.3
> 
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> 
> The git tag is release-3.3.3-RC0, commit d37586cbda3
> 
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> 
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> 
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> 
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
> 
> 
>     - The critical fixes which shipped in the 3.2.3 release.
>     -  CVEs in our code and dependencies
>     - Shaded client packaging issues.
>     - A switch from log4j to reload4j
> 
> 
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as well.
> 
> See the release notes for details.
> 
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
> 
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> -Steve
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Ayush Saxena <ay...@gmail.com>.

+1,
* Built from source
* Successful native build on Ubuntu 18.04
* Verified Checksums
(CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
* Successful RAT check
* Ran some basic HDFS shell commands
* Ran some basic YARN shell commands
* Browsed through UI (NN ,DN, RM, NM & JHS)
* Tried some commands on Hive using Hive built on hive-master
* Verified Signature: Says Good Signature but "This key is not certified
with a trusted signature!"
* Ran some MR example jobs(TeraGen, TeraSort, TeraValidate, WordCount,
WordMean & Pi)
* Version & commit hash seems correct in UI as well as in hadoop version
output.
* Browsed through the ChangeLog & Release Notes (One place mentions hadoop
3.4.0 though, but we can survive I suppose)
* Browsed through the documentation.

Thanx Steve for driving the release, Good Luck!!!

-Ayush



On Tue, 3 May 2022 at 16:54, Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Stack <st...@duboce.net>.

+1 (binding)

  * Signature: ok
  * Checksum : passed
  * Rat check (1.8.0_191): passed
   - mvn clean apache-rat:check
  * Built from source (1.8.0_191): failed
   - mvn clean install  -DskipTests
   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
-Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop clean install
  * Unit tests pass (1.8.0_191):
    - HDFS Tests passed (Didn't run more than this).

Deployed a ten node ha hdfs cluster with three namenodes and five
journalnodes. Ran a ten node hbase (older version of 2.5 branch built
against 3.3.2) against it. Tried a small verification job. Good. Ran a
bigger job with mild chaos. All seems to be working properly (recoveries,
logs look fine). Killed a namenode. Failover worked promptly. UIs look
good. Poked at the hdfs cli. Seems good.

S

On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Viraj Jasani <vj...@apache.org>.

+1 (non-binding),

With a minor change <https://github.com/apache/hadoop/pull/4272> in
hadoop-vote,

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

HDFS and MapReduce functional testing looks good.

As per PR#4268 <https://github.com/apache/hadoop/pull/4268>, except for a
few flakes, TestDistributedShell and TestCsiClient are consistently failing.


On Tue, May 3, 2022 at 4:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

+1 (binding)

* verified signature and checksum of the source tarball.
* built the source code on Rocky Linux 8 (x86_64) and OpenJDK 8
   by `mvn install -DskipTests -Pnative -Pdist`.
* launched pseudo distributed cluster with Kerberos security enabled and ran sample MR jobs.
* launched HA enabled 3-nodes docker cluster and ran sample MR jobs.
* launched pseudo distributed cluster and `spark-shell --master yarn`
   with spark-3.2.1-bin-without-hadoop and ran some tutorial code.
* built site documentation by `mvn site site:stage -Preleasedocs` and skimmed the contents.

Thanks,
Masatake Iwasaki

On 2022/05/03 20:18, Steve Loughran wrote:
> I have put together a release candidate (rc0) for Hadoop 3.3.3
> 
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> 
> The git tag is release-3.3.3-RC0, commit d37586cbda3
> 
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> 
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> 
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> 
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
> 
> 
>     - The critical fixes which shipped in the 3.2.3 release.
>     -  CVEs in our code and dependencies
>     - Shaded client packaging issues.
>     - A switch from log4j to reload4j
> 
> 
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as well.
> 
> See the release notes for details.
> 
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
> 
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> -Steve
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.

+1 (binding)

* verified signature and checksum of the source tarball.
* built the source code on Rocky Linux 8 (x86_64) and OpenJDK 8
   by `mvn install -DskipTests -Pnative -Pdist`.
* launched pseudo distributed cluster with Kerberos security enabled and ran sample MR jobs.
* launched HA enabled 3-nodes docker cluster and ran sample MR jobs.
* launched pseudo distributed cluster and `spark-shell --master yarn`
   with spark-3.2.1-bin-without-hadoop and ran some tutorial code.
* built site documentation by `mvn site site:stage -Preleasedocs` and skimmed the contents.

Thanks,
Masatake Iwasaki

On 2022/05/03 20:18, Steve Loughran wrote:
> I have put together a release candidate (rc0) for Hadoop 3.3.3
> 
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> 
> The git tag is release-3.3.3-RC0, commit d37586cbda3
> 
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> 
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> 
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> 
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
> 
> 
>     - The critical fixes which shipped in the 3.2.3 release.
>     -  CVEs in our code and dependencies
>     - Shaded client packaging issues.
>     - A switch from log4j to reload4j
> 
> 
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as well.
> 
> See the release notes for details.
> 
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
> 
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> -Steve
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.

+1 (binding)
Built tar from source in mac os and 1.8.0_161.   good

mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true


Able to access cloud s3 and abfs storage through command line.

Ran S3 and Abfs integration tests. All good.



On Tue, May 3, 2022 at 6:24 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>    - The critical fixes which shipped in the 3.2.3 release.
>    -  CVEs in our code and dependencies
>    - Shaded client packaging issues.
>    - A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>