You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Steve Loughran <st...@cloudera.com.INVALID> on 2023/02/27 17:58:33 UTC
[VOTE] Release Apache Hadoop 3.3.5 (RC2)
Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
We need anyone who can to verify the source and binary artifacts,
including those JARs staged on maven, the site documentation and the arm64
tar file.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
The git tag is release-3.3.5-RC2, commit 72f8c2a4888
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/
You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
This is off branch-3.3 and is the first big release since 3.3.2.
As to what changed since the RC1 attempt last week
1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
issues in maven 3.9.0)
4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).
The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.
Please try the release and vote. The vote will run for 5 days.
Steve and Mukund
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
well, lets see what others say.
we don't want to ship stuff with serious regression to hdfs.
I will highlight that I am completely fed up with doing this release and
really want to get it out the way -for which I depend on support from as
many other developers as possible.
Erik, right now what you can help by doing is test all the rest of the
release knowing that this issue exists and seeing if you can identify
anything else. That way this update will be the sole blocker and we can get
through that next RC with nothing else surfacing.
I had noticed that the arm64 release somehow missed out the native binaries
and was going to investigate that but didn't consider that a blocker… I was
just going to cut that artefact and, post Darcy, create a new arm64 release
using all the jars of the x86 build but replacing the x86 native libs with
the arm versions. This helps ensure that the JAR files in the wild all
match, which strikes me as a good thing.
Can I also encourage people in the HFS team to put their hand up and
volunteer for leading the next release, with a goal of getting something
out later this year.
On Thu, 2 Mar 2023 at 00:27, Erik Krogen <xk...@apache.org> wrote:
> Hi folks, apologies for being late to the release conversation, but I think
> we need to get HDFS-16923
> <https://issues.apache.org/jira/browse/HDFS-16923> into
> 3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>,
> which
> also went into 3.3.5, introduced an issue whereby Observer NameNodes will
> throw NPE upon any getListing call on a directory that doesn't exist. It
> will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
> for this and it has been merged to trunk/branch-3.3 as of a few minutes
> ago. I'd like to see about merging to branch-3.3.5 as well.
>
> Thanks for the consideration and sorry for not bringing this up in RC1 or
> earlier.
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
well, lets see what others say.
we don't want to ship stuff with serious regression to hdfs.
I will highlight that I am completely fed up with doing this release and
really want to get it out the way -for which I depend on support from as
many other developers as possible.
Erik, right now what you can help by doing is test all the rest of the
release knowing that this issue exists and seeing if you can identify
anything else. That way this update will be the sole blocker and we can get
through that next RC with nothing else surfacing.
I had noticed that the arm64 release somehow missed out the native binaries
and was going to investigate that but didn't consider that a blocker… I was
just going to cut that artefact and, post Darcy, create a new arm64 release
using all the jars of the x86 build but replacing the x86 native libs with
the arm versions. This helps ensure that the JAR files in the wild all
match, which strikes me as a good thing.
Can I also encourage people in the HFS team to put their hand up and
volunteer for leading the next release, with a goal of getting something
out later this year.
On Thu, 2 Mar 2023 at 00:27, Erik Krogen <xk...@apache.org> wrote:
> Hi folks, apologies for being late to the release conversation, but I think
> we need to get HDFS-16923
> <https://issues.apache.org/jira/browse/HDFS-16923> into
> 3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>,
> which
> also went into 3.3.5, introduced an issue whereby Observer NameNodes will
> throw NPE upon any getListing call on a directory that doesn't exist. It
> will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
> for this and it has been merged to trunk/branch-3.3 as of a few minutes
> ago. I'd like to see about merging to branch-3.3.5 as well.
>
> Thanks for the consideration and sorry for not bringing this up in RC1 or
> earlier.
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
well, lets see what others say.
we don't want to ship stuff with serious regression to hdfs.
I will highlight that I am completely fed up with doing this release and
really want to get it out the way -for which I depend on support from as
many other developers as possible.
Erik, right now what you can help by doing is test all the rest of the
release knowing that this issue exists and seeing if you can identify
anything else. That way this update will be the sole blocker and we can get
through that next RC with nothing else surfacing.
I had noticed that the arm64 release somehow missed out the native binaries
and was going to investigate that but didn't consider that a blocker… I was
just going to cut that artefact and, post Darcy, create a new arm64 release
using all the jars of the x86 build but replacing the x86 native libs with
the arm versions. This helps ensure that the JAR files in the wild all
match, which strikes me as a good thing.
Can I also encourage people in the HFS team to put their hand up and
volunteer for leading the next release, with a goal of getting something
out later this year.
On Thu, 2 Mar 2023 at 00:27, Erik Krogen <xk...@apache.org> wrote:
> Hi folks, apologies for being late to the release conversation, but I think
> we need to get HDFS-16923
> <https://issues.apache.org/jira/browse/HDFS-16923> into
> 3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>,
> which
> also went into 3.3.5, introduced an issue whereby Observer NameNodes will
> throw NPE upon any getListing call on a directory that doesn't exist. It
> will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
> for this and it has been merged to trunk/branch-3.3 as of a few minutes
> ago. I'd like to see about merging to branch-3.3.5 as well.
>
> Thanks for the consideration and sorry for not bringing this up in RC1 or
> earlier.
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
well, lets see what others say.
we don't want to ship stuff with serious regression to hdfs.
I will highlight that I am completely fed up with doing this release and
really want to get it out the way -for which I depend on support from as
many other developers as possible.
Erik, right now what you can help by doing is test all the rest of the
release knowing that this issue exists and seeing if you can identify
anything else. That way this update will be the sole blocker and we can get
through that next RC with nothing else surfacing.
I had noticed that the arm64 release somehow missed out the native binaries
and was going to investigate that but didn't consider that a blocker… I was
just going to cut that artefact and, post Darcy, create a new arm64 release
using all the jars of the x86 build but replacing the x86 native libs with
the arm versions. This helps ensure that the JAR files in the wild all
match, which strikes me as a good thing.
Can I also encourage people in the HFS team to put their hand up and
volunteer for leading the next release, with a goal of getting something
out later this year.
On Thu, 2 Mar 2023 at 00:27, Erik Krogen <xk...@apache.org> wrote:
> Hi folks, apologies for being late to the release conversation, but I think
> we need to get HDFS-16923
> <https://issues.apache.org/jira/browse/HDFS-16923> into
> 3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>,
> which
> also went into 3.3.5, introduced an issue whereby Observer NameNodes will
> throw NPE upon any getListing call on a directory that doesn't exist. It
> will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
> for this and it has been merged to trunk/branch-3.3 as of a few minutes
> ago. I'd like to see about merging to branch-3.3.5 as well.
>
> Thanks for the consideration and sorry for not bringing this up in RC1 or
> earlier.
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Hi folks, apologies for being late to the release conversation, but I think
we need to get HDFS-16923
<https://issues.apache.org/jira/browse/HDFS-16923> into
3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>, which
also went into 3.3.5, introduced an issue whereby Observer NameNodes will
throw NPE upon any getListing call on a directory that doesn't exist. It
will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
for this and it has been merged to trunk/branch-3.3 as of a few minutes
ago. I'd like to see about merging to branch-3.3.5 as well.
Thanks for the consideration and sorry for not bringing this up in RC1 or
earlier.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Hi folks, apologies for being late to the release conversation, but I think
we need to get HDFS-16923
<https://issues.apache.org/jira/browse/HDFS-16923> into
3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>, which
also went into 3.3.5, introduced an issue whereby Observer NameNodes will
throw NPE upon any getListing call on a directory that doesn't exist. It
will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
for this and it has been merged to trunk/branch-3.3 as of a few minutes
ago. I'd like to see about merging to branch-3.3.5 as well.
Thanks for the consideration and sorry for not bringing this up in RC1 or
earlier.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Hi folks, apologies for being late to the release conversation, but I think
we need to get HDFS-16923
<https://issues.apache.org/jira/browse/HDFS-16923> into
3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>, which
also went into 3.3.5, introduced an issue whereby Observer NameNodes will
throw NPE upon any getListing call on a directory that doesn't exist. It
will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
for this and it has been merged to trunk/branch-3.3 as of a few minutes
ago. I'd like to see about merging to branch-3.3.5 as well.
Thanks for the consideration and sorry for not bringing this up in RC1 or
earlier.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks.
now looking at a critical kerby CVE (
https://github.com/apache/hadoop/pull/5458) and revisited one for netty
from last week
i am never a fan of last-minute jar updates, but if we don't ship with them
we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18
months
On Mon, 6 Mar 2023 at 23:29, Erik Krogen <xk...@apache.org> wrote:
> > OK. Could you have a go with a (locally built) patch release
>
> Just validated the same on the latest HEAD of branch-3.3.5, which includes
> the two HDFS Jiras I mentioned plus one additional one:
>
> * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
> TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
> 55643692+slfan1989@users.noreply.github.com>]
> * d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
> will throw NPE if path does not exist (#5400) [ZanderXu <
> zanderxu@apache.org
> >]
> * 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
> Fix NPE when check the block location of empty directory (#5099)
> [zhengchenyu <zh...@gmail.com>]
> * 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
> connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
> stevel@cloudera.com>]
>
> On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > i looked at that test and wondered if it it was just being brittle to
> > time. I'm not a fan of those -there's one in abfs which is particularly
> bad
> > for me- maybe we could see if the test can be cut as it is quite a slow
> one
> >
> > On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> >
> > > A minor update on ITestS3AConcurrentOps#testParallelRename
> > >
> > > I was previously connected to a vpn due to which bandwidth was getting
> > > throttled earlier. Ran the test again today without vpn and had no
> issues
> > > (earlier only 40% of the overall putObject were able to get completed
> > > within timeout).
> > >
> > >
> > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > > >
> > > > > Thanks Steve. I see now that the branch cut was way back in October
> > so
> > > I
> > > > > definitely understand your frustration here!
> > > > >
> > > > > This made me realize that HDFS-16832
> > > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which
> resolves a
> > > > very
> > > > > similar issue as the aforementioned HDFS-16923, is also missing
> from
> > > the
> > > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > > before
> > > > > the initial 3.3.5 RC was made and I didn't notice the branch was
> cut.
> > > My
> > > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > > >
> > > >
> > > > thanks.
> > > >
> > > > >
> > > > > In the meantime, I tested for RC2 that a local cluster of NN +
> > standby
> > > +
> > > > > observer + QJM works as expected for some basic HDFS commands.
> > > > >
> > > >
> > > > OK. Could you have a go with a (locally built) patch release
> > > >
> > > > >
> > > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > > <st...@cloudera.com.invalid>
> > > > > wrote:
> > > > >
> > > > >> shipping broken hdfs isn't something we'd want to do, but if we
> can
> > be
> > > > >> confident that all other issues can be addressed in RC3 then I'd
> be
> > > > happy.
> > > > >>
> > > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > I will highlight that I am completely fed up with doing this
> > > release
> > > > >> and
> > > > >> >> really want to get it out the way -for which I depend on
> support
> > > from
> > > > >> as
> > > > >> >> many other developers as possible.
> > > > >> >
> > > > >> >
> > > > >> > hmm, I can feel the pain. I tried to find if there is any config
> > or
> > > > any
> > > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > > couldn't
> > > > >> find
> > > > >> > any. If someone does a getListing with needLocation and the file
> > > > doesn't
> > > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> > isn't
> > > > just
> > > > >> > the exception, AFAIK Observer reads have some logic around
> > handling
> > > > FNF
> > > > >> > specifically, that it attempts Active NN or something like that
> in
> > > > such
> > > > >> > cases, So, that will be broken as well for this use case.
> > > > >> >
> > > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > > side,
> > > > >> and
> > > > >> > it has already been too much work on your side, so you can argue
> > > that
> > > > it
> > > > >> > might not be a very frequent use case or so. It's your call.
> > > > >> >
> > > > >> > Just sharing, no intentions of saying you should do that, But as
> > an
> > > RM
> > > > >> > "nobody" can force you for a new iteration of a RC, it is gonna
> be
> > > > your
> > > > >> > call and discretion. As far as I know a release can not be
> vetoed
> > by
> > > > >> > "anybody" as per the apache by laws.(
> > > > >> >
> https://www.apache.org/legal/release-policy.html#release-approval
> > ).
> > > > >> Even
> > > > >> > our bylaws say that product release requires a Lazy Majority
> not a
> > > > >> > Consensus Approval.
> > > > >> >
> > > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > > you a
> > > > >> > pass, in case you are really in a state: ''Get me out of this
> > mess"
> > > > >> state,
> > > > >> > my basic validations on x86 & Aarch64 both are passing as of
> now,
> > > > >> couldn't
> > > > >> > reach the end for any of the RC's
> > > > >> >
> > > > >> > -Ayush
> > > > >> >
> > > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> >> While this RC is not going to be final, I just wanted to share
> > the
> > > > >> results
> > > > >> >> of the testing I have done so far with RC1 and RC2.
> > > > >> >>
> > > > >> >> * Signature: ok
> > > > >> >> * Checksum : ok
> > > > >> >> * Rat check (1.8.0_341): ok
> > > > >> >> - mvn clean apache-rat:check
> > > > >> >> * Built from source (1.8.0_341): ok
> > > > >> >> - mvn clean install -DskipTests
> > > > >> >> * Built tar from source (1.8.0_341): ok
> > > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > > >> -Dmaven.javadoc.skip=true
> > > > >> >>
> > > > >> >> * Built images using the tarball, installed and started all of
> > > Hdfs,
> > > > >> JHS
> > > > >> >> and Yarn components
> > > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > > Mapreduce
> > > > >> job
> > > > >> >> * Hdfs CRUD tests
> > > > >> >> * MapReduce wordcount job
> > > > >> >>
> > > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > > >> >>
> > > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> > ~960s.
> > > > >> This
> > > > >> >> is
> > > > >> >> consistently failing, looks like a recent regression.
> > > > >> >> I was also able to repro on trunk, will create Jira.
> > > > >> >>
> > > > >> >>
> > > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > > >> >> <st...@cloudera.com.invalid>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > > Hadoop
> > > > >> >> 3.3.5.
> > > > >> >> >
> > > > >> >> > We need anyone who can to verify the source and binary
> > artifacts,
> > > > >> >> > including those JARs staged on maven, the site documentation
> > and
> > > > the
> > > > >> >> arm64
> > > > >> >> > tar file.
> > > > >> >> >
> > > > >> >> > The RC is available at:
> > > > >> >> >
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > > >> >> >
> > > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > > >> >> >
> > > > >> >> > The maven artifacts are staged at
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > > >> >> >
> > > > >> >> > You can find my public key at:
> > > > >> >> >
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >> >> >
> > > > >> >> > Change log
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > > >> >> >
> > > > >> >> > Release notes
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > > >> >> >
> > > > >> >> > This is off branch-3.3 and is the first big release since
> > 3.3.2.
> > > > >> >> >
> > > > >> >> > As to what changed since the RC1 attempt last week
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > > there)
> > > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5
> index.md
> > > file
> > > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > > (creating
> > > > >> >> build
> > > > >> >> > issues in maven 3.9.0)
> > > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> > fixup.
> > > > >> >> (#5429)
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Note, because the arm64 binaries are built separately on a
> > > > different
> > > > >> >> > platform and JVM, their jar files may not match those of the
> > x86
> > > > >> >> > release -and therefore the maven artifacts. I don't think
> this
> > is
> > > > >> >> > an issue (the ASF actually releases source tarballs, the
> > binaries
> > > > are
> > > > >> >> > there for help only, though with the maven repo that's a bit
> > > > >> blurred).
> > > > >> >> >
> > > > >> >> > The only way to be consistent would actually untar the
> > > x86.tar.gz,
> > > > >> >> > overwrite its binaries with the arm stuff, retar, sign and
> push
> > > out
> > > > >> >> > for the vote. Even automating that would be risky.
> > > > >> >> >
> > > > >> >> > Please try the release and vote. The vote will run for 5
> days.
> > > > >> >> >
> > > > >> >> > Steve and Mukund
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks.
now looking at a critical kerby CVE (
https://github.com/apache/hadoop/pull/5458) and revisited one for netty
from last week
i am never a fan of last-minute jar updates, but if we don't ship with them
we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18
months
On Mon, 6 Mar 2023 at 23:29, Erik Krogen <xk...@apache.org> wrote:
> > OK. Could you have a go with a (locally built) patch release
>
> Just validated the same on the latest HEAD of branch-3.3.5, which includes
> the two HDFS Jiras I mentioned plus one additional one:
>
> * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
> TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
> 55643692+slfan1989@users.noreply.github.com>]
> * d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
> will throw NPE if path does not exist (#5400) [ZanderXu <
> zanderxu@apache.org
> >]
> * 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
> Fix NPE when check the block location of empty directory (#5099)
> [zhengchenyu <zh...@gmail.com>]
> * 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
> connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
> stevel@cloudera.com>]
>
> On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > i looked at that test and wondered if it it was just being brittle to
> > time. I'm not a fan of those -there's one in abfs which is particularly
> bad
> > for me- maybe we could see if the test can be cut as it is quite a slow
> one
> >
> > On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> >
> > > A minor update on ITestS3AConcurrentOps#testParallelRename
> > >
> > > I was previously connected to a vpn due to which bandwidth was getting
> > > throttled earlier. Ran the test again today without vpn and had no
> issues
> > > (earlier only 40% of the overall putObject were able to get completed
> > > within timeout).
> > >
> > >
> > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > > >
> > > > > Thanks Steve. I see now that the branch cut was way back in October
> > so
> > > I
> > > > > definitely understand your frustration here!
> > > > >
> > > > > This made me realize that HDFS-16832
> > > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which
> resolves a
> > > > very
> > > > > similar issue as the aforementioned HDFS-16923, is also missing
> from
> > > the
> > > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > > before
> > > > > the initial 3.3.5 RC was made and I didn't notice the branch was
> cut.
> > > My
> > > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > > >
> > > >
> > > > thanks.
> > > >
> > > > >
> > > > > In the meantime, I tested for RC2 that a local cluster of NN +
> > standby
> > > +
> > > > > observer + QJM works as expected for some basic HDFS commands.
> > > > >
> > > >
> > > > OK. Could you have a go with a (locally built) patch release
> > > >
> > > > >
> > > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > > <st...@cloudera.com.invalid>
> > > > > wrote:
> > > > >
> > > > >> shipping broken hdfs isn't something we'd want to do, but if we
> can
> > be
> > > > >> confident that all other issues can be addressed in RC3 then I'd
> be
> > > > happy.
> > > > >>
> > > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > I will highlight that I am completely fed up with doing this
> > > release
> > > > >> and
> > > > >> >> really want to get it out the way -for which I depend on
> support
> > > from
> > > > >> as
> > > > >> >> many other developers as possible.
> > > > >> >
> > > > >> >
> > > > >> > hmm, I can feel the pain. I tried to find if there is any config
> > or
> > > > any
> > > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > > couldn't
> > > > >> find
> > > > >> > any. If someone does a getListing with needLocation and the file
> > > > doesn't
> > > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> > isn't
> > > > just
> > > > >> > the exception, AFAIK Observer reads have some logic around
> > handling
> > > > FNF
> > > > >> > specifically, that it attempts Active NN or something like that
> in
> > > > such
> > > > >> > cases, So, that will be broken as well for this use case.
> > > > >> >
> > > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > > side,
> > > > >> and
> > > > >> > it has already been too much work on your side, so you can argue
> > > that
> > > > it
> > > > >> > might not be a very frequent use case or so. It's your call.
> > > > >> >
> > > > >> > Just sharing, no intentions of saying you should do that, But as
> > an
> > > RM
> > > > >> > "nobody" can force you for a new iteration of a RC, it is gonna
> be
> > > > your
> > > > >> > call and discretion. As far as I know a release can not be
> vetoed
> > by
> > > > >> > "anybody" as per the apache by laws.(
> > > > >> >
> https://www.apache.org/legal/release-policy.html#release-approval
> > ).
> > > > >> Even
> > > > >> > our bylaws say that product release requires a Lazy Majority
> not a
> > > > >> > Consensus Approval.
> > > > >> >
> > > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > > you a
> > > > >> > pass, in case you are really in a state: ''Get me out of this
> > mess"
> > > > >> state,
> > > > >> > my basic validations on x86 & Aarch64 both are passing as of
> now,
> > > > >> couldn't
> > > > >> > reach the end for any of the RC's
> > > > >> >
> > > > >> > -Ayush
> > > > >> >
> > > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> >> While this RC is not going to be final, I just wanted to share
> > the
> > > > >> results
> > > > >> >> of the testing I have done so far with RC1 and RC2.
> > > > >> >>
> > > > >> >> * Signature: ok
> > > > >> >> * Checksum : ok
> > > > >> >> * Rat check (1.8.0_341): ok
> > > > >> >> - mvn clean apache-rat:check
> > > > >> >> * Built from source (1.8.0_341): ok
> > > > >> >> - mvn clean install -DskipTests
> > > > >> >> * Built tar from source (1.8.0_341): ok
> > > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > > >> -Dmaven.javadoc.skip=true
> > > > >> >>
> > > > >> >> * Built images using the tarball, installed and started all of
> > > Hdfs,
> > > > >> JHS
> > > > >> >> and Yarn components
> > > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > > Mapreduce
> > > > >> job
> > > > >> >> * Hdfs CRUD tests
> > > > >> >> * MapReduce wordcount job
> > > > >> >>
> > > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > > >> >>
> > > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> > ~960s.
> > > > >> This
> > > > >> >> is
> > > > >> >> consistently failing, looks like a recent regression.
> > > > >> >> I was also able to repro on trunk, will create Jira.
> > > > >> >>
> > > > >> >>
> > > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > > >> >> <st...@cloudera.com.invalid>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > > Hadoop
> > > > >> >> 3.3.5.
> > > > >> >> >
> > > > >> >> > We need anyone who can to verify the source and binary
> > artifacts,
> > > > >> >> > including those JARs staged on maven, the site documentation
> > and
> > > > the
> > > > >> >> arm64
> > > > >> >> > tar file.
> > > > >> >> >
> > > > >> >> > The RC is available at:
> > > > >> >> >
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > > >> >> >
> > > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > > >> >> >
> > > > >> >> > The maven artifacts are staged at
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > > >> >> >
> > > > >> >> > You can find my public key at:
> > > > >> >> >
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >> >> >
> > > > >> >> > Change log
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > > >> >> >
> > > > >> >> > Release notes
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > > >> >> >
> > > > >> >> > This is off branch-3.3 and is the first big release since
> > 3.3.2.
> > > > >> >> >
> > > > >> >> > As to what changed since the RC1 attempt last week
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > > there)
> > > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5
> index.md
> > > file
> > > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > > (creating
> > > > >> >> build
> > > > >> >> > issues in maven 3.9.0)
> > > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> > fixup.
> > > > >> >> (#5429)
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Note, because the arm64 binaries are built separately on a
> > > > different
> > > > >> >> > platform and JVM, their jar files may not match those of the
> > x86
> > > > >> >> > release -and therefore the maven artifacts. I don't think
> this
> > is
> > > > >> >> > an issue (the ASF actually releases source tarballs, the
> > binaries
> > > > are
> > > > >> >> > there for help only, though with the maven repo that's a bit
> > > > >> blurred).
> > > > >> >> >
> > > > >> >> > The only way to be consistent would actually untar the
> > > x86.tar.gz,
> > > > >> >> > overwrite its binaries with the arm stuff, retar, sign and
> push
> > > out
> > > > >> >> > for the vote. Even automating that would be risky.
> > > > >> >> >
> > > > >> >> > Please try the release and vote. The vote will run for 5
> days.
> > > > >> >> >
> > > > >> >> > Steve and Mukund
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks.
now looking at a critical kerby CVE (
https://github.com/apache/hadoop/pull/5458) and revisited one for netty
from last week
i am never a fan of last-minute jar updates, but if we don't ship with them
we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18
months
On Mon, 6 Mar 2023 at 23:29, Erik Krogen <xk...@apache.org> wrote:
> > OK. Could you have a go with a (locally built) patch release
>
> Just validated the same on the latest HEAD of branch-3.3.5, which includes
> the two HDFS Jiras I mentioned plus one additional one:
>
> * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
> TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
> 55643692+slfan1989@users.noreply.github.com>]
> * d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
> will throw NPE if path does not exist (#5400) [ZanderXu <
> zanderxu@apache.org
> >]
> * 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
> Fix NPE when check the block location of empty directory (#5099)
> [zhengchenyu <zh...@gmail.com>]
> * 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
> connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
> stevel@cloudera.com>]
>
> On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > i looked at that test and wondered if it it was just being brittle to
> > time. I'm not a fan of those -there's one in abfs which is particularly
> bad
> > for me- maybe we could see if the test can be cut as it is quite a slow
> one
> >
> > On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> >
> > > A minor update on ITestS3AConcurrentOps#testParallelRename
> > >
> > > I was previously connected to a vpn due to which bandwidth was getting
> > > throttled earlier. Ran the test again today without vpn and had no
> issues
> > > (earlier only 40% of the overall putObject were able to get completed
> > > within timeout).
> > >
> > >
> > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > > >
> > > > > Thanks Steve. I see now that the branch cut was way back in October
> > so
> > > I
> > > > > definitely understand your frustration here!
> > > > >
> > > > > This made me realize that HDFS-16832
> > > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which
> resolves a
> > > > very
> > > > > similar issue as the aforementioned HDFS-16923, is also missing
> from
> > > the
> > > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > > before
> > > > > the initial 3.3.5 RC was made and I didn't notice the branch was
> cut.
> > > My
> > > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > > >
> > > >
> > > > thanks.
> > > >
> > > > >
> > > > > In the meantime, I tested for RC2 that a local cluster of NN +
> > standby
> > > +
> > > > > observer + QJM works as expected for some basic HDFS commands.
> > > > >
> > > >
> > > > OK. Could you have a go with a (locally built) patch release
> > > >
> > > > >
> > > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > > <st...@cloudera.com.invalid>
> > > > > wrote:
> > > > >
> > > > >> shipping broken hdfs isn't something we'd want to do, but if we
> can
> > be
> > > > >> confident that all other issues can be addressed in RC3 then I'd
> be
> > > > happy.
> > > > >>
> > > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > I will highlight that I am completely fed up with doing this
> > > release
> > > > >> and
> > > > >> >> really want to get it out the way -for which I depend on
> support
> > > from
> > > > >> as
> > > > >> >> many other developers as possible.
> > > > >> >
> > > > >> >
> > > > >> > hmm, I can feel the pain. I tried to find if there is any config
> > or
> > > > any
> > > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > > couldn't
> > > > >> find
> > > > >> > any. If someone does a getListing with needLocation and the file
> > > > doesn't
> > > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> > isn't
> > > > just
> > > > >> > the exception, AFAIK Observer reads have some logic around
> > handling
> > > > FNF
> > > > >> > specifically, that it attempts Active NN or something like that
> in
> > > > such
> > > > >> > cases, So, that will be broken as well for this use case.
> > > > >> >
> > > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > > side,
> > > > >> and
> > > > >> > it has already been too much work on your side, so you can argue
> > > that
> > > > it
> > > > >> > might not be a very frequent use case or so. It's your call.
> > > > >> >
> > > > >> > Just sharing, no intentions of saying you should do that, But as
> > an
> > > RM
> > > > >> > "nobody" can force you for a new iteration of a RC, it is gonna
> be
> > > > your
> > > > >> > call and discretion. As far as I know a release can not be
> vetoed
> > by
> > > > >> > "anybody" as per the apache by laws.(
> > > > >> >
> https://www.apache.org/legal/release-policy.html#release-approval
> > ).
> > > > >> Even
> > > > >> > our bylaws say that product release requires a Lazy Majority
> not a
> > > > >> > Consensus Approval.
> > > > >> >
> > > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > > you a
> > > > >> > pass, in case you are really in a state: ''Get me out of this
> > mess"
> > > > >> state,
> > > > >> > my basic validations on x86 & Aarch64 both are passing as of
> now,
> > > > >> couldn't
> > > > >> > reach the end for any of the RC's
> > > > >> >
> > > > >> > -Ayush
> > > > >> >
> > > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> >> While this RC is not going to be final, I just wanted to share
> > the
> > > > >> results
> > > > >> >> of the testing I have done so far with RC1 and RC2.
> > > > >> >>
> > > > >> >> * Signature: ok
> > > > >> >> * Checksum : ok
> > > > >> >> * Rat check (1.8.0_341): ok
> > > > >> >> - mvn clean apache-rat:check
> > > > >> >> * Built from source (1.8.0_341): ok
> > > > >> >> - mvn clean install -DskipTests
> > > > >> >> * Built tar from source (1.8.0_341): ok
> > > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > > >> -Dmaven.javadoc.skip=true
> > > > >> >>
> > > > >> >> * Built images using the tarball, installed and started all of
> > > Hdfs,
> > > > >> JHS
> > > > >> >> and Yarn components
> > > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > > Mapreduce
> > > > >> job
> > > > >> >> * Hdfs CRUD tests
> > > > >> >> * MapReduce wordcount job
> > > > >> >>
> > > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > > >> >>
> > > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> > ~960s.
> > > > >> This
> > > > >> >> is
> > > > >> >> consistently failing, looks like a recent regression.
> > > > >> >> I was also able to repro on trunk, will create Jira.
> > > > >> >>
> > > > >> >>
> > > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > > >> >> <st...@cloudera.com.invalid>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > > Hadoop
> > > > >> >> 3.3.5.
> > > > >> >> >
> > > > >> >> > We need anyone who can to verify the source and binary
> > artifacts,
> > > > >> >> > including those JARs staged on maven, the site documentation
> > and
> > > > the
> > > > >> >> arm64
> > > > >> >> > tar file.
> > > > >> >> >
> > > > >> >> > The RC is available at:
> > > > >> >> >
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > > >> >> >
> > > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > > >> >> >
> > > > >> >> > The maven artifacts are staged at
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > > >> >> >
> > > > >> >> > You can find my public key at:
> > > > >> >> >
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >> >> >
> > > > >> >> > Change log
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > > >> >> >
> > > > >> >> > Release notes
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > > >> >> >
> > > > >> >> > This is off branch-3.3 and is the first big release since
> > 3.3.2.
> > > > >> >> >
> > > > >> >> > As to what changed since the RC1 attempt last week
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > > there)
> > > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5
> index.md
> > > file
> > > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > > (creating
> > > > >> >> build
> > > > >> >> > issues in maven 3.9.0)
> > > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> > fixup.
> > > > >> >> (#5429)
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Note, because the arm64 binaries are built separately on a
> > > > different
> > > > >> >> > platform and JVM, their jar files may not match those of the
> > x86
> > > > >> >> > release -and therefore the maven artifacts. I don't think
> this
> > is
> > > > >> >> > an issue (the ASF actually releases source tarballs, the
> > binaries
> > > > are
> > > > >> >> > there for help only, though with the maven repo that's a bit
> > > > >> blurred).
> > > > >> >> >
> > > > >> >> > The only way to be consistent would actually untar the
> > > x86.tar.gz,
> > > > >> >> > overwrite its binaries with the arm stuff, retar, sign and
> push
> > > out
> > > > >> >> > for the vote. Even automating that would be risky.
> > > > >> >> >
> > > > >> >> > Please try the release and vote. The vote will run for 5
> days.
> > > > >> >> >
> > > > >> >> > Steve and Mukund
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks.
now looking at a critical kerby CVE (
https://github.com/apache/hadoop/pull/5458) and revisited one for netty
from last week
i am never a fan of last-minute jar updates, but if we don't ship with them
we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18
months
On Mon, 6 Mar 2023 at 23:29, Erik Krogen <xk...@apache.org> wrote:
> > OK. Could you have a go with a (locally built) patch release
>
> Just validated the same on the latest HEAD of branch-3.3.5, which includes
> the two HDFS Jiras I mentioned plus one additional one:
>
> * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
> TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
> 55643692+slfan1989@users.noreply.github.com>]
> * d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
> will throw NPE if path does not exist (#5400) [ZanderXu <
> zanderxu@apache.org
> >]
> * 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
> Fix NPE when check the block location of empty directory (#5099)
> [zhengchenyu <zh...@gmail.com>]
> * 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
> connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
> stevel@cloudera.com>]
>
> On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > i looked at that test and wondered if it it was just being brittle to
> > time. I'm not a fan of those -there's one in abfs which is particularly
> bad
> > for me- maybe we could see if the test can be cut as it is quite a slow
> one
> >
> > On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> >
> > > A minor update on ITestS3AConcurrentOps#testParallelRename
> > >
> > > I was previously connected to a vpn due to which bandwidth was getting
> > > throttled earlier. Ran the test again today without vpn and had no
> issues
> > > (earlier only 40% of the overall putObject were able to get completed
> > > within timeout).
> > >
> > >
> > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> > <stevel@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > > >
> > > > > Thanks Steve. I see now that the branch cut was way back in October
> > so
> > > I
> > > > > definitely understand your frustration here!
> > > > >
> > > > > This made me realize that HDFS-16832
> > > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which
> resolves a
> > > > very
> > > > > similar issue as the aforementioned HDFS-16923, is also missing
> from
> > > the
> > > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > > before
> > > > > the initial 3.3.5 RC was made and I didn't notice the branch was
> cut.
> > > My
> > > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > > >
> > > >
> > > > thanks.
> > > >
> > > > >
> > > > > In the meantime, I tested for RC2 that a local cluster of NN +
> > standby
> > > +
> > > > > observer + QJM works as expected for some basic HDFS commands.
> > > > >
> > > >
> > > > OK. Could you have a go with a (locally built) patch release
> > > >
> > > > >
> > > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > > <st...@cloudera.com.invalid>
> > > > > wrote:
> > > > >
> > > > >> shipping broken hdfs isn't something we'd want to do, but if we
> can
> > be
> > > > >> confident that all other issues can be addressed in RC3 then I'd
> be
> > > > happy.
> > > > >>
> > > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > I will highlight that I am completely fed up with doing this
> > > release
> > > > >> and
> > > > >> >> really want to get it out the way -for which I depend on
> support
> > > from
> > > > >> as
> > > > >> >> many other developers as possible.
> > > > >> >
> > > > >> >
> > > > >> > hmm, I can feel the pain. I tried to find if there is any config
> > or
> > > > any
> > > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > > couldn't
> > > > >> find
> > > > >> > any. If someone does a getListing with needLocation and the file
> > > > doesn't
> > > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> > isn't
> > > > just
> > > > >> > the exception, AFAIK Observer reads have some logic around
> > handling
> > > > FNF
> > > > >> > specifically, that it attempts Active NN or something like that
> in
> > > > such
> > > > >> > cases, So, that will be broken as well for this use case.
> > > > >> >
> > > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > > side,
> > > > >> and
> > > > >> > it has already been too much work on your side, so you can argue
> > > that
> > > > it
> > > > >> > might not be a very frequent use case or so. It's your call.
> > > > >> >
> > > > >> > Just sharing, no intentions of saying you should do that, But as
> > an
> > > RM
> > > > >> > "nobody" can force you for a new iteration of a RC, it is gonna
> be
> > > > your
> > > > >> > call and discretion. As far as I know a release can not be
> vetoed
> > by
> > > > >> > "anybody" as per the apache by laws.(
> > > > >> >
> https://www.apache.org/legal/release-policy.html#release-approval
> > ).
> > > > >> Even
> > > > >> > our bylaws say that product release requires a Lazy Majority
> not a
> > > > >> > Consensus Approval.
> > > > >> >
> > > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > > you a
> > > > >> > pass, in case you are really in a state: ''Get me out of this
> > mess"
> > > > >> state,
> > > > >> > my basic validations on x86 & Aarch64 both are passing as of
> now,
> > > > >> couldn't
> > > > >> > reach the end for any of the RC's
> > > > >> >
> > > > >> > -Ayush
> > > > >> >
> > > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> >> While this RC is not going to be final, I just wanted to share
> > the
> > > > >> results
> > > > >> >> of the testing I have done so far with RC1 and RC2.
> > > > >> >>
> > > > >> >> * Signature: ok
> > > > >> >> * Checksum : ok
> > > > >> >> * Rat check (1.8.0_341): ok
> > > > >> >> - mvn clean apache-rat:check
> > > > >> >> * Built from source (1.8.0_341): ok
> > > > >> >> - mvn clean install -DskipTests
> > > > >> >> * Built tar from source (1.8.0_341): ok
> > > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > > >> -Dmaven.javadoc.skip=true
> > > > >> >>
> > > > >> >> * Built images using the tarball, installed and started all of
> > > Hdfs,
> > > > >> JHS
> > > > >> >> and Yarn components
> > > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > > Mapreduce
> > > > >> job
> > > > >> >> * Hdfs CRUD tests
> > > > >> >> * MapReduce wordcount job
> > > > >> >>
> > > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > > >> >>
> > > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> > ~960s.
> > > > >> This
> > > > >> >> is
> > > > >> >> consistently failing, looks like a recent regression.
> > > > >> >> I was also able to repro on trunk, will create Jira.
> > > > >> >>
> > > > >> >>
> > > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > > >> >> <st...@cloudera.com.invalid>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > > Hadoop
> > > > >> >> 3.3.5.
> > > > >> >> >
> > > > >> >> > We need anyone who can to verify the source and binary
> > artifacts,
> > > > >> >> > including those JARs staged on maven, the site documentation
> > and
> > > > the
> > > > >> >> arm64
> > > > >> >> > tar file.
> > > > >> >> >
> > > > >> >> > The RC is available at:
> > > > >> >> >
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > > >> >> >
> > > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > > >> >> >
> > > > >> >> > The maven artifacts are staged at
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > > >> >> >
> > > > >> >> > You can find my public key at:
> > > > >> >> >
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >> >> >
> > > > >> >> > Change log
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > > >> >> >
> > > > >> >> > Release notes
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > > >> >> >
> > > > >> >> > This is off branch-3.3 and is the first big release since
> > 3.3.2.
> > > > >> >> >
> > > > >> >> > As to what changed since the RC1 attempt last week
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > > there)
> > > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5
> index.md
> > > file
> > > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > > (creating
> > > > >> >> build
> > > > >> >> > issues in maven 3.9.0)
> > > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> > fixup.
> > > > >> >> (#5429)
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Note, because the arm64 binaries are built separately on a
> > > > different
> > > > >> >> > platform and JVM, their jar files may not match those of the
> > x86
> > > > >> >> > release -and therefore the maven artifacts. I don't think
> this
> > is
> > > > >> >> > an issue (the ASF actually releases source tarballs, the
> > binaries
> > > > are
> > > > >> >> > there for help only, though with the maven repo that's a bit
> > > > >> blurred).
> > > > >> >> >
> > > > >> >> > The only way to be consistent would actually untar the
> > > x86.tar.gz,
> > > > >> >> > overwrite its binaries with the arm stuff, retar, sign and
> push
> > > out
> > > > >> >> > for the vote. Even automating that would be risky.
> > > > >> >> >
> > > > >> >> > Please try the release and vote. The vote will run for 5
> days.
> > > > >> >> >
> > > > >> >> > Steve and Mukund
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
> OK. Could you have a go with a (locally built) patch release
Just validated the same on the latest HEAD of branch-3.3.5, which includes
the two HDFS Jiras I mentioned plus one additional one:
* 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
55643692+slfan1989@users.noreply.github.com>]
* d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
will throw NPE if path does not exist (#5400) [ZanderXu <zanderxu@apache.org
>]
* 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
Fix NPE when check the block location of empty directory (#5099)
[zhengchenyu <zh...@gmail.com>]
* 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
stevel@cloudera.com>]
On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> i looked at that test and wondered if it it was just being brittle to
> time. I'm not a fan of those -there's one in abfs which is particularly bad
> for me- maybe we could see if the test can be cut as it is quite a slow one
>
> On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
>
> > A minor update on ITestS3AConcurrentOps#testParallelRename
> >
> > I was previously connected to a vpn due to which bandwidth was getting
> > throttled earlier. Ran the test again today without vpn and had no issues
> > (earlier only 40% of the overall putObject were able to get completed
> > within timeout).
> >
> >
> > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > >
> > > > Thanks Steve. I see now that the branch cut was way back in October
> so
> > I
> > > > definitely understand your frustration here!
> > > >
> > > > This made me realize that HDFS-16832
> > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > > very
> > > > similar issue as the aforementioned HDFS-16923, is also missing from
> > the
> > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > before
> > > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> > My
> > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > >
> > >
> > > thanks.
> > >
> > > >
> > > > In the meantime, I tested for RC2 that a local cluster of NN +
> standby
> > +
> > > > observer + QJM works as expected for some basic HDFS commands.
> > > >
> > >
> > > OK. Could you have a go with a (locally built) patch release
> > >
> > > >
> > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > <st...@cloudera.com.invalid>
> > > > wrote:
> > > >
> > > >> shipping broken hdfs isn't something we'd want to do, but if we can
> be
> > > >> confident that all other issues can be addressed in RC3 then I'd be
> > > happy.
> > > >>
> > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >>
> > > >> > I will highlight that I am completely fed up with doing this
> > release
> > > >> and
> > > >> >> really want to get it out the way -for which I depend on support
> > from
> > > >> as
> > > >> >> many other developers as possible.
> > > >> >
> > > >> >
> > > >> > hmm, I can feel the pain. I tried to find if there is any config
> or
> > > any
> > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > couldn't
> > > >> find
> > > >> > any. If someone does a getListing with needLocation and the file
> > > doesn't
> > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> isn't
> > > just
> > > >> > the exception, AFAIK Observer reads have some logic around
> handling
> > > FNF
> > > >> > specifically, that it attempts Active NN or something like that in
> > > such
> > > >> > cases, So, that will be broken as well for this use case.
> > > >> >
> > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > side,
> > > >> and
> > > >> > it has already been too much work on your side, so you can argue
> > that
> > > it
> > > >> > might not be a very frequent use case or so. It's your call.
> > > >> >
> > > >> > Just sharing, no intentions of saying you should do that, But as
> an
> > RM
> > > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > > your
> > > >> > call and discretion. As far as I know a release can not be vetoed
> by
> > > >> > "anybody" as per the apache by laws.(
> > > >> > https://www.apache.org/legal/release-policy.html#release-approval
> ).
> > > >> Even
> > > >> > our bylaws say that product release requires a Lazy Majority not a
> > > >> > Consensus Approval.
> > > >> >
> > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > you a
> > > >> > pass, in case you are really in a state: ''Get me out of this
> mess"
> > > >> state,
> > > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > > >> couldn't
> > > >> > reach the end for any of the RC's
> > > >> >
> > > >> > -Ayush
> > > >> >
> > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > wrote:
> > > >> >
> > > >> >> While this RC is not going to be final, I just wanted to share
> the
> > > >> results
> > > >> >> of the testing I have done so far with RC1 and RC2.
> > > >> >>
> > > >> >> * Signature: ok
> > > >> >> * Checksum : ok
> > > >> >> * Rat check (1.8.0_341): ok
> > > >> >> - mvn clean apache-rat:check
> > > >> >> * Built from source (1.8.0_341): ok
> > > >> >> - mvn clean install -DskipTests
> > > >> >> * Built tar from source (1.8.0_341): ok
> > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > >> -Dmaven.javadoc.skip=true
> > > >> >>
> > > >> >> * Built images using the tarball, installed and started all of
> > Hdfs,
> > > >> JHS
> > > >> >> and Yarn components
> > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > Mapreduce
> > > >> job
> > > >> >> * Hdfs CRUD tests
> > > >> >> * MapReduce wordcount job
> > > >> >>
> > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > >> >>
> > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> ~960s.
> > > >> This
> > > >> >> is
> > > >> >> consistently failing, looks like a recent regression.
> > > >> >> I was also able to repro on trunk, will create Jira.
> > > >> >>
> > > >> >>
> > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > >> >> <st...@cloudera.com.invalid>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > Hadoop
> > > >> >> 3.3.5.
> > > >> >> >
> > > >> >> > We need anyone who can to verify the source and binary
> artifacts,
> > > >> >> > including those JARs staged on maven, the site documentation
> and
> > > the
> > > >> >> arm64
> > > >> >> > tar file.
> > > >> >> >
> > > >> >> > The RC is available at:
> > > >> >> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > >> >> >
> > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > >> >> >
> > > >> >> > The maven artifacts are staged at
> > > >> >> >
> > > >> >>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > >> >> >
> > > >> >> > You can find my public key at:
> > > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >> >> >
> > > >> >> > Change log
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > >> >> >
> > > >> >> > Release notes
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > >> >> >
> > > >> >> > This is off branch-3.3 and is the first big release since
> 3.3.2.
> > > >> >> >
> > > >> >> > As to what changed since the RC1 attempt last week
> > > >> >> >
> > > >> >> >
> > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > there)
> > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> > file
> > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > (creating
> > > >> >> build
> > > >> >> > issues in maven 3.9.0)
> > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> fixup.
> > > >> >> (#5429)
> > > >> >> >
> > > >> >> >
> > > >> >> > Note, because the arm64 binaries are built separately on a
> > > different
> > > >> >> > platform and JVM, their jar files may not match those of the
> x86
> > > >> >> > release -and therefore the maven artifacts. I don't think this
> is
> > > >> >> > an issue (the ASF actually releases source tarballs, the
> binaries
> > > are
> > > >> >> > there for help only, though with the maven repo that's a bit
> > > >> blurred).
> > > >> >> >
> > > >> >> > The only way to be consistent would actually untar the
> > x86.tar.gz,
> > > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> > out
> > > >> >> > for the vote. Even automating that would be risky.
> > > >> >> >
> > > >> >> > Please try the release and vote. The vote will run for 5 days.
> > > >> >> >
> > > >> >> > Steve and Mukund
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
> OK. Could you have a go with a (locally built) patch release
Just validated the same on the latest HEAD of branch-3.3.5, which includes
the two HDFS Jiras I mentioned plus one additional one:
* 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
55643692+slfan1989@users.noreply.github.com>]
* d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
will throw NPE if path does not exist (#5400) [ZanderXu <zanderxu@apache.org
>]
* 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
Fix NPE when check the block location of empty directory (#5099)
[zhengchenyu <zh...@gmail.com>]
* 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
stevel@cloudera.com>]
On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> i looked at that test and wondered if it it was just being brittle to
> time. I'm not a fan of those -there's one in abfs which is particularly bad
> for me- maybe we could see if the test can be cut as it is quite a slow one
>
> On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
>
> > A minor update on ITestS3AConcurrentOps#testParallelRename
> >
> > I was previously connected to a vpn due to which bandwidth was getting
> > throttled earlier. Ran the test again today without vpn and had no issues
> > (earlier only 40% of the overall putObject were able to get completed
> > within timeout).
> >
> >
> > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > >
> > > > Thanks Steve. I see now that the branch cut was way back in October
> so
> > I
> > > > definitely understand your frustration here!
> > > >
> > > > This made me realize that HDFS-16832
> > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > > very
> > > > similar issue as the aforementioned HDFS-16923, is also missing from
> > the
> > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > before
> > > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> > My
> > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > >
> > >
> > > thanks.
> > >
> > > >
> > > > In the meantime, I tested for RC2 that a local cluster of NN +
> standby
> > +
> > > > observer + QJM works as expected for some basic HDFS commands.
> > > >
> > >
> > > OK. Could you have a go with a (locally built) patch release
> > >
> > > >
> > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > <st...@cloudera.com.invalid>
> > > > wrote:
> > > >
> > > >> shipping broken hdfs isn't something we'd want to do, but if we can
> be
> > > >> confident that all other issues can be addressed in RC3 then I'd be
> > > happy.
> > > >>
> > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >>
> > > >> > I will highlight that I am completely fed up with doing this
> > release
> > > >> and
> > > >> >> really want to get it out the way -for which I depend on support
> > from
> > > >> as
> > > >> >> many other developers as possible.
> > > >> >
> > > >> >
> > > >> > hmm, I can feel the pain. I tried to find if there is any config
> or
> > > any
> > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > couldn't
> > > >> find
> > > >> > any. If someone does a getListing with needLocation and the file
> > > doesn't
> > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> isn't
> > > just
> > > >> > the exception, AFAIK Observer reads have some logic around
> handling
> > > FNF
> > > >> > specifically, that it attempts Active NN or something like that in
> > > such
> > > >> > cases, So, that will be broken as well for this use case.
> > > >> >
> > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > side,
> > > >> and
> > > >> > it has already been too much work on your side, so you can argue
> > that
> > > it
> > > >> > might not be a very frequent use case or so. It's your call.
> > > >> >
> > > >> > Just sharing, no intentions of saying you should do that, But as
> an
> > RM
> > > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > > your
> > > >> > call and discretion. As far as I know a release can not be vetoed
> by
> > > >> > "anybody" as per the apache by laws.(
> > > >> > https://www.apache.org/legal/release-policy.html#release-approval
> ).
> > > >> Even
> > > >> > our bylaws say that product release requires a Lazy Majority not a
> > > >> > Consensus Approval.
> > > >> >
> > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > you a
> > > >> > pass, in case you are really in a state: ''Get me out of this
> mess"
> > > >> state,
> > > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > > >> couldn't
> > > >> > reach the end for any of the RC's
> > > >> >
> > > >> > -Ayush
> > > >> >
> > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > wrote:
> > > >> >
> > > >> >> While this RC is not going to be final, I just wanted to share
> the
> > > >> results
> > > >> >> of the testing I have done so far with RC1 and RC2.
> > > >> >>
> > > >> >> * Signature: ok
> > > >> >> * Checksum : ok
> > > >> >> * Rat check (1.8.0_341): ok
> > > >> >> - mvn clean apache-rat:check
> > > >> >> * Built from source (1.8.0_341): ok
> > > >> >> - mvn clean install -DskipTests
> > > >> >> * Built tar from source (1.8.0_341): ok
> > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > >> -Dmaven.javadoc.skip=true
> > > >> >>
> > > >> >> * Built images using the tarball, installed and started all of
> > Hdfs,
> > > >> JHS
> > > >> >> and Yarn components
> > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > Mapreduce
> > > >> job
> > > >> >> * Hdfs CRUD tests
> > > >> >> * MapReduce wordcount job
> > > >> >>
> > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > >> >>
> > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> ~960s.
> > > >> This
> > > >> >> is
> > > >> >> consistently failing, looks like a recent regression.
> > > >> >> I was also able to repro on trunk, will create Jira.
> > > >> >>
> > > >> >>
> > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > >> >> <st...@cloudera.com.invalid>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > Hadoop
> > > >> >> 3.3.5.
> > > >> >> >
> > > >> >> > We need anyone who can to verify the source and binary
> artifacts,
> > > >> >> > including those JARs staged on maven, the site documentation
> and
> > > the
> > > >> >> arm64
> > > >> >> > tar file.
> > > >> >> >
> > > >> >> > The RC is available at:
> > > >> >> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > >> >> >
> > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > >> >> >
> > > >> >> > The maven artifacts are staged at
> > > >> >> >
> > > >> >>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > >> >> >
> > > >> >> > You can find my public key at:
> > > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >> >> >
> > > >> >> > Change log
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > >> >> >
> > > >> >> > Release notes
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > >> >> >
> > > >> >> > This is off branch-3.3 and is the first big release since
> 3.3.2.
> > > >> >> >
> > > >> >> > As to what changed since the RC1 attempt last week
> > > >> >> >
> > > >> >> >
> > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > there)
> > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> > file
> > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > (creating
> > > >> >> build
> > > >> >> > issues in maven 3.9.0)
> > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> fixup.
> > > >> >> (#5429)
> > > >> >> >
> > > >> >> >
> > > >> >> > Note, because the arm64 binaries are built separately on a
> > > different
> > > >> >> > platform and JVM, their jar files may not match those of the
> x86
> > > >> >> > release -and therefore the maven artifacts. I don't think this
> is
> > > >> >> > an issue (the ASF actually releases source tarballs, the
> binaries
> > > are
> > > >> >> > there for help only, though with the maven repo that's a bit
> > > >> blurred).
> > > >> >> >
> > > >> >> > The only way to be consistent would actually untar the
> > x86.tar.gz,
> > > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> > out
> > > >> >> > for the vote. Even automating that would be risky.
> > > >> >> >
> > > >> >> > Please try the release and vote. The vote will run for 5 days.
> > > >> >> >
> > > >> >> > Steve and Mukund
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
> OK. Could you have a go with a (locally built) patch release
Just validated the same on the latest HEAD of branch-3.3.5, which includes
the two HDFS Jiras I mentioned plus one additional one:
* 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
55643692+slfan1989@users.noreply.github.com>]
* d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
will throw NPE if path does not exist (#5400) [ZanderXu <zanderxu@apache.org
>]
* 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
Fix NPE when check the block location of empty directory (#5099)
[zhengchenyu <zh...@gmail.com>]
* 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
stevel@cloudera.com>]
On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> i looked at that test and wondered if it it was just being brittle to
> time. I'm not a fan of those -there's one in abfs which is particularly bad
> for me- maybe we could see if the test can be cut as it is quite a slow one
>
> On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
>
> > A minor update on ITestS3AConcurrentOps#testParallelRename
> >
> > I was previously connected to a vpn due to which bandwidth was getting
> > throttled earlier. Ran the test again today without vpn and had no issues
> > (earlier only 40% of the overall putObject were able to get completed
> > within timeout).
> >
> >
> > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > >
> > > > Thanks Steve. I see now that the branch cut was way back in October
> so
> > I
> > > > definitely understand your frustration here!
> > > >
> > > > This made me realize that HDFS-16832
> > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > > very
> > > > similar issue as the aforementioned HDFS-16923, is also missing from
> > the
> > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > before
> > > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> > My
> > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > >
> > >
> > > thanks.
> > >
> > > >
> > > > In the meantime, I tested for RC2 that a local cluster of NN +
> standby
> > +
> > > > observer + QJM works as expected for some basic HDFS commands.
> > > >
> > >
> > > OK. Could you have a go with a (locally built) patch release
> > >
> > > >
> > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > <st...@cloudera.com.invalid>
> > > > wrote:
> > > >
> > > >> shipping broken hdfs isn't something we'd want to do, but if we can
> be
> > > >> confident that all other issues can be addressed in RC3 then I'd be
> > > happy.
> > > >>
> > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >>
> > > >> > I will highlight that I am completely fed up with doing this
> > release
> > > >> and
> > > >> >> really want to get it out the way -for which I depend on support
> > from
> > > >> as
> > > >> >> many other developers as possible.
> > > >> >
> > > >> >
> > > >> > hmm, I can feel the pain. I tried to find if there is any config
> or
> > > any
> > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > couldn't
> > > >> find
> > > >> > any. If someone does a getListing with needLocation and the file
> > > doesn't
> > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> isn't
> > > just
> > > >> > the exception, AFAIK Observer reads have some logic around
> handling
> > > FNF
> > > >> > specifically, that it attempts Active NN or something like that in
> > > such
> > > >> > cases, So, that will be broken as well for this use case.
> > > >> >
> > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > side,
> > > >> and
> > > >> > it has already been too much work on your side, so you can argue
> > that
> > > it
> > > >> > might not be a very frequent use case or so. It's your call.
> > > >> >
> > > >> > Just sharing, no intentions of saying you should do that, But as
> an
> > RM
> > > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > > your
> > > >> > call and discretion. As far as I know a release can not be vetoed
> by
> > > >> > "anybody" as per the apache by laws.(
> > > >> > https://www.apache.org/legal/release-policy.html#release-approval
> ).
> > > >> Even
> > > >> > our bylaws say that product release requires a Lazy Majority not a
> > > >> > Consensus Approval.
> > > >> >
> > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > you a
> > > >> > pass, in case you are really in a state: ''Get me out of this
> mess"
> > > >> state,
> > > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > > >> couldn't
> > > >> > reach the end for any of the RC's
> > > >> >
> > > >> > -Ayush
> > > >> >
> > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > wrote:
> > > >> >
> > > >> >> While this RC is not going to be final, I just wanted to share
> the
> > > >> results
> > > >> >> of the testing I have done so far with RC1 and RC2.
> > > >> >>
> > > >> >> * Signature: ok
> > > >> >> * Checksum : ok
> > > >> >> * Rat check (1.8.0_341): ok
> > > >> >> - mvn clean apache-rat:check
> > > >> >> * Built from source (1.8.0_341): ok
> > > >> >> - mvn clean install -DskipTests
> > > >> >> * Built tar from source (1.8.0_341): ok
> > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > >> -Dmaven.javadoc.skip=true
> > > >> >>
> > > >> >> * Built images using the tarball, installed and started all of
> > Hdfs,
> > > >> JHS
> > > >> >> and Yarn components
> > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > Mapreduce
> > > >> job
> > > >> >> * Hdfs CRUD tests
> > > >> >> * MapReduce wordcount job
> > > >> >>
> > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > >> >>
> > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> ~960s.
> > > >> This
> > > >> >> is
> > > >> >> consistently failing, looks like a recent regression.
> > > >> >> I was also able to repro on trunk, will create Jira.
> > > >> >>
> > > >> >>
> > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > >> >> <st...@cloudera.com.invalid>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > Hadoop
> > > >> >> 3.3.5.
> > > >> >> >
> > > >> >> > We need anyone who can to verify the source and binary
> artifacts,
> > > >> >> > including those JARs staged on maven, the site documentation
> and
> > > the
> > > >> >> arm64
> > > >> >> > tar file.
> > > >> >> >
> > > >> >> > The RC is available at:
> > > >> >> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > >> >> >
> > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > >> >> >
> > > >> >> > The maven artifacts are staged at
> > > >> >> >
> > > >> >>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > >> >> >
> > > >> >> > You can find my public key at:
> > > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >> >> >
> > > >> >> > Change log
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > >> >> >
> > > >> >> > Release notes
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > >> >> >
> > > >> >> > This is off branch-3.3 and is the first big release since
> 3.3.2.
> > > >> >> >
> > > >> >> > As to what changed since the RC1 attempt last week
> > > >> >> >
> > > >> >> >
> > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > there)
> > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> > file
> > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > (creating
> > > >> >> build
> > > >> >> > issues in maven 3.9.0)
> > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> fixup.
> > > >> >> (#5429)
> > > >> >> >
> > > >> >> >
> > > >> >> > Note, because the arm64 binaries are built separately on a
> > > different
> > > >> >> > platform and JVM, their jar files may not match those of the
> x86
> > > >> >> > release -and therefore the maven artifacts. I don't think this
> is
> > > >> >> > an issue (the ASF actually releases source tarballs, the
> binaries
> > > are
> > > >> >> > there for help only, though with the maven repo that's a bit
> > > >> blurred).
> > > >> >> >
> > > >> >> > The only way to be consistent would actually untar the
> > x86.tar.gz,
> > > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> > out
> > > >> >> > for the vote. Even automating that would be risky.
> > > >> >> >
> > > >> >> > Please try the release and vote. The vote will run for 5 days.
> > > >> >> >
> > > >> >> > Steve and Mukund
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
> OK. Could you have a go with a (locally built) patch release
Just validated the same on the latest HEAD of branch-3.3.5, which includes
the two HDFS Jiras I mentioned plus one additional one:
* 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
55643692+slfan1989@users.noreply.github.com>]
* d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
will throw NPE if path does not exist (#5400) [ZanderXu <zanderxu@apache.org
>]
* 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
Fix NPE when check the block location of empty directory (#5099)
[zhengchenyu <zh...@gmail.com>]
* 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
stevel@cloudera.com>]
On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> i looked at that test and wondered if it it was just being brittle to
> time. I'm not a fan of those -there's one in abfs which is particularly bad
> for me- maybe we could see if the test can be cut as it is quite a slow one
>
> On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
>
> > A minor update on ITestS3AConcurrentOps#testParallelRename
> >
> > I was previously connected to a vpn due to which bandwidth was getting
> > throttled earlier. Ran the test again today without vpn and had no issues
> > (earlier only 40% of the overall putObject were able to get completed
> > within timeout).
> >
> >
> > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> <stevel@cloudera.com.invalid
> > >
> > wrote:
> >
> > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> > >
> > > > Thanks Steve. I see now that the branch cut was way back in October
> so
> > I
> > > > definitely understand your frustration here!
> > > >
> > > > This made me realize that HDFS-16832
> > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > > very
> > > > similar issue as the aforementioned HDFS-16923, is also missing from
> > the
> > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > before
> > > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> > My
> > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > >
> > >
> > > thanks.
> > >
> > > >
> > > > In the meantime, I tested for RC2 that a local cluster of NN +
> standby
> > +
> > > > observer + QJM works as expected for some basic HDFS commands.
> > > >
> > >
> > > OK. Could you have a go with a (locally built) patch release
> > >
> > > >
> > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > <st...@cloudera.com.invalid>
> > > > wrote:
> > > >
> > > >> shipping broken hdfs isn't something we'd want to do, but if we can
> be
> > > >> confident that all other issues can be addressed in RC3 then I'd be
> > > happy.
> > > >>
> > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >>
> > > >> > I will highlight that I am completely fed up with doing this
> > release
> > > >> and
> > > >> >> really want to get it out the way -for which I depend on support
> > from
> > > >> as
> > > >> >> many other developers as possible.
> > > >> >
> > > >> >
> > > >> > hmm, I can feel the pain. I tried to find if there is any config
> or
> > > any
> > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > couldn't
> > > >> find
> > > >> > any. If someone does a getListing with needLocation and the file
> > > doesn't
> > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> isn't
> > > just
> > > >> > the exception, AFAIK Observer reads have some logic around
> handling
> > > FNF
> > > >> > specifically, that it attempts Active NN or something like that in
> > > such
> > > >> > cases, So, that will be broken as well for this use case.
> > > >> >
> > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > side,
> > > >> and
> > > >> > it has already been too much work on your side, so you can argue
> > that
> > > it
> > > >> > might not be a very frequent use case or so. It's your call.
> > > >> >
> > > >> > Just sharing, no intentions of saying you should do that, But as
> an
> > RM
> > > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > > your
> > > >> > call and discretion. As far as I know a release can not be vetoed
> by
> > > >> > "anybody" as per the apache by laws.(
> > > >> > https://www.apache.org/legal/release-policy.html#release-approval
> ).
> > > >> Even
> > > >> > our bylaws say that product release requires a Lazy Majority not a
> > > >> > Consensus Approval.
> > > >> >
> > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > you a
> > > >> > pass, in case you are really in a state: ''Get me out of this
> mess"
> > > >> state,
> > > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > > >> couldn't
> > > >> > reach the end for any of the RC's
> > > >> >
> > > >> > -Ayush
> > > >> >
> > > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> > wrote:
> > > >> >
> > > >> >> While this RC is not going to be final, I just wanted to share
> the
> > > >> results
> > > >> >> of the testing I have done so far with RC1 and RC2.
> > > >> >>
> > > >> >> * Signature: ok
> > > >> >> * Checksum : ok
> > > >> >> * Rat check (1.8.0_341): ok
> > > >> >> - mvn clean apache-rat:check
> > > >> >> * Built from source (1.8.0_341): ok
> > > >> >> - mvn clean install -DskipTests
> > > >> >> * Built tar from source (1.8.0_341): ok
> > > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > > >> -Dmaven.javadoc.skip=true
> > > >> >>
> > > >> >> * Built images using the tarball, installed and started all of
> > Hdfs,
> > > >> JHS
> > > >> >> and Yarn components
> > > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> > Mapreduce
> > > >> job
> > > >> >> * Hdfs CRUD tests
> > > >> >> * MapReduce wordcount job
> > > >> >>
> > > >> >> * Ran S3A tests with scale profile against us-west-2:
> > > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > > >> >>
> > > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after
> ~960s.
> > > >> This
> > > >> >> is
> > > >> >> consistently failing, looks like a recent regression.
> > > >> >> I was also able to repro on trunk, will create Jira.
> > > >> >>
> > > >> >>
> > > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > > >> >> <st...@cloudera.com.invalid>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Mukund and I have put together a release candidate (RC2) for
> > Hadoop
> > > >> >> 3.3.5.
> > > >> >> >
> > > >> >> > We need anyone who can to verify the source and binary
> artifacts,
> > > >> >> > including those JARs staged on maven, the site documentation
> and
> > > the
> > > >> >> arm64
> > > >> >> > tar file.
> > > >> >> >
> > > >> >> > The RC is available at:
> > > >> >> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > > >> >> >
> > > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > > >> >> >
> > > >> >> > The maven artifacts are staged at
> > > >> >> >
> > > >> >>
> > > >>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > >> >> >
> > > >> >> > You can find my public key at:
> > > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >> >> >
> > > >> >> > Change log
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > > >> >> >
> > > >> >> > Release notes
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > > >> >> >
> > > >> >> > This is off branch-3.3 and is the first big release since
> 3.3.2.
> > > >> >> >
> > > >> >> > As to what changed since the RC1 attempt last week
> > > >> >> >
> > > >> >> >
> > > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> > there)
> > > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> > file
> > > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > > (creating
> > > >> >> build
> > > >> >> > issues in maven 3.9.0)
> > > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE
> fixup.
> > > >> >> (#5429)
> > > >> >> >
> > > >> >> >
> > > >> >> > Note, because the arm64 binaries are built separately on a
> > > different
> > > >> >> > platform and JVM, their jar files may not match those of the
> x86
> > > >> >> > release -and therefore the maven artifacts. I don't think this
> is
> > > >> >> > an issue (the ASF actually releases source tarballs, the
> binaries
> > > are
> > > >> >> > there for help only, though with the maven repo that's a bit
> > > >> blurred).
> > > >> >> >
> > > >> >> > The only way to be consistent would actually untar the
> > x86.tar.gz,
> > > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> > out
> > > >> >> > for the vote. Even automating that would be risky.
> > > >> >> >
> > > >> >> > Please try the release and vote. The vote will run for 5 days.
> > > >> >> >
> > > >> >> > Steve and Mukund
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
i looked at that test and wondered if it it was just being brittle to
time. I'm not a fan of those -there's one in abfs which is particularly bad
for me- maybe we could see if the test can be cut as it is quite a slow one
On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> A minor update on ITestS3AConcurrentOps#testParallelRename
>
> I was previously connected to a vpn due to which bandwidth was getting
> throttled earlier. Ran the test again today without vpn and had no issues
> (earlier only 40% of the overall putObject were able to get completed
> within timeout).
>
>
> On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> >
> > > Thanks Steve. I see now that the branch cut was way back in October so
> I
> > > definitely understand your frustration here!
> > >
> > > This made me realize that HDFS-16832
> > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > very
> > > similar issue as the aforementioned HDFS-16923, is also missing from
> the
> > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> before
> > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> My
> > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > >
> >
> > thanks.
> >
> > >
> > > In the meantime, I tested for RC2 that a local cluster of NN + standby
> +
> > > observer + QJM works as expected for some basic HDFS commands.
> > >
> >
> > OK. Could you have a go with a (locally built) patch release
> >
> > >
> > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > > wrote:
> > >
> > >> shipping broken hdfs isn't something we'd want to do, but if we can be
> > >> confident that all other issues can be addressed in RC3 then I'd be
> > happy.
> > >>
> > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> > >>
> > >> > I will highlight that I am completely fed up with doing this
> release
> > >> and
> > >> >> really want to get it out the way -for which I depend on support
> from
> > >> as
> > >> >> many other developers as possible.
> > >> >
> > >> >
> > >> > hmm, I can feel the pain. I tried to find if there is any config or
> > any
> > >> > workaround which can dodge this HDFS issue, but unfortunately
> couldn't
> > >> find
> > >> > any. If someone does a getListing with needLocation and the file
> > doesn't
> > >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> > just
> > >> > the exception, AFAIK Observer reads have some logic around handling
> > FNF
> > >> > specifically, that it attempts Active NN or something like that in
> > such
> > >> > cases, So, that will be broken as well for this use case.
> > >> >
> > >> > Now, there is no denying the fact there is an issue on the HDFS
> side,
> > >> and
> > >> > it has already been too much work on your side, so you can argue
> that
> > it
> > >> > might not be a very frequent use case or so. It's your call.
> > >> >
> > >> > Just sharing, no intentions of saying you should do that, But as an
> RM
> > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > your
> > >> > call and discretion. As far as I know a release can not be vetoed by
> > >> > "anybody" as per the apache by laws.(
> > >> > https://www.apache.org/legal/release-policy.html#release-approval).
> > >> Even
> > >> > our bylaws say that product release requires a Lazy Majority not a
> > >> > Consensus Approval.
> > >> >
> > >> > So, you have a way out. You guys are 2 already and 1 I will give
> you a
> > >> > pass, in case you are really in a state: ''Get me out of this mess"
> > >> state,
> > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > >> couldn't
> > >> > reach the end for any of the RC's
> > >> >
> > >> > -Ayush
> > >> >
> > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> wrote:
> > >> >
> > >> >> While this RC is not going to be final, I just wanted to share the
> > >> results
> > >> >> of the testing I have done so far with RC1 and RC2.
> > >> >>
> > >> >> * Signature: ok
> > >> >> * Checksum : ok
> > >> >> * Rat check (1.8.0_341): ok
> > >> >> - mvn clean apache-rat:check
> > >> >> * Built from source (1.8.0_341): ok
> > >> >> - mvn clean install -DskipTests
> > >> >> * Built tar from source (1.8.0_341): ok
> > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > >> -Dmaven.javadoc.skip=true
> > >> >>
> > >> >> * Built images using the tarball, installed and started all of
> Hdfs,
> > >> JHS
> > >> >> and Yarn components
> > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> Mapreduce
> > >> job
> > >> >> * Hdfs CRUD tests
> > >> >> * MapReduce wordcount job
> > >> >>
> > >> >> * Ran S3A tests with scale profile against us-west-2:
> > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > >> >>
> > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> > >> This
> > >> >> is
> > >> >> consistently failing, looks like a recent regression.
> > >> >> I was also able to repro on trunk, will create Jira.
> > >> >>
> > >> >>
> > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > >> >> <st...@cloudera.com.invalid>
> > >> >> wrote:
> > >> >>
> > >> >> > Mukund and I have put together a release candidate (RC2) for
> Hadoop
> > >> >> 3.3.5.
> > >> >> >
> > >> >> > We need anyone who can to verify the source and binary artifacts,
> > >> >> > including those JARs staged on maven, the site documentation and
> > the
> > >> >> arm64
> > >> >> > tar file.
> > >> >> >
> > >> >> > The RC is available at:
> > >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > >> >> >
> > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > >> >> >
> > >> >> > The maven artifacts are staged at
> > >> >> >
> > >> >>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > >> >> >
> > >> >> > You can find my public key at:
> > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >> >> >
> > >> >> > Change log
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > >> >> >
> > >> >> > Release notes
> > >> >> >
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > >> >> >
> > >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> > >> >> >
> > >> >> > As to what changed since the RC1 attempt last week
> > >> >> >
> > >> >> >
> > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> there)
> > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> file
> > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > (creating
> > >> >> build
> > >> >> > issues in maven 3.9.0)
> > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> > >> >> (#5429)
> > >> >> >
> > >> >> >
> > >> >> > Note, because the arm64 binaries are built separately on a
> > different
> > >> >> > platform and JVM, their jar files may not match those of the x86
> > >> >> > release -and therefore the maven artifacts. I don't think this is
> > >> >> > an issue (the ASF actually releases source tarballs, the binaries
> > are
> > >> >> > there for help only, though with the maven repo that's a bit
> > >> blurred).
> > >> >> >
> > >> >> > The only way to be consistent would actually untar the
> x86.tar.gz,
> > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> out
> > >> >> > for the vote. Even automating that would be risky.
> > >> >> >
> > >> >> > Please try the release and vote. The vote will run for 5 days.
> > >> >> >
> > >> >> > Steve and Mukund
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
i looked at that test and wondered if it it was just being brittle to
time. I'm not a fan of those -there's one in abfs which is particularly bad
for me- maybe we could see if the test can be cut as it is quite a slow one
On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> A minor update on ITestS3AConcurrentOps#testParallelRename
>
> I was previously connected to a vpn due to which bandwidth was getting
> throttled earlier. Ran the test again today without vpn and had no issues
> (earlier only 40% of the overall putObject were able to get completed
> within timeout).
>
>
> On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> >
> > > Thanks Steve. I see now that the branch cut was way back in October so
> I
> > > definitely understand your frustration here!
> > >
> > > This made me realize that HDFS-16832
> > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > very
> > > similar issue as the aforementioned HDFS-16923, is also missing from
> the
> > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> before
> > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> My
> > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > >
> >
> > thanks.
> >
> > >
> > > In the meantime, I tested for RC2 that a local cluster of NN + standby
> +
> > > observer + QJM works as expected for some basic HDFS commands.
> > >
> >
> > OK. Could you have a go with a (locally built) patch release
> >
> > >
> > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > > wrote:
> > >
> > >> shipping broken hdfs isn't something we'd want to do, but if we can be
> > >> confident that all other issues can be addressed in RC3 then I'd be
> > happy.
> > >>
> > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> > >>
> > >> > I will highlight that I am completely fed up with doing this
> release
> > >> and
> > >> >> really want to get it out the way -for which I depend on support
> from
> > >> as
> > >> >> many other developers as possible.
> > >> >
> > >> >
> > >> > hmm, I can feel the pain. I tried to find if there is any config or
> > any
> > >> > workaround which can dodge this HDFS issue, but unfortunately
> couldn't
> > >> find
> > >> > any. If someone does a getListing with needLocation and the file
> > doesn't
> > >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> > just
> > >> > the exception, AFAIK Observer reads have some logic around handling
> > FNF
> > >> > specifically, that it attempts Active NN or something like that in
> > such
> > >> > cases, So, that will be broken as well for this use case.
> > >> >
> > >> > Now, there is no denying the fact there is an issue on the HDFS
> side,
> > >> and
> > >> > it has already been too much work on your side, so you can argue
> that
> > it
> > >> > might not be a very frequent use case or so. It's your call.
> > >> >
> > >> > Just sharing, no intentions of saying you should do that, But as an
> RM
> > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > your
> > >> > call and discretion. As far as I know a release can not be vetoed by
> > >> > "anybody" as per the apache by laws.(
> > >> > https://www.apache.org/legal/release-policy.html#release-approval).
> > >> Even
> > >> > our bylaws say that product release requires a Lazy Majority not a
> > >> > Consensus Approval.
> > >> >
> > >> > So, you have a way out. You guys are 2 already and 1 I will give
> you a
> > >> > pass, in case you are really in a state: ''Get me out of this mess"
> > >> state,
> > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > >> couldn't
> > >> > reach the end for any of the RC's
> > >> >
> > >> > -Ayush
> > >> >
> > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> wrote:
> > >> >
> > >> >> While this RC is not going to be final, I just wanted to share the
> > >> results
> > >> >> of the testing I have done so far with RC1 and RC2.
> > >> >>
> > >> >> * Signature: ok
> > >> >> * Checksum : ok
> > >> >> * Rat check (1.8.0_341): ok
> > >> >> - mvn clean apache-rat:check
> > >> >> * Built from source (1.8.0_341): ok
> > >> >> - mvn clean install -DskipTests
> > >> >> * Built tar from source (1.8.0_341): ok
> > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > >> -Dmaven.javadoc.skip=true
> > >> >>
> > >> >> * Built images using the tarball, installed and started all of
> Hdfs,
> > >> JHS
> > >> >> and Yarn components
> > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> Mapreduce
> > >> job
> > >> >> * Hdfs CRUD tests
> > >> >> * MapReduce wordcount job
> > >> >>
> > >> >> * Ran S3A tests with scale profile against us-west-2:
> > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > >> >>
> > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> > >> This
> > >> >> is
> > >> >> consistently failing, looks like a recent regression.
> > >> >> I was also able to repro on trunk, will create Jira.
> > >> >>
> > >> >>
> > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > >> >> <st...@cloudera.com.invalid>
> > >> >> wrote:
> > >> >>
> > >> >> > Mukund and I have put together a release candidate (RC2) for
> Hadoop
> > >> >> 3.3.5.
> > >> >> >
> > >> >> > We need anyone who can to verify the source and binary artifacts,
> > >> >> > including those JARs staged on maven, the site documentation and
> > the
> > >> >> arm64
> > >> >> > tar file.
> > >> >> >
> > >> >> > The RC is available at:
> > >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > >> >> >
> > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > >> >> >
> > >> >> > The maven artifacts are staged at
> > >> >> >
> > >> >>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > >> >> >
> > >> >> > You can find my public key at:
> > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >> >> >
> > >> >> > Change log
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > >> >> >
> > >> >> > Release notes
> > >> >> >
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > >> >> >
> > >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> > >> >> >
> > >> >> > As to what changed since the RC1 attempt last week
> > >> >> >
> > >> >> >
> > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> there)
> > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> file
> > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > (creating
> > >> >> build
> > >> >> > issues in maven 3.9.0)
> > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> > >> >> (#5429)
> > >> >> >
> > >> >> >
> > >> >> > Note, because the arm64 binaries are built separately on a
> > different
> > >> >> > platform and JVM, their jar files may not match those of the x86
> > >> >> > release -and therefore the maven artifacts. I don't think this is
> > >> >> > an issue (the ASF actually releases source tarballs, the binaries
> > are
> > >> >> > there for help only, though with the maven repo that's a bit
> > >> blurred).
> > >> >> >
> > >> >> > The only way to be consistent would actually untar the
> x86.tar.gz,
> > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> out
> > >> >> > for the vote. Even automating that would be risky.
> > >> >> >
> > >> >> > Please try the release and vote. The vote will run for 5 days.
> > >> >> >
> > >> >> > Steve and Mukund
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
i looked at that test and wondered if it it was just being brittle to
time. I'm not a fan of those -there's one in abfs which is particularly bad
for me- maybe we could see if the test can be cut as it is quite a slow one
On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> A minor update on ITestS3AConcurrentOps#testParallelRename
>
> I was previously connected to a vpn due to which bandwidth was getting
> throttled earlier. Ran the test again today without vpn and had no issues
> (earlier only 40% of the overall putObject were able to get completed
> within timeout).
>
>
> On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> >
> > > Thanks Steve. I see now that the branch cut was way back in October so
> I
> > > definitely understand your frustration here!
> > >
> > > This made me realize that HDFS-16832
> > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > very
> > > similar issue as the aforementioned HDFS-16923, is also missing from
> the
> > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> before
> > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> My
> > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > >
> >
> > thanks.
> >
> > >
> > > In the meantime, I tested for RC2 that a local cluster of NN + standby
> +
> > > observer + QJM works as expected for some basic HDFS commands.
> > >
> >
> > OK. Could you have a go with a (locally built) patch release
> >
> > >
> > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > > wrote:
> > >
> > >> shipping broken hdfs isn't something we'd want to do, but if we can be
> > >> confident that all other issues can be addressed in RC3 then I'd be
> > happy.
> > >>
> > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> > >>
> > >> > I will highlight that I am completely fed up with doing this
> release
> > >> and
> > >> >> really want to get it out the way -for which I depend on support
> from
> > >> as
> > >> >> many other developers as possible.
> > >> >
> > >> >
> > >> > hmm, I can feel the pain. I tried to find if there is any config or
> > any
> > >> > workaround which can dodge this HDFS issue, but unfortunately
> couldn't
> > >> find
> > >> > any. If someone does a getListing with needLocation and the file
> > doesn't
> > >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> > just
> > >> > the exception, AFAIK Observer reads have some logic around handling
> > FNF
> > >> > specifically, that it attempts Active NN or something like that in
> > such
> > >> > cases, So, that will be broken as well for this use case.
> > >> >
> > >> > Now, there is no denying the fact there is an issue on the HDFS
> side,
> > >> and
> > >> > it has already been too much work on your side, so you can argue
> that
> > it
> > >> > might not be a very frequent use case or so. It's your call.
> > >> >
> > >> > Just sharing, no intentions of saying you should do that, But as an
> RM
> > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > your
> > >> > call and discretion. As far as I know a release can not be vetoed by
> > >> > "anybody" as per the apache by laws.(
> > >> > https://www.apache.org/legal/release-policy.html#release-approval).
> > >> Even
> > >> > our bylaws say that product release requires a Lazy Majority not a
> > >> > Consensus Approval.
> > >> >
> > >> > So, you have a way out. You guys are 2 already and 1 I will give
> you a
> > >> > pass, in case you are really in a state: ''Get me out of this mess"
> > >> state,
> > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > >> couldn't
> > >> > reach the end for any of the RC's
> > >> >
> > >> > -Ayush
> > >> >
> > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> wrote:
> > >> >
> > >> >> While this RC is not going to be final, I just wanted to share the
> > >> results
> > >> >> of the testing I have done so far with RC1 and RC2.
> > >> >>
> > >> >> * Signature: ok
> > >> >> * Checksum : ok
> > >> >> * Rat check (1.8.0_341): ok
> > >> >> - mvn clean apache-rat:check
> > >> >> * Built from source (1.8.0_341): ok
> > >> >> - mvn clean install -DskipTests
> > >> >> * Built tar from source (1.8.0_341): ok
> > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > >> -Dmaven.javadoc.skip=true
> > >> >>
> > >> >> * Built images using the tarball, installed and started all of
> Hdfs,
> > >> JHS
> > >> >> and Yarn components
> > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> Mapreduce
> > >> job
> > >> >> * Hdfs CRUD tests
> > >> >> * MapReduce wordcount job
> > >> >>
> > >> >> * Ran S3A tests with scale profile against us-west-2:
> > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > >> >>
> > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> > >> This
> > >> >> is
> > >> >> consistently failing, looks like a recent regression.
> > >> >> I was also able to repro on trunk, will create Jira.
> > >> >>
> > >> >>
> > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > >> >> <st...@cloudera.com.invalid>
> > >> >> wrote:
> > >> >>
> > >> >> > Mukund and I have put together a release candidate (RC2) for
> Hadoop
> > >> >> 3.3.5.
> > >> >> >
> > >> >> > We need anyone who can to verify the source and binary artifacts,
> > >> >> > including those JARs staged on maven, the site documentation and
> > the
> > >> >> arm64
> > >> >> > tar file.
> > >> >> >
> > >> >> > The RC is available at:
> > >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > >> >> >
> > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > >> >> >
> > >> >> > The maven artifacts are staged at
> > >> >> >
> > >> >>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > >> >> >
> > >> >> > You can find my public key at:
> > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >> >> >
> > >> >> > Change log
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > >> >> >
> > >> >> > Release notes
> > >> >> >
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > >> >> >
> > >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> > >> >> >
> > >> >> > As to what changed since the RC1 attempt last week
> > >> >> >
> > >> >> >
> > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> there)
> > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> file
> > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > (creating
> > >> >> build
> > >> >> > issues in maven 3.9.0)
> > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> > >> >> (#5429)
> > >> >> >
> > >> >> >
> > >> >> > Note, because the arm64 binaries are built separately on a
> > different
> > >> >> > platform and JVM, their jar files may not match those of the x86
> > >> >> > release -and therefore the maven artifacts. I don't think this is
> > >> >> > an issue (the ASF actually releases source tarballs, the binaries
> > are
> > >> >> > there for help only, though with the maven repo that's a bit
> > >> blurred).
> > >> >> >
> > >> >> > The only way to be consistent would actually untar the
> x86.tar.gz,
> > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> out
> > >> >> > for the vote. Even automating that would be risky.
> > >> >> >
> > >> >> > Please try the release and vote. The vote will run for 5 days.
> > >> >> >
> > >> >> > Steve and Mukund
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
i looked at that test and wondered if it it was just being brittle to
time. I'm not a fan of those -there's one in abfs which is particularly bad
for me- maybe we could see if the test can be cut as it is quite a slow one
On Sat, 4 Mar 2023 at 18:28, Viraj Jasani <vj...@apache.org> wrote:
> A minor update on ITestS3AConcurrentOps#testParallelRename
>
> I was previously connected to a vpn due to which bandwidth was getting
> throttled earlier. Ran the test again today without vpn and had no issues
> (earlier only 40% of the overall putObject were able to get completed
> within timeout).
>
>
> On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> >
> > > Thanks Steve. I see now that the branch cut was way back in October so
> I
> > > definitely understand your frustration here!
> > >
> > > This made me realize that HDFS-16832
> > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > very
> > > similar issue as the aforementioned HDFS-16923, is also missing from
> the
> > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> before
> > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> My
> > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > >
> >
> > thanks.
> >
> > >
> > > In the meantime, I tested for RC2 that a local cluster of NN + standby
> +
> > > observer + QJM works as expected for some basic HDFS commands.
> > >
> >
> > OK. Could you have a go with a (locally built) patch release
> >
> > >
> > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > <st...@cloudera.com.invalid>
> > > wrote:
> > >
> > >> shipping broken hdfs isn't something we'd want to do, but if we can be
> > >> confident that all other issues can be addressed in RC3 then I'd be
> > happy.
> > >>
> > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> > >>
> > >> > I will highlight that I am completely fed up with doing this
> release
> > >> and
> > >> >> really want to get it out the way -for which I depend on support
> from
> > >> as
> > >> >> many other developers as possible.
> > >> >
> > >> >
> > >> > hmm, I can feel the pain. I tried to find if there is any config or
> > any
> > >> > workaround which can dodge this HDFS issue, but unfortunately
> couldn't
> > >> find
> > >> > any. If someone does a getListing with needLocation and the file
> > doesn't
> > >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> > just
> > >> > the exception, AFAIK Observer reads have some logic around handling
> > FNF
> > >> > specifically, that it attempts Active NN or something like that in
> > such
> > >> > cases, So, that will be broken as well for this use case.
> > >> >
> > >> > Now, there is no denying the fact there is an issue on the HDFS
> side,
> > >> and
> > >> > it has already been too much work on your side, so you can argue
> that
> > it
> > >> > might not be a very frequent use case or so. It's your call.
> > >> >
> > >> > Just sharing, no intentions of saying you should do that, But as an
> RM
> > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > your
> > >> > call and discretion. As far as I know a release can not be vetoed by
> > >> > "anybody" as per the apache by laws.(
> > >> > https://www.apache.org/legal/release-policy.html#release-approval).
> > >> Even
> > >> > our bylaws say that product release requires a Lazy Majority not a
> > >> > Consensus Approval.
> > >> >
> > >> > So, you have a way out. You guys are 2 already and 1 I will give
> you a
> > >> > pass, in case you are really in a state: ''Get me out of this mess"
> > >> state,
> > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > >> couldn't
> > >> > reach the end for any of the RC's
> > >> >
> > >> > -Ayush
> > >> >
> > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org>
> wrote:
> > >> >
> > >> >> While this RC is not going to be final, I just wanted to share the
> > >> results
> > >> >> of the testing I have done so far with RC1 and RC2.
> > >> >>
> > >> >> * Signature: ok
> > >> >> * Checksum : ok
> > >> >> * Rat check (1.8.0_341): ok
> > >> >> - mvn clean apache-rat:check
> > >> >> * Built from source (1.8.0_341): ok
> > >> >> - mvn clean install -DskipTests
> > >> >> * Built tar from source (1.8.0_341): ok
> > >> >> - mvn clean package -Pdist -DskipTests -Dtar
> > >> -Dmaven.javadoc.skip=true
> > >> >>
> > >> >> * Built images using the tarball, installed and started all of
> Hdfs,
> > >> JHS
> > >> >> and Yarn components
> > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> Mapreduce
> > >> job
> > >> >> * Hdfs CRUD tests
> > >> >> * MapReduce wordcount job
> > >> >>
> > >> >> * Ran S3A tests with scale profile against us-west-2:
> > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > >> >>
> > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> > >> This
> > >> >> is
> > >> >> consistently failing, looks like a recent regression.
> > >> >> I was also able to repro on trunk, will create Jira.
> > >> >>
> > >> >>
> > >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> > >> >> <st...@cloudera.com.invalid>
> > >> >> wrote:
> > >> >>
> > >> >> > Mukund and I have put together a release candidate (RC2) for
> Hadoop
> > >> >> 3.3.5.
> > >> >> >
> > >> >> > We need anyone who can to verify the source and binary artifacts,
> > >> >> > including those JARs staged on maven, the site documentation and
> > the
> > >> >> arm64
> > >> >> > tar file.
> > >> >> >
> > >> >> > The RC is available at:
> > >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> > >> >> >
> > >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> > >> >> >
> > >> >> > The maven artifacts are staged at
> > >> >> >
> > >> >>
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > >> >> >
> > >> >> > You can find my public key at:
> > >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >> >> >
> > >> >> > Change log
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> > >> >> >
> > >> >> > Release notes
> > >> >> >
> > >> >> >
> > >> >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> > >> >> >
> > >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> > >> >> >
> > >> >> > As to what changed since the RC1 attempt last week
> > >> >> >
> > >> >> >
> > >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma
> there)
> > >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md
> file
> > >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> > (creating
> > >> >> build
> > >> >> > issues in maven 3.9.0)
> > >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> > >> >> (#5429)
> > >> >> >
> > >> >> >
> > >> >> > Note, because the arm64 binaries are built separately on a
> > different
> > >> >> > platform and JVM, their jar files may not match those of the x86
> > >> >> > release -and therefore the maven artifacts. I don't think this is
> > >> >> > an issue (the ASF actually releases source tarballs, the binaries
> > are
> > >> >> > there for help only, though with the maven repo that's a bit
> > >> blurred).
> > >> >> >
> > >> >> > The only way to be consistent would actually untar the
> x86.tar.gz,
> > >> >> > overwrite its binaries with the arm stuff, retar, sign and push
> out
> > >> >> > for the vote. Even automating that would be risky.
> > >> >> >
> > >> >> > Please try the release and vote. The vote will run for 5 days.
> > >> >> >
> > >> >> > Steve and Mukund
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
A minor update on ITestS3AConcurrentOps#testParallelRename
I was previously connected to a vpn due to which bandwidth was getting
throttled earlier. Ran the test again today without vpn and had no issues
(earlier only 40% of the overall putObject were able to get completed
within timeout).
On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
>
> > Thanks Steve. I see now that the branch cut was way back in October so I
> > definitely understand your frustration here!
> >
> > This made me realize that HDFS-16832
> > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> very
> > similar issue as the aforementioned HDFS-16923, is also missing from the
> > RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> > the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > branch-3.3.5, so they are ready if/when an RC3 is cut.
> >
>
> thanks.
>
> >
> > In the meantime, I tested for RC2 that a local cluster of NN + standby +
> > observer + QJM works as expected for some basic HDFS commands.
> >
>
> OK. Could you have a go with a (locally built) patch release
>
> >
> > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> >> shipping broken hdfs isn't something we'd want to do, but if we can be
> >> confident that all other issues can be addressed in RC3 then I'd be
> happy.
> >>
> >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> > I will highlight that I am completely fed up with doing this release
> >> and
> >> >> really want to get it out the way -for which I depend on support from
> >> as
> >> >> many other developers as possible.
> >> >
> >> >
> >> > hmm, I can feel the pain. I tried to find if there is any config or
> any
> >> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> >> find
> >> > any. If someone does a getListing with needLocation and the file
> doesn't
> >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> just
> >> > the exception, AFAIK Observer reads have some logic around handling
> FNF
> >> > specifically, that it attempts Active NN or something like that in
> such
> >> > cases, So, that will be broken as well for this use case.
> >> >
> >> > Now, there is no denying the fact there is an issue on the HDFS side,
> >> and
> >> > it has already been too much work on your side, so you can argue that
> it
> >> > might not be a very frequent use case or so. It's your call.
> >> >
> >> > Just sharing, no intentions of saying you should do that, But as an RM
> >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> your
> >> > call and discretion. As far as I know a release can not be vetoed by
> >> > "anybody" as per the apache by laws.(
> >> > https://www.apache.org/legal/release-policy.html#release-approval).
> >> Even
> >> > our bylaws say that product release requires a Lazy Majority not a
> >> > Consensus Approval.
> >> >
> >> > So, you have a way out. You guys are 2 already and 1 I will give you a
> >> > pass, in case you are really in a state: ''Get me out of this mess"
> >> state,
> >> > my basic validations on x86 & Aarch64 both are passing as of now,
> >> couldn't
> >> > reach the end for any of the RC's
> >> >
> >> > -Ayush
> >> >
> >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >> >
> >> >> While this RC is not going to be final, I just wanted to share the
> >> results
> >> >> of the testing I have done so far with RC1 and RC2.
> >> >>
> >> >> * Signature: ok
> >> >> * Checksum : ok
> >> >> * Rat check (1.8.0_341): ok
> >> >> - mvn clean apache-rat:check
> >> >> * Built from source (1.8.0_341): ok
> >> >> - mvn clean install -DskipTests
> >> >> * Built tar from source (1.8.0_341): ok
> >> >> - mvn clean package -Pdist -DskipTests -Dtar
> >> -Dmaven.javadoc.skip=true
> >> >>
> >> >> * Built images using the tarball, installed and started all of Hdfs,
> >> JHS
> >> >> and Yarn components
> >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> >> job
> >> >> * Hdfs CRUD tests
> >> >> * MapReduce wordcount job
> >> >>
> >> >> * Ran S3A tests with scale profile against us-west-2:
> >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >> >>
> >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> >> This
> >> >> is
> >> >> consistently failing, looks like a recent regression.
> >> >> I was also able to repro on trunk, will create Jira.
> >> >>
> >> >>
> >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> >> <st...@cloudera.com.invalid>
> >> >> wrote:
> >> >>
> >> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> >> 3.3.5.
> >> >> >
> >> >> > We need anyone who can to verify the source and binary artifacts,
> >> >> > including those JARs staged on maven, the site documentation and
> the
> >> >> arm64
> >> >> > tar file.
> >> >> >
> >> >> > The RC is available at:
> >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >> >
> >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >> >
> >> >> > The maven artifacts are staged at
> >> >> >
> >> >>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >> >
> >> >> > You can find my public key at:
> >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >> >
> >> >> > Change log
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >> >
> >> >> > Release notes
> >> >> >
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >> >
> >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >> >
> >> >> > As to what changed since the RC1 attempt last week
> >> >> >
> >> >> >
> >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> (creating
> >> >> build
> >> >> > issues in maven 3.9.0)
> >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> >> (#5429)
> >> >> >
> >> >> >
> >> >> > Note, because the arm64 binaries are built separately on a
> different
> >> >> > platform and JVM, their jar files may not match those of the x86
> >> >> > release -and therefore the maven artifacts. I don't think this is
> >> >> > an issue (the ASF actually releases source tarballs, the binaries
> are
> >> >> > there for help only, though with the maven repo that's a bit
> >> blurred).
> >> >> >
> >> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> >> > for the vote. Even automating that would be risky.
> >> >> >
> >> >> > Please try the release and vote. The vote will run for 5 days.
> >> >> >
> >> >> > Steve and Mukund
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
A minor update on ITestS3AConcurrentOps#testParallelRename
I was previously connected to a vpn due to which bandwidth was getting
throttled earlier. Ran the test again today without vpn and had no issues
(earlier only 40% of the overall putObject were able to get completed
within timeout).
On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
>
> > Thanks Steve. I see now that the branch cut was way back in October so I
> > definitely understand your frustration here!
> >
> > This made me realize that HDFS-16832
> > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> very
> > similar issue as the aforementioned HDFS-16923, is also missing from the
> > RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> > the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > branch-3.3.5, so they are ready if/when an RC3 is cut.
> >
>
> thanks.
>
> >
> > In the meantime, I tested for RC2 that a local cluster of NN + standby +
> > observer + QJM works as expected for some basic HDFS commands.
> >
>
> OK. Could you have a go with a (locally built) patch release
>
> >
> > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> >> shipping broken hdfs isn't something we'd want to do, but if we can be
> >> confident that all other issues can be addressed in RC3 then I'd be
> happy.
> >>
> >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> > I will highlight that I am completely fed up with doing this release
> >> and
> >> >> really want to get it out the way -for which I depend on support from
> >> as
> >> >> many other developers as possible.
> >> >
> >> >
> >> > hmm, I can feel the pain. I tried to find if there is any config or
> any
> >> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> >> find
> >> > any. If someone does a getListing with needLocation and the file
> doesn't
> >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> just
> >> > the exception, AFAIK Observer reads have some logic around handling
> FNF
> >> > specifically, that it attempts Active NN or something like that in
> such
> >> > cases, So, that will be broken as well for this use case.
> >> >
> >> > Now, there is no denying the fact there is an issue on the HDFS side,
> >> and
> >> > it has already been too much work on your side, so you can argue that
> it
> >> > might not be a very frequent use case or so. It's your call.
> >> >
> >> > Just sharing, no intentions of saying you should do that, But as an RM
> >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> your
> >> > call and discretion. As far as I know a release can not be vetoed by
> >> > "anybody" as per the apache by laws.(
> >> > https://www.apache.org/legal/release-policy.html#release-approval).
> >> Even
> >> > our bylaws say that product release requires a Lazy Majority not a
> >> > Consensus Approval.
> >> >
> >> > So, you have a way out. You guys are 2 already and 1 I will give you a
> >> > pass, in case you are really in a state: ''Get me out of this mess"
> >> state,
> >> > my basic validations on x86 & Aarch64 both are passing as of now,
> >> couldn't
> >> > reach the end for any of the RC's
> >> >
> >> > -Ayush
> >> >
> >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >> >
> >> >> While this RC is not going to be final, I just wanted to share the
> >> results
> >> >> of the testing I have done so far with RC1 and RC2.
> >> >>
> >> >> * Signature: ok
> >> >> * Checksum : ok
> >> >> * Rat check (1.8.0_341): ok
> >> >> - mvn clean apache-rat:check
> >> >> * Built from source (1.8.0_341): ok
> >> >> - mvn clean install -DskipTests
> >> >> * Built tar from source (1.8.0_341): ok
> >> >> - mvn clean package -Pdist -DskipTests -Dtar
> >> -Dmaven.javadoc.skip=true
> >> >>
> >> >> * Built images using the tarball, installed and started all of Hdfs,
> >> JHS
> >> >> and Yarn components
> >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> >> job
> >> >> * Hdfs CRUD tests
> >> >> * MapReduce wordcount job
> >> >>
> >> >> * Ran S3A tests with scale profile against us-west-2:
> >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >> >>
> >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> >> This
> >> >> is
> >> >> consistently failing, looks like a recent regression.
> >> >> I was also able to repro on trunk, will create Jira.
> >> >>
> >> >>
> >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> >> <st...@cloudera.com.invalid>
> >> >> wrote:
> >> >>
> >> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> >> 3.3.5.
> >> >> >
> >> >> > We need anyone who can to verify the source and binary artifacts,
> >> >> > including those JARs staged on maven, the site documentation and
> the
> >> >> arm64
> >> >> > tar file.
> >> >> >
> >> >> > The RC is available at:
> >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >> >
> >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >> >
> >> >> > The maven artifacts are staged at
> >> >> >
> >> >>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >> >
> >> >> > You can find my public key at:
> >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >> >
> >> >> > Change log
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >> >
> >> >> > Release notes
> >> >> >
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >> >
> >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >> >
> >> >> > As to what changed since the RC1 attempt last week
> >> >> >
> >> >> >
> >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> (creating
> >> >> build
> >> >> > issues in maven 3.9.0)
> >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> >> (#5429)
> >> >> >
> >> >> >
> >> >> > Note, because the arm64 binaries are built separately on a
> different
> >> >> > platform and JVM, their jar files may not match those of the x86
> >> >> > release -and therefore the maven artifacts. I don't think this is
> >> >> > an issue (the ASF actually releases source tarballs, the binaries
> are
> >> >> > there for help only, though with the maven repo that's a bit
> >> blurred).
> >> >> >
> >> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> >> > for the vote. Even automating that would be risky.
> >> >> >
> >> >> > Please try the release and vote. The vote will run for 5 days.
> >> >> >
> >> >> > Steve and Mukund
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
A minor update on ITestS3AConcurrentOps#testParallelRename
I was previously connected to a vpn due to which bandwidth was getting
throttled earlier. Ran the test again today without vpn and had no issues
(earlier only 40% of the overall putObject were able to get completed
within timeout).
On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
>
> > Thanks Steve. I see now that the branch cut was way back in October so I
> > definitely understand your frustration here!
> >
> > This made me realize that HDFS-16832
> > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> very
> > similar issue as the aforementioned HDFS-16923, is also missing from the
> > RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> > the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > branch-3.3.5, so they are ready if/when an RC3 is cut.
> >
>
> thanks.
>
> >
> > In the meantime, I tested for RC2 that a local cluster of NN + standby +
> > observer + QJM works as expected for some basic HDFS commands.
> >
>
> OK. Could you have a go with a (locally built) patch release
>
> >
> > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> >> shipping broken hdfs isn't something we'd want to do, but if we can be
> >> confident that all other issues can be addressed in RC3 then I'd be
> happy.
> >>
> >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> > I will highlight that I am completely fed up with doing this release
> >> and
> >> >> really want to get it out the way -for which I depend on support from
> >> as
> >> >> many other developers as possible.
> >> >
> >> >
> >> > hmm, I can feel the pain. I tried to find if there is any config or
> any
> >> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> >> find
> >> > any. If someone does a getListing with needLocation and the file
> doesn't
> >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> just
> >> > the exception, AFAIK Observer reads have some logic around handling
> FNF
> >> > specifically, that it attempts Active NN or something like that in
> such
> >> > cases, So, that will be broken as well for this use case.
> >> >
> >> > Now, there is no denying the fact there is an issue on the HDFS side,
> >> and
> >> > it has already been too much work on your side, so you can argue that
> it
> >> > might not be a very frequent use case or so. It's your call.
> >> >
> >> > Just sharing, no intentions of saying you should do that, But as an RM
> >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> your
> >> > call and discretion. As far as I know a release can not be vetoed by
> >> > "anybody" as per the apache by laws.(
> >> > https://www.apache.org/legal/release-policy.html#release-approval).
> >> Even
> >> > our bylaws say that product release requires a Lazy Majority not a
> >> > Consensus Approval.
> >> >
> >> > So, you have a way out. You guys are 2 already and 1 I will give you a
> >> > pass, in case you are really in a state: ''Get me out of this mess"
> >> state,
> >> > my basic validations on x86 & Aarch64 both are passing as of now,
> >> couldn't
> >> > reach the end for any of the RC's
> >> >
> >> > -Ayush
> >> >
> >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >> >
> >> >> While this RC is not going to be final, I just wanted to share the
> >> results
> >> >> of the testing I have done so far with RC1 and RC2.
> >> >>
> >> >> * Signature: ok
> >> >> * Checksum : ok
> >> >> * Rat check (1.8.0_341): ok
> >> >> - mvn clean apache-rat:check
> >> >> * Built from source (1.8.0_341): ok
> >> >> - mvn clean install -DskipTests
> >> >> * Built tar from source (1.8.0_341): ok
> >> >> - mvn clean package -Pdist -DskipTests -Dtar
> >> -Dmaven.javadoc.skip=true
> >> >>
> >> >> * Built images using the tarball, installed and started all of Hdfs,
> >> JHS
> >> >> and Yarn components
> >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> >> job
> >> >> * Hdfs CRUD tests
> >> >> * MapReduce wordcount job
> >> >>
> >> >> * Ran S3A tests with scale profile against us-west-2:
> >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >> >>
> >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> >> This
> >> >> is
> >> >> consistently failing, looks like a recent regression.
> >> >> I was also able to repro on trunk, will create Jira.
> >> >>
> >> >>
> >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> >> <st...@cloudera.com.invalid>
> >> >> wrote:
> >> >>
> >> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> >> 3.3.5.
> >> >> >
> >> >> > We need anyone who can to verify the source and binary artifacts,
> >> >> > including those JARs staged on maven, the site documentation and
> the
> >> >> arm64
> >> >> > tar file.
> >> >> >
> >> >> > The RC is available at:
> >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >> >
> >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >> >
> >> >> > The maven artifacts are staged at
> >> >> >
> >> >>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >> >
> >> >> > You can find my public key at:
> >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >> >
> >> >> > Change log
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >> >
> >> >> > Release notes
> >> >> >
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >> >
> >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >> >
> >> >> > As to what changed since the RC1 attempt last week
> >> >> >
> >> >> >
> >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> (creating
> >> >> build
> >> >> > issues in maven 3.9.0)
> >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> >> (#5429)
> >> >> >
> >> >> >
> >> >> > Note, because the arm64 binaries are built separately on a
> different
> >> >> > platform and JVM, their jar files may not match those of the x86
> >> >> > release -and therefore the maven artifacts. I don't think this is
> >> >> > an issue (the ASF actually releases source tarballs, the binaries
> are
> >> >> > there for help only, though with the maven repo that's a bit
> >> blurred).
> >> >> >
> >> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> >> > for the vote. Even automating that would be risky.
> >> >> >
> >> >> > Please try the release and vote. The vote will run for 5 days.
> >> >> >
> >> >> > Steve and Mukund
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
A minor update on ITestS3AConcurrentOps#testParallelRename
I was previously connected to a vpn due to which bandwidth was getting
throttled earlier. Ran the test again today without vpn and had no issues
(earlier only 40% of the overall putObject were able to get completed
within timeout).
On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
>
> > Thanks Steve. I see now that the branch cut was way back in October so I
> > definitely understand your frustration here!
> >
> > This made me realize that HDFS-16832
> > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> very
> > similar issue as the aforementioned HDFS-16923, is also missing from the
> > RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> > the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > branch-3.3.5, so they are ready if/when an RC3 is cut.
> >
>
> thanks.
>
> >
> > In the meantime, I tested for RC2 that a local cluster of NN + standby +
> > observer + QJM works as expected for some basic HDFS commands.
> >
>
> OK. Could you have a go with a (locally built) patch release
>
> >
> > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> <st...@cloudera.com.invalid>
> > wrote:
> >
> >> shipping broken hdfs isn't something we'd want to do, but if we can be
> >> confident that all other issues can be addressed in RC3 then I'd be
> happy.
> >>
> >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> > I will highlight that I am completely fed up with doing this release
> >> and
> >> >> really want to get it out the way -for which I depend on support from
> >> as
> >> >> many other developers as possible.
> >> >
> >> >
> >> > hmm, I can feel the pain. I tried to find if there is any config or
> any
> >> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> >> find
> >> > any. If someone does a getListing with needLocation and the file
> doesn't
> >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> just
> >> > the exception, AFAIK Observer reads have some logic around handling
> FNF
> >> > specifically, that it attempts Active NN or something like that in
> such
> >> > cases, So, that will be broken as well for this use case.
> >> >
> >> > Now, there is no denying the fact there is an issue on the HDFS side,
> >> and
> >> > it has already been too much work on your side, so you can argue that
> it
> >> > might not be a very frequent use case or so. It's your call.
> >> >
> >> > Just sharing, no intentions of saying you should do that, But as an RM
> >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> your
> >> > call and discretion. As far as I know a release can not be vetoed by
> >> > "anybody" as per the apache by laws.(
> >> > https://www.apache.org/legal/release-policy.html#release-approval).
> >> Even
> >> > our bylaws say that product release requires a Lazy Majority not a
> >> > Consensus Approval.
> >> >
> >> > So, you have a way out. You guys are 2 already and 1 I will give you a
> >> > pass, in case you are really in a state: ''Get me out of this mess"
> >> state,
> >> > my basic validations on x86 & Aarch64 both are passing as of now,
> >> couldn't
> >> > reach the end for any of the RC's
> >> >
> >> > -Ayush
> >> >
> >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >> >
> >> >> While this RC is not going to be final, I just wanted to share the
> >> results
> >> >> of the testing I have done so far with RC1 and RC2.
> >> >>
> >> >> * Signature: ok
> >> >> * Checksum : ok
> >> >> * Rat check (1.8.0_341): ok
> >> >> - mvn clean apache-rat:check
> >> >> * Built from source (1.8.0_341): ok
> >> >> - mvn clean install -DskipTests
> >> >> * Built tar from source (1.8.0_341): ok
> >> >> - mvn clean package -Pdist -DskipTests -Dtar
> >> -Dmaven.javadoc.skip=true
> >> >>
> >> >> * Built images using the tarball, installed and started all of Hdfs,
> >> JHS
> >> >> and Yarn components
> >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> >> job
> >> >> * Hdfs CRUD tests
> >> >> * MapReduce wordcount job
> >> >>
> >> >> * Ran S3A tests with scale profile against us-west-2:
> >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >> >>
> >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> >> This
> >> >> is
> >> >> consistently failing, looks like a recent regression.
> >> >> I was also able to repro on trunk, will create Jira.
> >> >>
> >> >>
> >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> >> <st...@cloudera.com.invalid>
> >> >> wrote:
> >> >>
> >> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> >> 3.3.5.
> >> >> >
> >> >> > We need anyone who can to verify the source and binary artifacts,
> >> >> > including those JARs staged on maven, the site documentation and
> the
> >> >> arm64
> >> >> > tar file.
> >> >> >
> >> >> > The RC is available at:
> >> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >> >
> >> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >> >
> >> >> > The maven artifacts are staged at
> >> >> >
> >> >>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >> >
> >> >> > You can find my public key at:
> >> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >> >
> >> >> > Change log
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >> >
> >> >> > Release notes
> >> >> >
> >> >> >
> >> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >> >
> >> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >> >
> >> >> > As to what changed since the RC1 attempt last week
> >> >> >
> >> >> >
> >> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)"
> (creating
> >> >> build
> >> >> > issues in maven 3.9.0)
> >> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> >> (#5429)
> >> >> >
> >> >> >
> >> >> > Note, because the arm64 binaries are built separately on a
> different
> >> >> > platform and JVM, their jar files may not match those of the x86
> >> >> > release -and therefore the maven artifacts. I don't think this is
> >> >> > an issue (the ASF actually releases source tarballs, the binaries
> are
> >> >> > there for help only, though with the maven repo that's a bit
> >> blurred).
> >> >> >
> >> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> >> > for the vote. Even automating that would be risky.
> >> >> >
> >> >> > Please try the release and vote. The vote will run for 5 days.
> >> >> >
> >> >> > Steve and Mukund
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> Thanks Steve. I see now that the branch cut was way back in October so I
> definitely understand your frustration here!
>
> This made me realize that HDFS-16832
> <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
> similar issue as the aforementioned HDFS-16923, is also missing from the
> RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> branch-3.3.5, so they are ready if/when an RC3 is cut.
>
thanks.
>
> In the meantime, I tested for RC2 that a local cluster of NN + standby +
> observer + QJM works as expected for some basic HDFS commands.
>
OK. Could you have a go with a (locally built) patch release
>
> On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> shipping broken hdfs isn't something we'd want to do, but if we can be
>> confident that all other issues can be addressed in RC3 then I'd be happy.
>>
>> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > I will highlight that I am completely fed up with doing this release
>> and
>> >> really want to get it out the way -for which I depend on support from
>> as
>> >> many other developers as possible.
>> >
>> >
>> > hmm, I can feel the pain. I tried to find if there is any config or any
>> > workaround which can dodge this HDFS issue, but unfortunately couldn't
>> find
>> > any. If someone does a getListing with needLocation and the file doesn't
>> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
>> > the exception, AFAIK Observer reads have some logic around handling FNF
>> > specifically, that it attempts Active NN or something like that in such
>> > cases, So, that will be broken as well for this use case.
>> >
>> > Now, there is no denying the fact there is an issue on the HDFS side,
>> and
>> > it has already been too much work on your side, so you can argue that it
>> > might not be a very frequent use case or so. It's your call.
>> >
>> > Just sharing, no intentions of saying you should do that, But as an RM
>> > "nobody" can force you for a new iteration of a RC, it is gonna be your
>> > call and discretion. As far as I know a release can not be vetoed by
>> > "anybody" as per the apache by laws.(
>> > https://www.apache.org/legal/release-policy.html#release-approval).
>> Even
>> > our bylaws say that product release requires a Lazy Majority not a
>> > Consensus Approval.
>> >
>> > So, you have a way out. You guys are 2 already and 1 I will give you a
>> > pass, in case you are really in a state: ''Get me out of this mess"
>> state,
>> > my basic validations on x86 & Aarch64 both are passing as of now,
>> couldn't
>> > reach the end for any of the RC's
>> >
>> > -Ayush
>> >
>> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>> >
>> >> While this RC is not going to be final, I just wanted to share the
>> results
>> >> of the testing I have done so far with RC1 and RC2.
>> >>
>> >> * Signature: ok
>> >> * Checksum : ok
>> >> * Rat check (1.8.0_341): ok
>> >> - mvn clean apache-rat:check
>> >> * Built from source (1.8.0_341): ok
>> >> - mvn clean install -DskipTests
>> >> * Built tar from source (1.8.0_341): ok
>> >> - mvn clean package -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>> >>
>> >> * Built images using the tarball, installed and started all of Hdfs,
>> JHS
>> >> and Yarn components
>> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
>> job
>> >> * Hdfs CRUD tests
>> >> * MapReduce wordcount job
>> >>
>> >> * Ran S3A tests with scale profile against us-west-2:
>> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>> >>
>> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
>> This
>> >> is
>> >> consistently failing, looks like a recent regression.
>> >> I was also able to repro on trunk, will create Jira.
>> >>
>> >>
>> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> >> <st...@cloudera.com.invalid>
>> >> wrote:
>> >>
>> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> >> 3.3.5.
>> >> >
>> >> > We need anyone who can to verify the source and binary artifacts,
>> >> > including those JARs staged on maven, the site documentation and the
>> >> arm64
>> >> > tar file.
>> >> >
>> >> > The RC is available at:
>> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >> >
>> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >> >
>> >> > The maven artifacts are staged at
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >> >
>> >> > You can find my public key at:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > Change log
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >> >
>> >> > Release notes
>> >> >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >> >
>> >> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >> >
>> >> > As to what changed since the RC1 attempt last week
>> >> >
>> >> >
>> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> >> build
>> >> > issues in maven 3.9.0)
>> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> >> (#5429)
>> >> >
>> >> >
>> >> > Note, because the arm64 binaries are built separately on a different
>> >> > platform and JVM, their jar files may not match those of the x86
>> >> > release -and therefore the maven artifacts. I don't think this is
>> >> > an issue (the ASF actually releases source tarballs, the binaries are
>> >> > there for help only, though with the maven repo that's a bit
>> blurred).
>> >> >
>> >> > The only way to be consistent would actually untar the x86.tar.gz,
>> >> > overwrite its binaries with the arm stuff, retar, sign and push out
>> >> > for the vote. Even automating that would be risky.
>> >> >
>> >> > Please try the release and vote. The vote will run for 5 days.
>> >> >
>> >> > Steve and Mukund
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> Thanks Steve. I see now that the branch cut was way back in October so I
> definitely understand your frustration here!
>
> This made me realize that HDFS-16832
> <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
> similar issue as the aforementioned HDFS-16923, is also missing from the
> RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> branch-3.3.5, so they are ready if/when an RC3 is cut.
>
thanks.
>
> In the meantime, I tested for RC2 that a local cluster of NN + standby +
> observer + QJM works as expected for some basic HDFS commands.
>
OK. Could you have a go with a (locally built) patch release
>
> On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> shipping broken hdfs isn't something we'd want to do, but if we can be
>> confident that all other issues can be addressed in RC3 then I'd be happy.
>>
>> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > I will highlight that I am completely fed up with doing this release
>> and
>> >> really want to get it out the way -for which I depend on support from
>> as
>> >> many other developers as possible.
>> >
>> >
>> > hmm, I can feel the pain. I tried to find if there is any config or any
>> > workaround which can dodge this HDFS issue, but unfortunately couldn't
>> find
>> > any. If someone does a getListing with needLocation and the file doesn't
>> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
>> > the exception, AFAIK Observer reads have some logic around handling FNF
>> > specifically, that it attempts Active NN or something like that in such
>> > cases, So, that will be broken as well for this use case.
>> >
>> > Now, there is no denying the fact there is an issue on the HDFS side,
>> and
>> > it has already been too much work on your side, so you can argue that it
>> > might not be a very frequent use case or so. It's your call.
>> >
>> > Just sharing, no intentions of saying you should do that, But as an RM
>> > "nobody" can force you for a new iteration of a RC, it is gonna be your
>> > call and discretion. As far as I know a release can not be vetoed by
>> > "anybody" as per the apache by laws.(
>> > https://www.apache.org/legal/release-policy.html#release-approval).
>> Even
>> > our bylaws say that product release requires a Lazy Majority not a
>> > Consensus Approval.
>> >
>> > So, you have a way out. You guys are 2 already and 1 I will give you a
>> > pass, in case you are really in a state: ''Get me out of this mess"
>> state,
>> > my basic validations on x86 & Aarch64 both are passing as of now,
>> couldn't
>> > reach the end for any of the RC's
>> >
>> > -Ayush
>> >
>> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>> >
>> >> While this RC is not going to be final, I just wanted to share the
>> results
>> >> of the testing I have done so far with RC1 and RC2.
>> >>
>> >> * Signature: ok
>> >> * Checksum : ok
>> >> * Rat check (1.8.0_341): ok
>> >> - mvn clean apache-rat:check
>> >> * Built from source (1.8.0_341): ok
>> >> - mvn clean install -DskipTests
>> >> * Built tar from source (1.8.0_341): ok
>> >> - mvn clean package -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>> >>
>> >> * Built images using the tarball, installed and started all of Hdfs,
>> JHS
>> >> and Yarn components
>> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
>> job
>> >> * Hdfs CRUD tests
>> >> * MapReduce wordcount job
>> >>
>> >> * Ran S3A tests with scale profile against us-west-2:
>> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>> >>
>> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
>> This
>> >> is
>> >> consistently failing, looks like a recent regression.
>> >> I was also able to repro on trunk, will create Jira.
>> >>
>> >>
>> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> >> <st...@cloudera.com.invalid>
>> >> wrote:
>> >>
>> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> >> 3.3.5.
>> >> >
>> >> > We need anyone who can to verify the source and binary artifacts,
>> >> > including those JARs staged on maven, the site documentation and the
>> >> arm64
>> >> > tar file.
>> >> >
>> >> > The RC is available at:
>> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >> >
>> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >> >
>> >> > The maven artifacts are staged at
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >> >
>> >> > You can find my public key at:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > Change log
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >> >
>> >> > Release notes
>> >> >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >> >
>> >> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >> >
>> >> > As to what changed since the RC1 attempt last week
>> >> >
>> >> >
>> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> >> build
>> >> > issues in maven 3.9.0)
>> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> >> (#5429)
>> >> >
>> >> >
>> >> > Note, because the arm64 binaries are built separately on a different
>> >> > platform and JVM, their jar files may not match those of the x86
>> >> > release -and therefore the maven artifacts. I don't think this is
>> >> > an issue (the ASF actually releases source tarballs, the binaries are
>> >> > there for help only, though with the maven repo that's a bit
>> blurred).
>> >> >
>> >> > The only way to be consistent would actually untar the x86.tar.gz,
>> >> > overwrite its binaries with the arm stuff, retar, sign and push out
>> >> > for the vote. Even automating that would be risky.
>> >> >
>> >> > Please try the release and vote. The vote will run for 5 days.
>> >> >
>> >> > Steve and Mukund
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> Thanks Steve. I see now that the branch cut was way back in October so I
> definitely understand your frustration here!
>
> This made me realize that HDFS-16832
> <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
> similar issue as the aforementioned HDFS-16923, is also missing from the
> RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> branch-3.3.5, so they are ready if/when an RC3 is cut.
>
thanks.
>
> In the meantime, I tested for RC2 that a local cluster of NN + standby +
> observer + QJM works as expected for some basic HDFS commands.
>
OK. Could you have a go with a (locally built) patch release
>
> On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> shipping broken hdfs isn't something we'd want to do, but if we can be
>> confident that all other issues can be addressed in RC3 then I'd be happy.
>>
>> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > I will highlight that I am completely fed up with doing this release
>> and
>> >> really want to get it out the way -for which I depend on support from
>> as
>> >> many other developers as possible.
>> >
>> >
>> > hmm, I can feel the pain. I tried to find if there is any config or any
>> > workaround which can dodge this HDFS issue, but unfortunately couldn't
>> find
>> > any. If someone does a getListing with needLocation and the file doesn't
>> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
>> > the exception, AFAIK Observer reads have some logic around handling FNF
>> > specifically, that it attempts Active NN or something like that in such
>> > cases, So, that will be broken as well for this use case.
>> >
>> > Now, there is no denying the fact there is an issue on the HDFS side,
>> and
>> > it has already been too much work on your side, so you can argue that it
>> > might not be a very frequent use case or so. It's your call.
>> >
>> > Just sharing, no intentions of saying you should do that, But as an RM
>> > "nobody" can force you for a new iteration of a RC, it is gonna be your
>> > call and discretion. As far as I know a release can not be vetoed by
>> > "anybody" as per the apache by laws.(
>> > https://www.apache.org/legal/release-policy.html#release-approval).
>> Even
>> > our bylaws say that product release requires a Lazy Majority not a
>> > Consensus Approval.
>> >
>> > So, you have a way out. You guys are 2 already and 1 I will give you a
>> > pass, in case you are really in a state: ''Get me out of this mess"
>> state,
>> > my basic validations on x86 & Aarch64 both are passing as of now,
>> couldn't
>> > reach the end for any of the RC's
>> >
>> > -Ayush
>> >
>> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>> >
>> >> While this RC is not going to be final, I just wanted to share the
>> results
>> >> of the testing I have done so far with RC1 and RC2.
>> >>
>> >> * Signature: ok
>> >> * Checksum : ok
>> >> * Rat check (1.8.0_341): ok
>> >> - mvn clean apache-rat:check
>> >> * Built from source (1.8.0_341): ok
>> >> - mvn clean install -DskipTests
>> >> * Built tar from source (1.8.0_341): ok
>> >> - mvn clean package -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>> >>
>> >> * Built images using the tarball, installed and started all of Hdfs,
>> JHS
>> >> and Yarn components
>> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
>> job
>> >> * Hdfs CRUD tests
>> >> * MapReduce wordcount job
>> >>
>> >> * Ran S3A tests with scale profile against us-west-2:
>> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>> >>
>> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
>> This
>> >> is
>> >> consistently failing, looks like a recent regression.
>> >> I was also able to repro on trunk, will create Jira.
>> >>
>> >>
>> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> >> <st...@cloudera.com.invalid>
>> >> wrote:
>> >>
>> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> >> 3.3.5.
>> >> >
>> >> > We need anyone who can to verify the source and binary artifacts,
>> >> > including those JARs staged on maven, the site documentation and the
>> >> arm64
>> >> > tar file.
>> >> >
>> >> > The RC is available at:
>> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >> >
>> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >> >
>> >> > The maven artifacts are staged at
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >> >
>> >> > You can find my public key at:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > Change log
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >> >
>> >> > Release notes
>> >> >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >> >
>> >> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >> >
>> >> > As to what changed since the RC1 attempt last week
>> >> >
>> >> >
>> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> >> build
>> >> > issues in maven 3.9.0)
>> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> >> (#5429)
>> >> >
>> >> >
>> >> > Note, because the arm64 binaries are built separately on a different
>> >> > platform and JVM, their jar files may not match those of the x86
>> >> > release -and therefore the maven artifacts. I don't think this is
>> >> > an issue (the ASF actually releases source tarballs, the binaries are
>> >> > there for help only, though with the maven repo that's a bit
>> blurred).
>> >> >
>> >> > The only way to be consistent would actually untar the x86.tar.gz,
>> >> > overwrite its binaries with the arm stuff, retar, sign and push out
>> >> > for the vote. Even automating that would be risky.
>> >> >
>> >> > Please try the release and vote. The vote will run for 5 days.
>> >> >
>> >> > Steve and Mukund
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Sat, 4 Mar 2023 at 01:47, Erik Krogen <xk...@apache.org> wrote:
> Thanks Steve. I see now that the branch cut was way back in October so I
> definitely understand your frustration here!
>
> This made me realize that HDFS-16832
> <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
> similar issue as the aforementioned HDFS-16923, is also missing from the
> RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> branch-3.3.5, so they are ready if/when an RC3 is cut.
>
thanks.
>
> In the meantime, I tested for RC2 that a local cluster of NN + standby +
> observer + QJM works as expected for some basic HDFS commands.
>
OK. Could you have a go with a (locally built) patch release
>
> On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> shipping broken hdfs isn't something we'd want to do, but if we can be
>> confident that all other issues can be addressed in RC3 then I'd be happy.
>>
>> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > I will highlight that I am completely fed up with doing this release
>> and
>> >> really want to get it out the way -for which I depend on support from
>> as
>> >> many other developers as possible.
>> >
>> >
>> > hmm, I can feel the pain. I tried to find if there is any config or any
>> > workaround which can dodge this HDFS issue, but unfortunately couldn't
>> find
>> > any. If someone does a getListing with needLocation and the file doesn't
>> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
>> > the exception, AFAIK Observer reads have some logic around handling FNF
>> > specifically, that it attempts Active NN or something like that in such
>> > cases, So, that will be broken as well for this use case.
>> >
>> > Now, there is no denying the fact there is an issue on the HDFS side,
>> and
>> > it has already been too much work on your side, so you can argue that it
>> > might not be a very frequent use case or so. It's your call.
>> >
>> > Just sharing, no intentions of saying you should do that, But as an RM
>> > "nobody" can force you for a new iteration of a RC, it is gonna be your
>> > call and discretion. As far as I know a release can not be vetoed by
>> > "anybody" as per the apache by laws.(
>> > https://www.apache.org/legal/release-policy.html#release-approval).
>> Even
>> > our bylaws say that product release requires a Lazy Majority not a
>> > Consensus Approval.
>> >
>> > So, you have a way out. You guys are 2 already and 1 I will give you a
>> > pass, in case you are really in a state: ''Get me out of this mess"
>> state,
>> > my basic validations on x86 & Aarch64 both are passing as of now,
>> couldn't
>> > reach the end for any of the RC's
>> >
>> > -Ayush
>> >
>> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>> >
>> >> While this RC is not going to be final, I just wanted to share the
>> results
>> >> of the testing I have done so far with RC1 and RC2.
>> >>
>> >> * Signature: ok
>> >> * Checksum : ok
>> >> * Rat check (1.8.0_341): ok
>> >> - mvn clean apache-rat:check
>> >> * Built from source (1.8.0_341): ok
>> >> - mvn clean install -DskipTests
>> >> * Built tar from source (1.8.0_341): ok
>> >> - mvn clean package -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>> >>
>> >> * Built images using the tarball, installed and started all of Hdfs,
>> JHS
>> >> and Yarn components
>> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
>> job
>> >> * Hdfs CRUD tests
>> >> * MapReduce wordcount job
>> >>
>> >> * Ran S3A tests with scale profile against us-west-2:
>> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>> >>
>> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
>> This
>> >> is
>> >> consistently failing, looks like a recent regression.
>> >> I was also able to repro on trunk, will create Jira.
>> >>
>> >>
>> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> >> <st...@cloudera.com.invalid>
>> >> wrote:
>> >>
>> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> >> 3.3.5.
>> >> >
>> >> > We need anyone who can to verify the source and binary artifacts,
>> >> > including those JARs staged on maven, the site documentation and the
>> >> arm64
>> >> > tar file.
>> >> >
>> >> > The RC is available at:
>> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >> >
>> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >> >
>> >> > The maven artifacts are staged at
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >> >
>> >> > You can find my public key at:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > Change log
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >> >
>> >> > Release notes
>> >> >
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >> >
>> >> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >> >
>> >> > As to what changed since the RC1 attempt last week
>> >> >
>> >> >
>> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> >> build
>> >> > issues in maven 3.9.0)
>> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> >> (#5429)
>> >> >
>> >> >
>> >> > Note, because the arm64 binaries are built separately on a different
>> >> > platform and JVM, their jar files may not match those of the x86
>> >> > release -and therefore the maven artifacts. I don't think this is
>> >> > an issue (the ASF actually releases source tarballs, the binaries are
>> >> > there for help only, though with the maven repo that's a bit
>> blurred).
>> >> >
>> >> > The only way to be consistent would actually untar the x86.tar.gz,
>> >> > overwrite its binaries with the arm stuff, retar, sign and push out
>> >> > for the vote. Even automating that would be risky.
>> >> >
>> >> > Please try the release and vote. The vote will run for 5 days.
>> >> >
>> >> > Steve and Mukund
>> >> >
>> >>
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Thanks Steve. I see now that the branch cut was way back in October so I
definitely understand your frustration here!
This made me realize that HDFS-16832
<https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
similar issue as the aforementioned HDFS-16923, is also missing from the
RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
branch-3.3.5, so they are ready if/when an RC3 is cut.
In the meantime, I tested for RC2 that a local cluster of NN + standby +
observer + QJM works as expected for some basic HDFS commands.
On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> shipping broken hdfs isn't something we'd want to do, but if we can be
> confident that all other issues can be addressed in RC3 then I'd be happy.
>
> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>
> > I will highlight that I am completely fed up with doing this release and
> >> really want to get it out the way -for which I depend on support from as
> >> many other developers as possible.
> >
> >
> > hmm, I can feel the pain. I tried to find if there is any config or any
> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> find
> > any. If someone does a getListing with needLocation and the file doesn't
> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> > the exception, AFAIK Observer reads have some logic around handling FNF
> > specifically, that it attempts Active NN or something like that in such
> > cases, So, that will be broken as well for this use case.
> >
> > Now, there is no denying the fact there is an issue on the HDFS side, and
> > it has already been too much work on your side, so you can argue that it
> > might not be a very frequent use case or so. It's your call.
> >
> > Just sharing, no intentions of saying you should do that, But as an RM
> > "nobody" can force you for a new iteration of a RC, it is gonna be your
> > call and discretion. As far as I know a release can not be vetoed by
> > "anybody" as per the apache by laws.(
> > https://www.apache.org/legal/release-policy.html#release-approval). Even
> > our bylaws say that product release requires a Lazy Majority not a
> > Consensus Approval.
> >
> > So, you have a way out. You guys are 2 already and 1 I will give you a
> > pass, in case you are really in a state: ''Get me out of this mess"
> state,
> > my basic validations on x86 & Aarch64 both are passing as of now,
> couldn't
> > reach the end for any of the RC's
> >
> > -Ayush
> >
> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >
> >> While this RC is not going to be final, I just wanted to share the
> results
> >> of the testing I have done so far with RC1 and RC2.
> >>
> >> * Signature: ok
> >> * Checksum : ok
> >> * Rat check (1.8.0_341): ok
> >> - mvn clean apache-rat:check
> >> * Built from source (1.8.0_341): ok
> >> - mvn clean install -DskipTests
> >> * Built tar from source (1.8.0_341): ok
> >> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >>
> >> * Built images using the tarball, installed and started all of Hdfs, JHS
> >> and Yarn components
> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> job
> >> * Hdfs CRUD tests
> >> * MapReduce wordcount job
> >>
> >> * Ran S3A tests with scale profile against us-west-2:
> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >>
> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
> >> is
> >> consistently failing, looks like a recent regression.
> >> I was also able to repro on trunk, will create Jira.
> >>
> >>
> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >> wrote:
> >>
> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> 3.3.5.
> >> >
> >> > We need anyone who can to verify the source and binary artifacts,
> >> > including those JARs staged on maven, the site documentation and the
> >> arm64
> >> > tar file.
> >> >
> >> > The RC is available at:
> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >
> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >
> >> > The maven artifacts are staged at
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >
> >> > You can find my public key at:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > Change log
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >
> >> > Release notes
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >
> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >
> >> > As to what changed since the RC1 attempt last week
> >> >
> >> >
> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> >> build
> >> > issues in maven 3.9.0)
> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> (#5429)
> >> >
> >> >
> >> > Note, because the arm64 binaries are built separately on a different
> >> > platform and JVM, their jar files may not match those of the x86
> >> > release -and therefore the maven artifacts. I don't think this is
> >> > an issue (the ASF actually releases source tarballs, the binaries are
> >> > there for help only, though with the maven repo that's a bit blurred).
> >> >
> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> > for the vote. Even automating that would be risky.
> >> >
> >> > Please try the release and vote. The vote will run for 5 days.
> >> >
> >> > Steve and Mukund
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Thanks Steve. I see now that the branch cut was way back in October so I
definitely understand your frustration here!
This made me realize that HDFS-16832
<https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
similar issue as the aforementioned HDFS-16923, is also missing from the
RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
branch-3.3.5, so they are ready if/when an RC3 is cut.
In the meantime, I tested for RC2 that a local cluster of NN + standby +
observer + QJM works as expected for some basic HDFS commands.
On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> shipping broken hdfs isn't something we'd want to do, but if we can be
> confident that all other issues can be addressed in RC3 then I'd be happy.
>
> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>
> > I will highlight that I am completely fed up with doing this release and
> >> really want to get it out the way -for which I depend on support from as
> >> many other developers as possible.
> >
> >
> > hmm, I can feel the pain. I tried to find if there is any config or any
> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> find
> > any. If someone does a getListing with needLocation and the file doesn't
> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> > the exception, AFAIK Observer reads have some logic around handling FNF
> > specifically, that it attempts Active NN or something like that in such
> > cases, So, that will be broken as well for this use case.
> >
> > Now, there is no denying the fact there is an issue on the HDFS side, and
> > it has already been too much work on your side, so you can argue that it
> > might not be a very frequent use case or so. It's your call.
> >
> > Just sharing, no intentions of saying you should do that, But as an RM
> > "nobody" can force you for a new iteration of a RC, it is gonna be your
> > call and discretion. As far as I know a release can not be vetoed by
> > "anybody" as per the apache by laws.(
> > https://www.apache.org/legal/release-policy.html#release-approval). Even
> > our bylaws say that product release requires a Lazy Majority not a
> > Consensus Approval.
> >
> > So, you have a way out. You guys are 2 already and 1 I will give you a
> > pass, in case you are really in a state: ''Get me out of this mess"
> state,
> > my basic validations on x86 & Aarch64 both are passing as of now,
> couldn't
> > reach the end for any of the RC's
> >
> > -Ayush
> >
> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >
> >> While this RC is not going to be final, I just wanted to share the
> results
> >> of the testing I have done so far with RC1 and RC2.
> >>
> >> * Signature: ok
> >> * Checksum : ok
> >> * Rat check (1.8.0_341): ok
> >> - mvn clean apache-rat:check
> >> * Built from source (1.8.0_341): ok
> >> - mvn clean install -DskipTests
> >> * Built tar from source (1.8.0_341): ok
> >> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >>
> >> * Built images using the tarball, installed and started all of Hdfs, JHS
> >> and Yarn components
> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> job
> >> * Hdfs CRUD tests
> >> * MapReduce wordcount job
> >>
> >> * Ran S3A tests with scale profile against us-west-2:
> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >>
> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
> >> is
> >> consistently failing, looks like a recent regression.
> >> I was also able to repro on trunk, will create Jira.
> >>
> >>
> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >> wrote:
> >>
> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> 3.3.5.
> >> >
> >> > We need anyone who can to verify the source and binary artifacts,
> >> > including those JARs staged on maven, the site documentation and the
> >> arm64
> >> > tar file.
> >> >
> >> > The RC is available at:
> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >
> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >
> >> > The maven artifacts are staged at
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >
> >> > You can find my public key at:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > Change log
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >
> >> > Release notes
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >
> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >
> >> > As to what changed since the RC1 attempt last week
> >> >
> >> >
> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> >> build
> >> > issues in maven 3.9.0)
> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> (#5429)
> >> >
> >> >
> >> > Note, because the arm64 binaries are built separately on a different
> >> > platform and JVM, their jar files may not match those of the x86
> >> > release -and therefore the maven artifacts. I don't think this is
> >> > an issue (the ASF actually releases source tarballs, the binaries are
> >> > there for help only, though with the maven repo that's a bit blurred).
> >> >
> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> > for the vote. Even automating that would be risky.
> >> >
> >> > Please try the release and vote. The vote will run for 5 days.
> >> >
> >> > Steve and Mukund
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Thanks Steve. I see now that the branch cut was way back in October so I
definitely understand your frustration here!
This made me realize that HDFS-16832
<https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
similar issue as the aforementioned HDFS-16923, is also missing from the
RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
branch-3.3.5, so they are ready if/when an RC3 is cut.
In the meantime, I tested for RC2 that a local cluster of NN + standby +
observer + QJM works as expected for some basic HDFS commands.
On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> shipping broken hdfs isn't something we'd want to do, but if we can be
> confident that all other issues can be addressed in RC3 then I'd be happy.
>
> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>
> > I will highlight that I am completely fed up with doing this release and
> >> really want to get it out the way -for which I depend on support from as
> >> many other developers as possible.
> >
> >
> > hmm, I can feel the pain. I tried to find if there is any config or any
> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> find
> > any. If someone does a getListing with needLocation and the file doesn't
> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> > the exception, AFAIK Observer reads have some logic around handling FNF
> > specifically, that it attempts Active NN or something like that in such
> > cases, So, that will be broken as well for this use case.
> >
> > Now, there is no denying the fact there is an issue on the HDFS side, and
> > it has already been too much work on your side, so you can argue that it
> > might not be a very frequent use case or so. It's your call.
> >
> > Just sharing, no intentions of saying you should do that, But as an RM
> > "nobody" can force you for a new iteration of a RC, it is gonna be your
> > call and discretion. As far as I know a release can not be vetoed by
> > "anybody" as per the apache by laws.(
> > https://www.apache.org/legal/release-policy.html#release-approval). Even
> > our bylaws say that product release requires a Lazy Majority not a
> > Consensus Approval.
> >
> > So, you have a way out. You guys are 2 already and 1 I will give you a
> > pass, in case you are really in a state: ''Get me out of this mess"
> state,
> > my basic validations on x86 & Aarch64 both are passing as of now,
> couldn't
> > reach the end for any of the RC's
> >
> > -Ayush
> >
> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >
> >> While this RC is not going to be final, I just wanted to share the
> results
> >> of the testing I have done so far with RC1 and RC2.
> >>
> >> * Signature: ok
> >> * Checksum : ok
> >> * Rat check (1.8.0_341): ok
> >> - mvn clean apache-rat:check
> >> * Built from source (1.8.0_341): ok
> >> - mvn clean install -DskipTests
> >> * Built tar from source (1.8.0_341): ok
> >> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >>
> >> * Built images using the tarball, installed and started all of Hdfs, JHS
> >> and Yarn components
> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> job
> >> * Hdfs CRUD tests
> >> * MapReduce wordcount job
> >>
> >> * Ran S3A tests with scale profile against us-west-2:
> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >>
> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
> >> is
> >> consistently failing, looks like a recent regression.
> >> I was also able to repro on trunk, will create Jira.
> >>
> >>
> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >> wrote:
> >>
> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> 3.3.5.
> >> >
> >> > We need anyone who can to verify the source and binary artifacts,
> >> > including those JARs staged on maven, the site documentation and the
> >> arm64
> >> > tar file.
> >> >
> >> > The RC is available at:
> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >
> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >
> >> > The maven artifacts are staged at
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >
> >> > You can find my public key at:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > Change log
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >
> >> > Release notes
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >
> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >
> >> > As to what changed since the RC1 attempt last week
> >> >
> >> >
> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> >> build
> >> > issues in maven 3.9.0)
> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> (#5429)
> >> >
> >> >
> >> > Note, because the arm64 binaries are built separately on a different
> >> > platform and JVM, their jar files may not match those of the x86
> >> > release -and therefore the maven artifacts. I don't think this is
> >> > an issue (the ASF actually releases source tarballs, the binaries are
> >> > there for help only, though with the maven repo that's a bit blurred).
> >> >
> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> > for the vote. Even automating that would be risky.
> >> >
> >> > Please try the release and vote. The vote will run for 5 days.
> >> >
> >> > Steve and Mukund
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Thanks Steve. I see now that the branch cut was way back in October so I
definitely understand your frustration here!
This made me realize that HDFS-16832
<https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
similar issue as the aforementioned HDFS-16923, is also missing from the
RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
branch-3.3.5, so they are ready if/when an RC3 is cut.
In the meantime, I tested for RC2 that a local cluster of NN + standby +
observer + QJM works as expected for some basic HDFS commands.
On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> shipping broken hdfs isn't something we'd want to do, but if we can be
> confident that all other issues can be addressed in RC3 then I'd be happy.
>
> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
>
> > I will highlight that I am completely fed up with doing this release and
> >> really want to get it out the way -for which I depend on support from as
> >> many other developers as possible.
> >
> >
> > hmm, I can feel the pain. I tried to find if there is any config or any
> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> find
> > any. If someone does a getListing with needLocation and the file doesn't
> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> > the exception, AFAIK Observer reads have some logic around handling FNF
> > specifically, that it attempts Active NN or something like that in such
> > cases, So, that will be broken as well for this use case.
> >
> > Now, there is no denying the fact there is an issue on the HDFS side, and
> > it has already been too much work on your side, so you can argue that it
> > might not be a very frequent use case or so. It's your call.
> >
> > Just sharing, no intentions of saying you should do that, But as an RM
> > "nobody" can force you for a new iteration of a RC, it is gonna be your
> > call and discretion. As far as I know a release can not be vetoed by
> > "anybody" as per the apache by laws.(
> > https://www.apache.org/legal/release-policy.html#release-approval). Even
> > our bylaws say that product release requires a Lazy Majority not a
> > Consensus Approval.
> >
> > So, you have a way out. You guys are 2 already and 1 I will give you a
> > pass, in case you are really in a state: ''Get me out of this mess"
> state,
> > my basic validations on x86 & Aarch64 both are passing as of now,
> couldn't
> > reach the end for any of the RC's
> >
> > -Ayush
> >
> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> >
> >> While this RC is not going to be final, I just wanted to share the
> results
> >> of the testing I have done so far with RC1 and RC2.
> >>
> >> * Signature: ok
> >> * Checksum : ok
> >> * Rat check (1.8.0_341): ok
> >> - mvn clean apache-rat:check
> >> * Built from source (1.8.0_341): ok
> >> - mvn clean install -DskipTests
> >> * Built tar from source (1.8.0_341): ok
> >> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >>
> >> * Built images using the tarball, installed and started all of Hdfs, JHS
> >> and Yarn components
> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> job
> >> * Hdfs CRUD tests
> >> * MapReduce wordcount job
> >>
> >> * Ran S3A tests with scale profile against us-west-2:
> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >>
> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
> >> is
> >> consistently failing, looks like a recent regression.
> >> I was also able to repro on trunk, will create Jira.
> >>
> >>
> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> <st...@cloudera.com.invalid>
> >> wrote:
> >>
> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> 3.3.5.
> >> >
> >> > We need anyone who can to verify the source and binary artifacts,
> >> > including those JARs staged on maven, the site documentation and the
> >> arm64
> >> > tar file.
> >> >
> >> > The RC is available at:
> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >
> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >
> >> > The maven artifacts are staged at
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >
> >> > You can find my public key at:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > Change log
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >
> >> > Release notes
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >
> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >
> >> > As to what changed since the RC1 attempt last week
> >> >
> >> >
> >> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> >> build
> >> > issues in maven 3.9.0)
> >> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
> >> (#5429)
> >> >
> >> >
> >> > Note, because the arm64 binaries are built separately on a different
> >> > platform and JVM, their jar files may not match those of the x86
> >> > release -and therefore the maven artifacts. I don't think this is
> >> > an issue (the ASF actually releases source tarballs, the binaries are
> >> > there for help only, though with the maven repo that's a bit blurred).
> >> >
> >> > The only way to be consistent would actually untar the x86.tar.gz,
> >> > overwrite its binaries with the arm stuff, retar, sign and push out
> >> > for the vote. Even automating that would be risky.
> >> >
> >> > Please try the release and vote. The vote will run for 5 days.
> >> >
> >> > Steve and Mukund
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
shipping broken hdfs isn't something we'd want to do, but if we can be
confident that all other issues can be addressed in RC3 then I'd be happy.
On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> I will highlight that I am completely fed up with doing this release and
>> really want to get it out the way -for which I depend on support from as
>> many other developers as possible.
>
>
> hmm, I can feel the pain. I tried to find if there is any config or any
> workaround which can dodge this HDFS issue, but unfortunately couldn't find
> any. If someone does a getListing with needLocation and the file doesn't
> exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> the exception, AFAIK Observer reads have some logic around handling FNF
> specifically, that it attempts Active NN or something like that in such
> cases, So, that will be broken as well for this use case.
>
> Now, there is no denying the fact there is an issue on the HDFS side, and
> it has already been too much work on your side, so you can argue that it
> might not be a very frequent use case or so. It's your call.
>
> Just sharing, no intentions of saying you should do that, But as an RM
> "nobody" can force you for a new iteration of a RC, it is gonna be your
> call and discretion. As far as I know a release can not be vetoed by
> "anybody" as per the apache by laws.(
> https://www.apache.org/legal/release-policy.html#release-approval). Even
> our bylaws say that product release requires a Lazy Majority not a
> Consensus Approval.
>
> So, you have a way out. You guys are 2 already and 1 I will give you a
> pass, in case you are really in a state: ''Get me out of this mess" state,
> my basic validations on x86 & Aarch64 both are passing as of now, couldn't
> reach the end for any of the RC's
>
> -Ayush
>
> On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>
>> While this RC is not going to be final, I just wanted to share the results
>> of the testing I have done so far with RC1 and RC2.
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_341): ok
>> - mvn clean apache-rat:check
>> * Built from source (1.8.0_341): ok
>> - mvn clean install -DskipTests
>> * Built tar from source (1.8.0_341): ok
>> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Built images using the tarball, installed and started all of Hdfs, JHS
>> and Yarn components
>> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
>> * Hdfs CRUD tests
>> * MapReduce wordcount job
>>
>> * Ran S3A tests with scale profile against us-west-2:
>> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>>
>> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
>> is
>> consistently failing, looks like a recent regression.
>> I was also able to repro on trunk, will create Jira.
>>
>>
>> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>> wrote:
>>
>> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> 3.3.5.
>> >
>> > We need anyone who can to verify the source and binary artifacts,
>> > including those JARs staged on maven, the site documentation and the
>> arm64
>> > tar file.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >
>> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >
>> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >
>> > As to what changed since the RC1 attempt last week
>> >
>> >
>> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> build
>> > issues in maven 3.9.0)
>> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> (#5429)
>> >
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Steve and Mukund
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
shipping broken hdfs isn't something we'd want to do, but if we can be
confident that all other issues can be addressed in RC3 then I'd be happy.
On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> I will highlight that I am completely fed up with doing this release and
>> really want to get it out the way -for which I depend on support from as
>> many other developers as possible.
>
>
> hmm, I can feel the pain. I tried to find if there is any config or any
> workaround which can dodge this HDFS issue, but unfortunately couldn't find
> any. If someone does a getListing with needLocation and the file doesn't
> exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> the exception, AFAIK Observer reads have some logic around handling FNF
> specifically, that it attempts Active NN or something like that in such
> cases, So, that will be broken as well for this use case.
>
> Now, there is no denying the fact there is an issue on the HDFS side, and
> it has already been too much work on your side, so you can argue that it
> might not be a very frequent use case or so. It's your call.
>
> Just sharing, no intentions of saying you should do that, But as an RM
> "nobody" can force you for a new iteration of a RC, it is gonna be your
> call and discretion. As far as I know a release can not be vetoed by
> "anybody" as per the apache by laws.(
> https://www.apache.org/legal/release-policy.html#release-approval). Even
> our bylaws say that product release requires a Lazy Majority not a
> Consensus Approval.
>
> So, you have a way out. You guys are 2 already and 1 I will give you a
> pass, in case you are really in a state: ''Get me out of this mess" state,
> my basic validations on x86 & Aarch64 both are passing as of now, couldn't
> reach the end for any of the RC's
>
> -Ayush
>
> On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>
>> While this RC is not going to be final, I just wanted to share the results
>> of the testing I have done so far with RC1 and RC2.
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_341): ok
>> - mvn clean apache-rat:check
>> * Built from source (1.8.0_341): ok
>> - mvn clean install -DskipTests
>> * Built tar from source (1.8.0_341): ok
>> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Built images using the tarball, installed and started all of Hdfs, JHS
>> and Yarn components
>> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
>> * Hdfs CRUD tests
>> * MapReduce wordcount job
>>
>> * Ran S3A tests with scale profile against us-west-2:
>> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>>
>> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
>> is
>> consistently failing, looks like a recent regression.
>> I was also able to repro on trunk, will create Jira.
>>
>>
>> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>> wrote:
>>
>> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> 3.3.5.
>> >
>> > We need anyone who can to verify the source and binary artifacts,
>> > including those JARs staged on maven, the site documentation and the
>> arm64
>> > tar file.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >
>> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >
>> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >
>> > As to what changed since the RC1 attempt last week
>> >
>> >
>> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> build
>> > issues in maven 3.9.0)
>> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> (#5429)
>> >
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Steve and Mukund
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
shipping broken hdfs isn't something we'd want to do, but if we can be
confident that all other issues can be addressed in RC3 then I'd be happy.
On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> I will highlight that I am completely fed up with doing this release and
>> really want to get it out the way -for which I depend on support from as
>> many other developers as possible.
>
>
> hmm, I can feel the pain. I tried to find if there is any config or any
> workaround which can dodge this HDFS issue, but unfortunately couldn't find
> any. If someone does a getListing with needLocation and the file doesn't
> exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> the exception, AFAIK Observer reads have some logic around handling FNF
> specifically, that it attempts Active NN or something like that in such
> cases, So, that will be broken as well for this use case.
>
> Now, there is no denying the fact there is an issue on the HDFS side, and
> it has already been too much work on your side, so you can argue that it
> might not be a very frequent use case or so. It's your call.
>
> Just sharing, no intentions of saying you should do that, But as an RM
> "nobody" can force you for a new iteration of a RC, it is gonna be your
> call and discretion. As far as I know a release can not be vetoed by
> "anybody" as per the apache by laws.(
> https://www.apache.org/legal/release-policy.html#release-approval). Even
> our bylaws say that product release requires a Lazy Majority not a
> Consensus Approval.
>
> So, you have a way out. You guys are 2 already and 1 I will give you a
> pass, in case you are really in a state: ''Get me out of this mess" state,
> my basic validations on x86 & Aarch64 both are passing as of now, couldn't
> reach the end for any of the RC's
>
> -Ayush
>
> On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>
>> While this RC is not going to be final, I just wanted to share the results
>> of the testing I have done so far with RC1 and RC2.
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_341): ok
>> - mvn clean apache-rat:check
>> * Built from source (1.8.0_341): ok
>> - mvn clean install -DskipTests
>> * Built tar from source (1.8.0_341): ok
>> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Built images using the tarball, installed and started all of Hdfs, JHS
>> and Yarn components
>> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
>> * Hdfs CRUD tests
>> * MapReduce wordcount job
>>
>> * Ran S3A tests with scale profile against us-west-2:
>> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>>
>> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
>> is
>> consistently failing, looks like a recent regression.
>> I was also able to repro on trunk, will create Jira.
>>
>>
>> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>> wrote:
>>
>> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> 3.3.5.
>> >
>> > We need anyone who can to verify the source and binary artifacts,
>> > including those JARs staged on maven, the site documentation and the
>> arm64
>> > tar file.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >
>> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >
>> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >
>> > As to what changed since the RC1 attempt last week
>> >
>> >
>> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> build
>> > issues in maven 3.9.0)
>> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> (#5429)
>> >
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Steve and Mukund
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
shipping broken hdfs isn't something we'd want to do, but if we can be
confident that all other issues can be addressed in RC3 then I'd be happy.
On Fri, 3 Mar 2023 at 05:09, Ayush Saxena <ay...@gmail.com> wrote:
> I will highlight that I am completely fed up with doing this release and
>> really want to get it out the way -for which I depend on support from as
>> many other developers as possible.
>
>
> hmm, I can feel the pain. I tried to find if there is any config or any
> workaround which can dodge this HDFS issue, but unfortunately couldn't find
> any. If someone does a getListing with needLocation and the file doesn't
> exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> the exception, AFAIK Observer reads have some logic around handling FNF
> specifically, that it attempts Active NN or something like that in such
> cases, So, that will be broken as well for this use case.
>
> Now, there is no denying the fact there is an issue on the HDFS side, and
> it has already been too much work on your side, so you can argue that it
> might not be a very frequent use case or so. It's your call.
>
> Just sharing, no intentions of saying you should do that, But as an RM
> "nobody" can force you for a new iteration of a RC, it is gonna be your
> call and discretion. As far as I know a release can not be vetoed by
> "anybody" as per the apache by laws.(
> https://www.apache.org/legal/release-policy.html#release-approval). Even
> our bylaws say that product release requires a Lazy Majority not a
> Consensus Approval.
>
> So, you have a way out. You guys are 2 already and 1 I will give you a
> pass, in case you are really in a state: ''Get me out of this mess" state,
> my basic validations on x86 & Aarch64 both are passing as of now, couldn't
> reach the end for any of the RC's
>
> -Ayush
>
> On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
>
>> While this RC is not going to be final, I just wanted to share the results
>> of the testing I have done so far with RC1 and RC2.
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_341): ok
>> - mvn clean apache-rat:check
>> * Built from source (1.8.0_341): ok
>> - mvn clean install -DskipTests
>> * Built tar from source (1.8.0_341): ok
>> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Built images using the tarball, installed and started all of Hdfs, JHS
>> and Yarn components
>> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
>> * Hdfs CRUD tests
>> * MapReduce wordcount job
>>
>> * Ran S3A tests with scale profile against us-west-2:
>> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>>
>> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
>> is
>> consistently failing, looks like a recent regression.
>> I was also able to repro on trunk, will create Jira.
>>
>>
>> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> <st...@cloudera.com.invalid>
>> wrote:
>>
>> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> 3.3.5.
>> >
>> > We need anyone who can to verify the source and binary artifacts,
>> > including those JARs staged on maven, the site documentation and the
>> arm64
>> > tar file.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >
>> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >
>> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >
>> > As to what changed since the RC1 attempt last week
>> >
>> >
>> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> build
>> > issues in maven 3.9.0)
>> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> (#5429)
>> >
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Steve and Mukund
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Ayush Saxena <ay...@gmail.com>.
>
> I will highlight that I am completely fed up with doing this release and
> really want to get it out the way -for which I depend on support from as
> many other developers as possible.
hmm, I can feel the pain. I tried to find if there is any config or any
workaround which can dodge this HDFS issue, but unfortunately couldn't find
any. If someone does a getListing with needLocation and the file doesn't
exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
the exception, AFAIK Observer reads have some logic around handling FNF
specifically, that it attempts Active NN or something like that in such
cases, So, that will be broken as well for this use case.
Now, there is no denying the fact there is an issue on the HDFS side, and
it has already been too much work on your side, so you can argue that it
might not be a very frequent use case or so. It's your call.
Just sharing, no intentions of saying you should do that, But as an RM
"nobody" can force you for a new iteration of a RC, it is gonna be your
call and discretion. As far as I know a release can not be vetoed by
"anybody" as per the apache by laws.(
https://www.apache.org/legal/release-policy.html#release-approval). Even
our bylaws say that product release requires a Lazy Majority not a
Consensus Approval.
So, you have a way out. You guys are 2 already and 1 I will give you a
pass, in case you are really in a state: ''Get me out of this mess" state,
my basic validations on x86 & Aarch64 both are passing as of now, couldn't
reach the end for any of the RC's
-Ayush
On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> While this RC is not going to be final, I just wanted to share the results
> of the testing I have done so far with RC1 and RC2.
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
> - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
> - mvn clean install -DskipTests
> * Built tar from source (1.8.0_341): ok
> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> * Built images using the tarball, installed and started all of Hdfs, JHS
> and Yarn components
> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
> * Hdfs CRUD tests
> * MapReduce wordcount job
>
> * Ran S3A tests with scale profile against us-west-2:
> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>
> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
> consistently failing, looks like a recent regression.
> I was also able to repro on trunk, will create Jira.
>
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Ayush Saxena <ay...@gmail.com>.
>
> I will highlight that I am completely fed up with doing this release and
> really want to get it out the way -for which I depend on support from as
> many other developers as possible.
hmm, I can feel the pain. I tried to find if there is any config or any
workaround which can dodge this HDFS issue, but unfortunately couldn't find
any. If someone does a getListing with needLocation and the file doesn't
exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
the exception, AFAIK Observer reads have some logic around handling FNF
specifically, that it attempts Active NN or something like that in such
cases, So, that will be broken as well for this use case.
Now, there is no denying the fact there is an issue on the HDFS side, and
it has already been too much work on your side, so you can argue that it
might not be a very frequent use case or so. It's your call.
Just sharing, no intentions of saying you should do that, But as an RM
"nobody" can force you for a new iteration of a RC, it is gonna be your
call and discretion. As far as I know a release can not be vetoed by
"anybody" as per the apache by laws.(
https://www.apache.org/legal/release-policy.html#release-approval). Even
our bylaws say that product release requires a Lazy Majority not a
Consensus Approval.
So, you have a way out. You guys are 2 already and 1 I will give you a
pass, in case you are really in a state: ''Get me out of this mess" state,
my basic validations on x86 & Aarch64 both are passing as of now, couldn't
reach the end for any of the RC's
-Ayush
On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> While this RC is not going to be final, I just wanted to share the results
> of the testing I have done so far with RC1 and RC2.
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
> - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
> - mvn clean install -DskipTests
> * Built tar from source (1.8.0_341): ok
> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> * Built images using the tarball, installed and started all of Hdfs, JHS
> and Yarn components
> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
> * Hdfs CRUD tests
> * MapReduce wordcount job
>
> * Ran S3A tests with scale profile against us-west-2:
> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>
> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
> consistently failing, looks like a recent regression.
> I was also able to repro on trunk, will create Jira.
>
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Ayush Saxena <ay...@gmail.com>.
>
> I will highlight that I am completely fed up with doing this release and
> really want to get it out the way -for which I depend on support from as
> many other developers as possible.
hmm, I can feel the pain. I tried to find if there is any config or any
workaround which can dodge this HDFS issue, but unfortunately couldn't find
any. If someone does a getListing with needLocation and the file doesn't
exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
the exception, AFAIK Observer reads have some logic around handling FNF
specifically, that it attempts Active NN or something like that in such
cases, So, that will be broken as well for this use case.
Now, there is no denying the fact there is an issue on the HDFS side, and
it has already been too much work on your side, so you can argue that it
might not be a very frequent use case or so. It's your call.
Just sharing, no intentions of saying you should do that, But as an RM
"nobody" can force you for a new iteration of a RC, it is gonna be your
call and discretion. As far as I know a release can not be vetoed by
"anybody" as per the apache by laws.(
https://www.apache.org/legal/release-policy.html#release-approval). Even
our bylaws say that product release requires a Lazy Majority not a
Consensus Approval.
So, you have a way out. You guys are 2 already and 1 I will give you a
pass, in case you are really in a state: ''Get me out of this mess" state,
my basic validations on x86 & Aarch64 both are passing as of now, couldn't
reach the end for any of the RC's
-Ayush
On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> While this RC is not going to be final, I just wanted to share the results
> of the testing I have done so far with RC1 and RC2.
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
> - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
> - mvn clean install -DskipTests
> * Built tar from source (1.8.0_341): ok
> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> * Built images using the tarball, installed and started all of Hdfs, JHS
> and Yarn components
> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
> * Hdfs CRUD tests
> * MapReduce wordcount job
>
> * Ran S3A tests with scale profile against us-west-2:
> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>
> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
> consistently failing, looks like a recent regression.
> I was also able to repro on trunk, will create Jira.
>
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Ayush Saxena <ay...@gmail.com>.
>
> I will highlight that I am completely fed up with doing this release and
> really want to get it out the way -for which I depend on support from as
> many other developers as possible.
hmm, I can feel the pain. I tried to find if there is any config or any
workaround which can dodge this HDFS issue, but unfortunately couldn't find
any. If someone does a getListing with needLocation and the file doesn't
exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
the exception, AFAIK Observer reads have some logic around handling FNF
specifically, that it attempts Active NN or something like that in such
cases, So, that will be broken as well for this use case.
Now, there is no denying the fact there is an issue on the HDFS side, and
it has already been too much work on your side, so you can argue that it
might not be a very frequent use case or so. It's your call.
Just sharing, no intentions of saying you should do that, But as an RM
"nobody" can force you for a new iteration of a RC, it is gonna be your
call and discretion. As far as I know a release can not be vetoed by
"anybody" as per the apache by laws.(
https://www.apache.org/legal/release-policy.html#release-approval). Even
our bylaws say that product release requires a Lazy Majority not a
Consensus Approval.
So, you have a way out. You guys are 2 already and 1 I will give you a
pass, in case you are really in a state: ''Get me out of this mess" state,
my basic validations on x86 & Aarch64 both are passing as of now, couldn't
reach the end for any of the RC's
-Ayush
On Fri, 3 Mar 2023 at 08:41, Viraj Jasani <vj...@apache.org> wrote:
> While this RC is not going to be final, I just wanted to share the results
> of the testing I have done so far with RC1 and RC2.
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
> - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
> - mvn clean install -DskipTests
> * Built tar from source (1.8.0_341): ok
> - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> * Built images using the tarball, installed and started all of Hdfs, JHS
> and Yarn components
> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
> * Hdfs CRUD tests
> * MapReduce wordcount job
>
> * Ran S3A tests with scale profile against us-west-2:
> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>
> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
> consistently failing, looks like a recent regression.
> I was also able to repro on trunk, will create Jira.
>
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <stevel@cloudera.com.invalid
> >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> > 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> > 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> > 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> > issues in maven 3.9.0)
> > 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
While this RC is not going to be final, I just wanted to share the results
of the testing I have done so far with RC1 and RC2.
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
- mvn clean install -DskipTests
* Built tar from source (1.8.0_341): ok
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
* Built images using the tarball, installed and started all of Hdfs, JHS
and Yarn components
* Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
* Hdfs CRUD tests
* MapReduce wordcount job
* Ran S3A tests with scale profile against us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
consistently failing, looks like a recent regression.
I was also able to repro on trunk, will create Jira.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
While this RC is not going to be final, I just wanted to share the results
of the testing I have done so far with RC1 and RC2.
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
- mvn clean install -DskipTests
* Built tar from source (1.8.0_341): ok
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
* Built images using the tarball, installed and started all of Hdfs, JHS
and Yarn components
* Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
* Hdfs CRUD tests
* MapReduce wordcount job
* Ran S3A tests with scale profile against us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
consistently failing, looks like a recent regression.
I was also able to repro on trunk, will create Jira.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
While this RC is not going to be final, I just wanted to share the results
of the testing I have done so far with RC1 and RC2.
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
- mvn clean install -DskipTests
* Built tar from source (1.8.0_341): ok
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
* Built images using the tarball, installed and started all of Hdfs, JHS
and Yarn components
* Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
* Hdfs CRUD tests
* MapReduce wordcount job
* Ran S3A tests with scale profile against us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
consistently failing, looks like a recent regression.
I was also able to repro on trunk, will create Jira.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Viraj Jasani <vj...@apache.org>.
While this RC is not going to be final, I just wanted to share the results
of the testing I have done so far with RC1 and RC2.
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
- mvn clean install -DskipTests
* Built tar from source (1.8.0_341): ok
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
* Built images using the tarball, installed and started all of Hdfs, JHS
and Yarn components
* Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
* Hdfs CRUD tests
* MapReduce wordcount job
* Ran S3A tests with scale profile against us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
consistently failing, looks like a recent regression.
I was also able to repro on trunk, will create Jira.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)
Posted by Erik Krogen <xk...@apache.org>.
Hi folks, apologies for being late to the release conversation, but I think
we need to get HDFS-16923
<https://issues.apache.org/jira/browse/HDFS-16923> into
3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>, which
also went into 3.3.5, introduced an issue whereby Observer NameNodes will
throw NPE upon any getListing call on a directory that doesn't exist. It
will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
for this and it has been merged to trunk/branch-3.3 as of a few minutes
ago. I'd like to see about merging to branch-3.3.5 as well.
Thanks for the consideration and sorry for not bringing this up in RC1 or
earlier.
On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
> 1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> 2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> 3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
> issues in maven 3.9.0)
> 4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>