You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Masatake Iwasaki <iw...@oss.nttdata.com> on 2023/01/02 13:02:11 UTC
Re: [VOTE] Release Apache Hadoop 3.3.5
Thanks for your great effort for the new release, Steve and Mukund.
+1 while it would be nice if we can address missed Javadocs.
+ verified the signature and checksum.
+ built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
+ launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
+ created encryption zone, put and read files via httpfs.
+ ran example MR wordcount over encryption zone.
+ built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
# while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
+ skimmed the site contents.
- Javadocs are not contained (under r3.3.5/api).
# The issue can be reproduced even if I built site docs from the source.
Masatake Iwasaki
On 2022/12/22 4:28, Steve Loughran wrote:
> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>
> Given the time of year it's a bit unrealistic to run a 5 day vote and
> expect people to be able to test it thoroughly enough to make this the one
> we can ship.
>
> What we would like is for anyone who can to verify the tarballs, and test
> the binaries, especially anyone who can try the arm64 binaries. We've got
> the building of those done and now the build file will incorporate them
> into the release -but neither of us have actually tested it yet. Maybe I
> should try it on my pi400 over xmas.
>
> The maven artifacts are up on the apache staging repo -they are the ones
> from x86 build. Building and testing downstream apps will be incredibly
> helpful.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>
> The git tag is release-3.3.5-RC0, commit 3262495904d
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
> transitive CVEs under control -both genuine and false positive.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
> high-performance versions for file:// and s3a:// filesystems.
> file:// through java native io
> s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
> compatible systems validate these.
>
>
> Please try the release and vote on it, even though i don't know what is a
> good timeline here...i'm actually going on holiday in early jan. Mukund is
> around and so can drive the process while I'm offline.
>
> Assuming we do have another iteration, the RC1 will not be before mid jan
> for that reason
>
> Steve (and mukund)
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
>
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
>
I think it is because of what Ayush said. They are marked as done but not
fixed.
Also, some were marked with wrong fix versions, initially, we thought 3.3.9
but then decided to change to 3.3.5.
I have updated them and hopefully, in RC1 it will be fine.
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
I haven't got a chance to deep dive into HADOOP-18324
<https://issues.apache.org/jira/browse/HADOOP-18324> which is claimed to be
the reason for these failures. Most probably will try to check next week if
it is still there.
From the PR uploaded on HDFS-16853
<https://issues.apache.org/jira/browse/HDFS-16853> it looks like changing
or tweaking the cleanup logic itself rather than with playing with tests or
MiniDfsCluster, So, the clean up logic has issues but I still need to check
what is the impact of that, If I have a service and that terminates in a
non test setup, will the restart be an issue like these tests are facing,
my initial hunch was No. But I need to carefully check and see what is the
impact and what other issues it can cause. the original logic ain't
something which can be decoded with just a few seconds of cursory look.
++ @Owen O'Malley <ow...@gmail.com> is the original author of the
Hadoop Jira, maybe he can share some pointers about that.
-Ayush
On Thu, 5 Jan 2023 at 07:04, Chris Nauroth <cn...@apache.org> wrote:
> Is it a problem limited to MiniDFSCluster, or is it a broader problem of
> RPC client resource cleanup? The patch is changing connection close
> cleanup, so I assumed the latter. If so, then it could potentially impact
> applications integrating with the RPC clients.
>
> If the problem is limited to MiniDFSCluster and restarts within a single
> JVM, then I agree the impact is smaller. Then, we'd want to consider what
> downstream projects have tests that do restarts on a MiniDFSCluster.
>
> Chris Nauroth
>
>
> On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> mentioned anywhere in change log or release notes. Are they actually
> >> up-to-date?
> >
> >
> > I don't think there is any issue with the ReleaseNotes generation as such
> > but with the Resolution type of this ticket, It ain't marked as Fixed but
> > Done. The other ticket which is marked Done is also not part of the
> release
> > notes. [1]
> >
> > if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >
> > Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> > can happen outside a MiniDfsCluster setup? Just guessing from the
> > description in the ticket. It looked like when we did a restart of the
> > Namenode in the MiniDfsCluster, I guess that would be in the same single
> > JVM, and that is why a previous blocked thread caused issues with the
> > restart. That is what I understood, I haven't checked the code though.
> >
> > Second, In the same context, Being curious If this lands up being a
> > MiniDfsCluster only issue, do we still consider this a release blocker?
> Not
> > saying in a way it won't be serious, MiniDfsCluster is very widely used
> by
> > downstream projects and all, so just wanted to know....
> >
> > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> > veto, I am not sure if it will get done any time soon, so if the use case
> > is something required, I would suggest handling it at Hadoop itself. It
> > seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> > with 3.3.5 and it passed. Other than that Hive officially support only
> > Hadoop-3.3.1 and that too only in the last 4.x release[2]
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> > [2] https://issues.apache.org/jira/browse/HIVE-24484
> >
> > -Ayush
> >
> > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> >
> >> -1, because if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >> Otherwise, this RC was looking good:
> >>
> >> * Verified all checksums.
> >> * Verified all signatures.
> >> * Built from source, including native code on Linux.
> >> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >> -Drequire.zstd -DskipTests
> >> * Tests passed.
> >> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >> * Checked dependency tree to make sure we have all of the expected
> library
> >> updates that are mentioned in the release notes.
> >> * mvn -o dependency:tree
> >> * Farewell, S3Guard.
> >> * Confirmed that hadoop-openstack is now just a stub placeholder
> artifact
> >> with no code.
> >> * For ARM verification:
> >> * Ran "file <X>" on all native binaries in the ARM tarball to
> confirm
> >> they actually came out with ARM as the architecture.
> >> * Output of hadoop checknative -a on ARM looks good.
> >> * Ran a MapReduce job with the native bzip2 codec for compression,
> and
> >> it worked fine.
> >> * Ran a MapReduce job with YARN configured to use
> >> LinuxContainerExecutor and verified launching the containers through
> >> container-executor worked.
> >>
> >> My local setup didn't have the test failures mentioned by Viraj, though
> >> there was some flakiness with a few HDFS snapshot tests timing out.
> >>
> >> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> >> request tracking an upgrade attempt. It's looking like some amount of
> code
> >> changes are required:
> >>
> >> https://issues.apache.org/jira/browse/HIVE-26648
> >> https://github.com/apache/hive/pull/3744
> >>
> >> Chris Nauroth
> >>
> >>
> >> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> >>
> >> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> > mentioned anywhere in change log or release notes. Are they actually
> >> > up-to-date?
> >> >
> >> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> >> > <iw...@oss.nttdata.com> wrote:
> >> > >
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > >
> >> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> >> not
> >> > expected in hbase-shaded-with-hadoop-check-invariants.
> >> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> >> > HBASE-27292.
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >> > >
> >> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> >> > (check-jar-contents-for-stuff-with-hadoop) @
> >> > hbase-shaded-with-hadoop-check-invariants ---
> >> > > [ERROR] Found artifact with unexpected contents:
> >> >
> >>
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> >> > > Please check the following and either correct the build or
> >> update
> >> > > the allowed list with reasoning.
> >> > >
> >> > > com/
> >> > > com/sun/
> >> > > com/sun/jersey/
> >> > > com/sun/jersey/json/
> >> > > ...
> >> > >
> >> > >
> >> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> >> seem
> >> > to be problematic.
> >> > > Excluding them on hive-jdbc might be the fix.
> >> > >
> >> > > [ERROR] Failed to execute goal
> >> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> >> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >> >
> >>
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> >> > entry
> >> >
> >>
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> >> > java.lang.IllegalArgumentException: Unsupported class file major
> >> version 59
> >> > -> [Help 1]
> >> > > ...
> >> > >
> >> > >
> >> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> >> > > > Thanks for your great effort for the new release, Steve and
> Mukund.
> >> > > >
> >> > > > +1 while it would be nice if we can address missed Javadocs.
> >> > > >
> >> > > > + verified the signature and checksum.
> >> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> >> native
> >> > profile enabled.
> >> > > > + launched pseudo distributed cluster including kms and httpfs
> >> with
> >> > Kerberos and SSL enabled.
> >> > > > + created encryption zone, put and read files via httpfs.
> >> > > > + ran example MR wordcount over encryption zone.
> >> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux
> 8
> >> > (both x86_64 and aarch64).
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> >> 3.3.4
> >> > worked.
> >> > > > + skimmed the site contents.
> >> > > > - Javadocs are not contained (under r3.3.5/api).
> >> > > > # The issue can be reproduced even if I built site docs from
> >> the
> >> > source.
> >> > > >
> >> > > > Masatake Iwasaki
> >> > > >
> >> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> >> > > >> Mukund and I have put together a release candidate (RC0) for
> Hadoop
> >> > 3.3.5.
> >> > > >>
> >> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> >> and
> >> > > >> expect people to be able to test it thoroughly enough to make
> this
> >> > the one
> >> > > >> we can ship.
> >> > > >>
> >> > > >> What we would like is for anyone who can to verify the tarballs,
> >> and
> >> > test
> >> > > >> the binaries, especially anyone who can try the arm64 binaries.
> >> We've
> >> > got
> >> > > >> the building of those done and now the build file will
> incorporate
> >> > them
> >> > > >> into the release -but neither of us have actually tested it yet.
> >> > Maybe I
> >> > > >> should try it on my pi400 over xmas.
> >> > > >>
> >> > > >> The maven artifacts are up on the apache staging repo -they are
> the
> >> > ones
> >> > > >> from x86 build. Building and testing downstream apps will be
> >> > incredibly
> >> > > >> helpful.
> >> > > >>
> >> > > >> The RC is available at:
> >> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >> > > >>
> >> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >> > > >>
> >> > > >> The maven artifacts are staged at
> >> > > >>
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >> > > >>
> >> > > >> You can find my public key at:
> >> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > >>
> >> > > >> Change log
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >> > > >>
> >> > > >> Release notes
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >> > > >>
> >> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> >> > > >>
> >> > > >> Key changes include
> >> > > >>
> >> > > >> * Big update of dependencies to try and keep those reports of
> >> > > >> transitive CVEs under control -both genuine and false
> positive.
> >> > > >> * HDFS RBF enhancements
> >> > > >> * Critical fix to ABFS input stream prefetching for correct
> >> reading.
> >> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> >> > > >> high-performance versions for file:// and s3a:// filesystems.
> >> > > >> file:// through java native io
> >> > > >> s3a:// parallel GET requests.
> >> > > >> * This release includes Arm64 binaries. Please can anyone with
> >> > > >> compatible systems validate these.
> >> > > >>
> >> > > >>
> >> > > >> Please try the release and vote on it, even though i don't know
> >> what
> >> > is a
> >> > > >> good timeline here...i'm actually going on holiday in early jan.
> >> > Mukund is
> >> > > >> around and so can drive the process while I'm offline.
> >> > > >>
> >> > > >> Assuming we do have another iteration, the RC1 will not be before
> >> mid
> >> > jan
> >> > > >> for that reason
> >> > > >>
> >> > > >> Steve (and mukund)
> >> > > >>
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > > > For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> >> > > >
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >> > >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >> >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
I haven't got a chance to deep dive into HADOOP-18324
<https://issues.apache.org/jira/browse/HADOOP-18324> which is claimed to be
the reason for these failures. Most probably will try to check next week if
it is still there.
From the PR uploaded on HDFS-16853
<https://issues.apache.org/jira/browse/HDFS-16853> it looks like changing
or tweaking the cleanup logic itself rather than with playing with tests or
MiniDfsCluster, So, the clean up logic has issues but I still need to check
what is the impact of that, If I have a service and that terminates in a
non test setup, will the restart be an issue like these tests are facing,
my initial hunch was No. But I need to carefully check and see what is the
impact and what other issues it can cause. the original logic ain't
something which can be decoded with just a few seconds of cursory look.
++ @Owen O'Malley <ow...@gmail.com> is the original author of the
Hadoop Jira, maybe he can share some pointers about that.
-Ayush
On Thu, 5 Jan 2023 at 07:04, Chris Nauroth <cn...@apache.org> wrote:
> Is it a problem limited to MiniDFSCluster, or is it a broader problem of
> RPC client resource cleanup? The patch is changing connection close
> cleanup, so I assumed the latter. If so, then it could potentially impact
> applications integrating with the RPC clients.
>
> If the problem is limited to MiniDFSCluster and restarts within a single
> JVM, then I agree the impact is smaller. Then, we'd want to consider what
> downstream projects have tests that do restarts on a MiniDFSCluster.
>
> Chris Nauroth
>
>
> On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> mentioned anywhere in change log or release notes. Are they actually
> >> up-to-date?
> >
> >
> > I don't think there is any issue with the ReleaseNotes generation as such
> > but with the Resolution type of this ticket, It ain't marked as Fixed but
> > Done. The other ticket which is marked Done is also not part of the
> release
> > notes. [1]
> >
> > if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >
> > Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> > can happen outside a MiniDfsCluster setup? Just guessing from the
> > description in the ticket. It looked like when we did a restart of the
> > Namenode in the MiniDfsCluster, I guess that would be in the same single
> > JVM, and that is why a previous blocked thread caused issues with the
> > restart. That is what I understood, I haven't checked the code though.
> >
> > Second, In the same context, Being curious If this lands up being a
> > MiniDfsCluster only issue, do we still consider this a release blocker?
> Not
> > saying in a way it won't be serious, MiniDfsCluster is very widely used
> by
> > downstream projects and all, so just wanted to know....
> >
> > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> > veto, I am not sure if it will get done any time soon, so if the use case
> > is something required, I would suggest handling it at Hadoop itself. It
> > seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> > with 3.3.5 and it passed. Other than that Hive officially support only
> > Hadoop-3.3.1 and that too only in the last 4.x release[2]
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> > [2] https://issues.apache.org/jira/browse/HIVE-24484
> >
> > -Ayush
> >
> > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> >
> >> -1, because if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >> Otherwise, this RC was looking good:
> >>
> >> * Verified all checksums.
> >> * Verified all signatures.
> >> * Built from source, including native code on Linux.
> >> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >> -Drequire.zstd -DskipTests
> >> * Tests passed.
> >> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >> * Checked dependency tree to make sure we have all of the expected
> library
> >> updates that are mentioned in the release notes.
> >> * mvn -o dependency:tree
> >> * Farewell, S3Guard.
> >> * Confirmed that hadoop-openstack is now just a stub placeholder
> artifact
> >> with no code.
> >> * For ARM verification:
> >> * Ran "file <X>" on all native binaries in the ARM tarball to
> confirm
> >> they actually came out with ARM as the architecture.
> >> * Output of hadoop checknative -a on ARM looks good.
> >> * Ran a MapReduce job with the native bzip2 codec for compression,
> and
> >> it worked fine.
> >> * Ran a MapReduce job with YARN configured to use
> >> LinuxContainerExecutor and verified launching the containers through
> >> container-executor worked.
> >>
> >> My local setup didn't have the test failures mentioned by Viraj, though
> >> there was some flakiness with a few HDFS snapshot tests timing out.
> >>
> >> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> >> request tracking an upgrade attempt. It's looking like some amount of
> code
> >> changes are required:
> >>
> >> https://issues.apache.org/jira/browse/HIVE-26648
> >> https://github.com/apache/hive/pull/3744
> >>
> >> Chris Nauroth
> >>
> >>
> >> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> >>
> >> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> > mentioned anywhere in change log or release notes. Are they actually
> >> > up-to-date?
> >> >
> >> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> >> > <iw...@oss.nttdata.com> wrote:
> >> > >
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > >
> >> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> >> not
> >> > expected in hbase-shaded-with-hadoop-check-invariants.
> >> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> >> > HBASE-27292.
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >> > >
> >> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> >> > (check-jar-contents-for-stuff-with-hadoop) @
> >> > hbase-shaded-with-hadoop-check-invariants ---
> >> > > [ERROR] Found artifact with unexpected contents:
> >> >
> >>
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> >> > > Please check the following and either correct the build or
> >> update
> >> > > the allowed list with reasoning.
> >> > >
> >> > > com/
> >> > > com/sun/
> >> > > com/sun/jersey/
> >> > > com/sun/jersey/json/
> >> > > ...
> >> > >
> >> > >
> >> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> >> seem
> >> > to be problematic.
> >> > > Excluding them on hive-jdbc might be the fix.
> >> > >
> >> > > [ERROR] Failed to execute goal
> >> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> >> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >> >
> >>
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> >> > entry
> >> >
> >>
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> >> > java.lang.IllegalArgumentException: Unsupported class file major
> >> version 59
> >> > -> [Help 1]
> >> > > ...
> >> > >
> >> > >
> >> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> >> > > > Thanks for your great effort for the new release, Steve and
> Mukund.
> >> > > >
> >> > > > +1 while it would be nice if we can address missed Javadocs.
> >> > > >
> >> > > > + verified the signature and checksum.
> >> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> >> native
> >> > profile enabled.
> >> > > > + launched pseudo distributed cluster including kms and httpfs
> >> with
> >> > Kerberos and SSL enabled.
> >> > > > + created encryption zone, put and read files via httpfs.
> >> > > > + ran example MR wordcount over encryption zone.
> >> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux
> 8
> >> > (both x86_64 and aarch64).
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> >> 3.3.4
> >> > worked.
> >> > > > + skimmed the site contents.
> >> > > > - Javadocs are not contained (under r3.3.5/api).
> >> > > > # The issue can be reproduced even if I built site docs from
> >> the
> >> > source.
> >> > > >
> >> > > > Masatake Iwasaki
> >> > > >
> >> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> >> > > >> Mukund and I have put together a release candidate (RC0) for
> Hadoop
> >> > 3.3.5.
> >> > > >>
> >> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> >> and
> >> > > >> expect people to be able to test it thoroughly enough to make
> this
> >> > the one
> >> > > >> we can ship.
> >> > > >>
> >> > > >> What we would like is for anyone who can to verify the tarballs,
> >> and
> >> > test
> >> > > >> the binaries, especially anyone who can try the arm64 binaries.
> >> We've
> >> > got
> >> > > >> the building of those done and now the build file will
> incorporate
> >> > them
> >> > > >> into the release -but neither of us have actually tested it yet.
> >> > Maybe I
> >> > > >> should try it on my pi400 over xmas.
> >> > > >>
> >> > > >> The maven artifacts are up on the apache staging repo -they are
> the
> >> > ones
> >> > > >> from x86 build. Building and testing downstream apps will be
> >> > incredibly
> >> > > >> helpful.
> >> > > >>
> >> > > >> The RC is available at:
> >> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >> > > >>
> >> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >> > > >>
> >> > > >> The maven artifacts are staged at
> >> > > >>
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >> > > >>
> >> > > >> You can find my public key at:
> >> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > >>
> >> > > >> Change log
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >> > > >>
> >> > > >> Release notes
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >> > > >>
> >> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> >> > > >>
> >> > > >> Key changes include
> >> > > >>
> >> > > >> * Big update of dependencies to try and keep those reports of
> >> > > >> transitive CVEs under control -both genuine and false
> positive.
> >> > > >> * HDFS RBF enhancements
> >> > > >> * Critical fix to ABFS input stream prefetching for correct
> >> reading.
> >> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> >> > > >> high-performance versions for file:// and s3a:// filesystems.
> >> > > >> file:// through java native io
> >> > > >> s3a:// parallel GET requests.
> >> > > >> * This release includes Arm64 binaries. Please can anyone with
> >> > > >> compatible systems validate these.
> >> > > >>
> >> > > >>
> >> > > >> Please try the release and vote on it, even though i don't know
> >> what
> >> > is a
> >> > > >> good timeline here...i'm actually going on holiday in early jan.
> >> > Mukund is
> >> > > >> around and so can drive the process while I'm offline.
> >> > > >>
> >> > > >> Assuming we do have another iteration, the RC1 will not be before
> >> mid
> >> > jan
> >> > > >> for that reason
> >> > > >>
> >> > > >> Steve (and mukund)
> >> > > >>
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > > > For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> >> > > >
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >> > >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >> >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
I haven't got a chance to deep dive into HADOOP-18324
<https://issues.apache.org/jira/browse/HADOOP-18324> which is claimed to be
the reason for these failures. Most probably will try to check next week if
it is still there.
From the PR uploaded on HDFS-16853
<https://issues.apache.org/jira/browse/HDFS-16853> it looks like changing
or tweaking the cleanup logic itself rather than with playing with tests or
MiniDfsCluster, So, the clean up logic has issues but I still need to check
what is the impact of that, If I have a service and that terminates in a
non test setup, will the restart be an issue like these tests are facing,
my initial hunch was No. But I need to carefully check and see what is the
impact and what other issues it can cause. the original logic ain't
something which can be decoded with just a few seconds of cursory look.
++ @Owen O'Malley <ow...@gmail.com> is the original author of the
Hadoop Jira, maybe he can share some pointers about that.
-Ayush
On Thu, 5 Jan 2023 at 07:04, Chris Nauroth <cn...@apache.org> wrote:
> Is it a problem limited to MiniDFSCluster, or is it a broader problem of
> RPC client resource cleanup? The patch is changing connection close
> cleanup, so I assumed the latter. If so, then it could potentially impact
> applications integrating with the RPC clients.
>
> If the problem is limited to MiniDFSCluster and restarts within a single
> JVM, then I agree the impact is smaller. Then, we'd want to consider what
> downstream projects have tests that do restarts on a MiniDFSCluster.
>
> Chris Nauroth
>
>
> On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> mentioned anywhere in change log or release notes. Are they actually
> >> up-to-date?
> >
> >
> > I don't think there is any issue with the ReleaseNotes generation as such
> > but with the Resolution type of this ticket, It ain't marked as Fixed but
> > Done. The other ticket which is marked Done is also not part of the
> release
> > notes. [1]
> >
> > if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >
> > Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> > can happen outside a MiniDfsCluster setup? Just guessing from the
> > description in the ticket. It looked like when we did a restart of the
> > Namenode in the MiniDfsCluster, I guess that would be in the same single
> > JVM, and that is why a previous blocked thread caused issues with the
> > restart. That is what I understood, I haven't checked the code though.
> >
> > Second, In the same context, Being curious If this lands up being a
> > MiniDfsCluster only issue, do we still consider this a release blocker?
> Not
> > saying in a way it won't be serious, MiniDfsCluster is very widely used
> by
> > downstream projects and all, so just wanted to know....
> >
> > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> > veto, I am not sure if it will get done any time soon, so if the use case
> > is something required, I would suggest handling it at Hadoop itself. It
> > seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> > with 3.3.5 and it passed. Other than that Hive officially support only
> > Hadoop-3.3.1 and that too only in the last 4.x release[2]
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> > [2] https://issues.apache.org/jira/browse/HIVE-24484
> >
> > -Ayush
> >
> > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> >
> >> -1, because if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >> Otherwise, this RC was looking good:
> >>
> >> * Verified all checksums.
> >> * Verified all signatures.
> >> * Built from source, including native code on Linux.
> >> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >> -Drequire.zstd -DskipTests
> >> * Tests passed.
> >> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >> * Checked dependency tree to make sure we have all of the expected
> library
> >> updates that are mentioned in the release notes.
> >> * mvn -o dependency:tree
> >> * Farewell, S3Guard.
> >> * Confirmed that hadoop-openstack is now just a stub placeholder
> artifact
> >> with no code.
> >> * For ARM verification:
> >> * Ran "file <X>" on all native binaries in the ARM tarball to
> confirm
> >> they actually came out with ARM as the architecture.
> >> * Output of hadoop checknative -a on ARM looks good.
> >> * Ran a MapReduce job with the native bzip2 codec for compression,
> and
> >> it worked fine.
> >> * Ran a MapReduce job with YARN configured to use
> >> LinuxContainerExecutor and verified launching the containers through
> >> container-executor worked.
> >>
> >> My local setup didn't have the test failures mentioned by Viraj, though
> >> there was some flakiness with a few HDFS snapshot tests timing out.
> >>
> >> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> >> request tracking an upgrade attempt. It's looking like some amount of
> code
> >> changes are required:
> >>
> >> https://issues.apache.org/jira/browse/HIVE-26648
> >> https://github.com/apache/hive/pull/3744
> >>
> >> Chris Nauroth
> >>
> >>
> >> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> >>
> >> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> > mentioned anywhere in change log or release notes. Are they actually
> >> > up-to-date?
> >> >
> >> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> >> > <iw...@oss.nttdata.com> wrote:
> >> > >
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > >
> >> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> >> not
> >> > expected in hbase-shaded-with-hadoop-check-invariants.
> >> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> >> > HBASE-27292.
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >> > >
> >> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> >> > (check-jar-contents-for-stuff-with-hadoop) @
> >> > hbase-shaded-with-hadoop-check-invariants ---
> >> > > [ERROR] Found artifact with unexpected contents:
> >> >
> >>
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> >> > > Please check the following and either correct the build or
> >> update
> >> > > the allowed list with reasoning.
> >> > >
> >> > > com/
> >> > > com/sun/
> >> > > com/sun/jersey/
> >> > > com/sun/jersey/json/
> >> > > ...
> >> > >
> >> > >
> >> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> >> seem
> >> > to be problematic.
> >> > > Excluding them on hive-jdbc might be the fix.
> >> > >
> >> > > [ERROR] Failed to execute goal
> >> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> >> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >> >
> >>
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> >> > entry
> >> >
> >>
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> >> > java.lang.IllegalArgumentException: Unsupported class file major
> >> version 59
> >> > -> [Help 1]
> >> > > ...
> >> > >
> >> > >
> >> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> >> > > > Thanks for your great effort for the new release, Steve and
> Mukund.
> >> > > >
> >> > > > +1 while it would be nice if we can address missed Javadocs.
> >> > > >
> >> > > > + verified the signature and checksum.
> >> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> >> native
> >> > profile enabled.
> >> > > > + launched pseudo distributed cluster including kms and httpfs
> >> with
> >> > Kerberos and SSL enabled.
> >> > > > + created encryption zone, put and read files via httpfs.
> >> > > > + ran example MR wordcount over encryption zone.
> >> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux
> 8
> >> > (both x86_64 and aarch64).
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> >> 3.3.4
> >> > worked.
> >> > > > + skimmed the site contents.
> >> > > > - Javadocs are not contained (under r3.3.5/api).
> >> > > > # The issue can be reproduced even if I built site docs from
> >> the
> >> > source.
> >> > > >
> >> > > > Masatake Iwasaki
> >> > > >
> >> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> >> > > >> Mukund and I have put together a release candidate (RC0) for
> Hadoop
> >> > 3.3.5.
> >> > > >>
> >> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> >> and
> >> > > >> expect people to be able to test it thoroughly enough to make
> this
> >> > the one
> >> > > >> we can ship.
> >> > > >>
> >> > > >> What we would like is for anyone who can to verify the tarballs,
> >> and
> >> > test
> >> > > >> the binaries, especially anyone who can try the arm64 binaries.
> >> We've
> >> > got
> >> > > >> the building of those done and now the build file will
> incorporate
> >> > them
> >> > > >> into the release -but neither of us have actually tested it yet.
> >> > Maybe I
> >> > > >> should try it on my pi400 over xmas.
> >> > > >>
> >> > > >> The maven artifacts are up on the apache staging repo -they are
> the
> >> > ones
> >> > > >> from x86 build. Building and testing downstream apps will be
> >> > incredibly
> >> > > >> helpful.
> >> > > >>
> >> > > >> The RC is available at:
> >> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >> > > >>
> >> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >> > > >>
> >> > > >> The maven artifacts are staged at
> >> > > >>
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >> > > >>
> >> > > >> You can find my public key at:
> >> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > >>
> >> > > >> Change log
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >> > > >>
> >> > > >> Release notes
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >> > > >>
> >> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> >> > > >>
> >> > > >> Key changes include
> >> > > >>
> >> > > >> * Big update of dependencies to try and keep those reports of
> >> > > >> transitive CVEs under control -both genuine and false
> positive.
> >> > > >> * HDFS RBF enhancements
> >> > > >> * Critical fix to ABFS input stream prefetching for correct
> >> reading.
> >> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> >> > > >> high-performance versions for file:// and s3a:// filesystems.
> >> > > >> file:// through java native io
> >> > > >> s3a:// parallel GET requests.
> >> > > >> * This release includes Arm64 binaries. Please can anyone with
> >> > > >> compatible systems validate these.
> >> > > >>
> >> > > >>
> >> > > >> Please try the release and vote on it, even though i don't know
> >> what
> >> > is a
> >> > > >> good timeline here...i'm actually going on holiday in early jan.
> >> > Mukund is
> >> > > >> around and so can drive the process while I'm offline.
> >> > > >>
> >> > > >> Assuming we do have another iteration, the RC1 will not be before
> >> mid
> >> > jan
> >> > > >> for that reason
> >> > > >>
> >> > > >> Steve (and mukund)
> >> > > >>
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > > > For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> >> > > >
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >> > >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >> >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
I haven't got a chance to deep dive into HADOOP-18324
<https://issues.apache.org/jira/browse/HADOOP-18324> which is claimed to be
the reason for these failures. Most probably will try to check next week if
it is still there.
From the PR uploaded on HDFS-16853
<https://issues.apache.org/jira/browse/HDFS-16853> it looks like changing
or tweaking the cleanup logic itself rather than with playing with tests or
MiniDfsCluster, So, the clean up logic has issues but I still need to check
what is the impact of that, If I have a service and that terminates in a
non test setup, will the restart be an issue like these tests are facing,
my initial hunch was No. But I need to carefully check and see what is the
impact and what other issues it can cause. the original logic ain't
something which can be decoded with just a few seconds of cursory look.
++ @Owen O'Malley <ow...@gmail.com> is the original author of the
Hadoop Jira, maybe he can share some pointers about that.
-Ayush
On Thu, 5 Jan 2023 at 07:04, Chris Nauroth <cn...@apache.org> wrote:
> Is it a problem limited to MiniDFSCluster, or is it a broader problem of
> RPC client resource cleanup? The patch is changing connection close
> cleanup, so I assumed the latter. If so, then it could potentially impact
> applications integrating with the RPC clients.
>
> If the problem is limited to MiniDFSCluster and restarts within a single
> JVM, then I agree the impact is smaller. Then, we'd want to consider what
> downstream projects have tests that do restarts on a MiniDFSCluster.
>
> Chris Nauroth
>
>
> On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> mentioned anywhere in change log or release notes. Are they actually
> >> up-to-date?
> >
> >
> > I don't think there is any issue with the ReleaseNotes generation as such
> > but with the Resolution type of this ticket, It ain't marked as Fixed but
> > Done. The other ticket which is marked Done is also not part of the
> release
> > notes. [1]
> >
> > if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >
> > Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> > can happen outside a MiniDfsCluster setup? Just guessing from the
> > description in the ticket. It looked like when we did a restart of the
> > Namenode in the MiniDfsCluster, I guess that would be in the same single
> > JVM, and that is why a previous blocked thread caused issues with the
> > restart. That is what I understood, I haven't checked the code though.
> >
> > Second, In the same context, Being curious If this lands up being a
> > MiniDfsCluster only issue, do we still consider this a release blocker?
> Not
> > saying in a way it won't be serious, MiniDfsCluster is very widely used
> by
> > downstream projects and all, so just wanted to know....
> >
> > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> > veto, I am not sure if it will get done any time soon, so if the use case
> > is something required, I would suggest handling it at Hadoop itself. It
> > seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> > with 3.3.5 and it passed. Other than that Hive officially support only
> > Hadoop-3.3.1 and that too only in the last 4.x release[2]
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> > [2] https://issues.apache.org/jira/browse/HIVE-24484
> >
> > -Ayush
> >
> > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> >
> >> -1, because if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >> Otherwise, this RC was looking good:
> >>
> >> * Verified all checksums.
> >> * Verified all signatures.
> >> * Built from source, including native code on Linux.
> >> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >> -Drequire.zstd -DskipTests
> >> * Tests passed.
> >> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >> * Checked dependency tree to make sure we have all of the expected
> library
> >> updates that are mentioned in the release notes.
> >> * mvn -o dependency:tree
> >> * Farewell, S3Guard.
> >> * Confirmed that hadoop-openstack is now just a stub placeholder
> artifact
> >> with no code.
> >> * For ARM verification:
> >> * Ran "file <X>" on all native binaries in the ARM tarball to
> confirm
> >> they actually came out with ARM as the architecture.
> >> * Output of hadoop checknative -a on ARM looks good.
> >> * Ran a MapReduce job with the native bzip2 codec for compression,
> and
> >> it worked fine.
> >> * Ran a MapReduce job with YARN configured to use
> >> LinuxContainerExecutor and verified launching the containers through
> >> container-executor worked.
> >>
> >> My local setup didn't have the test failures mentioned by Viraj, though
> >> there was some flakiness with a few HDFS snapshot tests timing out.
> >>
> >> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> >> request tracking an upgrade attempt. It's looking like some amount of
> code
> >> changes are required:
> >>
> >> https://issues.apache.org/jira/browse/HIVE-26648
> >> https://github.com/apache/hive/pull/3744
> >>
> >> Chris Nauroth
> >>
> >>
> >> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> >>
> >> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> > mentioned anywhere in change log or release notes. Are they actually
> >> > up-to-date?
> >> >
> >> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> >> > <iw...@oss.nttdata.com> wrote:
> >> > >
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > >
> >> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> >> not
> >> > expected in hbase-shaded-with-hadoop-check-invariants.
> >> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> >> > HBASE-27292.
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >> > >
> >> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> >> > (check-jar-contents-for-stuff-with-hadoop) @
> >> > hbase-shaded-with-hadoop-check-invariants ---
> >> > > [ERROR] Found artifact with unexpected contents:
> >> >
> >>
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> >> > > Please check the following and either correct the build or
> >> update
> >> > > the allowed list with reasoning.
> >> > >
> >> > > com/
> >> > > com/sun/
> >> > > com/sun/jersey/
> >> > > com/sun/jersey/json/
> >> > > ...
> >> > >
> >> > >
> >> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> >> seem
> >> > to be problematic.
> >> > > Excluding them on hive-jdbc might be the fix.
> >> > >
> >> > > [ERROR] Failed to execute goal
> >> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> >> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >> >
> >>
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> >> > entry
> >> >
> >>
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> >> > java.lang.IllegalArgumentException: Unsupported class file major
> >> version 59
> >> > -> [Help 1]
> >> > > ...
> >> > >
> >> > >
> >> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> >> > > > Thanks for your great effort for the new release, Steve and
> Mukund.
> >> > > >
> >> > > > +1 while it would be nice if we can address missed Javadocs.
> >> > > >
> >> > > > + verified the signature and checksum.
> >> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> >> native
> >> > profile enabled.
> >> > > > + launched pseudo distributed cluster including kms and httpfs
> >> with
> >> > Kerberos and SSL enabled.
> >> > > > + created encryption zone, put and read files via httpfs.
> >> > > > + ran example MR wordcount over encryption zone.
> >> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux
> 8
> >> > (both x86_64 and aarch64).
> >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
> >> to
> >> > dependency change.
> >> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> >> 3.3.4
> >> > worked.
> >> > > > + skimmed the site contents.
> >> > > > - Javadocs are not contained (under r3.3.5/api).
> >> > > > # The issue can be reproduced even if I built site docs from
> >> the
> >> > source.
> >> > > >
> >> > > > Masatake Iwasaki
> >> > > >
> >> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> >> > > >> Mukund and I have put together a release candidate (RC0) for
> Hadoop
> >> > 3.3.5.
> >> > > >>
> >> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> >> and
> >> > > >> expect people to be able to test it thoroughly enough to make
> this
> >> > the one
> >> > > >> we can ship.
> >> > > >>
> >> > > >> What we would like is for anyone who can to verify the tarballs,
> >> and
> >> > test
> >> > > >> the binaries, especially anyone who can try the arm64 binaries.
> >> We've
> >> > got
> >> > > >> the building of those done and now the build file will
> incorporate
> >> > them
> >> > > >> into the release -but neither of us have actually tested it yet.
> >> > Maybe I
> >> > > >> should try it on my pi400 over xmas.
> >> > > >>
> >> > > >> The maven artifacts are up on the apache staging repo -they are
> the
> >> > ones
> >> > > >> from x86 build. Building and testing downstream apps will be
> >> > incredibly
> >> > > >> helpful.
> >> > > >>
> >> > > >> The RC is available at:
> >> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >> > > >>
> >> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >> > > >>
> >> > > >> The maven artifacts are staged at
> >> > > >>
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >> > > >>
> >> > > >> You can find my public key at:
> >> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > > >>
> >> > > >> Change log
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >> > > >>
> >> > > >> Release notes
> >> > > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >> > > >>
> >> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> >> > > >>
> >> > > >> Key changes include
> >> > > >>
> >> > > >> * Big update of dependencies to try and keep those reports of
> >> > > >> transitive CVEs under control -both genuine and false
> positive.
> >> > > >> * HDFS RBF enhancements
> >> > > >> * Critical fix to ABFS input stream prefetching for correct
> >> reading.
> >> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> >> > > >> high-performance versions for file:// and s3a:// filesystems.
> >> > > >> file:// through java native io
> >> > > >> s3a:// parallel GET requests.
> >> > > >> * This release includes Arm64 binaries. Please can anyone with
> >> > > >> compatible systems validate these.
> >> > > >>
> >> > > >>
> >> > > >> Please try the release and vote on it, even though i don't know
> >> what
> >> > is a
> >> > > >> good timeline here...i'm actually going on holiday in early jan.
> >> > Mukund is
> >> > > >> around and so can drive the process while I'm offline.
> >> > > >>
> >> > > >> Assuming we do have another iteration, the RC1 will not be before
> >> mid
> >> > jan
> >> > > >> for that reason
> >> > > >>
> >> > > >> Steve (and mukund)
> >> > > >>
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > > > For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> >> > > >
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >> > >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >> >
> >> >
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
Is it a problem limited to MiniDFSCluster, or is it a broader problem of
RPC client resource cleanup? The patch is changing connection close
cleanup, so I assumed the latter. If so, then it could potentially impact
applications integrating with the RPC clients.
If the problem is limited to MiniDFSCluster and restarts within a single
JVM, then I agree the impact is smaller. Then, we'd want to consider what
downstream projects have tests that do restarts on a MiniDFSCluster.
Chris Nauroth
On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> mentioned anywhere in change log or release notes. Are they actually
>> up-to-date?
>
>
> I don't think there is any issue with the ReleaseNotes generation as such
> but with the Resolution type of this ticket, It ain't marked as Fixed but
> Done. The other ticket which is marked Done is also not part of the release
> notes. [1]
>
> if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>
> Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> can happen outside a MiniDfsCluster setup? Just guessing from the
> description in the ticket. It looked like when we did a restart of the
> Namenode in the MiniDfsCluster, I guess that would be in the same single
> JVM, and that is why a previous blocked thread caused issues with the
> restart. That is what I understood, I haven't checked the code though.
>
> Second, In the same context, Being curious If this lands up being a
> MiniDfsCluster only issue, do we still consider this a release blocker? Not
> saying in a way it won't be serious, MiniDfsCluster is very widely used by
> downstream projects and all, so just wanted to know....
>
> Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> veto, I am not sure if it will get done any time soon, so if the use case
> is something required, I would suggest handling it at Hadoop itself. It
> seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> with 3.3.5 and it passed. Other than that Hive officially support only
> Hadoop-3.3.1 and that too only in the last 4.x release[2]
>
>
> [1]
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> [2] https://issues.apache.org/jira/browse/HIVE-24484
>
> -Ayush
>
> On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
>
>> -1, because if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>> Otherwise, this RC was looking good:
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests passed.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>> * Checked dependency tree to make sure we have all of the expected library
>> updates that are mentioned in the release notes.
>> * mvn -o dependency:tree
>> * Farewell, S3Guard.
>> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
>> with no code.
>> * For ARM verification:
>> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
>> they actually came out with ARM as the architecture.
>> * Output of hadoop checknative -a on ARM looks good.
>> * Ran a MapReduce job with the native bzip2 codec for compression, and
>> it worked fine.
>> * Ran a MapReduce job with YARN configured to use
>> LinuxContainerExecutor and verified launching the containers through
>> container-executor worked.
>>
>> My local setup didn't have the test failures mentioned by Viraj, though
>> there was some flakiness with a few HDFS snapshot tests timing out.
>>
>> Regarding Hive and Bouncy Castle, there is an existing issue and pull
>> request tracking an upgrade attempt. It's looking like some amount of code
>> changes are required:
>>
>> https://issues.apache.org/jira/browse/HIVE-26648
>> https://github.com/apache/hive/pull/3744
>>
>> Chris Nauroth
>>
>>
>> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>>
>> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> > mentioned anywhere in change log or release notes. Are they actually
>> > up-to-date?
>> >
>> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
>> > <iw...@oss.nttdata.com> wrote:
>> > >
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > >
>> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
>> not
>> > expected in hbase-shaded-with-hadoop-check-invariants.
>> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
>> > HBASE-27292.
>> > >
>> >
>> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>> > >
>> > > [INFO] --- exec-maven-plugin:1.6.0:exec
>> > (check-jar-contents-for-stuff-with-hadoop) @
>> > hbase-shaded-with-hadoop-check-invariants ---
>> > > [ERROR] Found artifact with unexpected contents:
>> >
>> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>> > > Please check the following and either correct the build or
>> update
>> > > the allowed list with reasoning.
>> > >
>> > > com/
>> > > com/sun/
>> > > com/sun/jersey/
>> > > com/sun/jersey/json/
>> > > ...
>> > >
>> > >
>> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
>> seem
>> > to be problematic.
>> > > Excluding them on hive-jdbc might be the fix.
>> > >
>> > > [ERROR] Failed to execute goal
>> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
>> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
>> >
>> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
>> > entry
>> >
>> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
>> > java.lang.IllegalArgumentException: Unsupported class file major
>> version 59
>> > -> [Help 1]
>> > > ...
>> > >
>> > >
>> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
>> > > > Thanks for your great effort for the new release, Steve and Mukund.
>> > > >
>> > > > +1 while it would be nice if we can address missed Javadocs.
>> > > >
>> > > > + verified the signature and checksum.
>> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
>> native
>> > profile enabled.
>> > > > + launched pseudo distributed cluster including kms and httpfs
>> with
>> > Kerberos and SSL enabled.
>> > > > + created encryption zone, put and read files via httpfs.
>> > > > + ran example MR wordcount over encryption zone.
>> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
>> > (both x86_64 and aarch64).
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
>> 3.3.4
>> > worked.
>> > > > + skimmed the site contents.
>> > > > - Javadocs are not contained (under r3.3.5/api).
>> > > > # The issue can be reproduced even if I built site docs from
>> the
>> > source.
>> > > >
>> > > > Masatake Iwasaki
>> > > >
>> > > > On 2022/12/22 4:28, Steve Loughran wrote:
>> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
>> > 3.3.5.
>> > > >>
>> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
>> and
>> > > >> expect people to be able to test it thoroughly enough to make this
>> > the one
>> > > >> we can ship.
>> > > >>
>> > > >> What we would like is for anyone who can to verify the tarballs,
>> and
>> > test
>> > > >> the binaries, especially anyone who can try the arm64 binaries.
>> We've
>> > got
>> > > >> the building of those done and now the build file will incorporate
>> > them
>> > > >> into the release -but neither of us have actually tested it yet.
>> > Maybe I
>> > > >> should try it on my pi400 over xmas.
>> > > >>
>> > > >> The maven artifacts are up on the apache staging repo -they are the
>> > ones
>> > > >> from x86 build. Building and testing downstream apps will be
>> > incredibly
>> > > >> helpful.
>> > > >>
>> > > >> The RC is available at:
>> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>> > > >>
>> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
>> > > >>
>> > > >> The maven artifacts are staged at
>> > > >>
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>> > > >>
>> > > >> You can find my public key at:
>> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>
>> > > >> Change log
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>> > > >>
>> > > >> Release notes
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>> > > >>
>> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
>> > > >>
>> > > >> Key changes include
>> > > >>
>> > > >> * Big update of dependencies to try and keep those reports of
>> > > >> transitive CVEs under control -both genuine and false positive.
>> > > >> * HDFS RBF enhancements
>> > > >> * Critical fix to ABFS input stream prefetching for correct
>> reading.
>> > > >> * Vectored IO API for all FSDataInputStream implementations, with
>> > > >> high-performance versions for file:// and s3a:// filesystems.
>> > > >> file:// through java native io
>> > > >> s3a:// parallel GET requests.
>> > > >> * This release includes Arm64 binaries. Please can anyone with
>> > > >> compatible systems validate these.
>> > > >>
>> > > >>
>> > > >> Please try the release and vote on it, even though i don't know
>> what
>> > is a
>> > > >> good timeline here...i'm actually going on holiday in early jan.
>> > Mukund is
>> > > >> around and so can drive the process while I'm offline.
>> > > >>
>> > > >> Assuming we do have another iteration, the RC1 will not be before
>> mid
>> > jan
>> > > >> for that reason
>> > > >>
>> > > >> Steve (and mukund)
>> > > >>
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
Is it a problem limited to MiniDFSCluster, or is it a broader problem of
RPC client resource cleanup? The patch is changing connection close
cleanup, so I assumed the latter. If so, then it could potentially impact
applications integrating with the RPC clients.
If the problem is limited to MiniDFSCluster and restarts within a single
JVM, then I agree the impact is smaller. Then, we'd want to consider what
downstream projects have tests that do restarts on a MiniDFSCluster.
Chris Nauroth
On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> mentioned anywhere in change log or release notes. Are they actually
>> up-to-date?
>
>
> I don't think there is any issue with the ReleaseNotes generation as such
> but with the Resolution type of this ticket, It ain't marked as Fixed but
> Done. The other ticket which is marked Done is also not part of the release
> notes. [1]
>
> if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>
> Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> can happen outside a MiniDfsCluster setup? Just guessing from the
> description in the ticket. It looked like when we did a restart of the
> Namenode in the MiniDfsCluster, I guess that would be in the same single
> JVM, and that is why a previous blocked thread caused issues with the
> restart. That is what I understood, I haven't checked the code though.
>
> Second, In the same context, Being curious If this lands up being a
> MiniDfsCluster only issue, do we still consider this a release blocker? Not
> saying in a way it won't be serious, MiniDfsCluster is very widely used by
> downstream projects and all, so just wanted to know....
>
> Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> veto, I am not sure if it will get done any time soon, so if the use case
> is something required, I would suggest handling it at Hadoop itself. It
> seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> with 3.3.5 and it passed. Other than that Hive officially support only
> Hadoop-3.3.1 and that too only in the last 4.x release[2]
>
>
> [1]
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> [2] https://issues.apache.org/jira/browse/HIVE-24484
>
> -Ayush
>
> On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
>
>> -1, because if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>> Otherwise, this RC was looking good:
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests passed.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>> * Checked dependency tree to make sure we have all of the expected library
>> updates that are mentioned in the release notes.
>> * mvn -o dependency:tree
>> * Farewell, S3Guard.
>> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
>> with no code.
>> * For ARM verification:
>> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
>> they actually came out with ARM as the architecture.
>> * Output of hadoop checknative -a on ARM looks good.
>> * Ran a MapReduce job with the native bzip2 codec for compression, and
>> it worked fine.
>> * Ran a MapReduce job with YARN configured to use
>> LinuxContainerExecutor and verified launching the containers through
>> container-executor worked.
>>
>> My local setup didn't have the test failures mentioned by Viraj, though
>> there was some flakiness with a few HDFS snapshot tests timing out.
>>
>> Regarding Hive and Bouncy Castle, there is an existing issue and pull
>> request tracking an upgrade attempt. It's looking like some amount of code
>> changes are required:
>>
>> https://issues.apache.org/jira/browse/HIVE-26648
>> https://github.com/apache/hive/pull/3744
>>
>> Chris Nauroth
>>
>>
>> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>>
>> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> > mentioned anywhere in change log or release notes. Are they actually
>> > up-to-date?
>> >
>> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
>> > <iw...@oss.nttdata.com> wrote:
>> > >
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > >
>> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
>> not
>> > expected in hbase-shaded-with-hadoop-check-invariants.
>> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
>> > HBASE-27292.
>> > >
>> >
>> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>> > >
>> > > [INFO] --- exec-maven-plugin:1.6.0:exec
>> > (check-jar-contents-for-stuff-with-hadoop) @
>> > hbase-shaded-with-hadoop-check-invariants ---
>> > > [ERROR] Found artifact with unexpected contents:
>> >
>> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>> > > Please check the following and either correct the build or
>> update
>> > > the allowed list with reasoning.
>> > >
>> > > com/
>> > > com/sun/
>> > > com/sun/jersey/
>> > > com/sun/jersey/json/
>> > > ...
>> > >
>> > >
>> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
>> seem
>> > to be problematic.
>> > > Excluding them on hive-jdbc might be the fix.
>> > >
>> > > [ERROR] Failed to execute goal
>> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
>> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
>> >
>> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
>> > entry
>> >
>> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
>> > java.lang.IllegalArgumentException: Unsupported class file major
>> version 59
>> > -> [Help 1]
>> > > ...
>> > >
>> > >
>> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
>> > > > Thanks for your great effort for the new release, Steve and Mukund.
>> > > >
>> > > > +1 while it would be nice if we can address missed Javadocs.
>> > > >
>> > > > + verified the signature and checksum.
>> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
>> native
>> > profile enabled.
>> > > > + launched pseudo distributed cluster including kms and httpfs
>> with
>> > Kerberos and SSL enabled.
>> > > > + created encryption zone, put and read files via httpfs.
>> > > > + ran example MR wordcount over encryption zone.
>> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
>> > (both x86_64 and aarch64).
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
>> 3.3.4
>> > worked.
>> > > > + skimmed the site contents.
>> > > > - Javadocs are not contained (under r3.3.5/api).
>> > > > # The issue can be reproduced even if I built site docs from
>> the
>> > source.
>> > > >
>> > > > Masatake Iwasaki
>> > > >
>> > > > On 2022/12/22 4:28, Steve Loughran wrote:
>> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
>> > 3.3.5.
>> > > >>
>> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
>> and
>> > > >> expect people to be able to test it thoroughly enough to make this
>> > the one
>> > > >> we can ship.
>> > > >>
>> > > >> What we would like is for anyone who can to verify the tarballs,
>> and
>> > test
>> > > >> the binaries, especially anyone who can try the arm64 binaries.
>> We've
>> > got
>> > > >> the building of those done and now the build file will incorporate
>> > them
>> > > >> into the release -but neither of us have actually tested it yet.
>> > Maybe I
>> > > >> should try it on my pi400 over xmas.
>> > > >>
>> > > >> The maven artifacts are up on the apache staging repo -they are the
>> > ones
>> > > >> from x86 build. Building and testing downstream apps will be
>> > incredibly
>> > > >> helpful.
>> > > >>
>> > > >> The RC is available at:
>> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>> > > >>
>> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
>> > > >>
>> > > >> The maven artifacts are staged at
>> > > >>
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>> > > >>
>> > > >> You can find my public key at:
>> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>
>> > > >> Change log
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>> > > >>
>> > > >> Release notes
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>> > > >>
>> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
>> > > >>
>> > > >> Key changes include
>> > > >>
>> > > >> * Big update of dependencies to try and keep those reports of
>> > > >> transitive CVEs under control -both genuine and false positive.
>> > > >> * HDFS RBF enhancements
>> > > >> * Critical fix to ABFS input stream prefetching for correct
>> reading.
>> > > >> * Vectored IO API for all FSDataInputStream implementations, with
>> > > >> high-performance versions for file:// and s3a:// filesystems.
>> > > >> file:// through java native io
>> > > >> s3a:// parallel GET requests.
>> > > >> * This release includes Arm64 binaries. Please can anyone with
>> > > >> compatible systems validate these.
>> > > >>
>> > > >>
>> > > >> Please try the release and vote on it, even though i don't know
>> what
>> > is a
>> > > >> good timeline here...i'm actually going on holiday in early jan.
>> > Mukund is
>> > > >> around and so can drive the process while I'm offline.
>> > > >>
>> > > >> Assuming we do have another iteration, the RC1 will not be before
>> mid
>> > jan
>> > > >> for that reason
>> > > >>
>> > > >> Steve (and mukund)
>> > > >>
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
Is it a problem limited to MiniDFSCluster, or is it a broader problem of
RPC client resource cleanup? The patch is changing connection close
cleanup, so I assumed the latter. If so, then it could potentially impact
applications integrating with the RPC clients.
If the problem is limited to MiniDFSCluster and restarts within a single
JVM, then I agree the impact is smaller. Then, we'd want to consider what
downstream projects have tests that do restarts on a MiniDFSCluster.
Chris Nauroth
On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> mentioned anywhere in change log or release notes. Are they actually
>> up-to-date?
>
>
> I don't think there is any issue with the ReleaseNotes generation as such
> but with the Resolution type of this ticket, It ain't marked as Fixed but
> Done. The other ticket which is marked Done is also not part of the release
> notes. [1]
>
> if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>
> Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> can happen outside a MiniDfsCluster setup? Just guessing from the
> description in the ticket. It looked like when we did a restart of the
> Namenode in the MiniDfsCluster, I guess that would be in the same single
> JVM, and that is why a previous blocked thread caused issues with the
> restart. That is what I understood, I haven't checked the code though.
>
> Second, In the same context, Being curious If this lands up being a
> MiniDfsCluster only issue, do we still consider this a release blocker? Not
> saying in a way it won't be serious, MiniDfsCluster is very widely used by
> downstream projects and all, so just wanted to know....
>
> Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> veto, I am not sure if it will get done any time soon, so if the use case
> is something required, I would suggest handling it at Hadoop itself. It
> seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> with 3.3.5 and it passed. Other than that Hive officially support only
> Hadoop-3.3.1 and that too only in the last 4.x release[2]
>
>
> [1]
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> [2] https://issues.apache.org/jira/browse/HIVE-24484
>
> -Ayush
>
> On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
>
>> -1, because if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>> Otherwise, this RC was looking good:
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests passed.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>> * Checked dependency tree to make sure we have all of the expected library
>> updates that are mentioned in the release notes.
>> * mvn -o dependency:tree
>> * Farewell, S3Guard.
>> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
>> with no code.
>> * For ARM verification:
>> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
>> they actually came out with ARM as the architecture.
>> * Output of hadoop checknative -a on ARM looks good.
>> * Ran a MapReduce job with the native bzip2 codec for compression, and
>> it worked fine.
>> * Ran a MapReduce job with YARN configured to use
>> LinuxContainerExecutor and verified launching the containers through
>> container-executor worked.
>>
>> My local setup didn't have the test failures mentioned by Viraj, though
>> there was some flakiness with a few HDFS snapshot tests timing out.
>>
>> Regarding Hive and Bouncy Castle, there is an existing issue and pull
>> request tracking an upgrade attempt. It's looking like some amount of code
>> changes are required:
>>
>> https://issues.apache.org/jira/browse/HIVE-26648
>> https://github.com/apache/hive/pull/3744
>>
>> Chris Nauroth
>>
>>
>> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>>
>> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> > mentioned anywhere in change log or release notes. Are they actually
>> > up-to-date?
>> >
>> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
>> > <iw...@oss.nttdata.com> wrote:
>> > >
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > >
>> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
>> not
>> > expected in hbase-shaded-with-hadoop-check-invariants.
>> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
>> > HBASE-27292.
>> > >
>> >
>> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>> > >
>> > > [INFO] --- exec-maven-plugin:1.6.0:exec
>> > (check-jar-contents-for-stuff-with-hadoop) @
>> > hbase-shaded-with-hadoop-check-invariants ---
>> > > [ERROR] Found artifact with unexpected contents:
>> >
>> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>> > > Please check the following and either correct the build or
>> update
>> > > the allowed list with reasoning.
>> > >
>> > > com/
>> > > com/sun/
>> > > com/sun/jersey/
>> > > com/sun/jersey/json/
>> > > ...
>> > >
>> > >
>> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
>> seem
>> > to be problematic.
>> > > Excluding them on hive-jdbc might be the fix.
>> > >
>> > > [ERROR] Failed to execute goal
>> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
>> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
>> >
>> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
>> > entry
>> >
>> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
>> > java.lang.IllegalArgumentException: Unsupported class file major
>> version 59
>> > -> [Help 1]
>> > > ...
>> > >
>> > >
>> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
>> > > > Thanks for your great effort for the new release, Steve and Mukund.
>> > > >
>> > > > +1 while it would be nice if we can address missed Javadocs.
>> > > >
>> > > > + verified the signature and checksum.
>> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
>> native
>> > profile enabled.
>> > > > + launched pseudo distributed cluster including kms and httpfs
>> with
>> > Kerberos and SSL enabled.
>> > > > + created encryption zone, put and read files via httpfs.
>> > > > + ran example MR wordcount over encryption zone.
>> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
>> > (both x86_64 and aarch64).
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
>> 3.3.4
>> > worked.
>> > > > + skimmed the site contents.
>> > > > - Javadocs are not contained (under r3.3.5/api).
>> > > > # The issue can be reproduced even if I built site docs from
>> the
>> > source.
>> > > >
>> > > > Masatake Iwasaki
>> > > >
>> > > > On 2022/12/22 4:28, Steve Loughran wrote:
>> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
>> > 3.3.5.
>> > > >>
>> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
>> and
>> > > >> expect people to be able to test it thoroughly enough to make this
>> > the one
>> > > >> we can ship.
>> > > >>
>> > > >> What we would like is for anyone who can to verify the tarballs,
>> and
>> > test
>> > > >> the binaries, especially anyone who can try the arm64 binaries.
>> We've
>> > got
>> > > >> the building of those done and now the build file will incorporate
>> > them
>> > > >> into the release -but neither of us have actually tested it yet.
>> > Maybe I
>> > > >> should try it on my pi400 over xmas.
>> > > >>
>> > > >> The maven artifacts are up on the apache staging repo -they are the
>> > ones
>> > > >> from x86 build. Building and testing downstream apps will be
>> > incredibly
>> > > >> helpful.
>> > > >>
>> > > >> The RC is available at:
>> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>> > > >>
>> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
>> > > >>
>> > > >> The maven artifacts are staged at
>> > > >>
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>> > > >>
>> > > >> You can find my public key at:
>> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>
>> > > >> Change log
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>> > > >>
>> > > >> Release notes
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>> > > >>
>> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
>> > > >>
>> > > >> Key changes include
>> > > >>
>> > > >> * Big update of dependencies to try and keep those reports of
>> > > >> transitive CVEs under control -both genuine and false positive.
>> > > >> * HDFS RBF enhancements
>> > > >> * Critical fix to ABFS input stream prefetching for correct
>> reading.
>> > > >> * Vectored IO API for all FSDataInputStream implementations, with
>> > > >> high-performance versions for file:// and s3a:// filesystems.
>> > > >> file:// through java native io
>> > > >> s3a:// parallel GET requests.
>> > > >> * This release includes Arm64 binaries. Please can anyone with
>> > > >> compatible systems validate these.
>> > > >>
>> > > >>
>> > > >> Please try the release and vote on it, even though i don't know
>> what
>> > is a
>> > > >> good timeline here...i'm actually going on holiday in early jan.
>> > Mukund is
>> > > >> around and so can drive the process while I'm offline.
>> > > >>
>> > > >> Assuming we do have another iteration, the RC1 will not be before
>> mid
>> > jan
>> > > >> for that reason
>> > > >>
>> > > >> Steve (and mukund)
>> > > >>
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
Is it a problem limited to MiniDFSCluster, or is it a broader problem of
RPC client resource cleanup? The patch is changing connection close
cleanup, so I assumed the latter. If so, then it could potentially impact
applications integrating with the RPC clients.
If the problem is limited to MiniDFSCluster and restarts within a single
JVM, then I agree the impact is smaller. Then, we'd want to consider what
downstream projects have tests that do restarts on a MiniDFSCluster.
Chris Nauroth
On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ay...@gmail.com> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> mentioned anywhere in change log or release notes. Are they actually
>> up-to-date?
>
>
> I don't think there is any issue with the ReleaseNotes generation as such
> but with the Resolution type of this ticket, It ain't marked as Fixed but
> Done. The other ticket which is marked Done is also not part of the release
> notes. [1]
>
> if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>
> Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> can happen outside a MiniDfsCluster setup? Just guessing from the
> description in the ticket. It looked like when we did a restart of the
> Namenode in the MiniDfsCluster, I guess that would be in the same single
> JVM, and that is why a previous blocked thread caused issues with the
> restart. That is what I understood, I haven't checked the code though.
>
> Second, In the same context, Being curious If this lands up being a
> MiniDfsCluster only issue, do we still consider this a release blocker? Not
> saying in a way it won't be serious, MiniDfsCluster is very widely used by
> downstream projects and all, so just wanted to know....
>
> Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> veto, I am not sure if it will get done any time soon, so if the use case
> is something required, I would suggest handling it at Hadoop itself. It
> seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> with 3.3.5 and it passed. Other than that Hive officially support only
> Hadoop-3.3.1 and that too only in the last 4.x release[2]
>
>
> [1]
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> [2] https://issues.apache.org/jira/browse/HIVE-24484
>
> -Ayush
>
> On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
>
>> -1, because if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>> Otherwise, this RC was looking good:
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests passed.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>> * Checked dependency tree to make sure we have all of the expected library
>> updates that are mentioned in the release notes.
>> * mvn -o dependency:tree
>> * Farewell, S3Guard.
>> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
>> with no code.
>> * For ARM verification:
>> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
>> they actually came out with ARM as the architecture.
>> * Output of hadoop checknative -a on ARM looks good.
>> * Ran a MapReduce job with the native bzip2 codec for compression, and
>> it worked fine.
>> * Ran a MapReduce job with YARN configured to use
>> LinuxContainerExecutor and verified launching the containers through
>> container-executor worked.
>>
>> My local setup didn't have the test failures mentioned by Viraj, though
>> there was some flakiness with a few HDFS snapshot tests timing out.
>>
>> Regarding Hive and Bouncy Castle, there is an existing issue and pull
>> request tracking an upgrade attempt. It's looking like some amount of code
>> changes are required:
>>
>> https://issues.apache.org/jira/browse/HIVE-26648
>> https://github.com/apache/hive/pull/3744
>>
>> Chris Nauroth
>>
>>
>> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>>
>> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> > mentioned anywhere in change log or release notes. Are they actually
>> > up-to-date?
>> >
>> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
>> > <iw...@oss.nttdata.com> wrote:
>> > >
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > >
>> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
>> not
>> > expected in hbase-shaded-with-hadoop-check-invariants.
>> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
>> > HBASE-27292.
>> > >
>> >
>> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>> > >
>> > > [INFO] --- exec-maven-plugin:1.6.0:exec
>> > (check-jar-contents-for-stuff-with-hadoop) @
>> > hbase-shaded-with-hadoop-check-invariants ---
>> > > [ERROR] Found artifact with unexpected contents:
>> >
>> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>> > > Please check the following and either correct the build or
>> update
>> > > the allowed list with reasoning.
>> > >
>> > > com/
>> > > com/sun/
>> > > com/sun/jersey/
>> > > com/sun/jersey/json/
>> > > ...
>> > >
>> > >
>> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
>> seem
>> > to be problematic.
>> > > Excluding them on hive-jdbc might be the fix.
>> > >
>> > > [ERROR] Failed to execute goal
>> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
>> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
>> >
>> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
>> > entry
>> >
>> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
>> > java.lang.IllegalArgumentException: Unsupported class file major
>> version 59
>> > -> [Help 1]
>> > > ...
>> > >
>> > >
>> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
>> > > > Thanks for your great effort for the new release, Steve and Mukund.
>> > > >
>> > > > +1 while it would be nice if we can address missed Javadocs.
>> > > >
>> > > > + verified the signature and checksum.
>> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
>> native
>> > profile enabled.
>> > > > + launched pseudo distributed cluster including kms and httpfs
>> with
>> > Kerberos and SSL enabled.
>> > > > + created encryption zone, put and read files via httpfs.
>> > > > + ran example MR wordcount over encryption zone.
>> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
>> > (both x86_64 and aarch64).
>> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
>> 3.3.4
>> > worked.
>> > > > + skimmed the site contents.
>> > > > - Javadocs are not contained (under r3.3.5/api).
>> > > > # The issue can be reproduced even if I built site docs from
>> the
>> > source.
>> > > >
>> > > > Masatake Iwasaki
>> > > >
>> > > > On 2022/12/22 4:28, Steve Loughran wrote:
>> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
>> > 3.3.5.
>> > > >>
>> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
>> and
>> > > >> expect people to be able to test it thoroughly enough to make this
>> > the one
>> > > >> we can ship.
>> > > >>
>> > > >> What we would like is for anyone who can to verify the tarballs,
>> and
>> > test
>> > > >> the binaries, especially anyone who can try the arm64 binaries.
>> We've
>> > got
>> > > >> the building of those done and now the build file will incorporate
>> > them
>> > > >> into the release -but neither of us have actually tested it yet.
>> > Maybe I
>> > > >> should try it on my pi400 over xmas.
>> > > >>
>> > > >> The maven artifacts are up on the apache staging repo -they are the
>> > ones
>> > > >> from x86 build. Building and testing downstream apps will be
>> > incredibly
>> > > >> helpful.
>> > > >>
>> > > >> The RC is available at:
>> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>> > > >>
>> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
>> > > >>
>> > > >> The maven artifacts are staged at
>> > > >>
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>> > > >>
>> > > >> You can find my public key at:
>> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >>
>> > > >> Change log
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>> > > >>
>> > > >> Release notes
>> > > >>
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>> > > >>
>> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
>> > > >>
>> > > >> Key changes include
>> > > >>
>> > > >> * Big update of dependencies to try and keep those reports of
>> > > >> transitive CVEs under control -both genuine and false positive.
>> > > >> * HDFS RBF enhancements
>> > > >> * Critical fix to ABFS input stream prefetching for correct
>> reading.
>> > > >> * Vectored IO API for all FSDataInputStream implementations, with
>> > > >> high-performance versions for file:// and s3a:// filesystems.
>> > > >> file:// through java native io
>> > > >> s3a:// parallel GET requests.
>> > > >> * This release includes Arm64 binaries. Please can anyone with
>> > > >> compatible systems validate these.
>> > > >>
>> > > >>
>> > > >> Please try the release and vote on it, even though i don't know
>> what
>> > is a
>> > > >> good timeline here...i'm actually going on holiday in early jan.
>> > Mukund is
>> > > >> around and so can drive the process while I'm offline.
>> > > >>
>> > > >> Assuming we do have another iteration, the RC1 will not be before
>> mid
>> > jan
>> > > >> for that reason
>> > > >>
>> > > >> Steve (and mukund)
>> > > >>
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
I don't think there is any issue with the ReleaseNotes generation as such
but with the Resolution type of this ticket, It ain't marked as Fixed but
Done. The other ticket which is marked Done is also not part of the release
notes. [1]
if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
Chris, I just had a very quick look at HDFS-16853, I am not sure if this
can happen outside a MiniDfsCluster setup? Just guessing from the
description in the ticket. It looked like when we did a restart of the
Namenode in the MiniDfsCluster, I guess that would be in the same single
JVM, and that is why a previous blocked thread caused issues with the
restart. That is what I understood, I haven't checked the code though.
Second, In the same context, Being curious If this lands up being a
MiniDfsCluster only issue, do we still consider this a release blocker? Not
saying in a way it won't be serious, MiniDfsCluster is very widely used by
downstream projects and all, so just wanted to know....
Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
veto, I am not sure if it will get done any time soon, so if the use case
is something required, I would suggest handling it at Hadoop itself. It
seems to be centric to Hive-3.x, I tried compiling the Hive master branch
with 3.3.5 and it passed. Other than that Hive officially support only
Hadoop-3.3.1 and that too only in the last 4.x release[2]
[1]
https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
[2] https://issues.apache.org/jira/browse/HIVE-24484
-Ayush
On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> -1, because if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
> Otherwise, this RC was looking good:
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Farewell, S3Guard.
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> My local setup didn't have the test failures mentioned by Viraj, though
> there was some flakiness with a few HDFS snapshot tests timing out.
>
> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> request tracking an upgrade attempt. It's looking like some amount of code
> changes are required:
>
> https://issues.apache.org/jira/browse/HIVE-26648
> https://github.com/apache/hive/pull/3744
>
> Chris Nauroth
>
>
> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> > mentioned anywhere in change log or release notes. Are they actually
> > up-to-date?
> >
> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> > <iw...@oss.nttdata.com> wrote:
> > >
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > >
> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> not
> > expected in hbase-shaded-with-hadoop-check-invariants.
> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> > HBASE-27292.
> > >
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> > >
> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> > (check-jar-contents-for-stuff-with-hadoop) @
> > hbase-shaded-with-hadoop-check-invariants ---
> > > [ERROR] Found artifact with unexpected contents:
> >
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > > Please check the following and either correct the build or
> update
> > > the allowed list with reasoning.
> > >
> > > com/
> > > com/sun/
> > > com/sun/jersey/
> > > com/sun/jersey/json/
> > > ...
> > >
> > >
> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> seem
> > to be problematic.
> > > Excluding them on hive-jdbc might be the fix.
> > >
> > > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> > entry
> >
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> > java.lang.IllegalArgumentException: Unsupported class file major version
> 59
> > -> [Help 1]
> > > ...
> > >
> > >
> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > > Thanks for your great effort for the new release, Steve and Mukund.
> > > >
> > > > +1 while it would be nice if we can address missed Javadocs.
> > > >
> > > > + verified the signature and checksum.
> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> native
> > profile enabled.
> > > > + launched pseudo distributed cluster including kms and httpfs
> with
> > Kerberos and SSL enabled.
> > > > + created encryption zone, put and read files via httpfs.
> > > > + ran example MR wordcount over encryption zone.
> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> > (both x86_64 and aarch64).
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> 3.3.4
> > worked.
> > > > + skimmed the site contents.
> > > > - Javadocs are not contained (under r3.3.5/api).
> > > > # The issue can be reproduced even if I built site docs from the
> > source.
> > > >
> > > > Masatake Iwasaki
> > > >
> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> > 3.3.5.
> > > >>
> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> and
> > > >> expect people to be able to test it thoroughly enough to make this
> > the one
> > > >> we can ship.
> > > >>
> > > >> What we would like is for anyone who can to verify the tarballs, and
> > test
> > > >> the binaries, especially anyone who can try the arm64 binaries.
> We've
> > got
> > > >> the building of those done and now the build file will incorporate
> > them
> > > >> into the release -but neither of us have actually tested it yet.
> > Maybe I
> > > >> should try it on my pi400 over xmas.
> > > >>
> > > >> The maven artifacts are up on the apache staging repo -they are the
> > ones
> > > >> from x86 build. Building and testing downstream apps will be
> > incredibly
> > > >> helpful.
> > > >>
> > > >> The RC is available at:
> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > > >>
> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > > >>
> > > >> The maven artifacts are staged at
> > > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > > >>
> > > >> You can find my public key at:
> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>
> > > >> Change log
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > > >>
> > > >> Release notes
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > > >>
> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > > >>
> > > >> Key changes include
> > > >>
> > > >> * Big update of dependencies to try and keep those reports of
> > > >> transitive CVEs under control -both genuine and false positive.
> > > >> * HDFS RBF enhancements
> > > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> > > >> high-performance versions for file:// and s3a:// filesystems.
> > > >> file:// through java native io
> > > >> s3a:// parallel GET requests.
> > > >> * This release includes Arm64 binaries. Please can anyone with
> > > >> compatible systems validate these.
> > > >>
> > > >>
> > > >> Please try the release and vote on it, even though i don't know what
> > is a
> > > >> good timeline here...i'm actually going on holiday in early jan.
> > Mukund is
> > > >> around and so can drive the process while I'm offline.
> > > >>
> > > >> Assuming we do have another iteration, the RC1 will not be before
> mid
> > jan
> > > >> for that reason
> > > >>
> > > >> Steve (and mukund)
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
I don't think there is any issue with the ReleaseNotes generation as such
but with the Resolution type of this ticket, It ain't marked as Fixed but
Done. The other ticket which is marked Done is also not part of the release
notes. [1]
if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
Chris, I just had a very quick look at HDFS-16853, I am not sure if this
can happen outside a MiniDfsCluster setup? Just guessing from the
description in the ticket. It looked like when we did a restart of the
Namenode in the MiniDfsCluster, I guess that would be in the same single
JVM, and that is why a previous blocked thread caused issues with the
restart. That is what I understood, I haven't checked the code though.
Second, In the same context, Being curious If this lands up being a
MiniDfsCluster only issue, do we still consider this a release blocker? Not
saying in a way it won't be serious, MiniDfsCluster is very widely used by
downstream projects and all, so just wanted to know....
Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
veto, I am not sure if it will get done any time soon, so if the use case
is something required, I would suggest handling it at Hadoop itself. It
seems to be centric to Hive-3.x, I tried compiling the Hive master branch
with 3.3.5 and it passed. Other than that Hive officially support only
Hadoop-3.3.1 and that too only in the last 4.x release[2]
[1]
https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
[2] https://issues.apache.org/jira/browse/HIVE-24484
-Ayush
On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> -1, because if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
> Otherwise, this RC was looking good:
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Farewell, S3Guard.
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> My local setup didn't have the test failures mentioned by Viraj, though
> there was some flakiness with a few HDFS snapshot tests timing out.
>
> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> request tracking an upgrade attempt. It's looking like some amount of code
> changes are required:
>
> https://issues.apache.org/jira/browse/HIVE-26648
> https://github.com/apache/hive/pull/3744
>
> Chris Nauroth
>
>
> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> > mentioned anywhere in change log or release notes. Are they actually
> > up-to-date?
> >
> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> > <iw...@oss.nttdata.com> wrote:
> > >
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > >
> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> not
> > expected in hbase-shaded-with-hadoop-check-invariants.
> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> > HBASE-27292.
> > >
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> > >
> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> > (check-jar-contents-for-stuff-with-hadoop) @
> > hbase-shaded-with-hadoop-check-invariants ---
> > > [ERROR] Found artifact with unexpected contents:
> >
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > > Please check the following and either correct the build or
> update
> > > the allowed list with reasoning.
> > >
> > > com/
> > > com/sun/
> > > com/sun/jersey/
> > > com/sun/jersey/json/
> > > ...
> > >
> > >
> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> seem
> > to be problematic.
> > > Excluding them on hive-jdbc might be the fix.
> > >
> > > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> > entry
> >
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> > java.lang.IllegalArgumentException: Unsupported class file major version
> 59
> > -> [Help 1]
> > > ...
> > >
> > >
> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > > Thanks for your great effort for the new release, Steve and Mukund.
> > > >
> > > > +1 while it would be nice if we can address missed Javadocs.
> > > >
> > > > + verified the signature and checksum.
> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> native
> > profile enabled.
> > > > + launched pseudo distributed cluster including kms and httpfs
> with
> > Kerberos and SSL enabled.
> > > > + created encryption zone, put and read files via httpfs.
> > > > + ran example MR wordcount over encryption zone.
> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> > (both x86_64 and aarch64).
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> 3.3.4
> > worked.
> > > > + skimmed the site contents.
> > > > - Javadocs are not contained (under r3.3.5/api).
> > > > # The issue can be reproduced even if I built site docs from the
> > source.
> > > >
> > > > Masatake Iwasaki
> > > >
> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> > 3.3.5.
> > > >>
> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> and
> > > >> expect people to be able to test it thoroughly enough to make this
> > the one
> > > >> we can ship.
> > > >>
> > > >> What we would like is for anyone who can to verify the tarballs, and
> > test
> > > >> the binaries, especially anyone who can try the arm64 binaries.
> We've
> > got
> > > >> the building of those done and now the build file will incorporate
> > them
> > > >> into the release -but neither of us have actually tested it yet.
> > Maybe I
> > > >> should try it on my pi400 over xmas.
> > > >>
> > > >> The maven artifacts are up on the apache staging repo -they are the
> > ones
> > > >> from x86 build. Building and testing downstream apps will be
> > incredibly
> > > >> helpful.
> > > >>
> > > >> The RC is available at:
> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > > >>
> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > > >>
> > > >> The maven artifacts are staged at
> > > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > > >>
> > > >> You can find my public key at:
> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>
> > > >> Change log
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > > >>
> > > >> Release notes
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > > >>
> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > > >>
> > > >> Key changes include
> > > >>
> > > >> * Big update of dependencies to try and keep those reports of
> > > >> transitive CVEs under control -both genuine and false positive.
> > > >> * HDFS RBF enhancements
> > > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> > > >> high-performance versions for file:// and s3a:// filesystems.
> > > >> file:// through java native io
> > > >> s3a:// parallel GET requests.
> > > >> * This release includes Arm64 binaries. Please can anyone with
> > > >> compatible systems validate these.
> > > >>
> > > >>
> > > >> Please try the release and vote on it, even though i don't know what
> > is a
> > > >> good timeline here...i'm actually going on holiday in early jan.
> > Mukund is
> > > >> around and so can drive the process while I'm offline.
> > > >>
> > > >> Assuming we do have another iteration, the RC1 will not be before
> mid
> > jan
> > > >> for that reason
> > > >>
> > > >> Steve (and mukund)
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
I don't think there is any issue with the ReleaseNotes generation as such
but with the Resolution type of this ticket, It ain't marked as Fixed but
Done. The other ticket which is marked Done is also not part of the release
notes. [1]
if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
Chris, I just had a very quick look at HDFS-16853, I am not sure if this
can happen outside a MiniDfsCluster setup? Just guessing from the
description in the ticket. It looked like when we did a restart of the
Namenode in the MiniDfsCluster, I guess that would be in the same single
JVM, and that is why a previous blocked thread caused issues with the
restart. That is what I understood, I haven't checked the code though.
Second, In the same context, Being curious If this lands up being a
MiniDfsCluster only issue, do we still consider this a release blocker? Not
saying in a way it won't be serious, MiniDfsCluster is very widely used by
downstream projects and all, so just wanted to know....
Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
veto, I am not sure if it will get done any time soon, so if the use case
is something required, I would suggest handling it at Hadoop itself. It
seems to be centric to Hive-3.x, I tried compiling the Hive master branch
with 3.3.5 and it passed. Other than that Hive officially support only
Hadoop-3.3.1 and that too only in the last 4.x release[2]
[1]
https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
[2] https://issues.apache.org/jira/browse/HIVE-24484
-Ayush
On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> -1, because if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
> Otherwise, this RC was looking good:
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Farewell, S3Guard.
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> My local setup didn't have the test failures mentioned by Viraj, though
> there was some flakiness with a few HDFS snapshot tests timing out.
>
> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> request tracking an upgrade attempt. It's looking like some amount of code
> changes are required:
>
> https://issues.apache.org/jira/browse/HIVE-26648
> https://github.com/apache/hive/pull/3744
>
> Chris Nauroth
>
>
> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> > mentioned anywhere in change log or release notes. Are they actually
> > up-to-date?
> >
> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> > <iw...@oss.nttdata.com> wrote:
> > >
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > >
> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> not
> > expected in hbase-shaded-with-hadoop-check-invariants.
> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> > HBASE-27292.
> > >
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> > >
> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> > (check-jar-contents-for-stuff-with-hadoop) @
> > hbase-shaded-with-hadoop-check-invariants ---
> > > [ERROR] Found artifact with unexpected contents:
> >
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > > Please check the following and either correct the build or
> update
> > > the allowed list with reasoning.
> > >
> > > com/
> > > com/sun/
> > > com/sun/jersey/
> > > com/sun/jersey/json/
> > > ...
> > >
> > >
> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> seem
> > to be problematic.
> > > Excluding them on hive-jdbc might be the fix.
> > >
> > > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> > entry
> >
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> > java.lang.IllegalArgumentException: Unsupported class file major version
> 59
> > -> [Help 1]
> > > ...
> > >
> > >
> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > > Thanks for your great effort for the new release, Steve and Mukund.
> > > >
> > > > +1 while it would be nice if we can address missed Javadocs.
> > > >
> > > > + verified the signature and checksum.
> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> native
> > profile enabled.
> > > > + launched pseudo distributed cluster including kms and httpfs
> with
> > Kerberos and SSL enabled.
> > > > + created encryption zone, put and read files via httpfs.
> > > > + ran example MR wordcount over encryption zone.
> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> > (both x86_64 and aarch64).
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> 3.3.4
> > worked.
> > > > + skimmed the site contents.
> > > > - Javadocs are not contained (under r3.3.5/api).
> > > > # The issue can be reproduced even if I built site docs from the
> > source.
> > > >
> > > > Masatake Iwasaki
> > > >
> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> > 3.3.5.
> > > >>
> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> and
> > > >> expect people to be able to test it thoroughly enough to make this
> > the one
> > > >> we can ship.
> > > >>
> > > >> What we would like is for anyone who can to verify the tarballs, and
> > test
> > > >> the binaries, especially anyone who can try the arm64 binaries.
> We've
> > got
> > > >> the building of those done and now the build file will incorporate
> > them
> > > >> into the release -but neither of us have actually tested it yet.
> > Maybe I
> > > >> should try it on my pi400 over xmas.
> > > >>
> > > >> The maven artifacts are up on the apache staging repo -they are the
> > ones
> > > >> from x86 build. Building and testing downstream apps will be
> > incredibly
> > > >> helpful.
> > > >>
> > > >> The RC is available at:
> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > > >>
> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > > >>
> > > >> The maven artifacts are staged at
> > > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > > >>
> > > >> You can find my public key at:
> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>
> > > >> Change log
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > > >>
> > > >> Release notes
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > > >>
> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > > >>
> > > >> Key changes include
> > > >>
> > > >> * Big update of dependencies to try and keep those reports of
> > > >> transitive CVEs under control -both genuine and false positive.
> > > >> * HDFS RBF enhancements
> > > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> > > >> high-performance versions for file:// and s3a:// filesystems.
> > > >> file:// through java native io
> > > >> s3a:// parallel GET requests.
> > > >> * This release includes Arm64 binaries. Please can anyone with
> > > >> compatible systems validate these.
> > > >>
> > > >>
> > > >> Please try the release and vote on it, even though i don't know what
> > is a
> > > >> good timeline here...i'm actually going on holiday in early jan.
> > Mukund is
> > > >> around and so can drive the process while I'm offline.
> > > >>
> > > >> Assuming we do have another iteration, the RC1 will not be before
> mid
> > jan
> > > >> for that reason
> > > >>
> > > >> Steve (and mukund)
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Ayush Saxena <ay...@gmail.com>.
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
I don't think there is any issue with the ReleaseNotes generation as such
but with the Resolution type of this ticket, It ain't marked as Fixed but
Done. The other ticket which is marked Done is also not part of the release
notes. [1]
if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
Chris, I just had a very quick look at HDFS-16853, I am not sure if this
can happen outside a MiniDfsCluster setup? Just guessing from the
description in the ticket. It looked like when we did a restart of the
Namenode in the MiniDfsCluster, I guess that would be in the same single
JVM, and that is why a previous blocked thread caused issues with the
restart. That is what I understood, I haven't checked the code though.
Second, In the same context, Being curious If this lands up being a
MiniDfsCluster only issue, do we still consider this a release blocker? Not
saying in a way it won't be serious, MiniDfsCluster is very widely used by
downstream projects and all, so just wanted to know....
Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
veto, I am not sure if it will get done any time soon, so if the use case
is something required, I would suggest handling it at Hadoop itself. It
seems to be centric to Hive-3.x, I tried compiling the Hive master branch
with 3.3.5 and it passed. Other than that Hive officially support only
Hadoop-3.3.1 and that too only in the last 4.x release[2]
[1]
https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
[2] https://issues.apache.org/jira/browse/HIVE-24484
-Ayush
On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cn...@apache.org> wrote:
> -1, because if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
> Otherwise, this RC was looking good:
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Farewell, S3Guard.
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file <X>" on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> My local setup didn't have the test failures mentioned by Viraj, though
> there was some flakiness with a few HDFS snapshot tests timing out.
>
> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> request tracking an upgrade attempt. It's looking like some amount of code
> changes are required:
>
> https://issues.apache.org/jira/browse/HIVE-26648
> https://github.com/apache/hive/pull/3744
>
> Chris Nauroth
>
>
> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> > mentioned anywhere in change log or release notes. Are they actually
> > up-to-date?
> >
> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> > <iw...@oss.nttdata.com> wrote:
> > >
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > >
> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> not
> > expected in hbase-shaded-with-hadoop-check-invariants.
> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> > HBASE-27292.
> > >
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> > >
> > > [INFO] --- exec-maven-plugin:1.6.0:exec
> > (check-jar-contents-for-stuff-with-hadoop) @
> > hbase-shaded-with-hadoop-check-invariants ---
> > > [ERROR] Found artifact with unexpected contents:
> >
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > > Please check the following and either correct the build or
> update
> > > the allowed list with reasoning.
> > >
> > > com/
> > > com/sun/
> > > com/sun/jersey/
> > > com/sun/jersey/json/
> > > ...
> > >
> > >
> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68
> seem
> > to be problematic.
> > > Excluding them on hive-jdbc might be the fix.
> > >
> > > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> > project hive-jdbc: Error creating shaded jar: Problem shading JAR
> >
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> > entry
> >
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> > java.lang.IllegalArgumentException: Unsupported class file major version
> 59
> > -> [Help 1]
> > > ...
> > >
> > >
> > > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > > Thanks for your great effort for the new release, Steve and Mukund.
> > > >
> > > > +1 while it would be nice if we can address missed Javadocs.
> > > >
> > > > + verified the signature and checksum.
> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with
> native
> > profile enabled.
> > > > + launched pseudo distributed cluster including kms and httpfs
> with
> > Kerberos and SSL enabled.
> > > > + created encryption zone, put and read files via httpfs.
> > > > + ran example MR wordcount over encryption zone.
> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> > (both x86_64 and aarch64).
> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop
> 3.3.4
> > worked.
> > > > + skimmed the site contents.
> > > > - Javadocs are not contained (under r3.3.5/api).
> > > > # The issue can be reproduced even if I built site docs from the
> > source.
> > > >
> > > > Masatake Iwasaki
> > > >
> > > > On 2022/12/22 4:28, Steve Loughran wrote:
> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> > 3.3.5.
> > > >>
> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote
> and
> > > >> expect people to be able to test it thoroughly enough to make this
> > the one
> > > >> we can ship.
> > > >>
> > > >> What we would like is for anyone who can to verify the tarballs, and
> > test
> > > >> the binaries, especially anyone who can try the arm64 binaries.
> We've
> > got
> > > >> the building of those done and now the build file will incorporate
> > them
> > > >> into the release -but neither of us have actually tested it yet.
> > Maybe I
> > > >> should try it on my pi400 over xmas.
> > > >>
> > > >> The maven artifacts are up on the apache staging repo -they are the
> > ones
> > > >> from x86 build. Building and testing downstream apps will be
> > incredibly
> > > >> helpful.
> > > >>
> > > >> The RC is available at:
> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > > >>
> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > > >>
> > > >> The maven artifacts are staged at
> > > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > > >>
> > > >> You can find my public key at:
> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >>
> > > >> Change log
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > > >>
> > > >> Release notes
> > > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > > >>
> > > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > > >>
> > > >> Key changes include
> > > >>
> > > >> * Big update of dependencies to try and keep those reports of
> > > >> transitive CVEs under control -both genuine and false positive.
> > > >> * HDFS RBF enhancements
> > > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > > >> * Vectored IO API for all FSDataInputStream implementations, with
> > > >> high-performance versions for file:// and s3a:// filesystems.
> > > >> file:// through java native io
> > > >> s3a:// parallel GET requests.
> > > >> * This release includes Arm64 binaries. Please can anyone with
> > > >> compatible systems validate these.
> > > >>
> > > >>
> > > >> Please try the release and vote on it, even though i don't know what
> > is a
> > > >> good timeline here...i'm actually going on holiday in early jan.
> > Mukund is
> > > >> around and so can drive the process while I'm offline.
> > > >>
> > > >> Assuming we do have another iteration, the RC1 will not be before
> mid
> > jan
> > > >> for that reason
> > > >>
> > > >> Steve (and mukund)
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
-1, because if I'm understanding the potential impact of HDFS-16853
correctly, then it's serious enough to fix before a release. (I could
change my vote if someone wants to make a case that it's not that serious.)
Otherwise, this RC was looking good:
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests passed.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
* Checked dependency tree to make sure we have all of the expected library
updates that are mentioned in the release notes.
* mvn -o dependency:tree
* Farewell, S3Guard.
* Confirmed that hadoop-openstack is now just a stub placeholder artifact
with no code.
* For ARM verification:
* Ran "file <X>" on all native binaries in the ARM tarball to confirm
they actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and
it worked fine.
* Ran a MapReduce job with YARN configured to use
LinuxContainerExecutor and verified launching the containers through
container-executor worked.
My local setup didn't have the test failures mentioned by Viraj, though
there was some flakiness with a few HDFS snapshot tests timing out.
Regarding Hive and Bouncy Castle, there is an existing issue and pull
request tracking an upgrade attempt. It's looking like some amount of code
changes are required:
https://issues.apache.org/jira/browse/HIVE-26648
https://github.com/apache/hive/pull/3744
Chris Nauroth
On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
> On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> <iw...@oss.nttdata.com> wrote:
> >
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >
> > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >
> > [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> > [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > Please check the following and either correct the build or update
> > the allowed list with reasoning.
> >
> > com/
> > com/sun/
> > com/sun/jersey/
> > com/sun/jersey/json/
> > ...
> >
> >
> > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> > Excluding them on hive-jdbc might be the fix.
> >
> > [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> > ...
> >
> >
> > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > Thanks for your great effort for the new release, Steve and Mukund.
> > >
> > > +1 while it would be nice if we can address missed Javadocs.
> > >
> > > + verified the signature and checksum.
> > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > > + created encryption zone, put and read files via httpfs.
> > > + ran example MR wordcount over encryption zone.
> > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > > + skimmed the site contents.
> > > - Javadocs are not contained (under r3.3.5/api).
> > > # The issue can be reproduced even if I built site docs from the
> source.
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/12/22 4:28, Steve Loughran wrote:
> > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> > >>
> > >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> > >> expect people to be able to test it thoroughly enough to make this
> the one
> > >> we can ship.
> > >>
> > >> What we would like is for anyone who can to verify the tarballs, and
> test
> > >> the binaries, especially anyone who can try the arm64 binaries. We've
> got
> > >> the building of those done and now the build file will incorporate
> them
> > >> into the release -but neither of us have actually tested it yet.
> Maybe I
> > >> should try it on my pi400 over xmas.
> > >>
> > >> The maven artifacts are up on the apache staging repo -they are the
> ones
> > >> from x86 build. Building and testing downstream apps will be
> incredibly
> > >> helpful.
> > >>
> > >> The RC is available at:
> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > >>
> > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > >>
> > >> The maven artifacts are staged at
> > >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > >>
> > >> You can find my public key at:
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >> Change log
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > >>
> > >> Release notes
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > >>
> > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > >>
> > >> Key changes include
> > >>
> > >> * Big update of dependencies to try and keep those reports of
> > >> transitive CVEs under control -both genuine and false positive.
> > >> * HDFS RBF enhancements
> > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > >> * Vectored IO API for all FSDataInputStream implementations, with
> > >> high-performance versions for file:// and s3a:// filesystems.
> > >> file:// through java native io
> > >> s3a:// parallel GET requests.
> > >> * This release includes Arm64 binaries. Please can anyone with
> > >> compatible systems validate these.
> > >>
> > >>
> > >> Please try the release and vote on it, even though i don't know what
> is a
> > >> good timeline here...i'm actually going on holiday in early jan.
> Mukund is
> > >> around and so can drive the process while I'm offline.
> > >>
> > >> Assuming we do have another iteration, the RC1 will not be before mid
> jan
> > >> for that reason
> > >>
> > >> Steve (and mukund)
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
-1, because if I'm understanding the potential impact of HDFS-16853
correctly, then it's serious enough to fix before a release. (I could
change my vote if someone wants to make a case that it's not that serious.)
Otherwise, this RC was looking good:
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests passed.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
* Checked dependency tree to make sure we have all of the expected library
updates that are mentioned in the release notes.
* mvn -o dependency:tree
* Farewell, S3Guard.
* Confirmed that hadoop-openstack is now just a stub placeholder artifact
with no code.
* For ARM verification:
* Ran "file <X>" on all native binaries in the ARM tarball to confirm
they actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and
it worked fine.
* Ran a MapReduce job with YARN configured to use
LinuxContainerExecutor and verified launching the containers through
container-executor worked.
My local setup didn't have the test failures mentioned by Viraj, though
there was some flakiness with a few HDFS snapshot tests timing out.
Regarding Hive and Bouncy Castle, there is an existing issue and pull
request tracking an upgrade attempt. It's looking like some amount of code
changes are required:
https://issues.apache.org/jira/browse/HIVE-26648
https://github.com/apache/hive/pull/3744
Chris Nauroth
On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
> On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> <iw...@oss.nttdata.com> wrote:
> >
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >
> > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >
> > [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> > [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > Please check the following and either correct the build or update
> > the allowed list with reasoning.
> >
> > com/
> > com/sun/
> > com/sun/jersey/
> > com/sun/jersey/json/
> > ...
> >
> >
> > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> > Excluding them on hive-jdbc might be the fix.
> >
> > [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> > ...
> >
> >
> > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > Thanks for your great effort for the new release, Steve and Mukund.
> > >
> > > +1 while it would be nice if we can address missed Javadocs.
> > >
> > > + verified the signature and checksum.
> > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > > + created encryption zone, put and read files via httpfs.
> > > + ran example MR wordcount over encryption zone.
> > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > > + skimmed the site contents.
> > > - Javadocs are not contained (under r3.3.5/api).
> > > # The issue can be reproduced even if I built site docs from the
> source.
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/12/22 4:28, Steve Loughran wrote:
> > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> > >>
> > >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> > >> expect people to be able to test it thoroughly enough to make this
> the one
> > >> we can ship.
> > >>
> > >> What we would like is for anyone who can to verify the tarballs, and
> test
> > >> the binaries, especially anyone who can try the arm64 binaries. We've
> got
> > >> the building of those done and now the build file will incorporate
> them
> > >> into the release -but neither of us have actually tested it yet.
> Maybe I
> > >> should try it on my pi400 over xmas.
> > >>
> > >> The maven artifacts are up on the apache staging repo -they are the
> ones
> > >> from x86 build. Building and testing downstream apps will be
> incredibly
> > >> helpful.
> > >>
> > >> The RC is available at:
> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > >>
> > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > >>
> > >> The maven artifacts are staged at
> > >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > >>
> > >> You can find my public key at:
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >> Change log
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > >>
> > >> Release notes
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > >>
> > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > >>
> > >> Key changes include
> > >>
> > >> * Big update of dependencies to try and keep those reports of
> > >> transitive CVEs under control -both genuine and false positive.
> > >> * HDFS RBF enhancements
> > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > >> * Vectored IO API for all FSDataInputStream implementations, with
> > >> high-performance versions for file:// and s3a:// filesystems.
> > >> file:// through java native io
> > >> s3a:// parallel GET requests.
> > >> * This release includes Arm64 binaries. Please can anyone with
> > >> compatible systems validate these.
> > >>
> > >>
> > >> Please try the release and vote on it, even though i don't know what
> is a
> > >> good timeline here...i'm actually going on holiday in early jan.
> Mukund is
> > >> around and so can drive the process while I'm offline.
> > >>
> > >> Assuming we do have another iteration, the RC1 will not be before mid
> jan
> > >> for that reason
> > >>
> > >> Steve (and mukund)
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
-1, because if I'm understanding the potential impact of HDFS-16853
correctly, then it's serious enough to fix before a release. (I could
change my vote if someone wants to make a case that it's not that serious.)
Otherwise, this RC was looking good:
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests passed.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
* Checked dependency tree to make sure we have all of the expected library
updates that are mentioned in the release notes.
* mvn -o dependency:tree
* Farewell, S3Guard.
* Confirmed that hadoop-openstack is now just a stub placeholder artifact
with no code.
* For ARM verification:
* Ran "file <X>" on all native binaries in the ARM tarball to confirm
they actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and
it worked fine.
* Ran a MapReduce job with YARN configured to use
LinuxContainerExecutor and verified launching the containers through
container-executor worked.
My local setup didn't have the test failures mentioned by Viraj, though
there was some flakiness with a few HDFS snapshot tests timing out.
Regarding Hive and Bouncy Castle, there is an existing issue and pull
request tracking an upgrade attempt. It's looking like some amount of code
changes are required:
https://issues.apache.org/jira/browse/HIVE-26648
https://github.com/apache/hive/pull/3744
Chris Nauroth
On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
> On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> <iw...@oss.nttdata.com> wrote:
> >
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >
> > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >
> > [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> > [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > Please check the following and either correct the build or update
> > the allowed list with reasoning.
> >
> > com/
> > com/sun/
> > com/sun/jersey/
> > com/sun/jersey/json/
> > ...
> >
> >
> > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> > Excluding them on hive-jdbc might be the fix.
> >
> > [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> > ...
> >
> >
> > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > Thanks for your great effort for the new release, Steve and Mukund.
> > >
> > > +1 while it would be nice if we can address missed Javadocs.
> > >
> > > + verified the signature and checksum.
> > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > > + created encryption zone, put and read files via httpfs.
> > > + ran example MR wordcount over encryption zone.
> > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > > + skimmed the site contents.
> > > - Javadocs are not contained (under r3.3.5/api).
> > > # The issue can be reproduced even if I built site docs from the
> source.
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/12/22 4:28, Steve Loughran wrote:
> > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> > >>
> > >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> > >> expect people to be able to test it thoroughly enough to make this
> the one
> > >> we can ship.
> > >>
> > >> What we would like is for anyone who can to verify the tarballs, and
> test
> > >> the binaries, especially anyone who can try the arm64 binaries. We've
> got
> > >> the building of those done and now the build file will incorporate
> them
> > >> into the release -but neither of us have actually tested it yet.
> Maybe I
> > >> should try it on my pi400 over xmas.
> > >>
> > >> The maven artifacts are up on the apache staging repo -they are the
> ones
> > >> from x86 build. Building and testing downstream apps will be
> incredibly
> > >> helpful.
> > >>
> > >> The RC is available at:
> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > >>
> > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > >>
> > >> The maven artifacts are staged at
> > >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > >>
> > >> You can find my public key at:
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >> Change log
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > >>
> > >> Release notes
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > >>
> > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > >>
> > >> Key changes include
> > >>
> > >> * Big update of dependencies to try and keep those reports of
> > >> transitive CVEs under control -both genuine and false positive.
> > >> * HDFS RBF enhancements
> > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > >> * Vectored IO API for all FSDataInputStream implementations, with
> > >> high-performance versions for file:// and s3a:// filesystems.
> > >> file:// through java native io
> > >> s3a:// parallel GET requests.
> > >> * This release includes Arm64 binaries. Please can anyone with
> > >> compatible systems validate these.
> > >>
> > >>
> > >> Please try the release and vote on it, even though i don't know what
> is a
> > >> good timeline here...i'm actually going on holiday in early jan.
> Mukund is
> > >> around and so can drive the process while I'm offline.
> > >>
> > >> Assuming we do have another iteration, the RC1 will not be before mid
> jan
> > >> for that reason
> > >>
> > >> Steve (and mukund)
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
>
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
>
I think it is because of what Ayush said. They are marked as done but not
fixed.
Also, some were marked with wrong fix versions, initially, we thought 3.3.9
but then decided to change to 3.3.5.
I have updated them and hopefully, in RC1 it will be fine.
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
>
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
>
I think it is because of what Ayush said. They are marked as done but not
fixed.
Also, some were marked with wrong fix versions, initially, we thought 3.3.9
but then decided to change to 3.3.5.
I have updated them and hopefully, in RC1 it will be fine.
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Mukund Madhav Thakur <mt...@cloudera.com.INVALID>.
>
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
>
I think it is because of what Ayush said. They are marked as done but not
fixed.
Also, some were marked with wrong fix versions, initially, we thought 3.3.9
but then decided to change to 3.3.5.
I have updated them and hopefully, in RC1 it will be fine.
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chris Nauroth <cn...@apache.org>.
-1, because if I'm understanding the potential impact of HDFS-16853
correctly, then it's serious enough to fix before a release. (I could
change my vote if someone wants to make a case that it's not that serious.)
Otherwise, this RC was looking good:
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests passed.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
* Checked dependency tree to make sure we have all of the expected library
updates that are mentioned in the release notes.
* mvn -o dependency:tree
* Farewell, S3Guard.
* Confirmed that hadoop-openstack is now just a stub placeholder artifact
with no code.
* For ARM verification:
* Ran "file <X>" on all native binaries in the ARM tarball to confirm
they actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and
it worked fine.
* Ran a MapReduce job with YARN configured to use
LinuxContainerExecutor and verified launching the containers through
container-executor worked.
My local setup didn't have the test failures mentioned by Viraj, though
there was some flakiness with a few HDFS snapshot tests timing out.
Regarding Hive and Bouncy Castle, there is an existing issue and pull
request tracking an upgrade attempt. It's looking like some amount of code
changes are required:
https://issues.apache.org/jira/browse/HIVE-26648
https://github.com/apache/hive/pull/3744
Chris Nauroth
On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <su...@apache.org> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
> On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> <iw...@oss.nttdata.com> wrote:
> >
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >
> > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >
> > [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> > [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > Please check the following and either correct the build or update
> > the allowed list with reasoning.
> >
> > com/
> > com/sun/
> > com/sun/jersey/
> > com/sun/jersey/json/
> > ...
> >
> >
> > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> > Excluding them on hive-jdbc might be the fix.
> >
> > [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> > ...
> >
> >
> > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > Thanks for your great effort for the new release, Steve and Mukund.
> > >
> > > +1 while it would be nice if we can address missed Javadocs.
> > >
> > > + verified the signature and checksum.
> > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > > + created encryption zone, put and read files via httpfs.
> > > + ran example MR wordcount over encryption zone.
> > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > > + skimmed the site contents.
> > > - Javadocs are not contained (under r3.3.5/api).
> > > # The issue can be reproduced even if I built site docs from the
> source.
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/12/22 4:28, Steve Loughran wrote:
> > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> > >>
> > >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> > >> expect people to be able to test it thoroughly enough to make this
> the one
> > >> we can ship.
> > >>
> > >> What we would like is for anyone who can to verify the tarballs, and
> test
> > >> the binaries, especially anyone who can try the arm64 binaries. We've
> got
> > >> the building of those done and now the build file will incorporate
> them
> > >> into the release -but neither of us have actually tested it yet.
> Maybe I
> > >> should try it on my pi400 over xmas.
> > >>
> > >> The maven artifacts are up on the apache staging repo -they are the
> ones
> > >> from x86 build. Building and testing downstream apps will be
> incredibly
> > >> helpful.
> > >>
> > >> The RC is available at:
> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > >>
> > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > >>
> > >> The maven artifacts are staged at
> > >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > >>
> > >> You can find my public key at:
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >> Change log
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > >>
> > >> Release notes
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > >>
> > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > >>
> > >> Key changes include
> > >>
> > >> * Big update of dependencies to try and keep those reports of
> > >> transitive CVEs under control -both genuine and false positive.
> > >> * HDFS RBF enhancements
> > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > >> * Vectored IO API for all FSDataInputStream implementations, with
> > >> high-performance versions for file:// and s3a:// filesystems.
> > >> file:// through java native io
> > >> s3a:// parallel GET requests.
> > >> * This release includes Arm64 binaries. Please can anyone with
> > >> compatible systems validate these.
> > >>
> > >>
> > >> Please try the release and vote on it, even though i don't know what
> is a
> > >> good timeline here...i'm actually going on holiday in early jan.
> Mukund is
> > >> around and so can drive the process while I'm offline.
> > >>
> > >> Assuming we do have another iteration, the RC1 will not be before mid
> jan
> > >> for that reason
> > >>
> > >> Steve (and mukund)
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chao Sun <su...@apache.org>.
Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
mentioned anywhere in change log or release notes. Are they actually
up-to-date?
On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
<iw...@oss.nttdata.com> wrote:
>
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
> [INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
> ...
>
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the source.
> >
> > Masatake Iwasaki
> >
> > On 2022/12/22 4:28, Steve Loughran wrote:
> >> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
> >>
> >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> >> expect people to be able to test it thoroughly enough to make this the one
> >> we can ship.
> >>
> >> What we would like is for anyone who can to verify the tarballs, and test
> >> the binaries, especially anyone who can try the arm64 binaries. We've got
> >> the building of those done and now the build file will incorporate them
> >> into the release -but neither of us have actually tested it yet. Maybe I
> >> should try it on my pi400 over xmas.
> >>
> >> The maven artifacts are up on the apache staging repo -they are the ones
> >> from x86 build. Building and testing downstream apps will be incredibly
> >> helpful.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >>
> >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >>
> >> The maven artifacts are staged at
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >>
> >> Release notes
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >> transitive CVEs under control -both genuine and false positive.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >> high-performance versions for file:// and s3a:// filesystems.
> >> file:// through java native io
> >> s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >> compatible systems validate these.
> >>
> >>
> >> Please try the release and vote on it, even though i don't know what is a
> >> good timeline here...i'm actually going on holiday in early jan. Mukund is
> >> around and so can drive the process while I'm offline.
> >>
> >> Assuming we do have another iteration, the RC1 will not be before mid jan
> >> for that reason
> >>
> >> Steve (and mukund)
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chao Sun <su...@apache.org>.
Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
mentioned anywhere in change log or release notes. Are they actually
up-to-date?
On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
<iw...@oss.nttdata.com> wrote:
>
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
> [INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
> ...
>
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the source.
> >
> > Masatake Iwasaki
> >
> > On 2022/12/22 4:28, Steve Loughran wrote:
> >> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
> >>
> >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> >> expect people to be able to test it thoroughly enough to make this the one
> >> we can ship.
> >>
> >> What we would like is for anyone who can to verify the tarballs, and test
> >> the binaries, especially anyone who can try the arm64 binaries. We've got
> >> the building of those done and now the build file will incorporate them
> >> into the release -but neither of us have actually tested it yet. Maybe I
> >> should try it on my pi400 over xmas.
> >>
> >> The maven artifacts are up on the apache staging repo -they are the ones
> >> from x86 build. Building and testing downstream apps will be incredibly
> >> helpful.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >>
> >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >>
> >> The maven artifacts are staged at
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >>
> >> Release notes
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >> transitive CVEs under control -both genuine and false positive.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >> high-performance versions for file:// and s3a:// filesystems.
> >> file:// through java native io
> >> s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >> compatible systems validate these.
> >>
> >>
> >> Please try the release and vote on it, even though i don't know what is a
> >> good timeline here...i'm actually going on holiday in early jan. Mukund is
> >> around and so can drive the process while I'm offline.
> >>
> >> Assuming we do have another iteration, the RC1 will not be before mid jan
> >> for that reason
> >>
> >> Steve (and mukund)
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks
pulling in a few of the recent changes which seem needed/important, now
wondering about the javadocs.
i will add a new probe for this in our automated release ant bulld so we
can't cut a release without that
https://github.com/steveloughran/validate-hadoop-client-artifacts
On Mon, 2 Jan 2023 at 15:47, Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>
are we adding some new dependencies from somewhere then? i never even knew
there was a com.sun.json module
hey, imagine if there was a single, standard, json library with a minimal
O/J mapping (strings, numbers, arrays and maps) -we'd be able to cut out
all of jackson, gson, jettison and maybe even avoid the eternal
jackson-databind CVE homework
> [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> ...
>
>
ahh, covered in https://issues.apache.org/jira/browse/HADOOP-17563 ... the
maven shade plugin needs to be updated to handle the multi-JVM JAR
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the
> source.
> >
> > Masatake Iwasaki
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks
pulling in a few of the recent changes which seem needed/important, now
wondering about the javadocs.
i will add a new probe for this in our automated release ant bulld so we
can't cut a release without that
https://github.com/steveloughran/validate-hadoop-client-artifacts
On Mon, 2 Jan 2023 at 15:47, Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>
are we adding some new dependencies from somewhere then? i never even knew
there was a com.sun.json module
hey, imagine if there was a single, standard, json library with a minimal
O/J mapping (strings, numbers, arrays and maps) -we'd be able to cut out
all of jackson, gson, jettison and maybe even avoid the eternal
jackson-databind CVE homework
> [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> ...
>
>
ahh, covered in https://issues.apache.org/jira/browse/HADOOP-17563 ... the
maven shade plugin needs to be updated to handle the multi-JVM JAR
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the
> source.
> >
> > Masatake Iwasaki
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks
pulling in a few of the recent changes which seem needed/important, now
wondering about the javadocs.
i will add a new probe for this in our automated release ant bulld so we
can't cut a release without that
https://github.com/steveloughran/validate-hadoop-client-artifacts
On Mon, 2 Jan 2023 at 15:47, Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>
are we adding some new dependencies from somewhere then? i never even knew
there was a com.sun.json module
hey, imagine if there was a single, standard, json library with a minimal
O/J mapping (strings, numbers, arrays and maps) -we'd be able to cut out
all of jackson, gson, jettison and maybe even avoid the eternal
jackson-databind CVE homework
> [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> ...
>
>
ahh, covered in https://issues.apache.org/jira/browse/HADOOP-17563 ... the
maven shade plugin needs to be updated to handle the multi-JVM JAR
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the
> source.
> >
> > Masatake Iwasaki
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
thanks
pulling in a few of the recent changes which seem needed/important, now
wondering about the javadocs.
i will add a new probe for this in our automated release ant bulld so we
can't cut a release without that
https://github.com/steveloughran/validate-hadoop-client-artifacts
On Mon, 2 Jan 2023 at 15:47, Masatake Iwasaki <iw...@oss.nttdata.com>
wrote:
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>
are we adding some new dependencies from somewhere then? i never even knew
there was a com.sun.json module
hey, imagine if there was a single, standard, json library with a minimal
O/J mapping (strings, numbers, arrays and maps) -we'd be able to cut out
all of jackson, gson, jettison and maybe even avoid the eternal
jackson-databind CVE homework
> [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> ...
>
>
ahh, covered in https://issues.apache.org/jira/browse/HADOOP-17563 ... the
maven shade plugin needs to be updated to handle the multi-JVM JAR
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the
> source.
> >
> > Masatake Iwasaki
> >
>
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chao Sun <su...@apache.org>.
Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
mentioned anywhere in change log or release notes. Are they actually
up-to-date?
On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
<iw...@oss.nttdata.com> wrote:
>
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
> [INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
> ...
>
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the source.
> >
> > Masatake Iwasaki
> >
> > On 2022/12/22 4:28, Steve Loughran wrote:
> >> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
> >>
> >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> >> expect people to be able to test it thoroughly enough to make this the one
> >> we can ship.
> >>
> >> What we would like is for anyone who can to verify the tarballs, and test
> >> the binaries, especially anyone who can try the arm64 binaries. We've got
> >> the building of those done and now the build file will incorporate them
> >> into the release -but neither of us have actually tested it yet. Maybe I
> >> should try it on my pi400 over xmas.
> >>
> >> The maven artifacts are up on the apache staging repo -they are the ones
> >> from x86 build. Building and testing downstream apps will be incredibly
> >> helpful.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >>
> >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >>
> >> The maven artifacts are staged at
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >>
> >> Release notes
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >> transitive CVEs under control -both genuine and false positive.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >> high-performance versions for file:// and s3a:// filesystems.
> >> file:// through java native io
> >> s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >> compatible systems validate these.
> >>
> >>
> >> Please try the release and vote on it, even though i don't know what is a
> >> good timeline here...i'm actually going on holiday in early jan. Mukund is
> >> around and so can drive the process while I'm offline.
> >>
> >> Assuming we do have another iteration, the RC1 will not be before mid jan
> >> for that reason
> >>
> >> Steve (and mukund)
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Chao Sun <su...@apache.org>.
Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
mentioned anywhere in change log or release notes. Are they actually
up-to-date?
On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
<iw...@oss.nttdata.com> wrote:
>
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
> [INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
>
> com/
> com/sun/
> com/sun/jersey/
> com/sun/jersey/json/
> ...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
> Excluding them on hive-jdbc might be the fix.
>
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
> ...
>
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> > + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> > + created encryption zone, put and read files via httpfs.
> > + ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> > + skimmed the site contents.
> > - Javadocs are not contained (under r3.3.5/api).
> > # The issue can be reproduced even if I built site docs from the source.
> >
> > Masatake Iwasaki
> >
> > On 2022/12/22 4:28, Steve Loughran wrote:
> >> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
> >>
> >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> >> expect people to be able to test it thoroughly enough to make this the one
> >> we can ship.
> >>
> >> What we would like is for anyone who can to verify the tarballs, and test
> >> the binaries, especially anyone who can try the arm64 binaries. We've got
> >> the building of those done and now the build file will incorporate them
> >> into the release -but neither of us have actually tested it yet. Maybe I
> >> should try it on my pi400 over xmas.
> >>
> >> The maven artifacts are up on the apache staging repo -they are the ones
> >> from x86 build. Building and testing downstream apps will be incredibly
> >> helpful.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >>
> >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >>
> >> The maven artifacts are staged at
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >>
> >> Release notes
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >> transitive CVEs under control -both genuine and false positive.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >> high-performance versions for file:// and s3a:// filesystems.
> >> file:// through java native io
> >> s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >> compatible systems validate these.
> >>
> >>
> >> Please try the release and vote on it, even though i don't know what is a
> >> good timeline here...i'm actually going on holiday in early jan. Mukund is
> >> around and so can drive the process while I'm offline.
> >>
> >> Assuming we do have another iteration, the RC1 will not be before mid jan
> >> for that reason
> >>
> >> Steve (and mukund)
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
[INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
Please check the following and either correct the build or update
the allowed list with reasoning.
com/
com/sun/
com/sun/jersey/
com/sun/jersey/json/
...
For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
Excluding them on hive-jdbc might be the fix.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
...
On 2023/01/02 22:02, Masatake Iwasaki wrote:
> Thanks for your great effort for the new release, Steve and Mukund.
>
> +1 while it would be nice if we can address missed Javadocs.
>
> + verified the signature and checksum.
> + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> + created encryption zone, put and read files via httpfs.
> + ran example MR wordcount over encryption zone.
> + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> + skimmed the site contents.
> - Javadocs are not contained (under r3.3.5/api).
> # The issue can be reproduced even if I built site docs from the source.
>
> Masatake Iwasaki
>
> On 2022/12/22 4:28, Steve Loughran wrote:
>> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>>
>> Given the time of year it's a bit unrealistic to run a 5 day vote and
>> expect people to be able to test it thoroughly enough to make this the one
>> we can ship.
>>
>> What we would like is for anyone who can to verify the tarballs, and test
>> the binaries, especially anyone who can try the arm64 binaries. We've got
>> the building of those done and now the build file will incorporate them
>> into the release -but neither of us have actually tested it yet. Maybe I
>> should try it on my pi400 over xmas.
>>
>> The maven artifacts are up on the apache staging repo -they are the ones
>> from x86 build. Building and testing downstream apps will be incredibly
>> helpful.
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>>
>> The git tag is release-3.3.5-RC0, commit 3262495904d
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Change log
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>>
>> Release notes
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>>
>> This is off branch-3.3 and is the first big release since 3.3.2.
>>
>> Key changes include
>>
>> * Big update of dependencies to try and keep those reports of
>> transitive CVEs under control -both genuine and false positive.
>> * HDFS RBF enhancements
>> * Critical fix to ABFS input stream prefetching for correct reading.
>> * Vectored IO API for all FSDataInputStream implementations, with
>> high-performance versions for file:// and s3a:// filesystems.
>> file:// through java native io
>> s3a:// parallel GET requests.
>> * This release includes Arm64 binaries. Please can anyone with
>> compatible systems validate these.
>>
>>
>> Please try the release and vote on it, even though i don't know what is a
>> good timeline here...i'm actually going on holiday in early jan. Mukund is
>> around and so can drive the process while I'm offline.
>>
>> Assuming we do have another iteration, the RC1 will not be before mid jan
>> for that reason
>>
>> Steve (and mukund)
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
[INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
Please check the following and either correct the build or update
the allowed list with reasoning.
com/
com/sun/
com/sun/jersey/
com/sun/jersey/json/
...
For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
Excluding them on hive-jdbc might be the fix.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
...
On 2023/01/02 22:02, Masatake Iwasaki wrote:
> Thanks for your great effort for the new release, Steve and Mukund.
>
> +1 while it would be nice if we can address missed Javadocs.
>
> + verified the signature and checksum.
> + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> + created encryption zone, put and read files via httpfs.
> + ran example MR wordcount over encryption zone.
> + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> + skimmed the site contents.
> - Javadocs are not contained (under r3.3.5/api).
> # The issue can be reproduced even if I built site docs from the source.
>
> Masatake Iwasaki
>
> On 2022/12/22 4:28, Steve Loughran wrote:
>> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>>
>> Given the time of year it's a bit unrealistic to run a 5 day vote and
>> expect people to be able to test it thoroughly enough to make this the one
>> we can ship.
>>
>> What we would like is for anyone who can to verify the tarballs, and test
>> the binaries, especially anyone who can try the arm64 binaries. We've got
>> the building of those done and now the build file will incorporate them
>> into the release -but neither of us have actually tested it yet. Maybe I
>> should try it on my pi400 over xmas.
>>
>> The maven artifacts are up on the apache staging repo -they are the ones
>> from x86 build. Building and testing downstream apps will be incredibly
>> helpful.
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>>
>> The git tag is release-3.3.5-RC0, commit 3262495904d
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Change log
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>>
>> Release notes
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>>
>> This is off branch-3.3 and is the first big release since 3.3.2.
>>
>> Key changes include
>>
>> * Big update of dependencies to try and keep those reports of
>> transitive CVEs under control -both genuine and false positive.
>> * HDFS RBF enhancements
>> * Critical fix to ABFS input stream prefetching for correct reading.
>> * Vectored IO API for all FSDataInputStream implementations, with
>> high-performance versions for file:// and s3a:// filesystems.
>> file:// through java native io
>> s3a:// parallel GET requests.
>> * This release includes Arm64 binaries. Please can anyone with
>> compatible systems validate these.
>>
>>
>> Please try the release and vote on it, even though i don't know what is a
>> good timeline here...i'm actually going on holiday in early jan. Mukund is
>> around and so can drive the process while I'm offline.
>>
>> Assuming we do have another iteration, the RC1 will not be before mid jan
>> for that reason
>>
>> Steve (and mukund)
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
[INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
Please check the following and either correct the build or update
the allowed list with reasoning.
com/
com/sun/
com/sun/jersey/
com/sun/jersey/json/
...
For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
Excluding them on hive-jdbc might be the fix.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
...
On 2023/01/02 22:02, Masatake Iwasaki wrote:
> Thanks for your great effort for the new release, Steve and Mukund.
>
> +1 while it would be nice if we can address missed Javadocs.
>
> + verified the signature and checksum.
> + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> + created encryption zone, put and read files via httpfs.
> + ran example MR wordcount over encryption zone.
> + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> + skimmed the site contents.
> - Javadocs are not contained (under r3.3.5/api).
> # The issue can be reproduced even if I built site docs from the source.
>
> Masatake Iwasaki
>
> On 2022/12/22 4:28, Steve Loughran wrote:
>> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>>
>> Given the time of year it's a bit unrealistic to run a 5 day vote and
>> expect people to be able to test it thoroughly enough to make this the one
>> we can ship.
>>
>> What we would like is for anyone who can to verify the tarballs, and test
>> the binaries, especially anyone who can try the arm64 binaries. We've got
>> the building of those done and now the build file will incorporate them
>> into the release -but neither of us have actually tested it yet. Maybe I
>> should try it on my pi400 over xmas.
>>
>> The maven artifacts are up on the apache staging repo -they are the ones
>> from x86 build. Building and testing downstream apps will be incredibly
>> helpful.
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>>
>> The git tag is release-3.3.5-RC0, commit 3262495904d
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Change log
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>>
>> Release notes
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>>
>> This is off branch-3.3 and is the first big release since 3.3.2.
>>
>> Key changes include
>>
>> * Big update of dependencies to try and keep those reports of
>> transitive CVEs under control -both genuine and false positive.
>> * HDFS RBF enhancements
>> * Critical fix to ABFS input stream prefetching for correct reading.
>> * Vectored IO API for all FSDataInputStream implementations, with
>> high-performance versions for file:// and s3a:// filesystems.
>> file:// through java native io
>> s3a:// parallel GET requests.
>> * This release includes Arm64 binaries. Please can anyone with
>> compatible systems validate these.
>>
>>
>> Please try the release and vote on it, even though i don't know what is a
>> good timeline here...i'm actually going on holiday in early jan. Mukund is
>> around and so can drive the process while I'm offline.
>>
>> Assuming we do have another iteration, the RC1 will not be before mid jan
>> for that reason
>>
>> Steve (and mukund)
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Posted by Masatake Iwasaki <iw...@oss.nttdata.com>.
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants.
Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
[INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
Please check the following and either correct the build or update
the allowed list with reasoning.
com/
com/sun/
com/sun/jersey/
com/sun/jersey/json/
...
For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be problematic.
Excluding them on hive-jdbc might be the fix.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hive-jdbc: Error creating shaded jar: Problem shading JAR /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 -> [Help 1]
...
On 2023/01/02 22:02, Masatake Iwasaki wrote:
> Thanks for your great effort for the new release, Steve and Mukund.
>
> +1 while it would be nice if we can address missed Javadocs.
>
> + verified the signature and checksum.
> + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled.
> + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled.
> + created encryption zone, put and read files via httpfs.
> + ran example MR wordcount over encryption zone.
> + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both x86_64 and aarch64).
> - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change.
> # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
> + skimmed the site contents.
> - Javadocs are not contained (under r3.3.5/api).
> # The issue can be reproduced even if I built site docs from the source.
>
> Masatake Iwasaki
>
> On 2022/12/22 4:28, Steve Loughran wrote:
>> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>>
>> Given the time of year it's a bit unrealistic to run a 5 day vote and
>> expect people to be able to test it thoroughly enough to make this the one
>> we can ship.
>>
>> What we would like is for anyone who can to verify the tarballs, and test
>> the binaries, especially anyone who can try the arm64 binaries. We've got
>> the building of those done and now the build file will incorporate them
>> into the release -but neither of us have actually tested it yet. Maybe I
>> should try it on my pi400 over xmas.
>>
>> The maven artifacts are up on the apache staging repo -they are the ones
>> from x86 build. Building and testing downstream apps will be incredibly
>> helpful.
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>>
>> The git tag is release-3.3.5-RC0, commit 3262495904d
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Change log
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>>
>> Release notes
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>>
>> This is off branch-3.3 and is the first big release since 3.3.2.
>>
>> Key changes include
>>
>> * Big update of dependencies to try and keep those reports of
>> transitive CVEs under control -both genuine and false positive.
>> * HDFS RBF enhancements
>> * Critical fix to ABFS input stream prefetching for correct reading.
>> * Vectored IO API for all FSDataInputStream implementations, with
>> high-performance versions for file:// and s3a:// filesystems.
>> file:// through java native io
>> s3a:// parallel GET requests.
>> * This release includes Arm64 binaries. Please can anyone with
>> compatible systems validate these.
>>
>>
>> Please try the release and vote on it, even though i don't know what is a
>> good timeline here...i'm actually going on holiday in early jan. Mukund is
>> around and so can drive the process while I'm offline.
>>
>> Assuming we do have another iteration, the RC1 will not be before mid jan
>> for that reason
>>
>> Steve (and mukund)
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org